Reverse Engineering the Xbox 360 SVOD file format

Foreward

Back in October of 2018 I began contributing to the open-source Xbox 360 Emulator research project Xenia. I was focusing on creating a new, cross-platform GUI for it when I learned that it wasn’t yet able to launch Games on Demand or Digital Download titles. There was a bit of groundwork in place, but it wasn’t operational.

I spent about 50 hours researching other projects and stepping through a debugger with my Jtagged Xbox 360, and was able to provide a working solution that seemingly covers all cases when working with SVOD systems. I have thus compiled my research into this post in hopes that someone with a similar problem can use my information. All code present in this blog post is available on Github, and at the time of writing, the code is still pending as a pull request.

What is SVOD?

SVOD, short for Secure Virtual Optical Drive, is an Xbox 360 filesystem implementation that is used for secure storage of Xbox 360 titles that require a large partition space. It is used for Games on Demand (GOD), games installed to an Xbox 360 console’s hard drive, and games requiring a filesystem larger than FATX’s 4GB limit. Much like STFS (Secure Transacted File System), it provides integrity checking through hash blocks. However, unlike STFS, SVOD is comprised of a header/manifest file and a “.data” folder, which the file system’s data split into ~166MB fragments. These fragments exist inside of the data folder, each suffixed with a number indicating the index of the data file.

Despite being fairly similar to STFS packages, SVOD systems require a drastically different implementation in order to successfully mount and read their contents. After 50+ hours of reverse engineering, I believe I have figured out the key components required to mount the file system and successfully read its contents, which I have subsequently contributed to the Xenia Xbox 360 Emulator research project.

The Header File

In order to mount an SVOD system, the header file must first be read, as it contains some important information required for calculating offsets. The Free60 project has already mapped out much of the header file and the SVOD descriptor associated with it. It uses the same header layout as an STFS package, but replaces the STFS descriptor with an SVOD descriptor.

Offset Length Type Information
0x00 0x01 byte Volume descriptor size (0x24)
0x01 0x01 byte Block Cache Element Count
0x02 0x01 byte Worker Thread Processor
0x03 0x01 byte Worker Thread Priority
0x04 0x14 byte[] Hash
0x18 0x01 byte Device features
0x19 0x03 uint24 Data block count
0x1C 0x03 uint24 Data block offset
0x1F 0x05 byte[] Padding/reserved

Most of these descriptor entries can be ignored, but Device featuresData block count and Data block offset are important. Each data fragment file contains blocks consisting of 0x800 bytes. These blocks are separated by hashes, making them discontinuous in memory. Thus, to access a specific file, we need to 1) Calculate the proper block, 2) Calculate the block’s offset in memory, and 3) map each file entry to its block range, so that we know which blocks contain that file’s data.

Mapping the SVOD Data Fragments

So, we’ve read the header file, extracted the important data from it, and (unless the header file is larger than 44kb and actually contains data, which is very uncommon) can essentially ignore it for the rest of the device’s lifespan. Now we want to open a handle for each data fragment. For Xenia, I used a standard library map with an integer as a key and the file handle as the value. This lets us easily select a file handle for the specified data fragment.

StfsContainerDevice::Error StfsContainerDevice::MapFiles() {
  // Map the file containing the STFS Header and read it.
  XELOGI("Mapping STFS Header File: %s", xe::to_string(local_path_).c_str());
  auto header_map = MappedMemory::Open(local_path_, MappedMemory::Mode::kRead);

  auto header_result = ReadHeaderAndVerify(header_map->data());
  if (header_result != Error::kSuccess) {
    XELOGE("Error reading STFS Header: %d", header_result);
    return header_result;
  }

  // If the STFS package is a single file, the header is self contained and
  // we don't need to map any extra files.
  // NOTE: data_file_count is 0 for STFS and 1 for SVOD
  if (header_.data_file_count <= 1) {
    XELOGI("STFS container is a single file.");
    mmap_.emplace(std::make_pair(0, std::move(header_map)));
    return Error::kSuccess;
  }

  // If the STFS package is multi-file, it is an SVOD system. We need to map
  // the files in the .data folder and can discard the header.
  auto data_fragment_path = local_path_ + L".data";
  if (!filesystem::PathExists(data_fragment_path)) {
    XELOGE("STFS container is multi-file, but path %s does not exist.",
           xe::to_string(data_fragment_path).c_str());
    return Error::kErrorFileMismatch;
  }

  // Ensure data fragment files are sorted
  auto fragment_files = filesystem::ListFiles(data_fragment_path);
  std::sort(fragment_files.begin(), fragment_files.end(),
            [](filesystem::FileInfo& left, filesystem::FileInfo& right) {
              return left.name < right.name;
            });

  if (fragment_files.size() != header_.data_file_count) {
    XELOGE("SVOD expecting %d data fragments, but %d are present.",
           header_.data_file_count, fragment_files.size());
    return Error::kErrorFileMismatch;
  }

  for (size_t i = 0; i < fragment_files.size(); i++) {
    auto file = fragment_files.at(i);
    auto path = xe::join_paths(file.path, file.name);
    auto data = MappedMemory::Open(path, MappedMemory::Mode::kRead);
    mmap_.emplace(std::make_pair(i, std::move(data)));
  }
  XELOGI("SVOD successfully mapped %d files.", fragment_files.size());
  return Error::kSuccess;
}

This C++ code does just that. It opens a file handle for each data fragment file and maps it to an index. One important thing to note is that we sort the files chronologically to ensure that each file is mapped to its proper index. This should never be a problem to begin with, but is a good thing to have in case any end-user edge cases arise with how the host operating system lists the files of the data directory.

Of course, file data isn’t the only thing stored in each block. Some blocks are responsible for storing the file system’s entry table as well. In fact, mapping out the file table is the first order of business, as each file entry maps the block that the file starts at. Before we can do that, however, we need to determine where the starting block is.

Locating the Starting Block

This step can be a bit confusing. SVOD systems can have multiple layouts. The most common of the two are the GDF (Game Disc File) layout and the XSF (Xbox Submission File) layout. Most Games on Demand and Installed Game SVOD systems use GDF, as they are already authored and are official releases. However, both homebrew and games converted with a third-party tool such as ISO2GOD use XSF layouts, as these are much easier to create.

Determining which layout a particular SVOD system uses requires a bit of logic. Generally, we’ll first want to check if the Device features field (in the SVOD descriptor) contains a flag indicating the EGDF layout. This flag has value xxxxxxxxxx. To check if this flag is present, we can just use the AND operator on the device features field, which will return a boolean value denoting whether or not a GDF layout is being used. If one is, in fact, being used, then the only data that needs to be skipped are the hash blocks, which we’ll dive into in a minute.

bool has_egdf_layout = features & kFeatureHasEnhancedGDFLayout;

If the SVOD system is not GDF, then it is likely XSF. Depending on what package is being loaded, there may be additional data to skip, such as the XSF header. The following code demonstrates how to check for each case:

StfsContainerDevice::Error StfsContainerDevice::ReadSVOD() {
  // SVOD Systems can have different layouts. The root block is
  // denoted by the magic "MICROSOFT*XBOX*MEDIA" and is always in
  // the first "actual" data fragment of the system.
  auto data = mmap_.at(0)->data();
  const char* MEDIA_MAGIC = "MICROSOFT*XBOX*MEDIA";

  // Check for EDGF layout
  auto layout = &header_.svod_volume_descriptor.layout_type;
  auto features = header_.svod_volume_descriptor.device_features;
  bool has_egdf_layout = features & kFeatureHasEnhancedGDFLayout;

  if (has_egdf_layout) {
    // The STFS header has specified that this SVOD system uses the EGDF layout.
    // We can expect the magic block to be located immediately after the hash
    // blocks. We also offset block address calculation by 0x1000 by shifting
    // block indices by +0x2.
    if (memcmp(data + 0x2000, MEDIA_MAGIC, 20) == 0) {
      base_offset_ = 0x0000;
      magic_offset_ = 0x2000;
      *layout = kEnhancedGDFLayout;
      XELOGI("SVOD uses an EGDF layout. Magic block present at 0x2000.");
    } else {
      XELOGE("SVOD uses an EGDF layout, but the magic block was not found.");
      return Error::kErrorFileMismatch;
    }
  } else if (memcmp(data + 0x12000, MEDIA_MAGIC, 20) == 0) {
    // If the SVOD's magic block is at 0x12000, it is likely using an XSF
    // layout. This is usually due to converting the game using a third-party
    // tool, as most of them use a nulled XSF as a template.

    base_offset_ = 0x10000;
    magic_offset_ = 0x12000;

    // Check for XSF Header
    const char* XSF_MAGIC = "XSF";
    if (memcmp(data + 0x2000, XSF_MAGIC, 3) == 0) {
      *layout = kXSFLayout;
      XELOGI("SVOD uses an XSF layout. Magic block present at 0x12000.");
      XELOGI("Game was likely converted using a third-party tool.");
    } else {
      *layout = kUnknownLayout;
      XELOGI("SVOD appears to use an XSF layout, but no header is present.");
      XELOGI("SVOD magic block found at 0x12000");
    }
  } else if (memcmp(data + 0xD000, MEDIA_MAGIC, 20) == 0) {
    // If the SVOD's magic block is at 0xD000, it most likely means that it is
    // a single-file system. The STFS Header is 0xB000 bytes , and the remaining
    // 0x2000 is from hash tables. In most cases, these will be STFS, not SVOD.

    base_offset_ = 0xB000;
    magic_offset_ = 0xD000;

    // Check for single file system
    if (header_.data_file_count == 1) {
      *layout = kSingleFileLayout;
      XELOGI("SVOD is a single file. Magic block present at 0xD000.");
    } else {
      *layout = kUnknownLayout;
      XELOGE(
          "SVOD is not a single file, but the magic block was found at "
          "0xD000.");
    }
  } else {
    XELOGE("Could not locate SVOD magic block.");
    return Error::kErrorReadError;
  }

  // Parse the root directory
  uint8_t* magic_block = data + magic_offset_;
  uint32_t root_block = xe::load(magic_block + 0x14);
  uint32_t root_size = xe::load(magic_block + 0x18);
  uint32_t root_creation_date = xe::load(magic_block + 0x1C);
  uint32_t root_creation_time = xe::load(magic_block + 0x20);
  uint64_t root_creation_timestamp =
      decode_fat_timestamp(root_creation_date, root_creation_time);

  auto root_entry = new StfsContainerEntry(this, nullptr, "", &mmap_);
  root_entry->attributes_ = kFileAttributeDirectory;
  root_entry->access_timestamp_ = root_creation_timestamp;
  root_entry->create_timestamp_ = root_creation_timestamp;
  root_entry->write_timestamp_ = root_creation_timestamp;
  root_entry_ = std::unique_ptr(root_entry);

  // Traverse all child entries
  return ReadEntrySVOD(root_block, 0, root_entry);
}

In the first case, we check for the presence of an XSF header, which should be located in the first data fragment file and in the location that we would normally expect the first block to be located at. Even if it is not found, we look for the start block magic string at 0x12000, as some shoddy homebrew may not actually include the XSF header but still be using that layout. Finally, the last case is even more rare, and only serves to launch some samples that were provided for indie game developers. In this case, the data is actually contained in the header file, so we skip the header (which is of size 0xB000) and the first two hash blocks to locate the magic string denoting the starting block.

If one of these three conditions are met, we should have successfully located the starting block (which contains the magic string "MICROSOFT*XBOX*MEDIA" and are now almost ready to parse the SVOD file table. Before we do, however, we need a way to calculate the addresses of the blocks we will encounter.

Calculating Block Offsets

When we parse the file table, we will be given the indices of blocks, but not the actual address or file that the block is present in. This can be a bit tricky, as different layouts will cause the calculated offset to be slightly different. Consider the following code:

void StfsContainerDevice::BlockToOffsetSVOD(size_t block, size_t* out_address,
                                            size_t* out_file_index) {
  // SVOD Systems use hash blocks for integrity checks. These hash blocks
  // cause blocks to be discontinuous in memory, and must be accounted for.
  //  - Each data block is 0x800 bytes in length
  //  - Every group of 0x198 data blocks is preceded a Level0 hash table.
  //    Level0 tables contain 0xCC hashes, each representing two data blocks.
  //    The total size of each Level0 hash table is 0x1000 bytes in length.
  //  - Every 0xA1C4 Level0 hash tables is preceded by a Level1 hash table.
  //    Level1 tables contain 0xCB hashes, each representing two Level0 hashes.
  //    The total size of each Level1 hash table is 0x1000 bytes in length.
  //  - Files are split into fragments of 0xA290000 bytes in length,
  //    consisting of 0x14388 data blocks, 0xCB Level0 hash tables, and 0x1
  //    Level1 hash table.

  const size_t BLOCK_SIZE = 0x800;
  const size_t HASH_BLOCK_SIZE = 0x1000;
  const size_t BLOCKS_PER_L0_HASH = 0x198;
  const size_t HASHES_PER_L1_HASH = 0xA1C4;
  const size_t BLOCKS_PER_FILE = 0x14388;
  const size_t MAX_FILE_SIZE = 0xA290000;
  const size_t BLOCK_OFFSET = header_.svod_volume_descriptor.data_block_offset;
  const SvodLayoutType LAYOUT = header_.svod_volume_descriptor.layout_type;

  // Resolve the true block address and file index
  size_t true_block = block - (BLOCK_OFFSET * 2);
  if (LAYOUT == kEnhancedGDFLayout) {
    // EGDF has an 0x1000 byte offset, which is two blocks
    true_block += 0x2;
  }

  size_t file_block = true_block % BLOCKS_PER_FILE;
  size_t file_index = true_block / BLOCKS_PER_FILE;
  size_t offset = 0;

  // Calculate offset caused by Level0 Hash Tables
  size_t level0_table_count = (file_block / BLOCKS_PER_L0_HASH) + 1;
  offset += level0_table_count * HASH_BLOCK_SIZE;

  // Calculate offset caused by Level1 Hash Tables
  size_t level1_table_count = (level0_table_count / HASHES_PER_L1_HASH) + 1;
  offset += level1_table_count * HASH_BLOCK_SIZE;

  // For single-file SVOD layouts, include the size of the header in the offset.
  if (LAYOUT == kSingleFileLayout) {
    offset += base_offset_;
  }

  size_t block_address = (file_block * BLOCK_SIZE) + offset;

  // If the offset causes the block address to overrun the file, round it.
  if (block_address >= MAX_FILE_SIZE) {
    file_index += 1;
    block_address %= MAX_FILE_SIZE;
    block_address += 0x2000;
  }

  *out_address = block_address;
  *out_file_index = file_index;
}

The code is pretty easy to follow and well documented, but in case you don’t quite understand, here’s a quick rundown:

Blocks are 0x800 bytes each and hash blocks are 0x1000 bytes each. Each group of 0x198 hash blocks is preceded by a Level-0 hash block, and each group of 0xA1C4 Level-0 hash blocks is preceded by a Level-1 hash block. To make this simpler, just consider that there will only be one Level-1 hash block per data file, and it will be the very first 0x1000 bytes of the file (in most cases). Likewise, there will be a Level-0 hash block immediately after the Level-1 hash block, also consisting of 0x1000 bytes. This means that for each data file in the SVOD system, the first real data block will be at offset 0x2000.

There are three problems with this, though. First, if the SVOD system contains an XSF header, the 0x2000 bytes of hash blocks will be preceded by a 0x10000 byte header, but only for the first fragment file. Second, if the SVOD system uses the GDF layout, for whatever reason, there is an additional 0x1000 byte offset when calculating the actual block address. If you know why, please do reply to this post with a comment! Lastly, if the SVOD system is self-contained in a single file, we need to skip the header, which is of size 0xB000. This is an extremely rare case, as most of the time a file like this will be STFS, not SVOD. However, it has occurred on a handful of rare files, most of which are SDK samples.

When we have the actual address of the block we want, we need to consider rounding it to the proper file. Files are limited to 0xA290000 bytes, so we can just use division and modulo operations to properly calculate the index of the data file that the block is included in.

The SVOD File Table

Now that we have the root block (denoted by the aforementioned magic string) and a way to calculate the true file and offset that the block resides in, we can begin to parse the file table. To do this, we treat the root block as the root directory. The file table acts as a tree structure, where each file table block is a leaf node either to the left or right hand side of the current node. In addition to this, a file table block consists of multiple file entries. We can distinguish the file in question using ordinals. Each entry has two fields which store ordinals for the left and right leaf nodes, both of which are 16-bit integers.  They are stored at block offsets 0x0 and 0x2 respectively, and we calculate the offset of the entry using size_t ordinal_offset = ordinal * 0x4;.

In addition to ordinals, the blocks contain other metadata that helps us parse its contents. Here is a table of the available metadata:

Offset Length Type Information
0x00 0x02 byte Left leaf ordinal
0x02 0x02 byte Right leaf ordinal
0x04 0x04 32-bit unsigned int File data start block index
0x08 0x04 32-bit unsigned int File data length (spanning all blocks)
0x0C 0x01 byte Attribute flags
0x0D 0x01 byte File name length
0x0E 0x** char[] Null-terminated file name

First, we have the index of the file’s starting block and its length. Given that each block is limited to 0x800 bytes, we can easily deduce the number of blocks a given file spans. Then, we have the length of the file name and its corresponding offset. We can simply cast this pointer to a char array (or string) and receive the proper file name. Lastly, we have the entry’s attribute flags. Simply put, if we AND the attributes with 0x10 and receive true as the returned value, the entry is another directory, and we scan its block(s) as we would any other directory.

If the entry is an actual file, we can just map out the blocks that make up the file. Consider the following code to achieve this:

StfsContainerDevice::Error StfsContainerDevice::ReadEntrySVOD(
    uint32_t block, uint32_t ordinal, StfsContainerEntry* parent) {
  // For games with a large amount of files, the ordinal offset can overrun
  // the current block and potentially hit a hash block.
  size_t ordinal_offset = ordinal * 0x4;
  size_t block_offset = ordinal_offset / 0x800;
  size_t true_ordinal_offset = ordinal_offset % 0x800;

  // Calculate the file & address of the block
  size_t entry_address, entry_file;
  BlockToOffsetSVOD(block + block_offset, &entry_address, &entry_file);
  entry_address += true_ordinal_offset;

  // Read block's descriptor
  auto data = mmap_.at(entry_file)->data() + entry_address;

  uint16_t node_l = xe::load(data + 0x00);
  uint16_t node_r = xe::load(data + 0x02);
  uint32_t data_block = xe::load(data + 0x04);
  uint32_t length = xe::load(data + 0x08);
  uint8_t attributes = xe::load(data + 0x0C);
  uint8_t name_length = xe::load(data + 0x0D);
  auto name = reinterpret_cast(data + 0x0E);
  auto name_str = std::string(name, name_length);

  // Read the left node
  if (node_l) {
    auto node_result = ReadEntrySVOD(block, node_l, parent);
    if (node_result != Error::kSuccess) {
      return node_result;
    }
  }

  // Read file & address of block's data
  size_t data_address, data_file;
  BlockToOffsetSVOD(data_block, &data_address, &data_file);

  // Create the entry
  // NOTE: SVOD entries don't have timestamps for individual files, which can
  //       cause issues when decrypting games. Using the root entry's timestamp
  //       solves this issues.
  auto entry = StfsContainerEntry::Create(this, parent, name_str, &mmap_);
  if (attributes & kFileAttributeDirectory) {
    // Entry is a directory
    entry->attributes_ = kFileAttributeDirectory | kFileAttributeReadOnly;
    entry->data_offset_ = 0;
    entry->data_size_ = 0;
    entry->block_ = block;
    entry->access_timestamp_ = root_entry_->create_timestamp();
    entry->create_timestamp_ = root_entry_->create_timestamp();
    entry->write_timestamp_ = root_entry_->create_timestamp();

    if (length) {
      // If length is greater than 0, traverse the directory's children
      auto directory_result = ReadEntrySVOD(data_block, 0, entry.get());
      if (directory_result != Error::kSuccess) {
        return directory_result;
      }
    }
  } else {
    // Entry is a file
    entry->attributes_ = kFileAttributeNormal | kFileAttributeReadOnly;
    entry->size_ = length;
    entry->allocation_size_ = xe::round_up(length, bytes_per_sector());
    entry->data_offset_ = data_address;
    entry->data_size_ = length;
    entry->block_ = data_block;
    entry->access_timestamp_ = root_entry_->create_timestamp();
    entry->create_timestamp_ = root_entry_->create_timestamp();
    entry->write_timestamp_ = root_entry_->create_timestamp();

    // Fill in all block records, sector by sector.
    if (entry->attributes() & X_FILE_ATTRIBUTE_NORMAL) {
      uint32_t block_index = data_block;
      size_t remaining_size = xe::round_up(length, 0x800);

      size_t last_record = -1;
      size_t last_offset = -1;
      while (remaining_size) {
        const size_t BLOCK_SIZE = 0x800;

        size_t offset, file_index;
        BlockToOffsetSVOD(block_index, &offset, &file_index);

        block_index++;
        remaining_size -= BLOCK_SIZE;

        if (offset - last_offset == 0x800) {
          // Consecutive, so append to last entry.
          entry->block_list_[last_record].length += BLOCK_SIZE;
          last_offset = offset;
          continue;
        }

        entry->block_list_.push_back({file_index, offset, BLOCK_SIZE});
        last_record = entry->block_list_.size() - 1;
        last_offset = offset;
      }
    }
  }

  parent->children_.emplace_back(std::move(entry));

  // Read the right node.
  if (node_r) {
    auto node_result = ReadEntrySVOD(block, node_r, parent);
    if (node_result != Error::kSuccess) {
      return node_result;
    }
  }

  return Error::kSuccess;
}

Conclusion

After following all of these steps, you should now be able to successfully map an SVOD system, mount it, and read from it. This write-up does not cover writing to it, but that wasn’t the goal either. The Xenia project aims to accurately emulate the console by reading from the system, so this is more than enough for the time being.

I would like to thank the Velocity project for providing some insight into how they managed to read from SVOD systems (although it is incomplete in some regards and does not successfully handle all cases thrown at it). If a daring reader would like to fix its implementation using the research provided in this post, you would be doing the community a great service, and would likely learn something too.

If you have any questions or comments, please reach out in the comments section!

Leave a Reply