For Fall of the Samurai, we’ve made a number of changes to the savegame and startpos formats (chiefly compression) in order to optimise them and reduce their size. An unfortunate side effect of this that we’ve identified is that community-generated modding tools to read these files will no longer function.
We’re really keen to see the great community-crafted modding tools that are out there keep working though, and in the spirit of that we’d like to keep the you chaps in the loop on this.
Thanks kindly to Alan and Ed, our campaign programmers, who have taken the time out to put this info together. I’ll hand them over to you and they will talk you through it.
Shogun 2 save games (and startpos files) now use compression and some format changes to achieve smaller file sizes. Note that uncompressed files, in either previous format or new format, still work.
Several simple mechanisms have been used to reduce data size and improve compressibility. ESF files now start 0xABCA if using these mechanisms.
A new mechanism for section end positions and section block sizes replaces the 4 bytes used previously with from 1 to 5 bytes; each byte has high bit set if there is more to come, with the remaining bits used for size. (ie read a byte, AND with 127, add to result. If high bit was set, shift result by 7, repeat with next byte.) The result is now a delta rather than absolute position, which is relative to the position after the last byte; for section blocks, the end position and block count are both written this way, and the position delta is relative to the end of both of these.
Also, for sections, the type id, section id and version information is written either as 2 or 4 bytes, rather than always 4 as previously. Type id high bit set indicates section; next bit indicates section block; next bit indicates old-style section id and version (written as 16 bit unsigned integer for id then 8 bit unsigned integer for version). If this bit is not set, then read next byte. Version is in the next four bits of the type id – (type_id & 31) / 2 – section id is in the remaining bits – (type_id & 1) * 256 + next_byte.
New type ids supported:
18: bool true (no additional data)
19: bool false (no additional data)
20: unsigned integer 0 (no additional data)
21: unsigned integer 1 (no additional data)
22: unsigned integer written as 8 bit (1 byte follows)
23: unsigned integer written as 16 bit (2 bytes follow)
24: unsigned integer written as 24 bit (3 bytes follow)
25: integer 0 (no additional data)
26: integer written as 8 bit (1 byte follows)
27: integer written as 16 bit (2 bytes follow)
28: integer written as 24 bit (3 bytes follow)
29: float 0 (no additional data)
Equivalent block type ids are these ORed with 64.
Compressed save game ESFs simply contain an entire, loadable ESF file that has been compressed, along with the information required to decompress this data.
Shogun 2 uses LZMA compression for ESF savegames. See http://www.7-zip.org/sdk.html for details.
Compressed save games have an uncompressed section called SAVE_GAME_HEADER (startpos files have an additional CAMPAIGN_PREOPEN_MAP_INFO section), followed by a section called COMPRESSED_DATA. The COMPRESSED_DATA section contains a section named COMPRESSED_DATA_INFO, which contains a 4-byte unsigned integer (the decompressed size of the compressed data), and a 5-byte block containing the information required by LZMA to decompress the data (often called ‘encode properties’). The COMPRESSED_DATA section also contains a large byte block containing the actual compressed data.
Decompressing this data (using the decompressed size, encode properties and compressed data block) will yield a complete ESF file which can be loaded directly by Shogun 2.