itch.io's formatting doesn't behave itself when it comes to code. Because of this, here it is in plain HTML.
This tutorial outlines the process of packing and unpacking all the resources required for a game (be it textures, scripts or level data) into a single file, to make it easier to distribute, and more importantly easier for people to download and play. Though this method is most applicable to games, it could also be used to pack resources such as icons and fonts for more general-purpose applications. Note that this method is unsuitable for packing dynamic libraries, at least, not without writing custom code to load the libraries.
The method outlined in this tutorial is one of many, and doesn't do anything fancy such as compression, which makes it unsuitable for packing the resources for large games. That said, for the majority of indie games where the resources total a size of about a gigabyte, it works perfectly and is rather fast.
The general gist of it is this:
- Build the game into an executable.
- Write a specific sequence of bytes at the end of the executable.
- Pack all the resources into a single file, with some kind of index that outlines the offsets of each individual file.
- Append the resource package onto the end of the executable.
- To read back the data, simply read from the end of the executable file and look for the byte sequence that we wrote earlier. The index can then be used to lookup the data offset for the raw data of each file.
The code examples in this tutorial are written in C, but this method can be utilised in any language that has a somewhat complete file API.
Packing Files
Packing the files is the easier portion of this process, so that's what will be covered first.
The first step is to write a specific sequence of bytes to the end of the built executable. Before doing this, it's a good idea to check if these bytes already exist or not, that way you don't add more when you don't need to. That detail has been omitted from the following example for simplicity. This sequence of bytes should be something that is extremely unlikely to appear anywhere else in the resulting file, either in the executable's code or the game's resources. I normally use a 64-bit integer generated by the ELF-hash algorithm, from a string such as "game data". If all your game is storing is text, simply writing a null character will work just fine - but in most cases games rely on more than just text; it's likely that a texture, for example, will contain some null bytes itself, which will mess with the unpacker.
The code for this generally goes something like this:
FILE* file = fopen("./my_executable", "r+");
fseek(file, 0, SEEK_END);
unsigned long long bs = elf_hash("game data");
fwrite(&bs, sizeof(bs), 1, file);
fclose(file);
It's not a bad idea to make a small, stand-alone program to do this part, that way you can invoke it as a post-build command from your build system. If you're using the hashing method of generating the byte sequence, just make sure that your game uses the same hash function for loading in the package.
Now it's time to actually do the packing. I usually combine this with the program that adds the byte sequence, though you could easily build it in to your level export process as well, if you have one.
To do this, first an index of the byte offsets and sizes for all the files needs to be generated. Unfortunately, there's no real clean way to do this other than iterating all the files to be packed, fopen'ing them and using fseek and ftell to calculate their sizes. This index is not dissimilar to a table in Lua or a dictionary in Python, with the file names being the keys, and their offsets being the values. Instead of storing the actual string of the file names, I normally just store hashes of them, which slightly reduces the time it takes to lookup a file in the index.
The code to pack all of the files looks like this:
const unsigned int buffer_size = 2048;
unsigned char buffer[buffer_size];
FILE* output = fopen("./output.res", "w");
unsigned int index_element_size = sizeof(unsigned int) * 3;
/* Calculate and write the size of the index. This will
* be used to offset the data blob in the read process. */
unsigned int index_size = index_element_size * file_count;
fwrite(&index_size, sizeof(index_size), 1, output);
unsigned int index_offset = sizeof(unsigned int);
unsigned int current_blob_offset = 0;
/* Write all the files to the output file. */
for (int i = 0; i < file_count; i++) {
unsigned int file_size;
unsigned int name_hash;
name_hash = elf_hash(files[i]);
FILE* file = fopen(files[i], "r");
fseek(file, 0, SEEK_END);
file_size = ftell(file);
/* Write the file into the index */
fseek(output, index_element_size * i + index_offset, SEEK_SET);
fwrite(&name_hash, sizeof(name_hash), 1, output);
fwrite(& current_blob_offset, sizeof(current_blob_offset), 1, output);
fwrite(&file_size, sizeof(file_size), 1, output);
/* Copy the file's data into the data blob */
fseek(output, current_blob_offset + index_size + index_offset, SEEK_SET);
fseek(file, 0, SEEK_SET);
for (unsigned int ii = 0; ii < file_size; ii += buffer_size) {
unsigned int bytes_read = fread(buffer, 1, buffer_size, file);
fwrite(buffer, bytes_read, 1, output);
}
fclose(file);
current_blob_offset += file_size;
}
fclose(output);
Instead of copying the file all at once, a buffer is used to copy 2048 byte portions of it at a time. This is because some resources, such as textures, can be quite large, and it would be slow to load the entire thing into memory at once. For conciseness, error checking has been left out, but you always want to be checking that file handles open successfully in any real world application.
Once you have all of the resources packed into a single file, it's trivial to simply append that file onto the end of the executable. Again, we want to use a buffer to copy the package, since the package could be gigabytes in size.
const unsigned int buffer_size = 2048;
unsigned char buffer[buffer_size];
FILE* exec_file = fopen("./my_executable", "r+");
FILE* package_file = fopen("./output.res", "r");
fseek(exec_file, 0, SEEK_END);
fseek(package_file, 0, SEEK_END);
unsigned int package_size = ftell(package_file);
fseek(package_file, 0, SEEK_SET);
for (unsigned int i = 0; i < package_size; i += buffer_size) {
unsigned int bytes_read = fread(buffer, 1, buffer_size, package_file);
fwrite(buffer, bytes_read, 1, exec_file);
}
fclose(exec_file);
fclose(package_file);
And with that, the packing portion of the process is done. If you did everything correctly, you now have a game executable that contains all of its resources.
Reading The Resources
Now for the less trivial part; reading the resources back in from the packed executable. You probably want to set up a system so that only release builds will read from the package, since having the files separate can be advantageous for development, if only because it allows you to edit the resources individually more quickly, without having to repack the executable every time you make an edit.
The first step of the process of reading is for the executable to fopen itself. To do this, you need the path of the executable. On Unix-like systems, this can be done by calling fopen on argv[0] passed from main. On Windows it's generally more involved to get the path of the executable, since running an application through the command line doesn't require you to specify it's extension, so there's a chance that argv[0] won't get the actual path. I usually use the GetModuleFileNameA function from the Win32 API, passing NULL for the hModule parameter.
After the program has opened itself, it needs to look for the byte sequence that was written to it in the first step, as that indicates the end of the executable and the start of the package. Unfortunately, I haven't found a way to do this that's clean and particularly fast other than simply checking every byte in the file until it is found. Luckily, it only needs to be done once at the start of the program.
The following function looks for the byte sequence and returns an offset relative to the start of the file of the 8 bytes after the sequence, which is the start of the index, since the byte sequence was written as a 64-bit integer.
unsigned int find_byte_sequence(const char* file, unsigned long long sequence) {
FILE* file = fopen(file, "r");
fseek(file, 0, SEEK_END);
int file_size = ftell(file);
for (int i = file_size - 1; i >= 0; i--) {
unsigned long long u64;
fseek(file, i, SEEK_SET);
fread(&u64, sizeof(u64), 1, file);
if (u64 == sequence) {
fclose(file);
return i + sizeof(u64);
}
}
fclose(file);
return 0;
}
After the byte offset of the package is known, separate files can be read in. As with the packing, error checking has been omitted, but you absolutely want to add it in. Note the use of goto in the following function.
void load_resource(const char* filename, unsigned char** buffer,
unsigned int* size, unsigned int sequence_offset) {
unsigned int filename_hash = elf_hash(filename);
FILE* exec_file = fopen(get_exec_name(), "r");
fseek(exec_file, sequence_offset, SEEK_SET);
unsigned int index_size;
unsigned int index_size_size = sizeof(unsigned int);
fread(&index_size, sizeof(index_size), 1, exec_file);
unsigned int index_element_size = sizeof(unsigned int) * 3;
unsigned int read_offset, read_size;
for (unsigned int i = 0; i < index_size; i += index_element_size) {
unsigned int hash, offset, size;
fread(&hash, sizeof(hash), 1, exec_file);
fread(&offset, sizeof(offset), 1, exec_file);
fread(&size, sizeof(size), 1, exec_file);
if (hash == filename_hash) {
read_offset = index_size_size + index_size + sequence_offset + offset;
read_size = size;
goto read;
}
}
/* The file was not found in the index */
*buffer = NULL;
*size = 0;
fclose(exec_file);
return;
read:
/* The file was found, read the data at it's offset */
*buffer = malloc(read_size);
*size = read_size;
fseek(exec_file, read_offset, SEEK_SET);
fread(*buffer, read_size, 1, exec_file);
fclose(exec_file);
}
It's a good idea to wrap this function in a preprocessor block so that it will only load from the package in release builds, and instead load from separate files in debug builds.