What's new

N64 ROM compression methods?

BitRain

New member
I heard about Nintendo specific ROMs using the Yaz0 compression method in N64 ROMs but games made by other developers and non Nintendo titles use a different compression method.
Does anybody know what they are? Rr have any documentation on them?
 

zoinkity

New member
For a quick list of common things (in other words, what comes to mind):
gzip/zlib
lzo1x
pkzip
lzh/lha
lz77 in many, many variants
lzari
rnc
lzb
rle
huffman, traditional and variants

There's more than that, but those are ones that come to mind. Also, there's a plethora of variations of these used. For instance, one company inverted the long backtrack flag in lzo. Basically, if there was a compressor that was open-source at the time you can expect it was implemented in one title or another.

Incidentally, Yaz and Yay are interchangable formats. The only difference is in storage. Also, in 64DD games the '0' and '1' in Ya* and MIO are read to determine if the archive source is cart of disk respectively.
 

zoinkity

New member
The one in Super Smash Brothers? Decompression yes, but I don't think we did compression.

At its heart it's just a bitwise LZ scheme, but the backtrack and length tables are huffman encoded.

[edit]
I should probably check notes before posting. The format is similiar to the vpk format used in e-reader data. Both are bitwise LZ huffman schemes, but I'll have to run a few tests to know for certain how close they really are. It's not direct compatibility at any rate.
 
Last edited:

AIO

New member
Ya the one in Super Smash Brothers.

I tried reading the de/compression algorithms for different formats and got lost very fast! Oh well lol. I suck at algorithms ;/ .

At least I can look at the decompressed data though :) .
 

zoinkity

New member
Okay, it is compatible. Use level 2.
Tested using a decompressed SSB file recompressed via nvpktool. Works on reinsertion in ROM and via the before-mentioned decompression tool. Most of the time you'll want to use method 0 instead of 1, but if the file is mostly 16bit image(s) method 1 might work out better for you. It's really setting the sample size is all.
 

AIO

New member
That is great news! I've run into a problem though. Both decompressors extract the data into many files. How would I use nvpktool with that? I tried compressing one big file but I fail I guess ;/ .
 

zoinkity

New member
That's because the game's data is stored in many small files, some (but not all) individually vpk compressed.

The typical method for editing is to alter a file, compress just that file, then reinsert it where it came from. Of course, there's the small, game-specific issue of having to update filetables/addresses/shifting unrelated files to make it fit, which is why neither tool automates this ;*)
I could go into detail on their filesystem and how to shift/rebuild the bugger, but I do not have the time to make an inserter or (oh dear)editor. The basics are already written into the NALE extraction file at lines 72 and 2365. As a note, you will have to alter a few hardcoded addresses to music and the table start points. Everything else is relatively self-contained.
 

AIO

New member
Oh interesting. So basically I just have to reinsert the specific files I edit. It makes a lot of sense now.

Thanks for all your help! So the hard part is making sure the data fits in and everything is properly aligned. That will be a pain if I have to manually alter some hardcoded addresses. Good thing I'm not looking to do large scale changes. Just small ones like altering character stats.

Edit: So I decompressed with Universal Game Decompressor and I tried re-compressing 634446.bin and according to that file tool, the compressed size is 2C8, but I got 2D6 after compressing it with level 2. Keep in mind I didn't edit 634446.bin. If I used Level 2 method 1, then the size became 2AF. My question is, what's the simplest way to deal with file size difference?
 
Last edited:

zoinkity

New member
Edit: So I decompressed with Universal Game Decompressor and I tried re-compressing 634446.bin and according to that file tool, the compressed size is 2C8, but I got 2D6 after compressing it with level 2. Keep in mind I didn't edit 634446.bin. If I used Level 2 method 1, then the size became 2AF. My question is, what's the simplest way to deal with file size difference?

Unfortunately, since they don't use the same compression algorithm as the original you will probably wind up with different filesizes.

The usual method--even used by professional companies--is to compress the same file using different methods and choose the smallest. They'll usually use a small wrapper program or script so they don't need to do it by hand, but there's really no good way to predict the compressed size of a file without compressing it* Redundancy increases compressability, so when possible reuse values.

You're probably going to hit at least one case where the files only grow. At that point you'll have to alter the filetable. I don't remember off the top of my head how involved that will be in this case, so let me get back to you on how to do that.

*(File compression and complexity are somewhat interrelated topics. The incredibly important LZ77 scheme came from a complexity mapping program LZ76 before it. Interestingly LZ76 was a more complicated program, but the detected levels of complexity roughly correspond to the compression ratios of LZ77.)
 

AIO

New member
Alright. So as I thought, smaller = better. So in the case that I get it to be smaller than the original, what do I do to about the size gap?

Lol good idea with the redundancy. It reminds me of my failed attempt to modify the compressed data by hand. I will keep in mind to intentionally reuse values, when space is needed.
 

zoinkity

New member
Alright. So as I thought, smaller = better. So in the case that I get it to be smaller than the original, what do I do to about the size gap?

I had meant to test if it doesn't care about having more data and just quits when it hits the endcode (and get the filetable details together), but something kinda came up.
If you don't want to wait for me to write back, copy the file in with a hex editor and run it either under emulation or with a flashcart. If the given file doesn't break the game when it loads, then you don't need to correct sizes ;*)
 

AIO

New member
I'm patient and have very little experience with this stuff. I will give it a try though. Also I'm kinda confused on one thing though. When i use subdrag's decompressor, it's quite different from when i use your midway dec. Will either one work fine? Or is Subdrag's incomplete? Yours seems to extract more out of it. I just don't know what to do with those .req files.

Let me know what the results are, if you end up testing this!
 

zoinkity

New member
Or is Subdrag's incomplete? Yours seems to extract more out of it.

SubDrag's decompressor only extracts compressed files. It's automated though, so he can handle a lot more games than Midwaydec at this point. Midwaydec needs filelists for each game, but that also means it can extract absolutely everything inside one.
So...
Yeah, Midwaydec grabs everything you put a line in it for.

Tested, and it looks like it will just decompress a file up to a certain filesize and ignore any extra data falling after it. So, if it's shorter you don't need to do anything unless you want to.

If you do need to edit the table...
VPK files only fall in a certain part of the SSB ROM. That filetable is at 0x1AC870.
Entry format: 0xC each
0x0 4 offset to data from end of table [1B2C6C]; compressed if negative
0x4 2 -1 if not present
offset in words in decompressed file for filetable (*4 for actual)
0x6 2 compressed size in words (*4 for actual)
0x8 2 -1 if not present
offset in words in decompressed file for resource data (*4 for actual)
0xA 2 decompressed size in words (*4 for actual)

If you're moving entries around you only need to change the values at +0 and +6. Take the actual size of the compressed file and divide it by four for the value at +6. Setting the offset takes two steps. Subtract 0x1B2C6C from its position in the ROM for the base value. If it's VPK compressed set 0x80000000. If you look at the table it's probably obvious what they're doing though.

I've been bogged down by an unexpected project but I'll try to write a little file inserter for you when I get some time. It'll be in python, but, well, aegh...
 

AIO

New member
Awesome! Thanks again man! This week I'll try following your instructions.

No rush lol. That would be great to have a file inserter.
 

Tcll

Complexity == Fun >:3
hey zoinkitty, been hearing alot about your knowledge of the N64's compression types. :)

I'm the developer of Universal Model Converter, and have been very interested in learning about this stuff, though havn't had the time to get to it.

think you could help me (and by that I mean my group) out with putting hacks back into SSB64?? :)
(the recompression part is the issue)
 

AIO

New member
hey zoinkitty, been hearing alot about your knowledge of the N64's compression types. :)

I'm the developer of Universal Model Converter, and have been very interested in learning about this stuff, though havn't had the time to get to it.

think you could help me (and by that I mean my group) out with putting hacks back into SSB64?? :)
(the recompression part is the issue)
I finally figured out it, so we're good now. I could have done it earlier, but got caught up in other things. I did as zoinkity said and tried reusing values when possible. That helps with getting the size down.

Now what's left to do is to automate the task better :king: .
 

zoinkity

New member
I thought I had PM'd this before, but apparently not. Sorry about that.
http://www.mediafire.com/view/z0r73k6zdc1e4d9/SSB.py

Well, it's really crude but it does get the job done. Should run on a standard python install 3.3 or higher without trouble. This only allows you to replace files found in the filetable (the ones in the filelist named "resource") and can't directly compress them yet. So, you're still going to have to compress them with nvpktool (or whatever you're using) before importing them. Currently only offset correction tables for the US version (and only for the files that are moving) are included.

That said, it does correct all the offsets and encode entries properly. Tested output works on console without a hitch. It's a heck of a lot faster than doing it by hand.

"Help/Usage" should give you a rundown of how to use it. Basically, load a ROM, double left-click an entry to fiddle with it. Toggle the checkmarks to add or remove that part of the entry, and enter the offset or file you want to replace it with there. If you don't change something it won't change.

I'll be busy for a while, but if there's any serious errors let me know. I'll try to add compression in at a future time and more file replacement support.
 

AIO

New member
Man I'm such a procrastinator. But finally got around to using this tool. Thanks a lot. It works and is convenient :) .
 

Top