N64 Texture Image Extraction

[BGNG]

Within the past 24 hours, I've hacked the Albanian dialtone out of the track representation binary data of F-Zero X. I'm preparing to create a course editor program, much like the Expansion Kit and the fabled "MetEdit"... The only thing that I'm lacking is the actual texture data that the game maps to the track.

I ask for information of the texture images stored in F-Zero X, preferably a specification of the binary encoding. I not only want to obtain the textures, but I want my program to extract them from the ROM that will have custom tracks patched to it... both to save size in the binary execuatble, and in case the user changed the textures.

Any help would be appreciated. This isn't information you can just pull out of someone's pocket... (Well, I guess that depends on what they have in their pocket at the time)

[Quvack]

Wow sounds like one hell of a project :) I've been hacking various roms a little bit for the n64 but havent looked at F-Zero from that perspective, so I'm not sure if the textures are compressed or not. Have you tried opening the rom in a program like Sprite Viewer? It supports some N64 format, so if you look through that, changing formats and things, if your lucky they'll be uncompressed aswell.

Programming wise I guess you'll either have to load them the way the game does by finding where its pointers to the textures are or link to all of the locations manually?

Regardless best of luck with that project :) If you need any help I can try a little though my rom hacking skills are rather basic currently, well not up to level editing anyways.

[BGNG]

Thanks for the tip about Sprite Viewer. I did get the "Bottom of the world" textures, the little character icons, several fonts, and... I found a TON of images used in the Course Edit feature that's apparently built into the game, yet you simply cannot access it without the expansion... Time to do some OTHER hacking...

However, things like the track textures (what I want), character portraits on the "set your engine setting" screen, the title screen images... Those are obviously compressed, but I don't know how...

Can anyone give me some pointers as to how they may be compressed? Perhaps by some form of LZW or RLE?

[Quvack]

Your welcome :)

Theres course edit textures within the normal rom? Interesting! I'll have to have a look thru the rom when I get a bit more time on my hands :)

I really have no idea what sort of compression the textures would be using, sometimes you can tell by going thru the rom and looking for various things that may hint at the formatting used, othertimes you'll have to use a debugger and breakpoint when various things are accessed so you can track down the de-compression routine. It's likely it'd be a variation of one of the common ones though, this is all guessing though, you might have some luck with one called 'Zelda Extractor' its been able to rip a few things out of some games for me, not just zelda. Again i'll look into it when I have a little time :)

The textures might not be compressed still though, its a pain finding the stuff sometimes when they wont show up correctly unless you have the right formatting and width, etc. set up! Just takes time really if its like that.

[MasterPhW]

There is a great prog where you can extract textures of all nintendo consoles (okay... not the NGC... but all others) it's called SNES edit and It's by a german coder...
HP:http://snesedit.romhack.de/
(hope I'm allowed to post this link, if not, delete it!

[BGNG]

Okay, I think I found what I'm looking for... And of course: it's probably harder than it should be.

I found another graphics viewer called Tile Molester, which is SnowBro's enhancement to the infamous Tile Layer Pro. Tile Molester provides a number of codecs for displaying images, and I've found that images in the game can go down in the resolution as far as 4-bit Linear images.

Browsing through the file this way, I re-noticed something I'd noticed before: black bars with white bars inside. Not knowing what these were, I counted them: 26. That's the number of tracks in the game. I take a look at the offset, and slap myself in the forhead. This is a graphical representation of the track data I've already hacked in its entirety; the whole focus of this project.
__________

Following this data, however, is a number of odd occurances. It starts out as a relatively white bar, which turns into a bunch of oscillating "tornadoes" similar in appearance to Windows Media Player's "Neon Highway" visualization. These baffled me since I first started looking through the ROM for images.

I think I know what they are. Looking further into the file, I found three relatively large "white bar-tornado" combos, followed by 33 smaller ones, followed by a HUGE number of tiny ones.

Think of those little icons that show up on the left side of the screen during a GP with the top 6 racers' faces in them. Those are uncompressed in the file; I already found them. And I also saw three more in a different spot of the file, which appear to be enhanced versions of Captain Falcon, Samurai Goroh and Jody Summer. The Goroh version is EXACTLY the same as the image seen in IGN's screenshots of the Expansion Kit.

So I added it all up... Three big tornadoes, three full-screen Title Screen images... 33 smaller ones, 33 racer portraits (Accelleration/Max Speed Screen)... Too-many-to-count tiny ones... too-many-to-count course textures. I sure as heck hope I'm right.
__________

If my suspicion is correct, and these are indeed a collection of compressed images, then I need only the standard under which these images are encoded.

Does anybody, instead of having references to programs, have any tips as to the potential encoding scheme of these images? I'll try LZW ans RLE (common in GIF and TGA, respectively), but if they don't work, I'll want to try something else. It can't be too complicated; because if it were, it would bog down the load times.

[BGNG]

Looks to me like all the compressed images start with the ANSI word MIO0

Ring any bells to anyone?
__________

I found but one reference on Google about this, and I translated it from Russian. From what I can tell, this is the same encoding Super Mario 64 used (MIO being an abbreviation of Mario), so... Anyone know how that works?

[Anton]

"MIO0" - this is what SuperMario uses like a compression type.
And ZeldaExtractor supports "Yaz0" method.

Quvack - could you tell me please what other games works with ZeldaExtractor except Zelda64 ones? I have a compressing utility for Yaz0 and i'm interested what oter games use this type.

[BGNG]

Yes, I've already deducted that part... I'm interested in how it works.

[TheBench]

Can anyone post the exact compression information for MIO0 and Yaz0 please?

[Anton]

Sorry, I don't know.

[BGNG]

Um, yes, that's... putting it rather... bluntly, TheBench. Indeed, I am searching for exactly the same thing. Check your PMs, TheBench. I've made a proposal you may want to consider.

I hereby request that unless any given member has exact information on the binary specifications of MIO0 and/or Yaz0 (NOT links to programs that can read them), that he not post in this thread whatsoever. This is a picky subject, and there's no need to keep it floating around if the solution cannot be provided; because (no offense to Anton, you've been a good sport) posting anything but the requested information is a disservice to the cause.

I've contacted LaC, a contributing member to the N64 emulation scene and co-author of the emulator NEmu, at Dextrose and I'm awaiting a response. I have other plans on how to obtain this information from other people, but all those efforts may fail. If it comes right down to it, I'll royally sink my razor-sharp teeth into F-Zero X until it cries mercy and gives me what I want.

Additionally... If anyone else is interested in joining TheBench and I in our cause, give me a PM. As soon as I get this stuff figured out, there won't be a soul out there who won't hear about it. This is a slight bit of torture I don't think anyone more needs to tollerate.

[minkster]

y are u hacking into fzero anyways? Just wondering if its a hobby or sumthing counting on i dont no anything about hacking or coding :P

[BGNG]

Read my first post in this topic. And please, honor my request: no more replies in this thread.

[BGNG]

Okay, comments are now welcome in this thread, for I have figured out MIO0 with the help of Quvack. In this post, I have included my documentation on the encoding, which is presented as a binary specification. I have also provided a Visual Basic source code module that I developed for decompressing MIO0.

Both files are included with the MIO0.zip attatched to this post.

EDIT:
Here's an example of how to use the module:


Dim RetVal As Integer
RetVal = mioDecodeMIO0("F-Zero X.n64", &H2E01E9, "Test")
MsgBox mioGetErrorString(RetVal)

If you just happen to have F-Zero X, non-byte-swapped (aka Z64), then that will successfully extract the Master splash screen (shown below) and store it as raw pixel data. The pixels are 16-bit, RGBA at 320×240. For some reason, the image is offset at the second byte, the image is shifted one pixel left and the first pixel doesn't seem to be there.

[Quvack]

lol once you set to something it doesnt sound like it takes you that long to work it out! congratulations :) I'm gonna be messing around with this a lot!

[BGNG]

Well, like I said in my first post... It only took me a matter of hours to completely hack the track representation data in F-Zero X. I've been stuck for a week and a half without textures... But now I have them, so the project can officially be underway.

I'm preparing to write the graphics engine for the editor. It will pull textures straight from the ROM, so any game modders will be able to work with the actual in-game graphics.

Anyhoo... As soon as I feel like it, I'll do this decoding of Yaz0 just the same, if only to help out TheBench in his cause.

EDIT:
Okay, I've updated the modMIO0.bas file. You can still go back and get the old one to find out why. The way I did things, it read all the data from the hard disk, one byte at a time, and did the same for writing. I've modified the procedure to work solely in memory, and it now takes a mere fraction of the time to complete itself.

I've also made it so the decoder can work for the developer and simply render to a buffer in the developer's program for easy access without going to and from the file system. Here're the new examples:

This code, just like before, reads from a file and writes to a file.

Dim RetVal As Integer
RetVal = mioDecodeMIO0("F-Zero X.n64", &H23EC51, "Big Blue.tex")
MsgBox mioGetErrorString(RetVal)

This code uses the lowerish-level memory version to do the same thing:

Dim RetVal As Integer, Buffer() As Byte
Open "F-Zero X.n64" For Binary Access Read As #1
RetVal = mioMemDecode(1, &H23EC51, Buffer())
Close #1: Open "Big Blue.tex" For Output As #2
Print #2, StrConv(Buffer(), 64);
Close #2
MsgBox mioGetErrorString(RetVal)

Note that the "For Output" and "StrConv" were used solely for the purpose of resetting the length of any existing "Big Blue.tex"... If you wanted to use "For Binary Access Write," you would use the statement: Put #2, 1, Buffer()

Keep in mind, though, that mioMemDecode HAS to have the file opened for Binary and the Read option set. Opening "For Input" will not work. Additionally, the buffer that will recieve the decompressed MIO0 data MUST NOT have predefined dimensions, and must be of the Byte data type. Do like I did, and simply say "Buffer() As Byte"

Attatched is the modified VB code and an MSPaint arrangement of the textures found in the Big Blue section. (Well... The very last image, of the water that shows up below the course, is actually an image that's uncompressed in a different part of the ROM)

[Quvack]

This kind of information is going to help out a hell of a lot when it comes to hacking some n64 games, especially when MIO0 and Yaz0 are ones that are used in a number of roms from what I remember. I'm gonna go through a few of the games I'm interested in and see what I can find in regards to what compression schemes they use.
Compression has been a thing that has plagued rom hacking for a long time so the more information about the formats out there the better, especially with nice lil tools like this!

[TheBench]

Good work! Interesting stuff. You should run through every rom if you get a chance, just see which use MIO0. I'm seeing if I can apply it.

[Anton]

this is already exist an uncompressor, for mio0 games, i guess:
http://www.dextrose.com/index.php?s=3&m=8&f=270#f270
readme says that "This tool has been tested on SM64 (English) and Pilot Wings 64 (English)".
I tried to extract data from mario with this tool (and found fonts in one of extracted files). Can edit them (for transtation) but can't compress it back to be able to insert it to rom.

would be very COOL to have a proggy with the reverse process - compressor....
are there any?

[BGNG]

would be very COOL to have a proggy with the reverse process - compressor.... are there any?

Not yet. I'll get right on that. (-:

Oh, and that's the very program I used to experiment. Without it, I don't know if I'd have been able to crack MIO0.

[BGNG]

Alright, bit of an update... I've been slacking off a bit, but I have an update nonetheless. My main reason for posting this particular reply is for Anton, who's been wanting an encoder. I've written an encoder, and it works. However, for an unknown reason, it's anyone's guess as to if the resulting file will be greater than or less than the data in a ROM. It compressed Big Blue track textures in F-Zero X more than the data in the ROM, but it made a larger file for the Master Splash Screen than what was in the ROM... And for the life of me, I can't figure out why...

But it makes MIO0 files, Anton. You can perchance use it to make MIO0 data for Super Mario 64 and they may be compressed enough to use. I compiled it as a Windows EXE in case you don't have Visual Basic, and that's in the ZIP file. You'll need the VB6 runtime to use it, but that can be found at Microsoft.com if need be. NOTE: The nature of encoding LZ77 is very CPU-intensive, and takes a long time to process. In a worst-case scenario, the encoder needs to look at every single byte in the file, one-at-a-time, so be patient. I've included a progress indicator, but that's why it takes so long.

In other news... I've modified the decoder program once more, making it even faster. On my computer, it's near-instantaneous when extracting those beloved track textures that caused me to start this topic in the first place. Working with some MipMapping in OpenGL, I bypassed the need to resize the textures, and now I can use them immediately from the ROM image.

The mioDecodeMIO0 and mioMemDecode functions still use the same syntax, so I won't re-post that. If you're interested in using the encoder, though, here's what you might do:

Dim RetVal As Integer, Buffer() As Byte
Open "Something.blah" For Binary Access Read As #1
mioMemEncode 1, Buffer()
Close #1: Open "Something.mio0" For Output As #2
Print #2, StrConv(Buffer(), 64);
Close #2

I haven't made a mioEncodeMIO0 function, because I'm not yet satisfied with the performance of mioMemEncode... But without any further ado, here's the ZIP:

[BGNG]

Woo hoo! Here's the update you all may concievably have been waiting for!

I've completely wrapped up my MIO0 experiences, and this topic can now successfully come to a close.

It turns out I multiplied by 2 a few too many times when writing the encoder. I take that out, and it works like a charm. However, in F-Zero X at least, the encoder magically doesn't compress data quite as much as Nintendo did. It's always just a few bytes larger... Not much to shake a stick at, but enough to throw off an inline offset... I'll have to find out why it does that.

No, there aren't any more speed upgrades for the decoder... I'm pretty sure it's as fast as VB can make it right now.

Here's a list of the prototypes of the codecing functions:

Function mioDecodeMIO0(InputFile As String, DataOffset, OutputFile As String) As Integer
Function mioEncodeMIO0(InputFile As String, OutputFile As String) As Integer
Function mioGetErrorString(ErrNum) As String
Function mioMemDecode(FileNum, DataOffset, Buffer() As Byte) As Integer
Function mioMemEncode(FNum, Buffer() As Byte) As Integer

The "Horrible MIO0 Encoder" has been changed to "MIO0 Codec Kit," because I made a fully-functional little tool out of it. I abbreviated it M0CK, which is kinda approperiate. Note: The "Offset" field is 0-based and decimal.

In addition, I reformatted the documentation "MIO0.txt" into a non-typoed, color-parsed "MIO0.html"! So, basically, this post wraps up everything I've been doing since I started the topic.

Contents of the ZIP:
Within the ZIP file attatched to this post, there is a compiled Windows EXE of M0CK (which needs the Microsoft VB6 runtime, availible at their site), the latest version of the modMIO0 Visual Basic module, the newly-formatted MIO0 HTML documentation file, and a ZIP file containing the source code to M0CK.

And below is a picture of M0CK, just to let people see if it's what they want before they download it.

[TheBench]

Well what's your algorithm for moving something into this LZ format?

[BGNG]

That's all explained in the HTML file I put in that ZIP.

[TheBench]

That's how the LZ works, but not how you decide which are largest parts to repeat, etc.

[BGNG]

The last paragraph says to take the next 18 bytes of the file and compare it with the previous 4096 to find the longest match. After that, a Length/Distance pair can be made to compress it. It's, logically, the same thing as decoding, only backwards.

[Anton]

Cool!
Thanks!
I'll try it in couple next days.

Ah, a question. When I'll figure out how all this works - can I put this tool to my site (zelda64rus)? (full credits to you, for sure!)
And I'd be very appreciate for a "readme" file (what games tested/supported, some examples of usage, etc)

Thanks again!

[Anton]

BGNG - could you please give me your e-mail - Me and one another guy, interested in translation of Mario would like to contact you..
It seems that there is an error in compression code...

Here what I did:

1. Uncopressed Mario rom (not byteswapped) with N64 uncompressor.
2. Found a file with fonts (00108a40)
3. Extract png image with N64GFX
4. Edited it a bit in photoshop
5. Imported edited png image to this file (00108a40)
6. Compressed it with your tool
6. via "RomPut" tool (also tried via copy paste in hex-editor) inserted compressed 00108a40 back into rom.

when loading rom in Ultrahle - the "super mario 64" logo starts to appear - and then the game hangs.

So I think this is a small error in code... I'm not a programmer, but I can't see any other answer..

Any help from you would be appreciated.
Thnx.

[Guru64]

Sound cool! Can you post some screenshots of your level editor?

[Anton]

This is not a level editor - this is a uncompression/compression tool for games that use Mio0 compression (mario, pilot wings, f-zero x, etc)

[BGNG]

Heh... Aren't I bashful... Um.... Yeah, I forgot all about M0CK, Anton. I'll get right back to work on that. Sorry for keeping you waiting.

Also, I originally sought MIO0 so that I could make an F-Zero X level editor, which is currently in production. I'm still fighting against the mysterious math Nintendo used with the twist vectors, but I did take some screenshots when hacking the course data:

[rajkosto]

i have a prob with this.It's regarding packing data back into the rom.i used N64UNC to get all the images decompressed (i dunno why your doesn;t work,it always says error,even if i enter offset) then i open the first one (00108A40) edit it in Tile Molester (just made the coin a diferent colour).Then i use ur proggy to compress it.Use RomPut to add it back to the rom (non byte swapped off course) at the same offset as teh files name (00108A40).Anyways,when i byteswap it(to load up in emu) then load up i get mising models,reandom lock ups,and corrupted/white textures and BSP data all over the place.What am i doing wrong ?

[Anton]

You do ok, this is just an error in code in BGNG's compression tool, I guess.

BGNG - would you correct this at last, please? :)

[rajkosto]

hmm...maybe it;s just how compression works...if u change something then the file size is gotta be diferent....i dunno...i just wanna mess with mario images...

[Anton]

hmm...maybe it;s just how compression works...if u change something then the file size is gotta be diferent....i dunno...i just wanna mess with mario images...

This doc (by me) has some info about n64 editing, file sizes, etc :)
http://www.shedevr.org.ru/zelda64rus/n64translation_how-to_eng.zip

[rajkosto]

great,all i wanna is to mess with mario,so plz fix this ok

[rajkosto]

YAY ! MARIO'S HEAD (NUMBER OF LIVES) IS NOW A STAR !

without any errors.What i did is compare the original rom with my rom (with inserted compressed image) and since the new compressed image was SMALLER than the original,there were some left over bits from the original one.So i just hex edited all that remaining bits to 0's till the next MIO0 header.It worked ! Quite tedious but hey hey it worked.BUT what to do if file is bigger ? then i have no idea....U could make your MOCK select the compressed file,uncompress it,then after editing,when u compress it,tries to do it the same size as the original if not then just append 0's at the end.

[rajkosto]

actualy i just tried
and i don't think those extra leftover bits from the orgininal matter,it messed up if the new files is LARGER than the orig... :homestar:

[Anton]

actualy i just tried
and i don't think those extra leftover bits from the orgininal matter,it messed up if the new files is LARGER than the orig... :homestar:

You MUST have new file smaller. In any case.
Just try to edit it more accurate to make it smaller in size.
If it's larger - it overwrights part of a next header, and you'll see garbage in game then.

[rajkosto]

stupid non resizable roms....

why don't they have like a file system or sumting...make much easier to edit...

also why does SPrite Viewer always say Not enough memory when loading zelda rom ?

[rajkosto]

also how do u extract stuff using this thing ?

[MasterPhW]

Mmh... I wonder why there isn't a GFX plugin out for extracting textures like in the Gamecube (dolphin) one! That would be great!
Anton and BGNG you seems to me very able in texture hacking of N64 games, why not contacting Rice and trying to help him in inserting an texture dumber into his plugin for N64 Games? Would be great! I'm using it often with my GameCube games and often recognizes some "secret" textures I've never seen! Probably there are some in the N64 roms, too?

[Anton]

There are plugins that extracts textures, already.
Textures are in very bad quality, btw. (I tried to extract some for Q3 Arena map - but didn't finish it. And textures are ugly (looks like ~15x15 pixels (or so), 72 dpi - very ugly) :)

I didn't edit textures. I extracted only graphics needed for Zelda translation to russian.

"also how do u extract stuff using this thing ?"
You'll need to uncompress rom, then search for graphics in uncompressed files (with Sprite Viewer or Tile Molester), then extract *.png image with n64gfx from those files.

[rajkosto]

why can't i extract the textures from the ROM directly using M0CK ?
Cuz the uncompressor thingy gives garble at the end of each file

[Anton]

why can't i extract the textures from the ROM directly using M0CK ?
Cuz the uncompressor thingy gives garble at the end of each file


Ask BGNG :)

[rajkosto]

there is a texture dumper plugin...even one that dumps both geometry and textures...just search the board for it.... :party:

[rajkosto]

there is a texture dumper plugin...even one that dumps both geometry and textures...just search the board for it.... :party:

EDIT :

Actualyy here it is http://www.emutalk.net/showthread.php?t=21092

SV viewer doesn't work :bye3: All the colours are messed up

[BGNG]

I found this old bookmark in my web browser called "EmuTalk" and LOW AND BEHOLD: this topic was brought back to life. (-:

Sorry about any waiting. I don't often check to see what new stuff is here in this forum.
__________

Yes, MIO0's compressor is flawed. The reason it doesn't compress fully is because there's something in the coding that compresses incorrectly altogether. It's probably just a small problem, but I've been unable to find it. I think it's a logical error that I didn't account for in the coding in the first place.

I've already spoken to HyperHacker about this. If anyone wants to have a functional compressor, the documentation of the MIO0 format is included with the M0CK program. I am unable to make a sufficient compressor, but there are other programmers out there in the world. You'll have to find one, as I am unwilling to try to correct M0CK.
__________

And in terms of image hacking: the new files that you insert into the ROM must be the same size or smaller, else it will take up data in the ROM wich is legitimately used for other code/resources.

And Sprite Viewer's colors ARE messed up, I believe... There is another program written by SnowBro in Java called Tile Molester, which is certainly worth the download. It's the new version of his classic Tile Edit which was used to edit the graphics in old NES titles like Metroid to make new games. (SnowBro also created the infamous MetEdit)
__________

Lesse here... "why can't i extract the textures from the ROM directly using M0CK?" - You can. Just type in the beginning offset of the MIO0 data in the ROM... M0CK doesn't automatically search through the ROM for MIO0 headers, which may be what you're asking... because I simply didn't code it to do so.

"Cuz the uncompressor thingy gives garble at the end of each file" - Not sure what you're talking about here. All the files I ever decompressed did the same job as the other decompressor I was using (which I didn't make)
__________

As for the rest... I have no life's passion. Hacking is simply a hobby; nothing I would want to do full-time. Same goes with programming and video games in general for that matter.

I'm still willing to help out, though. If you have any questions on whatever, go ahead and ask and I'll find out what I can in order to send you on your way. I'm just not willing to be the one to actually provide any solutions if it requires extensive work on my part. (Selfish? Lazy? Nah. I just don't want to be caught up in it)

[Anton]

Sad...

[SubDrag]

Have you tried decompressing the compressed files you make, and also output each step of the decompression at a time if there was a bug? That way, you can see whether it was a problem with algorithm, or a problem with the implementation.

If you're really going to give up when you're so close, I'll implement the compressor next week, assuming it's not a specifications error.

[BGNG]

The files I tested on do in fact decompress to the source files. Though, since there definitely is error in the code, I wouldn't be surprised if that didn't always happen.

I've been over the code a zillion times. It does EXACLTY what I planned for it to do. It's the fact that there's something I'm not planning out correctly to make it work right. Don't know what, though. But since my main purpose for ever researching MIO0 in the first place was to make a decoder (which works) to extract the texture images from F-Zero X, my work is done. This is simply MORE than I had intended to do.

And the specification is documented in that HTML file included with M0CK. If you get to understanding that really well, you can make a good compressor.

And when working on a compressor yourself, keep in mind that compressing one large chunk of data could concievably mark out the beginning of an even longer chunk for the next match, ultimately making a larger file. I'm CERTAIN my programming doesn't account for that as well as it should.

[SubDrag]

If you find a file that doesn't work, decompress and see where the problem arises.

Anyways, how I would/will implement this encoder is by doing the following:

Write three bytes

Start the looping stuff:
Variable for amount back, size

Start at position 0, loop until curPos - 2 (while < (curPos - 2))
Loop at each position till curPos, compare to next data one by one

Replace the amount back and size variables with larger size/amt back if size to replace is larger, as you go (if was match)

after done looping:
if size < 3, write next byte
else write the stuff for amount back and size (that 12-bit pair)


Finish when file is done (reached input size)

Of course, as you go, you'll have to update the 1/0 compress/decompress tally.
And the offets will be written last, as you write file. I wouldn't worry too much about the large chunk of writing marking out an even longer chunk; in most cases, it'll only be a few bytes. If you want to handle that, your algorithm will be more complicated, and take a bunch of passes I'd imagine. Perhaps your problem arises with the last few bytes?

And considering I believe (haven't looked, but in most cases) since the data uses a palette of some sort (you haven't discussed the image format), with your input image you can turn down the # of colors and fit nicely.

I thought you were working on a level editor for F-Zero? If so, you'd need compression. I was hoping for that...

[BGNG]

I, failing to write a compressor upon attempt, cannot guarantee any "proper" method for implementing one, as I was unable to do so myself. But basically, yeah. Write three bytes and keep checking back in the already-compressed SOURCE file for more matches. "BEWARE OF CHAIN CHOMP" for example, would compress to "BEWARE OF CHAINOMP" because of the " CH" match.

And the problem with my algorithm causes it to fail throughout the entire file. I did most of the testing with it using the third splash screen for F-Zero X (the cross-section of the Blue Falcon), and even early in the file there were some differences between MY compressed file and the one used in the ROM.

The texture images I needed are MIO0 compressed, and once decompressed are encoded with standard 16-bit RGBA... that is, [rrrr rggg] [ggbb bbba] for the bits. Indeed, cutting down the NUMBER of colors used in the image will cause the same color to be repeated more frequently, allowing the final image to be compressed, even when changed.

However, I have no need to do that in the level editor. The level data itself is stored uncompressed in the ROM. All I need to do is change the bytes for that. In order to make a graphical interface for the program, I needed textures to map to the polygons I'd draw, and what better textures than the ones actually used in the game?

I am able to extract those texture images from the ROM to use in my program, so (says the title of this very thread) my quest is over. This codec utility is just extra nick-nack that resulted from my research.

[SubDrag]

Just because you compress your file and it does not equal the one they have, does NOT mean it is compressed wrong. It simply means that their compression algorithm was different. As long as it uses the same specifications, there is no need to worry. Since you're making a level editor, why not allow users to create their own textures and insert them?

[BGNG]

I'm near-certain the compressor I made is indeed incorrect. The only application I've seen attempted for it thus-far was one of Anton's associates changing the font of Super Mario 64 to Russian, where the file turned out smaller, but execution of the code messed up. That might have been because of the header checksum, which I brought to his (gottaX) attention but never got a reply.

And as far as my project, it's a level editor, not programming overhaul. I intended to create something like the F-Zero X Expansion Kit for the 64DD which was released solely in Japan, and the best way for me to do that would be to hack the ROM in similar fashion to other level editors (the only of which I have used was MetEdit). Making changable textures would be a taxing implementation at best, especially because my project sports proprietary course data files for transport and patching to template ROMs.

[SubDrag]

I've used your program to decompress some of the MIO0 encoding from F-Zero X, where is the palette? I see a bunch of indexes, but where's the palette?

[BGNG]

There is no palette in F-Zero X, to my knowledge. All images I've seen used in the game (and I believe I've looked through them ALL) are either 16-bit RGBA or 8-Bit alpha greyscale. Here's the utility module I made for my project. The particular procedure you want is called LoadTextureMIO0. In there, "Dest" is the pointer to a UserControl I made to easily implement OpenGL in my programming.

I've attatched the module to this post.

Lesse here... I've constructed that utility to extract textures directly from the ROM, so you'll need the F-Zero X ROM in Z64 (non byte-swapped) format. I've not yet implemented support for V64 yet. Then, use these offsets (remember that Visual Basic starts at 1 instead of 0) to locate the track textures:


&H235131 'Mute City
&H239A81 'Port Town
&H23EC51 'Big Blue
&H243D91 'Sand Ocean
&H24A271 'Devil's Forest
&H2507F1 'White Land
&H255101 'Sector
&H259601 'Red Canyon
&H25F361 'Fire Field
&H266C21 'Silence
&H26D781 'Ending
That is... File offset &H235131 in Visual Basic corresponds to 0x235130 in C++.

So, to extract all track textures for any given track set (locale, venue, world, etc.) where in this example, Mute City is used:

Dim FBuff() As Byte
Open "F-Zero X.n64" For Binary Access Read As #1
mioMemDecode 1, &H235131, FBuff()
Close #1
For I = 0 To 6: LoadTextureMIO0 FBuff, CDbl(4096) * I + 1, 64, 32, I, True, Form1.GLComet1: Next
LoadTextureMIO0 FBuff, CDbl(4096) * 7 + 1, 64, 32, 7, True, Form1.GLComet1
For I = 8 To 10: LoadTextureMIO0 FBuff, CDbl(4096) * I + 1, 64, 32, I, True, Form1.GLComet1: Next
And GLComet1 is that UserControl for OpenGL that I made.
__________

Also in that module:

The CCSpline procedure takes an array of doubles and interpolates them into a Closed Cubic Spline, which is (I believe) the same way the tracks are connected in the game itself. It takes the array PIn() and makes an interpolated loop which is stored in POut(). You would need to call this 3 times, one for the X coordinates, and once for Y and once for Z.

The FloatF() and IntF() functions accept 4- and 2-byte values and converts them from Big-Endian to memory, which Visual Basic stores as Little-Endian. Because they're Big-Endian in the ROM, I needed to make a compensation for my project. I have not yet written functions that take a value in memory and convert them to Big-Endian, however.
__________

I know this is more than you need, but this is a working example of how the textures can be loaded from MIO0 and converted into bitmaps, which in this instance are used in OpenGL.

[SubDrag]

I'm programming it in C++ (ugh VB). One thing I'd like to know as well, is whether there is any indicator that lets you know if the data is 8-bit or 16-bit, or is that in an unknown image table or on the calls to load the textures themselves, more likely? Goldeneye let you know in the header of the image.

I'll probably do a quick re-write in C++ of the decompression routine, and have it convert to bitmaps directly, as well.

Oh, and those pointers you gave me, are they to some sort of model data or textures? Are they interspersed, and both in MIO0? Gimme a location of a 16-bit RGBA if you don't mind, something I could find in the game, and I'll try decompressing then recompressing something when its done; and know if it worked.

[BGNG]

"(ugh VB)"? Explain why you said that and I'll consider answering your questions.

[SubDrag]

Once you use C++, VB hurts the eyes. My main is question is the image. How do you know its dimensionss, 8-bit or 16-bit, etc

[BGNG]

I am well familiar with both.

If you don't feel like converting the code to C++, I'll do that for you. I was just making sure you didn't have any unjust bias against VB like so many do. I personally know quite a number of programming languages including variations of C, BASIC, Java, and Assembly. BASIC is probably the easiest to use, and it's adequate for most of my purposes, so I use that one the most.
__________

For videogame programming, each resource has exactly one (1) function. In the case of these images, they are used solely as textures for one part of the game. So if you were the programmer, it should be fairly obvious that you know exactly where you'll be using said images; meaning you'll also know offhand the dimensions and pixel depth of each of them.

F-Zero X is no exception to that rule. Each image's dimensions are hard-coded into the game since the images are raw binary data. It's an old-fashioned technique, but it's nonetheless faster than deriving the dimensions from the resource itself.

Like I mentioned in my previous post, these are the actual offsets of the track texture images in the F-Zero X ROM:

0x235130 //Mute City
0x239A80 //Port Town
0x23EC50 //Big Blue
0x243D90 //Sand Ocean
0x24A270 //Devil's Forest
0x2507F0 //White Land
0x255100 //Sector
0x259600 //Red Canyon
0x25F360 //Fire Field
0x266C20 //Silence
0x26D780 //Ending
All textures for any given track set are included in one file. The pixels for these images are stored left-to-right, top-to-bottom in 16-bit RGBA format. These particular resources have 10 images at 128×64 pixels, and 1 image at 256×64 pixels. In the order that they appear in the file: there are 7 128×64 images, then the 256×64 image, then three more 128×64 images.

The 256×64 image, for example, is the texture for track that has high walls.

[SubDrag]

I'm getting less data when I decompress than that amount.
0xBFC0 is decompressed size using M0CK of at least two level sets (only tried 2).

128 x 64 = 0x2000 in hex. Two bytes per pixel, = 0x4000

256 x 64 = 0x4000 in hex. Two bytes per pixel, = 0x8000

0xBFC0 is what M0CK gives me
Anything I'm doing wrong?

Beginning ones look like 64 x 32...decompress identically to your before.

Using that hypothesis, and seeing your 128 x 16, works great.
Ahh I see, you gave me the dimensions in bytes, which is counter-intuitive a bit. Got it. One more image after this, just didn't fit into 10 attachment limit.

OK now time to work on the decompressor and then finally compressor now that I can test my results.

Next edit: My decompressor implemented...work on compressor starting, after lunch.

[BGNG]

Woah. Talk about a memory lapse... Yeah, I was all wrong with those figures. The images are of 64×32 pixels (times 10) and 128×16 pixels (times one)... I have no idea where I got those other numbers. (I'm not as think as you confused I am)

Also, at the end of those files, you'll notice miniature versions (I think 16×8 pixels) of the larger images. Those are used in the Edit Course feature, which I believe to be hard-coded into F-Zero X, but cannot be accessed without the F-Zero X Expansion Kit via N64DD.

Edit:
I have documentation on the MIPS Assembly opcodes (binary data and execution) for the R4300 RISC processor that the Nintendo 64 uses. If you would like said documentation, you would probably be able to write a disassembler and crack open a few games (like Mario Kart 64) and take the compressor code out of there...

...if you feel like it.

[SubDrag]

Compression works nicely. I don't know why yours isn't working. I haven't tried putting in a user image, but this recompresses the textures from big blue, and I put them into the fire level. I guess I had a lot of experience with the ridiculous amounts of hours I put into successfully getting Goldeneye's compression. As to your question, I have the documentation on MIPS; I'm quite an experienced gameshark hacker. I have all the tools I need, Nemu is great for breakpointing and was instrumental in some of the Goldeneye stuff.

Since this is one chunk of data, have to write some stuff that will properly allow insertion of user images. I had no idea how simple F-Zero was, barely uses any textures at all.

Oh, and my compression was about 0x200 worse than theirs, not a huge deal, and especially since user images you can turn the colors down and you'll be fine.

[BGNG]

Wow... A delightful conversation... I don't get too many of these.

Well, that's the same problem I was having; my compressor was generating larger files than those used in the ROM. That means that (obviously) it wasn't compressing as well as it could. To this day I can't figure out what Nintendo did differently, but it works better. It drove all the sense out of me, so I have since given up.
__________

You've probably noticed in your hacking up there that even though the track textures were Big Blue, the bottom (the lava stuff) was still Fire Field. The images for the bottoms of the worlds are NOT MIO0 data. They are uncompressed RGBA starting at offset 0x004CFE50 in the ROM... or it might be 0x004CFE4E... Tile Molester's offset display is confusing. It's just a few bytes from those locations if it isn't either of them.

Also... Like I mentioned in my very first post in this thread, I've completely cracked every byte of the course data resources in the ROM. I've documented the file format and I've attatched the specification to this post.

Look through that document a bit. The course data starts at offset 0x002AD1E0 in the ROM, and all courses are exactly the same size. They are all in order, from Mute City - Figure 8 through Big Hand - Dangerous Curves. There are two more after Big Hand: Death Race and Ending, in that order.

After looking through that document, you should be able to see that changing byte 0x002B3062 (Z64 format) from 08 to 02 will change the internal track set of the Fire Field course from the Fire Field textures to the Big Blue textures.

[SubDrag]

Took a look at the file, looks like you've mapped it out nicely. I still can't get over the shock of how simple it is, the whole thing, compared to other games. This is your project though, I just wanted to get MIO0 going, still have to just finish up the little compression thing, but that's it, then back to my other projects. The hitch with these textures is that they're in the sets of 11+, as you found, which is a huge nuisance for editing textures, especially since our compression algorithms are not as efficient. When they're individual, it's a lot better.

Just for reference, the MIO0 in other games, such as mario or mario kart, are they similar in that they use a lot of textures in one bank?

It would be cool to see a track editor though. Heh, I never really played this game much back in the day, it's kind of funny how every level is essentially the same, just recolor the texture bank and change the level a bit.

[BGNG]

Actually, there's a bit of an update you may want to know about. gottaX (Anton's affiliate) actually DID reply to me the other day. One of the members (SHEDEVR) in his crew was able to create a "flexible parsing" functionality for my MIO0 compression code. He has succeeded in making a compressor which makes files SMALLER than those seen in Nintendo's ROMs. So whatever Nintendo did to make a compressor better than mine, SHEDEVR did it better than them.

I have not been able to obtain the program he created, however. I will keep on that and keep you updated.
__________

I know in Mario Kart 64 that the ghost saves for a memory pak are stored as compressed MIO0 data. So I know that Mario Kart 64 has a compression algorithm in it somewhere. As for course textures or what-not, I really don't know. But chances are, they are individual. F-Zero X is the only place I've seen "libraries" of textures in that manner.

[SubDrag]

Don't need the program, just the algorithm. But I guess I can abort the project and concentrate on GE since he has the working one. I'm sure if we thought about it, we could think up a better LZ algorithm

[CaH4e3]

I have some corrections ;) My nickname is not SHEDEVR ;) Here you can find source and binary files for encoder/decoder... Since previous version I've improved compression ratio for some bytes more. But sources has been used for some research, it have some unnecessary code and rectilinear algorithm... It need some improvements for faster packing like binary- or suffix three applying. Enjoy ;)

http://cah4e3.deruspsx.com/upload/mio0pack.rar

[BGNG]

Lookin' tastey. I haven't had a chance to test it, but the coding is sound.

And what is it about these emulation utilities that are always coded for DOS? It isn't very hard to set up a window with a dialog box.

[SubDrag]

I have some corrections ;) My nickname is not SHEDEVR ;) Here you can find source and binary files for encoder/decoder... Since previous version I've improved compression ratio for some bytes more. But sources has been used for some research, it have some unnecessary code and rectilinear algorithm... It need some improvements for faster packing like binary- or suffix three applying. Enjoy ;)

http://cah4e3.deruspsx.com/upload/mio0pack.rar

Do you mind explaining how your algorithm for LZ encoding works?

[HyperHacker]

o_o

Sorry to bring up an old thread like this, but wow. I just coded one of these myself a few days ago, and got it so close to having as small output as Nintendo's... (Source code here (http://board.acmlm.org/thread.php?id=10435); includes some info on Mario Kart's compression code.) How did you get it to work? I figured out most of the optimization but I couldn't get that one last part right...

[BGNG]

Looks like CaH4e3 up there did it. I was never able to. Look through his code to see what you can find.

[HyperHacker]

Hmm, I looked at the code, didn't help much without any form of comments or documentation. <_< It actually doesn't seem to be compressing as well as Nintendo's in all cases, though. For example, a file at 0x641F70 in Mario Kart 64 (US) is compressed to 908 bytes in the ROM, but extracting it and compressing it with this, it comes out to around 910.