[$threadinfo[title]]

[Elpenguino]

Is there any way to extract the files from within these .pak files? Attempting to extract files manually with a hex editor has been nothing but a pain so far.

so far, I've noticed that the first bytes in each one is 00 03 00 05 00 00 00 00 00 00 00...
then a byte containing the number of entries...

[thakis]

Hmm... thakis might be of help. He's the master of GameCube file formats. :D

Who could resist these kind words... ;-)

I've taken a look at metroids pak files. I understand the header well enough to write a program to split a pak file into several files. Maybe I'll post it in the next few days, but it's probably not of much use because the files contained in a pak are probably in some custom format as well.

[thakis]

Texture conversion kinda works

[Elpenguino]

oooh. good work so far, thakis.

[thakis]

I don't have the time to continue this, though, because I want to continue to work on my bmd viewer.

If anyone wants to continue working on this, tell me...here's a list of what I've done so far:

- A .pak extractor that splits a pak file into all the files contained in them. Some of the contained files are compressed, a decompressor is still needed.
- A tool to convert a metroid texture file (.txtr) to a super mario sunshine texture file (.bti) (which can then be converted to a dds file with another tool).


Interesting things left to do:
- A metroid model viewer (.cmdl viewer)
- A metroid level viewer (.mrea)

Viewers for about 25 other file types.

[Elpenguino]

thakis, if you would...could you email me the source and any data you know about? You should still have my email address... Thanks in advance.

[thakis]

I'll post everything here, if someone else wants to to take a look...

pak file format
-------------

The pak file format starts with a file header and a list that stores file name entries for some of the files in the pak (not for all). After that, there's a file header for each file stored in the pak file. This header stores the type of a file (called 'tag' in the source, for some reason it's stored in the file name entries as well), a unique identifier for this file (if the file has a name in the filename table, then the name has the same id), where in the pak this file is stored and a flag that says if the file is compressed or not (I'm not completely sure on this - but it's quite likely. Files which don't have this flag set look "normal" in a hex editor, files which have this file set look "compressed").

mpakdump takes a pak file and extracts all files found in it. Files are not decompressed, though - you can see in the filename if a file is compressed (compressed files start with 1, uncompressed files with 0). The filename table in the pak file is ignored.


File types
---------

There a several file types in a pak file. .txtr files are texture files. mtexdump takes an uncompressed (!) txtr file and converts it to a bti file (that's the image format used by mario sunshine, it required the least amount of code to convert to this format. You can use btidump to convert bti to dds, which is supported by most image viewers).



Here's a list of the other file types together with some notes:

STRG - strings (unicode or japanes shift-jis)
CMDL - model
TXTR - texture
CSKR
HINT
FRME
ANCS
ANIM
AFSM
DGRP
EVNT
FONT
CINF
MAPA uncompressed files start with 0xdeadd00d
MAPW uncompressed files start with 0xdeadf00d
MLVL uncompressed files start with 0xdeafbabe
MREA - level geometry, uncompressed files start with 0xdeadbeef
PART - particles?
PATH
SCAN
SAVW
SWHC
WPSC
ELSC
DPSC
CRSC
DUMB
CTWK - tweak
CSNG - midi data


Stuff to do:
- write a decompressor for the compressed files
- write viewers for the more interesting file types (mrea, followed by cmdl imo)


EDIT (three months later ;-) ): I wrote a decompressor for the compressed files, it is attached (mdecomp20051222.zip, source included). It takes a compressed file and decompresses it (if the input file is called "1_....", the output file is called "01_....").

[ShizZy]

Interesting... I'll take a look when I get a spare chance, no guarentees though.

[Elpenguino]

here's a few formats from metroid prime 2 that may also be in metroid prime:
CLSN
CSPP
PLTT
ANMS
MADF
SAVA
SPSC
SRSC
DCLN
AGSC - Scan data?
ATBL
HMAP
PTLA
STLC
EGMC
RULE
FSM2
MAPU

also...that texture converter doesn't seem to work with metroid prime 2. it keeps producing textures that are either 128 bytes or more than 10 MB in size. I do not have a metroid prime iso at the moment, so I cannot compare the formats myself.

[thakis]

HMAP - probably a height map and similar to .ymap in sms
FSM2 - sounds a lot like finite state machine ;-)
MAPU - probably similar to MAPA and MAPW (what are the first 4 bytes in an uncompressed file?)

Be sure that you only give uncompressed txtr files to mtexdump...(I don't have m2, so I can't take a look either ;-))

[Elpenguino]

MAPU begins with 0xABCD EF01.
there's one in GGuiSys.pak that contains strings such as "Sand", "Swamp", "Cliff", "TempleHub", and "TempleInt".

oh, and none of the textures seem to work. either they're all compressed, or they use a different format...

[thakis]

If their filename starts with 1, they are compressed. Otherwise, they are not.

[Elpenguino]

alright, I never noticed that. so the uncompressed textures do indeed work with the program. heh...

[Elpenguino]

interesting note:
uncompressed .RULE files begin with the string 'RULE'
uncompressed .SCAN files begin with the string 'SCAN'
uncompressed .FONT files should begin with the string 'FONT'
uncompressed .CMDL files should begin with 0xDEADBABE.
uncompressed .MAPU files begin with 0xABCDEF01
uncompressed .SAVW files begin with 0xC001D00D
uncompressed .STRG files begin with 0x87654321
uncompressed .PTLA files begin with 0xDEAFBEEF
uncompressed .MREA files begin with 0xDEADBEEF
uncompressed .MLVL files begin with 0xDEAFBABE
uncompressed .MAPW files begin with 0xDEADF00D
uncompressed .MAPA files begin with 0xDEAFD00D
that's all that I've found so far.

[DarkTitan]

so what program will open a model in a CMDL file?

[Elpenguino]

as of right now, none.

but eventually, the format will be figured out and a viewer will be written.

[DarkTitan]

alright, I never noticed that. so the uncompressed textures do indeed work with the program. heh...

so the files that start with a 1 are already compressed (how can i view the texture if its already compressed)? Does this mean the program that thakis posted will not work with these. also will the bti dds converter program work on a compressed file (the files starting with 1, then put into the bti converter program)?

[Elpenguino]

all files that begin with 1_ use an (as of now) unknown compression format. there is nothing we can do with these right now.

so no, textures that begin with 1 cannot be used with that texture converter.

[SarahHarp]

Just found this thread today, because just recently I've gotten into cube emulation, because I wanted to see if I could get data out of the Prime games.. and appearnly it's possible.. kinda. So I'm just wondering if anyone is still working on this?
And I would LOVE to help.. only thing is that am not too sure on how to make programs that read unknown files. Though I have made programs before (heck, I even make video games.) so... yeah.
I've also been dieing to have the samus model from this game.. ho man..! So if anyone has gotton the CMDL file cracked please inform me : )

-Sarah

[thakis]

Hey Sarah, you could try to reverse the CMDL format yourself. If you're experienced with 3d graphics, it's really not that hard. Get yourself a hex editor (I use frhed with some patches) and look if you can find a small (uncompressed) CMDL file. Check if you can see some byte blocks that look like floats, this could be coordinates. Write a program that reads theses floats and displays them as points - repeat until it looks correct. Then get back to us ;-)

I did the same with super mario sunshine files, search the "szs format explained" thread for bmdview2 + source and take a look at vtx1.h/cpp to see how mario stores vertex data - there really is not much to it.

[SarahHarp]

Ok thanks, (Heh I was hoping you'd respond, your an inspiration :) )

[SarahHarp]

What would happen if there wern't any uncompressed files? All the CMDL's start with 1 (=compressed) O_o; Would I have to uncompress them, how would you do that?

[thakis]

Hmm, that won't be easy.
You could compare compressed and uncompressed files of the same file type (for example, "ELSC" in Metroid1). You know that the compressed file has a fixed header, so try to make up a decompression scheme that yields this known header. The compression method could even be some public compression format (zlib, ...).
Another approach would be to disassemble the .dol (dol is a gamecube .exe), step through it in an emulator (dolwin is open source, I don't know if dolphin comes with a debugger) until you find the piece of code that does the decompression and copy it. (yagcd has documentation on the .dol file format).

Good luck ;-)

[BlueFalcon7]

i read somewhere recently about file extensions that windows uses a 3 letter file extension and mac uses a 4 letter file extensions (with exceptions like .dmg and .mp3) so i wonder if metroid prime was made with a mac and then archived for purposes of reducing header clutter, and so it can go into an iso and also for file type help, theres filext.com

[thakis]

Metroid itself doesn't use any file extensions, they are added by mpakdump. And it's not true that windows uses 3 letter extensions - back in the DOS days, only three letters were supported. Nowadays, three letters are the most common, but there are four or more letters as well (.html, .class, .manifest, ...). And .png files have the extension .png on Macs as well.

[BlueFalcon7]

thats sort of what i meant, i was not clear that windows also uses 4 letters but i read in the index of the windows XP help and support center and it says "In the Macintosh environment, a four-character sequence that identifies the type of a Macintosh file. The Macintosh Finder uses the file type and file creator to determine the appropriate desktop icon for that file."

[thakis]

Hey,

I had a some spare time today, and I managed to write a decompressor for the compressed files. It was rather easy, the files are simply compressed with zlib. I attached it to my post which has the other two programs (mpakdump, mtexdump) attached as well (8th post in this thread).

The .CMDL format doesn't look to hard to reverse once it's uncompressed, so give it a shot...if you are interested in computer game file formats, this is a really good opportunity to get started :flowers:

EDIT: if you see a file which has 0x78da near the beginning, it's pretty certainly a zlib file (with maximal compression).

[Elpenguino]

...and there was much rejoicing. good work, thakis.

[SarahHarp]

Wow, I knew I should've tried zlib... though I thought it couldn't be that easy...

Anyways thank you so much ^_^

--Sarah

[boxing]

I don't have any MP1 files for testing but I do have MP2. The mpakdump tool still works. But the 1_***.txtr files are no longer compressed the same way it seems. The first 4 bytes are still the decompression size but it's like there's no other header info. Any ideas? Or a single MP1 1_***.txtr file for comparision that I could use to see the differences?

[thakis]

Get mdecomp, drop 1_***.txtr on it, and you get a 01_***.txtr that should work with mtexdump.

[Dirtie]

Hmm, do you reckon your pak decompressor could work for Ultimate Spiderman as well?

[thakis]

Try it. I don't think it'll work, though.

[fraug]

Has anyone tried to pull the sound effects from these files at all?

[Elpenguino]

mdecomp works with metroid prime, but not metroid prime 2...

[SarahHarp]

That's correct. They seem to have used a differnt decompression algorithm. However I'm mainly concerned with the first MP so you'll just have to wait till I make it there : ) (Even if you did uncompress mp2 stuff, I'm dont think there's much you can do with it, as everything (assuming cmdl's are differnt aswell) is in a differnt format (think of the TXTRs... ) You could always try to decompress it yourself ; )

Oh and no, the sound effect area hasn't been touched. I looked into the matter for a quick bit, and I noticed they are basicly arranged by type (all Thardus sounds are in one file, all Flaahgra sounds are in another, etc).. I'm pretty sure sound files use the AGSC file type.)

[boxing]

Hey,

So I started playing with MP1 data with the tools in this thread. Cool stuff. I'm mostly interested in the textures right now.

mpackdump -> mdecomp 1_* -> mtexdump -> btidump

I've found that all the textures with no mipmaps come out incorrectly swizzled. The colors look correct but the 8x8 tile arrangement is not.

Example.

Incorrect
Metroid2.pak -> 01_1a40e64a.TXTR is 256x256 with 1 MipMap, 32,780 bytes


Correct
Metroid2.pak -> 01_1a4a2d1a.TXTR is 256x256 with 6 MipMaps, 43,692 bytes

I've played with the code and I'm not sure at which step is the problem. My *guess* is it's the bti->dds step since there's some liberties taken assuming mipmips and flags.

Thakis? Have a chance to have a look?

Cheers

[thakis]

Hey, thanks for the detailed bug report :-)

I'll take a look.

edit: Are all 1-mipmap-images broken? Btw, that looks like a problem I had with some of windwaker's world map bti images...

[boxing]

Hey Thakis,

Yup all 1 mipmap textures are broken. The 8x8 data blocks look correct but the arrangement of them doesn't seem proper. I hope it's the same fix as you mentioned. It would be nice to keep playing today!

Cheers

[thakis]

Some textures are in a different format than dxt1. I added support for all formats I found to mtexdump, I'm going to post an updated version soon. I found a small bug in btidump during the process as well. I am now able to convert _all_ .TXTR files found in mp1 to dds without a crash :-P

dxt1 compressed files without mipmaps still don't work, though.

edit: if someone cares, here's a table of mp1's image formats:
//0 - i4
//1 - i8
//2 - i4a4
//3 - i8a8
//4 - idx4
//5 - idx8
//(there's no file with format 6)
//7 - r5g6b5
//8 - rgb5a3
//9 - r8g8b8a8
//10 - dxt1

Format 4 doesn't completely work atm, though (the rest works). Also, it seems as if all mp1 textures are stored upside down.

[SarahHarp]

Excellent work, Thakis : )

[BlueFalcon7]

i have seen image gallery 1 and 3 in the game, i never got image gallery 2, i cant wait to look at it on my PC, and anything on metroid prime 2 textures? also anyone ever notice how close metroid prime is to halo PC with the halo .map files, someone should just make a spark edit for metroid prime

[boxing]

Cool update. I'm still more curious about the single mipmap DXT1 textures... I still play with the swizzling every once in a while but can't find the proper unpack.

Hopefully soon.

[thakis]

Here's the updated texture converter..."soon" is a long time sometimes :-P

edit: txtr files in format 4 don't work with this version... :-|

[BoggyB]

*delurk*

Nice program thakis! I'm currently writing a PAK explorer thingy in VB based on your code. It's not much at the moment - all it does is list the files in there. I plan to add a hex viewer and dumper to it, and maybe a texture viewer... see how it goes.

One thing I've come across while coding: did you know there are duplicate entries in the pak files? For example, in Metroid1.Pak on Metroid Prime, entry id 0x00392179 tag TXTR is listed twice. It appears that your program overwrites the file extracted from the first entry with the later one. I only found out as I was using the id as a unique key.

[thakis]

I didn't notice it, or I can't remember that I did notice. Do the two keys refer to different data chunks as well?

[BoggyB]

They refer to different sections in the file. Haven't checked if the contents of the two locations are actually different.

[erix920]

Can you guys work on one for mp2? I've been dieing for a converter!

[SarahHarp]

Once the compression algorithm for MP2 has been cracked, work can begin on it (actaully, I'm pretty sure it uses the identical file structure as MP1) so basicly, once the decompression has been finished, the viewers and everything used for the first Prime will work for it aswell.

[BoggyB]

Argh, the cable modem ate my last post.

Basically, the .PAK format in MP2 is the same, and can be extracted with the tools for MP1. I think it still uses zlib compression as well. However, the format of the files inside is not the same. I haven't checked that many, but I know that the TXTR and STRG formats are different to how they were in MP1.

[erix920]

Nice, when you try to decompress MP2 txtr files it says error in the decompressor. However the pakextarctor works beautiful.

[BoggyB]

Hmm. I could have sworn it was standard zlib, but then again I may be mixing up files from the two versions.

[Elpenguino]

It isn't. I don't have a clue what it is, though. I'm not exactly a genius when it comes to compression formats.

[erix920]

The one thing that annoying me is there's is like 40 seperate texture for one thing, example, samus's gun...
http://gmod.vamphost.com/images/models.JPG

[SarahHarp]

That's becasue each of them must be overlayed over a group (or groups). It's just the way Metroid Prime does things I suppose. It might be because the models mainly all use 256x256 sized texture, combing all the textures for a particular object might result in a really large image size, so they minimize size by having smaller, but more textures.
SSBM also does this, but to an even greater extent. A typical character model from MP has about 8 groups, using about 14 textures (which includes shaders, specular maps, etc), while in smash bros, the trophie Young Link has 52 groups, using 39 textures.


Lol and that's not the best example you posted, because I only see 3 textures, not 40 ; )

[erix920]

There are atleast 40 files in that folder. There may be 3 groups buts there arn't three textures.. Can't wait for that mp2 extractor :).

[Someone!]

Super Smash Bros Melee doesn't have PAK's and stuff...
How can I extract models and textures from there?

[SarahHarp]

Textures are located inside the model files. Models files (though incomplete still) are read by my Smash Bros viewer. Texture support is only at about 80% though.

[Someone!]

Where can I find the viewer?
Is there an exporter? Beacuse I can't do it with GCMTool (unknown formats and such, dunno which ones are the models)

[SarahHarp]

The Metroid viewer is still under development. You're going to have to wait (for how long? I'm not sure) Until I get the texture placement sorted. Then there's not much left to do with the models, therefore the viewer can be released.

[Psychomax]

How different are the .pak files in Metroid Prime 3? Is there an extractor for those yet?

[antidote]

I looked into it and they do look similar, but i haven't actually tried to figure out how it works exactly yet (due to my lack of experience) I think it may be possible to make minor modifications to the current decompressor. Sarah harp has apparently cracked this format already so ask him.

[Phazonaddict]

Sarah harp has apparently cracked this format already so ask him.
Her :P
But yeah, she has, she had the MP3 Samus loaded pu so that would insinuate she had it working. She has not been on in quite a while though...

[antidote]

Her :P
But yeah, she has, she had the MP3 Samus loaded pu so that would insinuate she had it working. She has not been on in quite a while though...

Hate to break it to you but Sarah Harp is male.

I know where he shows up the most, and is currently still active, however I'm not going to tell you where he goes as that is not my business.

[Phazonaddict]

I thought I recall SarahHarp being referred... oh fail on my behalf.

[Psychomax]

Yeah, I PM'd him a few weeks ago, and still no response. Maybe he's just going to wait until he releases the CMDL viewer; didn't he say he wanted to make it a texture viewer as well?

[antidote]

What he told me was that he is tired of the people on this forum constantly nagging him for a release and that he was to the point of abandoning the project and just leave it as is and unreleased. So our best bet is to just leave him alone. When he's ready he'll release it.

[Psychomax]

Oh, I guess I'm not the only one PMing him, then :I

Ok, I can wait. There are people here who have been waiting longer than me, anyways.

edit: This may be kind of a general question, but where would I start if I wanted to make the new .pak extractor myself?

[antidote]

Break down the file, find out how it works by comparing to known formats and seeing the differences between them, use thakis' source from his .pak decompressor as a base, then compare it to other files from the same game (I'm assuming MP3 here) and then try to vary the source a bit to compensate for the differences and possible solutions for those difference. Although the compression scheme may be different from MP and MP2:E as is the trend.

[Ness64]

Hello,i want to start into Emulation too and have several questions:
I think you use C++ or C# to make these programms?
I am litle bit confused so...
What do i need to get those textures ? Do i need the *.gcm of prime? Or the *.iso file?
And it would be really helpull to make it like a tutorial because its not that easy to understand!

Oh and if its a iso file,did you ripped of it with this sever thingy programm to acces ngc to computer or not? DONT ask how only if you did it with the server programm ;)


Sorry if i didnt notice some sentences where you described to to use them...
MFG NESS

[Sercio]

It doesnt matter. The image can be .gcm or .iso both are exact the same! There are tools to explore and extract those images so you can maybe browse trough the files and find some textures... Search for GC-tool !
Good luck!

[Phazonaddict]

I think you use C++ or C# to make these programms?

What do i need to get those textures ? Do i need the *.gcm of prime? Or the *.iso file?

Oh and if its a iso file,did you ripped of it with this sever thingy programm to acces ngc to computer or not? DONT ask how only if you did it with the server programm ;)

If I recall, the programs were made using C++. However, I don't know anything about C#, but I assume it'd easily be doable with that too if it's anything like C/C++.


You need to break the .gcm/.iso (same thing, different name) up into the game files.
Inside that there are .pak files.
The utility in this thread breaks a .pak up into the individual files within.
Each file will start with 1_ or 0_. If it starts with a 1_ you need to use the decompressor program somewhere around this section.
Finally, there is another program to change .TXTR to .bti files. Then you can change the .bti to .dds, if I recall. BAM, you then have the textures. (again, somewhere in this section)



There are multiple ways to backup your games, some involving the Gamecube, some involving the Wii, some involving JUST your PC. However, I don't think we're allowed to discuss ripping methods here, but you can easily find them online.

[Ness64]

There are multiple ways to backup your games, some involving the Gamecube, some involving the Wii, some involving JUST your PC. However, I don't think we're allowed to discuss ripping methods here, but you can easily find them online.

I said dont say how you rip them.. ;)
So i think i did not bad things lol

Ok to these programmers...
I would want to help too but i cant figure out your source code :down:
What does it do in the code? Could someone explain it? Would be really nice...

PS: BTW there doesnt exist model extractors..? I read that it have to be programmed?