PDA

View Full Version : Back to Metroid Prime hacking



interdpth
September 10th, 2009, 08:09
Hello, my name is interdpth, and i've been scouring through the net for data on those CMDL's and stuff. So far I can just draw the points of a model but I can do that pretty good as you can see
http://interdpth.arc-nova.org/SAMUS.png

But that's all I got so far.
I have the basic CMDL header that shalted guy posted. I see the unknown section has patterns in it, i've had a little help from Sarah Harp, and hoping to get more.

If anyone has any data they can share. PLEASE DO. This is my first 3D app, you may know me for my MZM and MF editors Double Helix and Zero Fission.

So help? XD


I found how to reference the texture files. But that's my only progress so far. Still better than nothing.

Sercio
September 10th, 2009, 22:07
Saraharp didnt finish his job. There is no release of his viewer for over 2 years now :(
Curious if something might happen...

interdpth
September 10th, 2009, 23:25
The more help I get, the sooner my own can be released. Currently it just reads CMDL's compressed or uncompressed it auto decompresses and reads just the first segment. If anyone wants to help, I can show my CMDL header and tell how I read it. @_@

ANY HELP is appreciated especially how to tell what a segment is and stuff i'll be here all week. =p

I'll have it loading directly from a pak when I can get the models displaying, than directly from the games! ^_^

Fluesopp
September 12th, 2009, 12:14
Have you looked at the code for Thakis' BMD viewer? It might give you a good idea of how to do the rendering.

interdpth
September 14th, 2009, 03:33
Yeah, i've looked, i'll be using it for rendering help.

But first I need to get the data figured out @_@

and I can't get in touch with the almighty Thakis i've heard about.


Edit:

If anyone could give me details on how to load textures and map them, that'd be lovely :)

revel8n
September 30th, 2009, 22:24
Hello All,
New here but hopefully my first post will be worth the read.

i have been interested in game file format reverse engineering for a while now, and after getting accustomed to things over on the xentax.com forums, i finally got around to looking into some of the various nintendo related games.

Since Metroid Prime was one of the games i had intended on getting into at some point in the future anyway, with the renewed (continued?) interest in the topic, i offered to help interdpth out with his endeavor to reverse the formats. Having started a little before the beginning of last week we have had a good bit of progress on things, so i am here with my first post to update everyone on how things are going.

So far the CMDL file is mostly done with concern to determining where the vertices, normals, uv, and index information resides. Models can currently be rendered without textures. The texture file information is known and a version that supports at least preliminary support for them should hopefully be done in the near future. For the moment the main things missing from this format is the determination of the texture stage information for blending, texture coordinate usage in multitexture, and some of the data that determines the number of indices present in the index data.

There is also a start on the CINF and CSKR skeleton and skinning information data, but this still needs a bit of testing with the model data, and there is a version of the CSKR format that contains some extra information that still needs to be deciphered as it is not present in all CSKR files. Hopefully the ANCS and ANIM formats will come to follow soon after.

Still trying to get more accustomed to the way the gamecube and revolution handle things as this is the first gamecube format i have worked with. But hopefully things will come together nicely in the end.

i have probably missed some things, but oh well, heh.

http://img177.imageshack.us/img177/5093/2782648e.th.png (http://img177.imageshack.us/i/2782648e.png/)
http://img183.imageshack.us/img183/3332/578c5d12.th.png (http://img183.imageshack.us/i/578c5d12.png/)
http://img62.imageshack.us/img62/9397/202ac74c.th.png (http://img62.imageshack.us/i/202ac74c.png/)
http://img62.imageshack.us/img62/6879/1e637fcb.th.png (http://img62.imageshack.us/i/1e637fcb.png/)
http://img96.imageshack.us/img96/9264/success.th.png (http://img96.imageshack.us/i/success.png/)
http://img183.imageshack.us/img183/3609/b2b41738.th.png (http://img183.imageshack.us/i/b2b41738.png/)

interdpth
October 1st, 2009, 10:30
Basic texture support has been added.
:)

Some textures like Samus show as all color...
Should have that fixed soon I hope.

I think we'll have a release within the month guaranteed, as revl8n we still need to determine blending and all that, but I think we'll get it done soon, but check out this pirate in the mean time. :)
http://i83.photobucket.com/albums/j318/interdpth/pirate-1.png

MasterPhW
October 1st, 2009, 13:36
It's nice to see, that there is still interest in Metroid hacking and that you already archived that much stuff.
Your IRC chat is also funny. xD
Keep up the good work!

thakis
October 3rd, 2009, 01:53
Good job, guys :-)

revel8n
October 5th, 2009, 00:57
Just a quick update on the status of textures. They work! For the most part anyway. Still a few formats to work out. Now to work out the blending and other assignment flags as it appears some of the textures are used to specify specular or other appearance attributes for rendering.

http://img59.imageshack.us/img59/1799/textures.png

antidote
October 7th, 2009, 02:29
Looks interesting, when do you think you'll be able to post the program?

interdpth
October 7th, 2009, 05:07
Possibly after we get this segment figured out and the mesh headers, I'd like to not release it until we have basic support for all the files in all the games, but, revl8n has his own say, he can release whenever he's ready as I can too. So who knows?

But it will be released, and it will be open source! ^_^

if someone could come on and tell us how the material segment is laid out that'd be really awesome, the stuff after what texture what segment uses. >_> Attribute data and the like

antidote
October 7th, 2009, 19:32
Sarah! oh Sarah! *picks up rock*

hmm... sorry can't find him...

Wish I knew how to help you, i'll take a look but I know next to nothing about image formats.

interdpth
October 12th, 2009, 09:31
Got blending and all that stuff working. So new screenshots.
Tomorrow i'll take a look at material defs seeing as how I now know lighting is controlled in there.

The viewer is officially called MPxViewer.
Revl8n is taking a look at the animation stuff right now, hopefully we can finish CMDL soon, and move on to other things.

There's a surprise at the end XD

http://i83.photobucket.com/albums/j318/interdpth/betagravity.pngBeta Gravity
http://i83.photobucket.com/albums/j318/interdpth/insideship.png Yes, there is more to the inside, it's just that circle textured. We never see inside it...I wonder why they felt the need to texture it
http://i83.photobucket.com/albums/j318/interdpth/planet.png
http://i83.photobucket.com/albums/j318/interdpth/powersuit.png
http://i83.photobucket.com/albums/j318/interdpth/varia.png

Basic MP2 support.
Material Definitions and other bugs are plaguing MPxViewer for now, support will be added of course. :)
http://i83.photobucket.com/albums/j318/interdpth/lololol.png

Fluesopp
October 14th, 2009, 13:50
MP2 support? Nice! Keep up the good work!
Any plans for MP3 too?

antidote
October 14th, 2009, 20:47
*Drools* blending

Well looking good keep up the excellent work

interdpth
October 15th, 2009, 00:12
Yes of course there will be MP3 support! ^_^

Honestly i've had enough of the CMDL format, we'll probably get back to eventually, but models render amazingly.

But in this update, you'll see that we have a tree view now, it needs a bit more work. XD

But raw texture viewer is now implemented, and a skeleton viewer is too! ^_^

http://i83.photobucket.com/albums/j318/interdpth/Chozo.png
http://i83.photobucket.com/albums/j318/interdpth/Flaahgra.png
http://i83.photobucket.com/albums/j318/interdpth/Morphball.png
http://i83.photobucket.com/albums/j318/interdpth/Skeleton.png Samus's skeleto
http://i83.photobucket.com/albums/j318/interdpth/TextureViewer.png some cool looking texture.

antidote
October 15th, 2009, 19:56
*GLOMPS* I luv you Interdpth!

interdpth
October 26th, 2009, 08:15
Update:
http://i83.photobucket.com/albums/j318/interdpth/Wee-1.png

We also have some support for MREA's but this is the one I wanted to show. ;)

antidote
October 27th, 2009, 06:54
mp3 lzo compression = cracked



//version 2.0 (20061001)
//by thakis

//decompresses compressed metroid prime files.
//metroid prime 1 uses zlib compression,
//metroid prime 2 uses segmented LZO1x (thanks to SkippyJr
// for pointing this out)

//zlib decompression is handled by http://www.zlib.net/,
//LZO code (lzo.h/c) was taken from ffmpegs libavcodec.

#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <vector>
#include <string>
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include "zlib.h"
#include "lzo.h"

using namespace std;

typedef unsigned char u8;
typedef unsigned short u16;
typedef unsigned int u32;

char* fname;
vector<u8> dst;

inline void toWORD(u16& d)
{
u8 w1 = d & 0xff;
u8 w2 = (d >> 8) & 0xff;
d = (w1 << 8) | w2;
}

inline void toDWORD(u32& d)
{
u8 w1 = d & 0xff;
u8 w2 = (d >> 8) & 0xff;
u8 w3 = (d >> 16) & 0xff;
u8 w4 = d >> 24;
d = (w1 << 24) | (w2 << 16) | (w3 << 8) | w4;
}

#pragma pack(push, 1)

struct HeaderMP1_2
{
u32 uncompSize;
//mp1 has 0x78da (zlib stream with maximal compression)
//mp2 has something else :-P (lzo doesn't seem to have a header)
u16 compressionMethod;
};

struct HeaderMP3
{
char id[4];
u32 unk; // version maybe?
u32 unk2; // Hash?
u32 uncompSize;
};


#pragma pack(pop)
int decompressZLib(u8* source, int sourceSize, u8* dest, int dstSize)
{
int ret;
z_stream strm;

// allocate decompression state
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = sourceSize;
strm.next_in = source;
ret = inflateInit(&strm);
if (ret != Z_OK)
return ret;

//decompress input
strm.avail_out = dstSize;
strm.next_out = dest;
ret = inflate(&strm, Z_NO_FLUSH);
assert(ret != Z_STREAM_ERROR); /* state not clobbered */
switch (ret)
{
case Z_NEED_DICT:
ret = Z_DATA_ERROR; /* and fall through */
case Z_DATA_ERROR:
case Z_MEM_ERROR:
(void)inflateEnd(&strm);
return ret;
}

assert(strm.avail_out == 0);
assert(ret == Z_STREAM_END);

// clean up and return
(void)inflateEnd(&strm);
return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
}

int decompressLZO(u8* source, int sourceSize, u8* dest, u32& dstSize)
{
int size = dstSize;
int result = lzo1x_decode(dest, &size, source, &sourceSize);
dstSize = size;
return result;
}

void WriteFile(FILE * inFile, u32 uncompSize, int size = 0)
{
if (size == 0)
{
u32 bytesLeft = uncompSize;
while (bytesLeft > 0)
{
u16 segmentSize;
fread(&segmentSize, sizeof(segmentSize), 1, inFile);
toWORD(segmentSize);
vector<u8> src(segmentSize);
fread(&src[0], 1, segmentSize, inFile);

int result = decompressLZO(&src[0], segmentSize,
&dst[uncompSize - bytesLeft], bytesLeft);

if ((result & LZO_ERROR) != 0)
{
cerr << "Failed to decompress \'" << fname << "\'" << endl;
system("pause");
return;
}
}
}
else
{
vector<u8> src(size - 4);
fread(&src[0], 1, size - 4, inFile);
fclose(inFile);

if (decompressZLib(&src[0], size - 4, &dst[0], uncompSize) != Z_OK)
{
cerr << "Failed to decompress \'" << fname << "\'" << endl;
return;
}
}
//write buffer to output file
string inName = fname, outName;
string::size_type pos = inName.rfind('\\');
if (pos != string::npos)
outName = inName.substr(0, pos + 1) + "0" + inName.substr(pos + 1);
else
outName = "0" + inName;

FILE* outFile = fopen(outName.c_str(), "wb");
if (outFile == NULL)
return;

fwrite(&dst[0], 1, uncompSize, outFile);
fclose(outFile);
}

void decompressMP1_2()
{
FILE* inFile = fopen(fname, "rb");

//get file size
fseek(inFile, 0, SEEK_END);
int size = ftell(inFile);
fseek(inFile, 0, SEEK_SET);

//read file
HeaderMP1_2 h;
fread(&h, sizeof(h), 1, inFile);
toDWORD(h.uncompSize);
toWORD(h.compressionMethod);
fseek(inFile, -2, SEEK_CUR);

//decompress to buffer
dst.resize(h.uncompSize);
if (h.compressionMethod == 0x78da)
WriteFile(inFile, h.uncompSize, size);
else
WriteFile(inFile, h.uncompSize);
}

void decompressMP3()
{
FILE* inFile = fopen(fname, "rb");

HeaderMP3 h;

fread(&h, sizeof(h), 1, inFile);
toDWORD(h.unk);
toDWORD(h.unk2);
toDWORD(h.uncompSize);

dst.resize(h.uncompSize);

WriteFile(inFile, h.uncompSize);
}

int main(int argc, char* argv[])
{
if (argc < 2)
return EXIT_FAILURE;

FILE* inFile = fopen(argv[1], "rb");
if (inFile == NULL)
return EXIT_FAILURE;
fname = argv[1];
u16 tmp;
fread(&tmp, sizeof(tmp), 1, inFile);
toWORD(tmp);
fclose(inFile);
switch (tmp)
{
case 0x434d:
decompressMP3();
break;
default:
decompressMP1_2();
break;
}

return EXIT_SUCCESS;
}


currently the code is rather rough because i JUST cracked it and haven't had a chance to clean up the source and add support for the other formats back. so currently it can only decompress metroid prime 3 files

EDIT:
cleaned and reorginized
Download here: mDecomp.zip (http://www.mediafire.com/download.php?mdzd4dnggcd)
EDIT2:
Hmmm it seems that TXTR files use a different compression scheme, i'll have to look into that.

revel8n
October 27th, 2009, 16:01
Thanks for the update. i had started but hadn't gotten around to finishing looking into this after my update the mpakdump source code. i'll take a look at things and see if i can see what all is going on.


And well since i'm here, i guess i'll:
http://img193.imageshack.us/img193/8229/mreatextured.png

antidote
October 27th, 2009, 18:12
it was rather simple XD, all they really did was add the ID and version and remove the compression method (well version 1 anyway)

EDIT:
Pirate Frigate FTW!

antidote
October 29th, 2009, 07:40
We have cracked All 3 versions of metroid prime 3's Compression

so for your viewing pleasure:
http://img408.imageshack.us/img408/8739/zerosuit.png

EDIT:
as well as this:
mdecomp.zip (1.29 MB) (http://www.mediafire.com/?0zzwtndjndu)

interdpth
October 30th, 2009, 19:06
My answer to that is with a question.

Why?

antidote
October 30th, 2009, 19:51
um sure but as interdpth asked why?

interdpth
October 31st, 2009, 08:05
Check out the mdecomp source earlier in the thread, it's super easy to reverse. :)

Good luck on your translating, eventually we'll have a compressor and stuff, again good luck!

revel8n
October 31st, 2009, 10:29
Just an added note: The game pak file have flags in the file entry list that specify whether a file is compressed or not. As such it may not be necessary to actually re-compress any files in some cases. As for textures the mtexdump source should be easy to reverse as well to allow for recreation of modified textures and the like.

We will more than likely end up doing tests with recreated pak files as well at some point so it's not too far fetched to as about the possibility of recreating patched pak files. Will have to see how things go, heh.

Hope your translation goes well.

interdpth
October 31st, 2009, 18:30
Well I should have a pak creator up tonight ^^

So tests will be sooner rather than later

antidote
November 1st, 2009, 07:44
well the pak creator fell through due to unforeseen problems, however i did fix a bug in MPakDump, it now properly reads and dumps the Name Table, now don't think this means that every file has a name, it's only for certain files.

interdpth
November 1st, 2009, 19:01
i shan't give up on the pakcreator.

Once VS is installed i'll get back to work

antidote
November 2nd, 2009, 00:35
here is the Modified MPakDump with the MP3 Name Table fix, you'll notice a few changes straight away.
mpakdump.zip (117.93 KB) (http://www.mediafire.com/download.php?k2gxd5yygu0)

Fluesopp
November 5th, 2009, 14:44
This looks absolutely fantastic. Do you have export options available? Maya, 3Ds?

interdpth
November 5th, 2009, 22:07
Due to possible copyright stuff, I don't think we'll be making an exporter maybe an importer XD

But it will be open source so anyone can do whatever :)

pmix2005
November 9th, 2009, 06:14
how much time you need to release this amazing tool?

antidote
November 10th, 2009, 04:20
once it's stable and we have support for most of the formats, Inter and revel8n are doing the programming, I'm helping, rather dismally, to crack the formats, currently the SCLY section of MREA files are giving us a run for our money and that is whats being concentrated on by inter and myself. I'm also reorganizing the code for the utilities so that they can be used in a more efficient manner in the viewer.

antidote
November 20th, 2009, 07:57
Well, we're still around and i'm still working on MP, i took a break from cracking SCLY and looked into FRME and discovered that it is a rather simple format.
it's not quite done YET but here is what i have.



//--------------------------------------
//--- 010 Editor v3.0.6 Binary Template
//
// File:
// Author:
// Revision:
// Purpose:
//--------------------------------------
#include "common-types.bt"

// #pragma byteorder(big_endian)
BigEndian();

// mark used bytes with a light green background
SetBackColor(cLtGreen);
struct FRME_FILE
{
struct FRME_HEADER
{
uint32 unknown0;
uint32 unknown1;
uint32 unknown2;
uint32 unknown3;
uint32 unknownCount;
}fileHeader;

struct FRME_DATA
{
local uint32 i = 0;
for (i = 0; i < fileHeader.unknownCount; i++)
{
struct
{
char tag[4];
//uint8 unk1;
string Name1;
//uint8 unk2;
string Name2;
uint16 unknown1[2];
uint32 unknown2[4];
uint32 unknown3;
switch(tag)
{
case "BWIG":
uint8 unknown4;
uint16 unknownArray[unknown4];
float4 unknown5;
uint8 data[0x32];
break;
case "CAMR":
float4 translateValue;
uint32 unknown5[2];
uint8 unknown6;
switch(unknown6)
{
case 1:
uint8 unknown7;
break;
case 2:
uint8 unknownArray[2];
break;
case 69:
uint32 unknown7[2];
break;
}
uint16 unknown[0x1F];
break;
case "ENRG":
uint8 data[0x47];
break;
case "GRUP":
uint8 unknown4;
uint16 unknown5[2];
uint8 unknown6;
if (unknown5[1] != 0)
float3 unknown7;
else
uint16 unknown7[5];

uint8 unknown8[0x36];
//uint16 unknown8;
//uint8 data[0x48];
break;
case "HWIG":
uint8 data[0x43];
break;
case "IMGP":
uint8 data[0xA7];
break;
case "LITE":
uint8 data[0x63];
break;
case "MODL":
uint32 CMDL;
uint32 unknown4[2];
uint8 unknown5;
uint16 unk[unknown5];
float4 translateValue[2];
uint32 unknown6[8];
uint16 unknown7;
break;
case "METR":
Printf("%s Here %i\n", tag, i);
uint16 unknown4;
uint32 unknown5;
uint8 data[0x47];
break;
case "SLGP":
uint8 data[0x53];
break;
case "TBGP":
uint8 data[0x66];
break;
case "TXPN":
float4 translateValue[2];
uint16 unknown4;
uint32 unknown5[0x08];
uint32 unknown6[2];
uint32 FONT <format = hex>;
uint32 unknown7[2];
uint8 unknownArrayCount;
uint16 unknownArray[unknownArrayCount];
float4 translateValue2[2];
uint8 data[0x22];
break;

}
}widgetInfo;
}
}fileData;

};

struct FRME_FILE fileInfo;

Psychomax
January 4th, 2010, 04:26
the download links for antidote's new versions of mpakdump and mdecomp seem to have expired. Does anyone else have them that can reupload them?

antidote
January 5th, 2010, 21:23
here you go http://antidote.metroidconstruction.com/

Psychomax
January 6th, 2010, 07:12
Awesome, thanks! Is there another place to get the new version of mdecomp, too? The source code on the other page says that it doesn't work with texture files

Andyflower
January 7th, 2010, 07:17
I will get a new dump of the iso, then head to the store this weekend to get me some verbatim dual-layer discs. I used a memorex, it was the last one I had, and didn't think twice about it, 'cause my SSBB memorex works. I'll burn at 2.4x next time, too.

interdpth
January 8th, 2010, 20:27
I'll be uploading the source to MPxViewer to googlecode within the next day or so :)

So anyone will be able to use it.

interdpth
January 8th, 2010, 21:36
MPxViewer SVN (http://code.google.com/p/mpxviewer/)

SVN for it :D

antidote
January 29th, 2010, 23:15
Recheck my site, I have added MDecompress.

interdpth
July 24th, 2010, 09:34
I have resumed partial work on mpxviewer.

So yay

flarespire
April 16th, 2011, 04:12
ok ive been trying to get this to work but im having NO success at all, im trying to make a CMDL to OBJ converter, now whatever ive tried all i get is a spikey vertex mess and im in desperate need of help, if anyone gets this please assist me....