hello,
guess i'll chime in here. i have plans to get back to this format as well as the Metroid Prime and Brawl formats again soon.
You want to look for patterns within the binary data of the files. Compare multiple files and see if there are any consistent similarities. It is often best to start with small files of the same type, and if necessary comparing against a medium to large file to see if any of those assumed consistencies can be confirmed. Header information will usually be contained at the beginning of the file so you can often make some good assumptions from the first 10s of bytes or so.
For formats that are completely unknown you will have to start making assumptions about values within the files, then take steps to attempt to verify or deny the viability of those assumptions. Scan the file for things you can readily prove....counts of data entries, consistent placement of data patterns, etc. i.e., If you find a section with a list of string data, attempt to count the number of strings manually, then check for that count somewhere in the determined header information. The same goes for data patterns.
Since the file needs to be read by the game there is more than likely going to be some way for it to determine where data is located within the file. The most common way of doing this is by explicitly listing an offset somewhere withing the header or other data structures contained within the file. This is usually the biggest back and forth area when it comes to attempting to interpret the data, because given the nature of file formats you are likely to have to decide whether a file represents a count of some sort or an offset within the file. As far as offsets are concerned you have at least on piece of information right off the bat that help only slightly in validating offsets, that's the file size. If a value is larger then the possible file size, it is highly unlikely it is a valid offset (if its reasonably close, and you have suspicions the file is compressed, it could be the uncompressed size, but that's for another topic, heh). Another common thing that can help identify file offsets is the tendency for data to be stored aligned to the size of various computer data type sizes and quite often power of 2 boundaries. This is not always the case, but it can at least initially rule out a lot of values that do not fall within even numbers. 4, 8, and 16 byte alignments are quite common.
Since the files should usually follow some sort of structured layout, once you have assumed or determined a file offset, time to see whats there. And since multiple file should follow the same or very similar structure, you should be able to find the same or similarly inclined offset in another file of the same type. Then its on to comparing the data at those locations, and the cycle repeats, with things being verified, disproved, leading to new assumptions being made, thus more verifications, and so on. Not all formats include every count or offset, and it can often be inferred from the counts that are known, and the sizes of the data itself, will have to experiment in some cases to determine the difference.
Determining the intent of data values you find after you have exhausted offset and count validations, can be a bit tricky. Any hex editor worth using should allow for you to visually see how data at a particular offset is/would be interpreted when assumed to be various data types (bytes, shorts, longs, floats, doubles, etc.), though i guess if you want you could always work it out by hand or in your head if you mind works that way, heh. Having an idea of the type of data you are looking for and how it may or should be structured can be very helpful here. in particular for this case, the gamecube/wii has a very particularly way of receiving its data for rendering, and may be something you want to look into should you be inclined to look into more games on these systems.
Sorry to have been so long winded in getting to this point. Now as to the Melee formats in question. The *.dat file start off pretty similarly in terms of structure. The header is 0x20 (32) bytes long, and all file offset are relative to the end of the header (thus they all need 32-bytes added to them). Determining this comes from file comparisons or even better if you can find any related loading code in the game executable. Also of note is that all the data values are in Big Endian, thus you should be able to read them fairly easily from within a hex editor, but unless you are on a computer using the same format, the data will need to be byte swapped before it can be interpreted correctly.
If you look at the first 4 bytes of the files, and start off assuming it is offset related, you will notice from your first round of of offset validation checks that it is exactly the same as the file size. Cannot really have data located at the very end of the file, so it is probably not an offset, but since it is so close to (exactly) the actual size of the file, it is probably not a count of some sort either, but the actual file size stored explicitly in the header data. The next 4 byte value after that you might notice is smaller, yet quite close to the previous file size value. You could then attempt to go to that value offset in the file and see whats there, but you have to make sure to take into account the fact that all offsets are relative to the end of the header, so adding 32 to the value will probably be a bit more useful to finding a proper location to investigate. You need to also make sure to take this into account when investigating any other assumed offsets you want to look into, as well as if you want to check whether or not some data you have found in the file has its offset listed somewhere else.
Like i said, i have not gotten back to finishing this format yet, so there are still a number of things i need to look into. i may have a bit more information, i will see what i can dig up.
Hope that helps.
