MD5 codes of good roms (idea for n64 emulators)

[Hacktarux]

I am currently writing the ini file format of mupen64 and searching what info should i put into it. I think that it should be usefull if the emulator can detect good and bad dumps. And maybe, other emulator authors can be interested by implementing it in their emu (actually it is more usefull in other emu than mine :D ).

Here, you'll find the md5 codes of almost all the good roms (thanks smiff for the help :) )

[Hacktarux]

And source code of the little utility that have generated this file.

[Slougi]

Sounds nice, just one question:
All those codes were for the z64 format, what if the rom is in another one? Wouldn't that change the MD5 code completely?

[Hacktarux]

My utility swap roms to z64 format before calculating the MD5 code.

[Slougi]

What i meant is that if the end user has say, a rom in V64 format, will it still recognize it?

[Hacktarux]

If an emu authors want to implement it, he will have to do the same : byteswap to z64 format and calculate the md5. (Or modify the md5 source code so that it swap each word before using it) It is not hard to do but it depends on how the loading process is actually done. This process is done internally, there's no need to change the actual format of the file in the hard drive.

For example the smiff's conker's bad fur day rom in z64 format has the same code as mine in v64 format. (it is one of the roms i have verified).

I don't think it will be long to calculate it, but i think the end user have to be able to disable it.

[Slougi]

Ah sorry, I didn't get your point correctly! /slaps self
So basically the rom would always get byteswapped to z64 and after that compared to the MD5 code? How would that work for rom browsers?

[Hacktarux]

I depends on how it is implemented on the emu :
- it can be check when the rom is actually loaded (the emu warn the user and ask if he wants to continue)
- it can be done when the user change the directory
- it can be done by an option included in the menu that check a file or a directory.

Or ... any other suggestion ?

[flow``]

hacktarux..


i've got the current n64 rom set (-2 unreleased)

if you want to give me a binary (since i have no idea how to compile.. i'm a non-coder :P) i'd be happy to generate a list for you.

[Hacktarux]

The binary is included in the source code (you need the .exe and the z.dll file). The utility browse automatically all the subdirectory and zip files.

[Doomulation]

So basiclly, what you're saying is that, the CRC should always be "right" byteswapped?

[Hacktarux]

No, what i meant is that my utility give the same CRC with all the known rom format (of course it has to be good roms to have the same CRC !)

[flow``]

that wont work.

many bad dumps have the same crc's as their good dumps.

if you dont believe me, get all of the 1080 bad dumps for example and load up pj64 - it gives the same entry for most of the bad dumps since they all share the same crc..

[Hacktarux]

No !!!

I don't use the CRC stored in the rom. I calculate the MD5 fingerprint of the whole rom. If you change only one bit in the rom, the result is compleatly different. I'm sure that this is how goodrom identify the roms but the algorithm is not publically available and it is not implementable in the emulators.

If you don't believe me, try my program on different bad dumps !

edit: my answer to Doomulation was ambiguous because i used the word CRC where i had to use MD5 :(

[flow``]

once i get a bigger hdd and am able to unzip them all (~27 gigs) i'll give it a try and send you the list :)

[Hacktarux]

My program perfectly handle roms in zip file !
It will just take more time...

[flow``]

hrm.. and its crashing when i double click the executable in my rom directory :/

the only files i took out of the zip were the exe, z.dll, and i put in my zlib.dll (1.1.4) just to make sure

[Slougi]

run it via command prompt, u need to specify the rom directory, that's why it seems to crash. really it just gives the error msg and closes, so windows in it's stupidity closes the dos window as well.

[Hacktarux]

yes, it's just a little command line tool, once it will be run in a few rom collection, it will be useless, so i have not taken time to make a gui.
It will produce a log file (n64roms_md5.txt) in the directory where you launched it.

[Cyberman]

Originally posted by Hacktarux
yes, it's just a little command line tool, once it will be run in a few rom collection, it will be useless, so i have not taken time to make a gui.
It will produce a log file (n64roms_md5.txt) in the directory where you launched it.

I wonder if a 64 bit CRC would be a good public substitute for the MD5 codes? I am leery of using algorythms that are patented for things that are GPL or integrated into something that's free. :)

The big problem with CRC's is picking the right start word and the polynomial for generating the CRC. although 64 bits will take 8 tables of 256 elements to create and might be slow it should have adequate Uniqueness for rom sizes up to 4gig.. 32 CRCS are only guarenteed unique to 64K that's why you can have bad roms with the same CRC's as good roms because the ROM images are large enough the probability of finding a match is significantly increased.

I'm not sure why the hell Nonmindo used 32 bit CRC's for such large roms (sigh).

Cyb

[Slougi]

I currently have my roms on cd's and not much hdd space, if i do em one cd a time can i just copy the stuff in the text files over into a single one?

[flow``]

ok..

will just the good dumps, pds, and jfg [f1] be enough for you?

or do you want the whole collectoin (-2 unreleased)

[flow``]

here ya go

[bluef]

64 bit CRC wouldn't be a great substitute. But if you're worried about that then there are plenty of other algorithms around.

MD5 is licensed freely by RSA however and is even implemented in the GNU application md5sum (part of textutils i think) and is also used in the open source ids tripwire. All they ask is that you acknowledge that you have used their algorithm.

[Slougi]

Here are the MD5 codes for my (U) roms

[Slougi]

and (E), (PD), and (J)

[Hacktarux]

OK, i think that i have enough codes for good roms, i have not check all the things, but it seems that everything is right : a good rom has always the same code and if only one bit is altered, the code is totally different.

I don't think we need to have bad roms code. I you see a reason, let me know.

And cyberman:
Usually, i don't use patent code but i don't see any problem in this case. This algorithm is used in many free products and it is free even in a commercial product. And one last thing, i am using an independant implementation that can be found in sourceforge.

[Doomulation]

It appears to me that every rom has a diffrent code.
It would be the same to take the CRC code and check.

[flow``]

no...

many bad dumps share the same crc values are there corresponding good dumps..

[Doomulation]

Oh...
Bah! Too confusing, I'll just use the old fasion way of using the CRC :P

[Hacktarux]

Originally posted by Doomulation
Oh...
Bah! Too confusing, I'll just use the old fasion way of using the CRC :P

I don't think you understand what i am trying to make.

I am not searching to know how to identify a game, i am searching how to detect if a rom is a good or bad dump. I think it is usefull to integrate it in emulators because the goodn64 tool is not easy to understand by a newbie. Maybe, i'm wrong but i can see in this message board that it's the origine of many questions.

[Cyberman]

Originally posted by Hacktarux


I don't think you understand what i am trying to make.

I am not searching to know how to identify a game, i am searching how to detect if a rom is a good or bad dump. I think it is usefull to integrate it in emulators because the goodn64 tool is not easy to understand by a newbie. Maybe, i'm wrong but i can see in this message board that it's the origine of many questions.
Hmmm how about a 'good' plugin for that? IE an optional plugin for emulators that allows them to send the data for a rom to the plugin and indentify if that rom is good or not. This prevents having to have each emulator author code it into there emulator. Simplifies using the MD5 code as well since the plugin would do all the detection and calculation.

Cyb

[flow``]

no.. the old fashion way was usingt he rom headers! :P

(nemu, pagan, uhle, corn, and a few other oldies)

[Hacktarux]

Originally posted by Cyberman

Hmmm how about a 'good' plugin for that? IE an optional plugin for emulators that allows them to send the data for a rom to the plugin and indentify if that rom is good or not. This prevents having to have each emulator author code it into there emulator. Simplifies using the MD5 code as well since the plugin would do all the detection and calculation.

Cyb

I thought that the codes could be integrated in the current ini files. To calculate the message digest, there's only 3 line to add in the code :
md5_init(&state);
md5_append(&state, (const md5_byte_t *)buf, size);
md5_finish(&state, digest);
The app needs to be linked to the md5 library and buf is the rom in the z64 format.

But if an emu author ask me to make a dll i can do one :P

Generally, i prefer plugins, but for this, i don't know what is better : loading a plugin and calling the function will probably need more lines of code than implementing it directly.

[pj64er]

dont do plugin, what other alternative can there be to Detecting/Sorting by GoodRoms? the point of a plugin is to allow many options, ie gfx (jabo, glide64, daedalus, linker's, orkin's, 1964's, TR64's)

[Doomulation]

Originally posted by Hacktarux


I don't think you understand what i am trying to make.

I am not searching to know how to identify a game, i am searching how to detect if a rom is a good or bad dump. I think it is usefull to integrate it in emulators because the goodn64 tool is not easy to understand by a newbie. Maybe, i'm wrong but i can see in this message board that it's the origine of many questions.
I very well understand what you are trying to make, but it is still very confusing to me.
My utility as you can see below, has the ability to check if it's a good dump through a database with CRCs. And at least one of my users are missing like 20 known n64 roms, so that's no problem.

[Hacktarux]

Hey doom, i don't have anything against you, but your crc check is totally useless. I have just check your utility and it just extracts the crc code stored in the rom header. + you have made useless work because these code were in the pj64 rdb file. So your utility just check 8 bytes of the rom, if these 8 bytes are right, you assume that all the other bytes are right !

MD5 technic checks every bytes of the rom, if you change one bit in the file, the result code is totally different.

[Doomulation]

Originally posted by Hacktarux
Hey doom, i don't have anything against you, but your crc check is totally useless. I have just check your utility and it just extracts the crc code stored in the rom header. + you have made useless work because these code were in the pj64 rdb file. So your utility just check 8 bytes of the rom, if these 8 bytes are right, you assume that all the other bytes are right !

MD5 technic checks every bytes of the rom, if you change one bit in the file, the result code is totally different.
True, I could do it your way, but I am confused on how it works.
It would be good to just compile a damn DLL of which could check this and return true or fals depending on whether its good or not.

and btw: pj's rdb for example has a list of if it good or bad, for example. By checking the CRC!
The rom-header CRC, nothing like yours.

[Hacktarux]

I know that pj64 can detect some bad dumps but generally it can't. Maybe i will make a dll, but if not, here is what you have to do :
you load the rom in a buffer.
then you byteswap the rom so it is in z64 format : it has to begin with 0x80 0x37 0x12 0x40
you calculate the md5 code of this buffer (there's many dlls to do that available on the net in the language you want)
and then you compare the result with the codes you can find in the files posted in this thread

Hope it helps you to understand that.

[Doomulation]

Originally posted by Hacktarux
I know that pj64 can detect some bad dumps but generally it can't. Maybe i will make a dll, but if not, here is what you have to do :
you load the rom in a buffer.
then you byteswap the rom so it is in z64 format : it has to begin with 0x80 0x37 0x12 0x40
you calculate the md5 code of this buffer (there's many dlls to do that available on the net in the language you want)
and then you compare the result with the codes you can find in the files posted in this thread

Hope it helps you to understand that.
I understand THAT.
But generally, what I don't understand is how to calculate the code.

And about the DLL, it can call c/c++ dlls. Just needn't do much trouble like saying FileP for argument of something that can do as a long...

[Hacktarux]

Here, is a md5 library in visual basic, if you don't understand how to use that, i don't know what to say....

[icepir8]

Hacktarux:

Great Idea. who is going to build the list of MD5s for good roms?

[Reznor007]

Originally posted by Doomulation

I understand THAT.
But generally, what I don't understand is how to calculate the code.

And about the DLL, it can call c/c++ dlls. Just needn't do much trouble like saying FileP for argument of something that can do as a long...

Just do a search for MD5 on Yahoo or something, and you can probably find something about the algorithm.

[Hacktarux]

Originally posted by icepir8
Hacktarux:

Great Idea. who is going to build the list of MD5s for good roms?

Smiff has already done one, you can find this list on the first post of this thread. You can also find on this thread two lists built by flow and slougi, i have to check if there is any difference between these files, i don't think there's any difference but it has to be checked.

[Doomulation]

So I'm just gonna call MD5File with a filename and it returns the MD5 code?

And that code is CLEAR as hell. I don't understand how anybody CAN'T understand it.

[rob in london]

doomulation i think you need to buffer it instead of calling it with the filename, as unless you byte-swap it so they're all in the same file format, the same rom will generate different codes depending on how the data is arranged

[Doomulation]

Originally posted by rob in london
doomulation i think you need to buffer it instead of calling it with the filename, as unless you byte-swap it so they're all in the same file format, the same rom will generate different codes depending on how the data is arranged
Yeah, well, I was wondering 'cuz there was no explanation.

[Doomulation]

Hacktarux,
you've got a problem.

The process takes TOO much time! Takes forever just to load about 40 roms in the utility using it.

And secondly, trying this on Yoshi's Story (J) [!] returned a value of which i couldn't find in the list attached at the start.

[Hacktarux]

What you mean by forever ?
Is it with your vb utility or with my little command line tool ?
Remember that Smiff and others have launched it on ~1000 roms and maybe they have to wait a few hours but they have not waited forever.

I have scan 50 roms over a lan and it takes about 10 mins.

For the Yoshi's Story problem, i have to ask again if it is with my tool or with your program that you obtain this value ? If it is with my tool, i'm almost sure that you have a bad dump. If it is with your program, check that you have all the bytes loaded and that you have correctly byteswapped it in your program.

[StrmnNrmn]

Calculating an nd5 digest of each rom seems like an excellent method or verifying good and bad dumps to me. True, it's slow (especially over a full set of roms), but you can be quite clever and cache the results in a database, so you only need to calculate the results once (or until the rom changes on disk).

[Doomulation]

Originally posted by StrmnNrmn
Calculating an nd5 digest of each rom seems like an excellent method or verifying good and bad dumps to me. True, it's slow (especially over a full set of roms), but you can be quite clever and cache the results in a database, so you only need to calculate the results once (or until the rom changes on disk).
I've also thought about caching it, though, it still takes forever to load. And that must be changed.
Just loading an application and you have wait hours before you can use it? No thanks.

Hacktarux: I did it all with your tool. And yoshi's story was (J) [!], which means a good dump.

[Hacktarux]

Originally posted by Doomulation

I've also thought about caching it, though, it still takes forever to load. And that must be changed.
Just loading an application and you have wait hours before you can use it? No thanks.

Hacktarux: I did it all with your tool. And yoshi's story was (J) [!], which means a good dump.

Firstly, we don't have to process all the roms before running the emu. We just have to check a rom when it is loaded and when the code is not in the cache. (If we want to add an option to check all the roms, we have to warn the user that it will take time)

Secondly (J) [!] doesn't mean that it's a good dump. It only means that someone have checked the rom with goodn64 and it was a good dump. But the file may have been altered since by transmission errors...

[StrmnNrmn]

Why can't you just calculate the md5 digest when you need it - i.e. when the user starts to run the rom? The interface could just show "n/a" for roms that had not been checked. You could then add an option to scan all roms ("Warning, this operation may take several minutes"..), or check them using a lower-priority thread in the background.

But really, if you want to know for sure whether a rom is a good or bad dump, I don't see any option but performing a hash of every single byte of the file. What's going to take the time is the reading from disk (and decompression if zipped) - not the choice of hash function (md5, crc64 or whatever).

[Doomulation]

Originally posted by StrmnNrmn
Why can't you just calculate the md5 digest when you need it - i.e. when the user starts to run the rom? The interface could just show "n/a" for roms that had not been checked. You could then add an option to scan all roms ("Warning, this operation may take several minutes"..), or check them using a lower-priority thread in the background.

But really, if you want to know for sure whether a rom is a good or bad dump, I don't see any option but performing a hash of every single byte of the file. What's going to take the time is the reading from disk (and decompression if zipped) - not the choice of hash function (md5, crc64 or whatever).
Actually, dave gave me a c++-compiled DLL of which can perform the same thing to compare the whole file in a speed of about 40 roms in 2 seconds when unzipped.

When zipped, however, it took 260 seconds.
And about the md5...visual basic does not support multi-thread. VB .net does however, which is what I current have and is trying to learn.

And I did think of using what you said StrmnNrmn, by using a thread in the background to do this work, and to try to check a special rom when chosen to, and updating the gui when one rom has been checked.

This feature requires me to have a CRC database, but that's no problem.

[Hacktarux]

I need some explanation about your dll ! For me such performances are impossible.

proof:
to execute a hash algorithm whatever it is and how much you optimize it, it needs to read all the bytes of each roms.

If we consider that each roms have 16 Mb length on average. Then 40 roms * 16 = 640 Mb : nearly the size of a cdrom.

And you say that you read a whole cdrom in 2 seconds !!!!!!!

Congratulation doom, your hard drive is able to burn a cdrom at 1083x :D

Try to calculate the maximum speed in nero and you'll see that it is impossible....

[Doomulation]

Originally posted by Hacktarux
I need some explanation about your dll ! For me such performances are impossible.

proof:
to execute a hash algorithm whatever it is and how much you optimize it, it needs to read all the bytes of each roms.

If we consider that each roms have 16 Mb length on average. Then 40 roms * 16 = 640 Mb : nearly the size of a cdrom.

And you say that you read a whole cdrom in 2 seconds !!!!!!!

Congratulation doom, your hard drive is able to burn a cdrom at 1083x :D

Try to calculate the maximum speed in nero and you'll see that it is impossible....
Dave reported this. I haven't tested myself. Ask dave.

[ShadowPrince]

Seems,that he mean only time needed for calculating md5 sum,Hack.Without loading rom time,althought even this not explain record time of 2 sec :).May be 20 sec ?

Anyway i compiled the list of good codes for everyone who's
intrested.Based on the great project 64 rdb file. (slightly changed)
If someone have more complete list of roms with goodn64 sums ,than please leave me private message.(i can't make goodn64 work on my machine).Best to post here scan list of goodn64.

I have two problems with current list :

1.[FC70E272-08FFE7AA-C:50]
Good Name=Spacestation Silicon Valley (E) OR (F) [!]
Good Code=
2.[2F700DCD-176CC5C9-C:45]
Good Name=Turok - Dinosaur Hunter (U) (V1.1) OR (v1.2) [!]
Good Code=
If someone have those 4 roms,please post here their checksums in md5 format and goodn64 format.