Rice's Plugin Source <discussion>

[toe_head2001]

This is a really cool plugin update, thanks!

 

[mudlord]

Are there patent issues on using MD5?

That is something I'm not sure about.

MD5 is just plane amazingly acurate, it is however extremely intensive on the CPU.

I agree with ya there, though Jabo's plugin does texture dumping and loading pretty quick...

If Jabo can provide the format information, I don't think conversion is impossible

That's exactly the whole trouble. Jabo's texture packs use only MD5 for texture identification for hi-res info. There is no real format specs. Everything is done by the MD5 checksums, and so there is no data specifying in the filenames whether its a CI texture or not. Thats because the MD5s are all unique, like you elaborated on about MD5's strengths. And so, the checksums are the only identifying things (which I can see why then Jabo made the format simple for his plugins, since there is only one method, MD5, and only one image format, PNG).

There is no easy solution, the portability issue brings up the morass of the library DLL needed for texture loading and dumping as well.

The good news is, I made it no longer a bloated library . It now only loads the formats we use (BMP/JPG/PNG), and all other format support is removed. Now, its actually quite petite and slim. And the specs for BMGLib talk about cross platform compatibility being quite good.

I was thinking that it would offload creating the XML information reguarding the files and allow in real time a way of seeing what images are being dumped in game. IE a tree of the data and file list to allow you to pause the game and examine the textures dumped. That might help people with making TP's also.

Oh alright

Wow, 1 to 10 ^12% chance? Last, I looked, 255 ^ 4 equals roughly 10 ^ 9 (4 228 250 625)!

Holy crap! I guess then there will be zero issues with texture usage then....heheheheheh.

 

[Legend]

Hey Mudlord, are you planning on helping to get Rice's Video compatable with PJ 1.7 again. I'm not sure what happened with this but as you probably know, fullscreen hasn't worked for awhile and causes a full system lock. I'm just not sure where Zilmer and yourself stand on this issue so that's why I'm inquiring. Thanks as always.

Also, I noticed in Brunswick Pro Bowling, the floors are missing with your "stable" version. With identical settings, tried the original Rice's build and it works perfect.

 

[Cyberman]

Wow, 1 to 10 ^12% chance? Last, I looked, 255 ^ 4 equals roughly 10 ^ 9 (4 228 250 625)!

1 billion = 9 digits
1 trillion = 12 digits. Please keep your numbers correct.
Also it's not 255^4 it's 256^4 which is the same as 2^32 which is 4294967296.
Just because you know something doesn't mean you shouldn't think about what's being said first .

1 billion is about the limit of CRC32 I'm parinoid enough that I would use it for small spans. 2^25 leaves enough headroom in probabalistic issues of the CRC algorythm. IE ~32meg.

That's exactly the whole trouble. Jabo's texture packs use only MD5 for texture identification for hi-res info. There is no real format specs. Everything is done by the MD5 checksums, and so there is no data specifying in the filenames whether its a CI texture or not. Thats because the MD5s are all unique, like you elaborated on about MD5's strengths. And so, the checksums are the only identifying things (which I can see why then Jabo made the format simple for his plugins, since there is only one method, MD5, and only one image format, PNG).

Hmmm if the MD5 is computed off the same textures AND you can generate a set of CRC's on the same data then it's a non issue. As long as the information that you are generating the check information on is the same it will be consistant. If the information changes then things will be different and incompatible.

I suppose the MD5 data can be dumped out with the CRC data? Then one can take the TP's made for Jabo's plugin and compare the data with the XML database information generated by Rice's new and improved plugin. That's the reason I wanted to stream the data out that way. It might make porting JABO's TP's easier. What size MD5 is he using?

Cyb

 

[Doomulation]

1 billion = 9 digits
1 trillion = 12 digits. Please keep your numbers correct.

1 milliard = 9 digits
1 billion = 12 digits
Apparently the Americans twisted the word "billion" to replace milliard and to bump down the whole scale. In many other languages, milliard comes between million and billion.

Also it's not 255^4 it's 256^4 which is the same as 2^32 which is 4294967296.

My mistake, yes... I'm used to numbers 0 through 255 because that's what you'll get on a computer. But of course that's 256 different possibilities.

1 billion is about the limit of CRC32...

1 milliard And about 4 milliard possibilities.

 

[Cyberman]

You mean this?
If I remember correctly it was not used in US English because it (the use of the milliard system) introduced even more irregularity to the language. IE it created more inconsistancy (which is just what everyone needs inconsistant mathematical terms). I suppose from now on Giga should be used hence forth. At least a googleplex is still a googolplex

Cyb

 

[mudlord]

Hmmm if the MD5 is computed off the same textures AND you can generate a set of CRC's on the same data then it's a non issue. As long as the information that you are generating the check information on is the same it will be consistant. If the information changes then things will be different and incompatible.

I suppose the MD5 data can be dumped out with the CRC data? Then one can take the TP's made for Jabo's plugin and compare the data with the XML database information generated by Rice's new and improved plugin. That's the reason I wanted to stream the data out that way. It might make porting JABO's TP's easier. What size MD5 is he using?

Yeah, I can agree with you there. As long as we try to output the same MD5's as Jabo's, then this might occur. However, I haven't got a clue on how Jabo performs his texturing, and I'm sure he has his reasons for not disclosing info like that. Which means basically, the hardest part is getting the MD5's the same.

But then we have your idea of the data base for cataloging textures to help simplify the process of porting them. Which means, for those logging functions, I need to check out the real nitty gritty of how best to do it it seems. I would imagine of making changes to the basic texturing core to do this right in both APIs.

As for info on the size, he made a very vague remark of "a simple md5, nothing fancy". Which means, my guess its the default 128-bit hash size. However, I do want to respect his wishes of not having his texture pack format a standard, which means this functionality will be import only, when and if its added.

 

[Sirmatto]

A heads up, on this thread Hacktarux was reminded of when he originally promised to release his port of the Rice code So again, he says he'll clean it up and release it, which may or may not help you out, Mudlord. But maybe someone will take it and merge the Linux code into yours.

 

[jackschmidt]

Are there patent issues on using MD5?

As far as I know, md5 is open. The algorithm is freely available and there are tons of code posted online that do md5. That makes either the patent open or not patentable anymore.

 

[Cyberman]

Yeah, I can agree with you there. As long as we try to output the same MD5's as Jabo's, then this might occur. However, I haven't got a clue on how Jabo performs his texturing, and I'm sure he has his reasons for not disclosing info like that. Which means basically, the hardest part is getting the MD5's the same.

Rice's plugin does it with the data that's loaded right? (IE when the program attempts to load it from the ROM into the texture RAM).

But then we have your idea of the data base for cataloging textures to help simplify the process of porting them. Which means, for those logging functions, I need to check out the real nitty gritty of how best to do it it seems. I would imagine of making changes to the basic texturing core to do this right in both APIs.

Well I was thinking of dumping the CRC/MD5 data and use a seperate program to handle that. IE a simple CRC32 MD5128 filepath for each texture dumped. I'm not sure which format is best for dumping etc. in terms of images.

As for info on the size, he made a very vague remark of "a simple md5, nothing fancy". Which means, my guess its the default 128-bit hash size. However, I do want to respect his wishes of not having his texture pack format a standard, which means this functionality will be import only, when and if its added.

Are you against a more databased oriented approach? I've been looking at SQLite (which is a VERY small database engine). It might make packaging the whole texture thing MUCH much simpler (and rather easy to play with).
The images can be stored as BLOBs (Binary Large OBject) and all the information about the textures IE CRC32 MD5 sum etc can be stored with them. Empty BLOB references mean use the old texture for example if there is a Database match. It also means later one can add images as well.
Let me see this leaves it as 2 databases 1 set of BLOB information (CRC MD5 BLOB) and 1 set of texture information (CRC MD5 height width depth and anything else useful such as original file dump path etc.). The idea is that one won't have to think about putting the textures in the right directory and finding the image file that needs replaced would be a snap as well. End result is fewer file fiddling and simplifying the plugin's view of things. Interesting thing about SQLite is it's quite small. You can prune it down to 100K I've header. Package it with both the texture manager and the plugin things could be quite easy. Old TP's can be converted using the texture manager likely.

As for indexed images (IE paletted images) how does Rice's plugin handle those?

Cyb

 

[mudlord]

A heads up, on this thread Hacktarux was reminded of when he originally promised to release his port of the Rice code So again, he says he'll clean it up and release it, which may or may not help you out, Mudlord. But maybe someone will take it and merge the Linux code into yours.

thats good news!

As far as I know, md5 is open. The algorithm is freely available and there are tons of code posted online that do md5. That makes either the patent open or not patentable anymore.

Ok. Great.

Rice's plugin does it with the data that's loaded right? (IE when the program attempts to load it from the ROM into the texture RAM).

for that, it does it when the textures are dumped and loaded. I'm not sure if it processes it on TMEM, yet.

Well I was thinking of dumping the CRC/MD5 data and use a seperate program to handle that. IE a simple CRC32 MD5128 filepath for each texture dumped. I'm not sure which format is best for dumping etc. in terms of images.

Oh alright, thanks for clearing it up.

Personally, I'm not against the approach of using a database format for using textures since there obviously seems to be great benefits for the end users. And for the indexed images questions, there seems to be two different criteria when dumping: the plain CI images (in BMP), and the CI by RGBA (PNG). What seems redundant to me is though why shouldnt both criteria be both PNG? As for the specifics on palleted image use, i will have to get back to you on that.

 

[Legend]

The crashing problems started with build 47, and its threading changes.

So, as consolation, versions before that particular build will work.
for that bowling bug, try this. it fixes dkr and zelda, so it could potentially fix your bugs.

The BMG file is not valid on either PJ 1.7 and 1.6. I have to use the old BMG and then it doesn't see you version. Please check it out.

Also, are you saying that because of the new threading changes it would involve alot of change to RiceVideo to get to work.

EDIT:Also, the "Development" one does not boot roms, it is selectable in the vid plugin section, but does not work.

 

[jackschmidt]

And for the indexed images questions, there seems to be two different criteria when dumping: the plain CI images (in BMP), and the CI by RGBA (PNG). What seems redundant to me is though why shouldnt both criteria be both PNG?

I'd like to participate but I'm not familiar with Rice's code and I am no N64 emulator developer so pardon me if it's off base and I sound like a crackpot. Wouldn't bmp produce totally uncompressed images that would at least yield images without sacrificing any detail? I do think there are subtle differences between PNG and BMP though I cannot remember it right now.

I don't know if this helps but this link gives a comparison between some of the different image formats.
http://www.iceteks.com/articles.php/imageformats/1

 

[mudlord]

The BMG file is not valid on either PJ 1.7 and 1.6. I have to use the old BMG and then it doesn't see you version. Please check it out.

That is plain weird. The new BMGLib is not compatible with old rice video versions, however that doesnt excuse it not working on your system. It all works for me...have you got the MSVC2005 runtimes installed? I'' probably compile a different version. And I did test that bowling game, and it works perfect now...like the old rice video builds...

Also, are you saying that because of the new threading changes it would involve alot of change to RiceVideo to get to work.

Probably, it might just be a single fix, OR it might require completely overhauling how the plugin functions just for PJ64 1.7....

EDIT:Also, the "Development" one does not boot roms, it is selectable in the vid plugin section, but does not work.

Odd....works for me fine when using shaders and otherwise...Might as well compile another version for that too....

 

[mudlord]

Wouldn't bmp produce totally uncompressed images that would at least yield images without sacrificing any detail? I do think there are subtle differences between PNG and BMP though I cannot remember it right now.

The thing is, PNG is 100% lossless. So, PNG will yeild the exact same quality as BMP, and we also enjoy compression from it, which will make texture packs smaller . And then since PNG supports colour indexing, that could mean BMP will be made redundant.

 

[jackschmidt]

The thing is, PNG is 100% lossless. So, PNG will yeild the exact same quality as BMP, and we also enjoy compression from it, which will make texture packs smaller . And then since PNG supports colour indexing, that could mean BMP will be made redundant.

You bring up a good point. I thought PNG was lossy. My bad. It does seem BMP is a bit redundant. Maybe PNG was an after-thought implementation?

 

[Legend]

The "Stable" release has no BMGlib so the plugin is not selectable. The "Development" plugin with its DX9 file is also not selectable in the plugins, whereas before it was-so it worst now. Also, your info about "update 8" is the same as the info for "update 7".

 

[mudlord]

The "Stable" release has no BMGlib so the plugin is not selectable. The "Development" plugin with its DX9 file is also not selectable in the plugins, whereas before it was-so it worst now. Also, your info about "update 8" is the same as the info for "update 7".

I believe you are mistaken..

Those DLLs no longer require BMGLib.DLL. To put it in plain English, BMGLib is inside the plugins. Thus, the plugins still use BMGLib, however, the end user no longer needs to manually put the DLL in the emulator folder, since ALL BMGLib code is now inside the plugin, and not as a external component anymore.

Also, for the DX9 plugin, you must install a very recent version of DirectX or put that d3dx9_30.dll in the emulator root directory, like Direct64.

As for versioning, it would be helpful to note that those are two completely different code branches. One uses a pure DX9 base, while the other (the stable branch and the one in the main "trunk"), uses the same basic codebase as Rice's last public release.

 

[Cyberman]

Personally, I'm not against the approach of using a database format for using textures since there obviously seems to be great benefits for the end users. And for the indexed images questions, there seems to be two different criteria when dumping: the plain CI images (in BMP), and the CI by RGBA (PNG). What seems redundant to me is though why shouldnt both criteria be both PNG? As for the specifics on palleted image use, i will have to get back to you on that.

First the primary motivation using the database approach is too make manipulation of the data easier, which makes the end users happier. Dealing with 5000 textures is a pain. If they are all neatly packed as BLOB information and retrieved for the database program from a big database I think things will be easier all around. Less disk space, less disk fragmentation, fewer files, less mess all these benifit the programmer and the user. Downside is that SQLite is a 2.5m code chunk (yes 2.5 meg .C file), but it's not likely you need to do much with that.

At the high level end of things the database files can be created for each ROM the plugin sees and the images can be dumped. I know duke nukem zero hour had 6000 images and I only made it half way through the ROM. I suspect much of this can be eliminated by tacking an incrementing index to each image record. Merging the format to say dumping only PNG and allowing JPG and PNG for HR textures is likely a good idea. For HR textures the program just needs to look for a database in the highrez texture directory with the correct name. If an image isn't in that database then it's loaded normally. Obviously when a game is started this database will need to be spun through to find all 'hit' values for the CRC or MD5.

Anyhow I am babbling again.

Cyb

 

[Legend]

OK, I now have Microsoft Visual C++ 2005 redistributable installed. I have all the right files in the right place and are using the DirectX9.0c. I still cannot even see the dlls in the plugin selection. Niether the stable or the dev. in any emu, even after a restart of the computer. What am I missing here? Does no one else have this issue? Do I need to sign up and get hold of Visual C++2005 SP1 or something?

EDIT: I can select the DEBUG version just fine, though when trying to load a rom I recieve an error message:"Cannot initialize DX8, please check Direct settings".

EDIT2 - I just also installed .NET framework 3.0 and the very latest DirectX runtime components and I still cannot view either the "Stable" or the "Development" files in the plugin chooser. I also still get the same error when trying to boot the "debug".