"Texture Packs" Format ?

[darknior]

Hello everybody

A member of forum write :

Jabo and I have discussed this at length, and it was decided it is not possible.
There are several reasons why:

1) Jabo's plugin uses MD5 checksums, Rice's uses CRC32
2) Jabo's plugin uses non power-of-two textures, Rice's only supports textures that are in a power of 2
3) Due to how the plugins emulate the N64 graphics,its unfeasible for things to be perfect.

So. basically, what you are asking is near to impossible. Sorry.

And i don't understand very well ...
Because i think he answer my question of why there are two formats of PACKs ?

With files like :
0AE21369ED692A6A5F29B13F798E8639.png
or
MARIOKART64#2C95F02E#2#1#6B24A530_ciByRGBA.png

The second format, work good on Project64 6.1.1 B10 with RiceVideoPluggin !
But the first format, i don't understand
He talk about Jabo's on Project64 1.7 Beta but it don't propose me to use the HiresTextures

Who can explain me how to use these Packs ?
Or if it is realy impossible to have one and convert to other one format ?

Thanks

 

[mudlord]

And i don't understand very well ...
Because i think he answer my question of why there are two formats of PACKs ?

With files like :
0AE21369ED692A6A5F29B13F798E8639.png
or
MARIOKART64#2C95F02E#2#1#6B24A530_ciByRG BA.png

The second format, work good on Project64 6.1.1 B10 with RiceVideoPluggin !
But the first format, i don't understand
He talk about Jabo's on Project64 1.7 Beta but it don't propose me to use the HiresTextures

Who can explain me how to use these Packs ?
Or if it is realy impossible to have one and convert to other one format ?

Well the answer is quite complicated to understand. But I'll do my best to explain:

First of all,the biggest technical hurdle is the idenfying method use for the textures. For example the sequence:

0AE21369ED692A6A5F29B13F798E8639

is known as a MD5 checksum. Its a special identifier used to uniquely identify a texture. Due to how MD5 works, its insanely accurate, and can be used to pick up the tiniest changes in files. Jabo's plugin uses this method, solely out of simplicity, because a MD5 checksum is so accurate, that there is no other identifiers needed. And also, Jabo's plugin uses only RGBA format PNG's, in its texture format. And RGBA's can be used as well for normal RGB textures (which is exactly how I compress PNG screenshots with the current version of my plugin), in Jabo's plugin.

Rice's, however, is a bit more complicated.

MARIOKART64#2C95F02E#2#1#6B24A530_ciByRG BA.png

The 1st bit is the internal ROM name. Second bit is the CRC32, and then theres the other part of the CRC32 checksum, followed by the texture format. Rice's plugin uses a variety of different formats, but I'm hoping to simplify things, to make it easier on me and the retexturers. The biggest problem is this identifier method. Its not feasible to go through each texture and change it because it can take quite a long time to convert textures, because of these identifiers. These identifiers are completely different to each other, because of these checksum methods. Secondly, we have another technical issue, and that is the texture format. Jabo's plugin uses only PNG textures. Rice's plugin (currently...:matrix: ),
uses:

* BMP (but this can be ditched altogether, when I port more internal things to use RGBA (like the debugger code),and rewrite the hi res code
* JPG
* PNG (the best format to use)

Which means, the format of the textures HAS to be PNG, and nothing else.

On top of that,we have yet another technical hurdle. And that is the business of non power-of-two textures....

This is another big problem. Due to texture mapping in Rice's plugin, all textures are handled as powers of two. The Nintendo 64, however, uses non-power-of-two textures in cases. Jabo's plugin, also uses non power of two textures. The problem is, going through ALL textures and manually resizing will take a huge amount of work. Add on top of that how different emulation is in both plugins,and we have got quite a problem on our hands.

Which leads me to this: Its not possible currently to convert texture formats between the two different plugins in a easy fashion. I'd say its not worth the effort at all.


~whew....~

 

[SubDrag]

I didn't think the N64 did support (or uses?) non-power of 2 widths? Well, in reality more like non-power of 4 widths. If you've got an stored image that is Width 0x15 x Height 0x15, the game will need to pad the texture to be 0x18 x 0x15, then it will be used. I may be misinformed, or at least Goldeneye does it this way. It's possible it's just for simplicity, but not sure.

 

[darknior]

Thanks "mudlord" for your answer
It is not realy that i want to know but it is interesting lol

I read many more about yesterday also on the board of BETA TEST PJ64 1.7 and i finaly understand where is the problem !
BUT i understand also that there is not solution to convert PACKs at this time

It is not good, because it cut the scene in two parts, PJ64 1.6 or 1.7 !!!
And the work is more slowly like that for everybody ...

 

[Xenobond]

Converting between the two is a time intensive task. You'd have to redump the textures for whatever version you are going to, and then match up the new filenames and formats. I had done this with my perfect dark pack, and although I didn't have a lot of textures made at the time, it was still fairly hectic.

 

[Terranigma]

Yeah, this is going to be annoying for the texture makers, but hey, nothing's ever easy. I wish you guys the best of luck.

 

[mudlord]

Well, in reality more like non-power of 4 widths. If you've got an stored image that is Width 0x15 x Height 0x15, the game will need to pad the texture to be 0x18 x 0x15, then it will be used. I may be misinformed, or at least Goldeneye does it this way. It's possible it's just for simplicity, but not sure.

Hmmm, never knew that..Thanks for letting me know. It could be a microcode thing, too, in how the games handle texture management. But I know Jabo's plugin uses NPOT textures (he said so himself when commenting on current texture issues in his plugin on the official 1.7 beta forum).

 

[Cyberman]

Hmmm, never knew that..Thanks for letting me know. It could be a microcode thing, too, in how the games handle texture management. But I know Jabo's plugin uses NPOT textures (he said so himself when commenting on current texture issues in his plugin on the official 1.7 beta forum).

Jabo is using Direct X correct? PowerOfTwo Textures are primarily an issue with OpenGL? I seem to remember OpenGL liking those better.

Cyb

 

[mudlord]

Jabo is using Direct X correct? PowerOfTwo Textures are primarily an issue with OpenGL? I seem to remember OpenGL liking those better.

Jabo is using DX8, and polling some driver caps for NPOT textures, as well as shaders. With OGL , your right, there is a custom GL extension for handling non power of two textures, but it seems Rice wrote the texture handling code at a time where these hardware features were unavailable....So your right . With OGL, its much less of a mess doing NPOT stuff than DX...

 

[Toasty]

If you had the original texture packs, one ripped by Rice's plugin and one ripped by Jabo's plugin, wouldn't it be possible to automate the conversion of a hi-res pack from one naming convention to the other? For example, a program could compare the original textures ripped from both plugins, then when it finds a match it could look through the hi-res pack for a file with the same name as one of the original textures and rename it to the corresponding (ripped by the other plugin) texture's name. Or am I missing something?

 

[Cyberman]

If you had the original texture packs, one ripped by Rice's plugin and one ripped by Jabo's plugin, wouldn't it be possible to automate the conversion of a hi-res pack from one naming convention to the other? For example, a program could compare the original textures ripped from both plugins, then when it finds a match it could look through the hi-res pack for a file with the same name as one of the original textures and rename it to the corresponding (ripped by the other plugin) texture's name. Or am I missing something?

You indeed are. How do you corelate the data such that you know this texture is MD5 38978923747012320 and this texture CRC 2332122F are indeed the exact same texture. Currently you can't it's being discused a lot and there are some possibilities.

Cyb

 

[Toasty]

By re-hashing the image data from both original texture packs with the same hash algorithm. Let's say you had two original texture packs (these aren't real names, just fake ones to be used as an example):

Ripped by Rice's plugin:
ZELDA#12345678#1#1#12345678.png
ZELDA#12345678#1#2#12345678.png
ZELDA#12345678#1#3#12345678.png
ZELDA#12345678#1#4#12345678.png
ZELDA#12345678#1#5#12345678.png

Ripped by Jabo's plugin:
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE1.png
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE2.png
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE3.png
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE4.png
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE5.png

Let's say you also have a hi-res pack for Rice's plugin with these textures:
ZELDA#12345678#1#2#12345678.png
ZELDA#12345678#1#4#12345678.png

So, couldn't a program hash image data from both of the original texture packs (using an arbitrary hash algorithm) to map the different names to each other? For example (the stuff in italics is what the program would compute):

Ripped by Rice's plugin:
ZELDA#12345678#1#1#12345678.png this image data has a hash of 9876543210 using hash algorithm XXX
ZELDA#12345678#1#2#12345678.png this image data has a hash of 9876543211 using hash algorithm XXX
ZELDA#12345678#1#3#12345678.png this image data has a hash of 9876543212 using hash algorithm XXX
ZELDA#12345678#1#4#12345678.png this image data has a hash of 9876543213 using hash algorithm XXX
ZELDA#12345678#1#5#12345678.png this image data has a hash of 9876543214 using hash algorithm XXX

Ripped by Jabo's plugin:
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE1.png this image data has a hash of 9876543210 using hash algorithm XXX
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE2.png this image data has a hash of 9876543212 using hash algorithm XXX
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE3.png this image data has a hash of 9876543214 using hash algorithm XXX
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE4.png this image data has a hash of 9876543213 using hash algorithm XXX
ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE5.png this image data has a hash of 9876543211 using hash algorithm XXX

Match the generated hashes:
ZELDA#12345678#1#1#12345678.png from Rice's plugin corresponds to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE1.png from Jabo's plugin
ZELDA#12345678#1#2#12345678.png from Rice's plugin corresponds to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE5.png from Jabo's plugin
ZELDA#12345678#1#3#12345678.png from Rice's plugin corresponds to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE2.png from Jabo's plugin
ZELDA#12345678#1#4#12345678.png from Rice's plugin corresponds to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE4.png from Jabo's plugin
ZELDA#12345678#1#5#12345678.png from Rice's plugin corresponds to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE3.png from Jabo's plugin

Rename files in hi-res texture pack:
Rename ZELDA#12345678#1#2#12345678.png to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE5.png
Rename ZELDA#12345678#1#4#12345678.png to ABCDEFABCDEFABCDEFABCDEFABCDEFABCDE4.png

Would that work? It seems to me that so long as the textures ripped by both plugins are the same (even though their names are unrelated) the original textures ripped through both plugins should be able to act as a 'dictionary' for translating names from one convention to the other.

 

[Mollymutt]

Would that work? It seems to me that so long as the textures ripped by both plugins are the same (even though their names are unrelated) the original textures ripped through both plugins should be able to act as a 'dictionary' for translating names from one convention to the other.

The textures are not the same. Jabo's dump everything to one folder, including the ones that in Rice's have a second texture with the alpha layer to match. Jabo's do not have alpha layers, so I don't think your scenario would work. This of course is coming from someone who has no idea what all of that mumbo jumbo you wrote meant.

 

[darknior]

I read it can work, but not for all the pictures because like the two pluggins don't use the same code for translate texture to PNG, some time they're can be different and the program will not working
But if it make already 50 or 70% of the work, it is already fantastic !!!

 

[Toasty]

The textures are not the same. Jabo's dump everything to one folder, including the ones that in Rice's have a second texture with the alpha layer to match. Jabo's do not have alpha layers, so I don't think your scenario would work. This of course is coming from someone who has no idea what all of that mumbo jumbo you wrote meant.

Yeah, I wasn't sure how to explain it. Maybe pseudo-code will convey the idea better. (See below.) If I understand what you were saying, couldn't the program just open both files (color and alpha) from Rice's plugin and combine them together before hashing the image data?

Function ConvertRiceToJabo(RiceOriginalTextures, JaboOriginalTextures, RiceHiResTextures)
[INDENT]RiceNameToHashMap = New Map
JaboHashToNameMap = New Map

ForEach Texture in RiceOriginalTextures
[INDENT]ImageData = Texture.ReadImageFromFile()
HashCode = ArbitraryHashAlgorithm(ImageData)
RiceHashToNameMap[Texture.Name] = HashCode[/INDENT]

ForEach Texture in JaboOriginalTextures
[INDENT]ImageData = Texture.ReadImageFromFile()
HashCode = ArbitraryHashAlgorithm(ImageData)
JaboHashToNameMap[HashCode] = [Texture.Name][/INDENT]

ForEach Texture in RiceHiResTextures
[INDENT]OriginalHashCode = RiceNameToHashMap[Texture.Name]
JaboName = JaboHashToNameMap[OriginalHashCode]
Texture.RenameTo(JaboName)[/INDENT][/INDENT]

 

[mudlord]

so long as the textures ripped by both plugins are the same

.....And theres one of the biggest technical hurdles. Overcoming that, this might be a reality.........Ideas like this sound wonderful on paper, but when you think about it, they are harder than they seem. And when you think of the variables involved, things get much more complicated....I might shed some light on the subject more l8r.

Ciao.....for now..

 

[mudlord]

Okay...

well, this might burst some people's bubbles, but I have a little something to show you how difficult things can be if people want to pursue this idea of a program interconverting between tex formats.

I have attached a simple MD5 calculator I programmed last night. Use this on a Jabo-format texture, and you'll notice something.....

Try it on a Rice-format texture that is about the same as a Jabo-format texture,and notice the MD5 sum given......olice:

 

[Toasty]

One problem is that I don't have access to Jabo's new plugin. (I don't suppose someone would be willing to post a few samples of textures ripped by Jabo's plugin and their Rice-ripped counterparts?) The hash program you posted is kind of crude though. (No offense intended - I know you have a lot of expertise in this area.) It only hashes raw data, so different results could be produced, even if the images themselves are identical, but simply formatted differently.

For example, if I have a PNG and convert it to a 32-bit BMP, the BMP will contain the same image data as the PNG, but the raw binary data that makes up the file will be very different. Even two identical images that are formatted the same way could potentially have differences that don't impact image data. If a program were to actually interpret image data from these files (actually decode the RGBA data for each pixel) and hash that information, it would overcome the deficiency of getting different hashes for identical images.

If image data is not consistent between plugins, (as in, textures from one plugin actually have different color information than ones ripped from the other plugin,) I can see how this would cause problems though. (Is this the case?)

 

[mudlord]

(I don't suppose someone would be willing to post a few samples of textures ripped by Jabo's plugin and their Rice-ripped counterparts?)

Sure thing. I'll post some samples of Djipi's cel shaded OOT texture packs....

The hash program you posted is kind of crude though. (No offense intended - I know you have a lot of expertise in this area.) It only hashes raw data, so different results could be produced, even if the images themselves are identical, but simply formatted differently.

No offense taken ^_^. I know its crude, it only took around 2-3 mins for the base from start to finish, then a while longer to research and implement a working and accurate MD5 algorithm. Indeed, the teeniest changes in files can be easily picked up with MD5, thats just the nature of how the hashs are processed and calculated.

For example, if I have a PNG and convert it to a 32-bit BMP, the BMP will contain the same image data as the PNG, but the raw binary data that makes up the file will be very different.

Yep, very true. Thats the beauty of MD5...

If a program were to actually interpret image data from these files (actually decode the RGBA data for each pixel) and hash that information, it would overcome the deficiency of getting different hashes for identical images.

Ah, I see where you think my app is crude. . I didn't have time to implement the PNG decompression algorithms (too tired), but maybe thats something I can look into at a later date. But theres another important thing I think most retextures and end-users are missing...and I'll elaborate in the next section.

If image data is not consistent between plugins, (as in, textures from one plugin actually have different color information than ones ripped from the other plugin,) I can see how this would cause problems though. (Is this the case?)

Yes, this is exactly the case. The different plugins have different combiner emulation, different blender emulation, different texture cache implementation, different framebuffer emulation (Jabo does all framebuffers in hardware currently), different texture mapping code (a major one currently) and there's the major snag of how the checksums are calculated. Which is basically: is image checksums are done in the TMEM OR in RDRAM. There's some other problems too, which are inherent to the actual emulation. Which means, things can get very complicated if we want exact checksums. And with MD5, which is VERY accurate, things get even harder to get perfect.....

 

[Toasty]

Hmm nuts. Well it was worth a shot. I'll look forward to seeing a few samples as it will give me something tangible on which to try ideas. Thanks for taking time to explain the technicalities. Automating redundant/tedious tasks is one of my favorite things to do, so this interests me a lot.