Spotless - find duplicate textures easily

[taotao]

What it is
Spotless is a tool I wrote some time ago to find duplicate textures more easily.

How it works
Simply unpack the .zip inside the folder of the game you want to check.
If you double-click the .exe it it'll generate a file called log.htm which lists all the locations where the same texture has been found.

Known problems
Color indexed files with the same DRAM CRC but with different Palette CRCs are currently identified as duplicates.

Plans for future versions

• don't list textures which differ only by palette crc (should be easy)
• make the program run from the hires-folder by allowing to choose which game to scan, maybe with a simple GUI or as command line tool

Download
Spotless 0.3.1

Any comments and improvement suggestions are welcome.

 

[NES_player4LIFE]

win7 ultimate error... python31.dll missing.

 

[MasterPhW]

The tools sounds like a great idea (and gets me into trouble to finally write a proper tool guide for high res texturing and sticky it...), but it would be really awesome if it also could move the duplicate files into a folder, like f1, f2, f3, f4... fx, so they are presorted already. A lot of the work for retexturing is sorting all files and it could really speed up the sorting this way.
Couldn't test it, because same error message and my python build environment isn't installed here yet.

 

[microdev]

Color indexed files have an additional part in the filename which I don't know what it means.

Which part are you refering to? Do you mean the palette CRC? (the second CRC value in the texture name)

 

[taotao]

win7 ultimate error... python31.dll missing.

Ok, I guess that's embarrassing. I thought py2exe creates executables which are independent from python. There has to be a way but apparently it isn't as simple as I thought.
So I guess for now this needs Python3.1 to be installed.
I'll see what I can do about it.

The tools sounds like a great idea (and gets me into trouble to finally write a proper tool guide for high res texturing and sticky it...), but it would be really awesome if it also could move the duplicate files into a folder, like f1, f2, f3, f4... fx, so they are presorted already.

It's a good idea, I'll think about it. This should be easy to implement.

Which part are you refering to? Do you mean the palette CRC? (the second CRC value in the texture name)

I suppose that's the one.
The thing is, this palette CRC isn't mentioned in Rice's readme where every part is explained and there doesn't seem to be an actual explanation about the dump formats.
So am I correct in assuming that two files with the same CRC but with different palette CRCs don't count as duplicates?

Thanks for the comments.

 

[taotao]

I updated the Download-Link in the first post.
The program now comes with python31.dll which was apparently built by py2exe as well.
I don't know if that fixes it though since I currently don't have a system without Python installed to try this.
So any feedback if this works now will be highly appreciated, thanks.

 

[MasterPhW]

It does "work" now (means, no error message about missing libraries), BUT and that's a big one:
The log.htm is empty., even it's launched in my unsorted MyGlide texture folder.

No texture files
python31.dll

 

[taotao]

Are you sure that there are duplicate textures in that folder?
If there are none it only lists files which aren't textures. The .exe itself is handled as an exception and is not listed. python31.dll isn't handled as exception yet so it will still be listed.
I guess I should implement something like 'No duplicates found'.

Maybe you could make sure that you try this somewhere where you place duplicate textures on purpose to see if it still doesn't give any output.

Thanks for the help so far.

 

[MasterPhW]

Are you sure that there are duplicate textures in that folder?
If there are none it only lists files which aren't textures. The .exe itself is handled as an exception and is not listed. python31.dll isn't handled as exception yet so it will still be listed.
I guess I should implement something like 'No duplicates found'.

Maybe you could make sure that you try this somewhere where you place duplicate textures on purpose to see if it still doesn't give any output.

Thanks for the help so far.

I've checked my already sorted folder and now it work, so it seems, that the folder really had no duplicates. So I have to excuse the "error" report, but I'm wondering about the missing image in the log.

  Spoiler:

Is that the way it meant to be?

 

[taotao]

I've checked my already sorted folder and now it work, so it seems, that the folder really had no duplicates. So I have to excuse the "error" report, but I'm wondering about the missing image in the log.

Which Browser did you use to view the log?
As said in the first post I tried this with Internet Explorer and Firefox but Firefox won't display any image that contains '#' in the filename. This might also be an issue for other browsers.

Thanks for your help.

 

[MasterPhW]

Which Browser did you use to view the log?
As said in the first post I tried this with Internet Explorer and Firefox but Firefox won't display any image that contains '#' in the filename. This might also be an issue for other browsers.

Thanks for your help.

Yeah, same issue with Opera 11.01. I've reported the bug, hope to see it fixed in the upcoming release 11.10 and will report back. IE is displaying it without problems. Sorry for overlooking it in the first post.

 

[microdev]

The thing is, this palette CRC isn't mentioned in Rice's readme where every part is explained and there doesn't seem to be an actual explanation about the dump formats.

Here is the explanation I added as comment to the source code of 1964Video (in TextureFilters.cpp):

		/*
			
			(CASTLEVANIA2)#(58E2333F)#(2)#(0)#(D7A5C6D9)_ciByRGBA.png
			(------1-----)#(---2----)#(3)#(4)#(---5----)_ciByRGBA.png

			1. Internal ROM name
			2. The DRAM CRC
			3. The image pixel size (8b=0, 16b=1, 24b=2, 32b=3)
			4. The texture format (RGBA=0, YUV=1, CI=2, IA=3, I=4)
			5. The palette CRC

			<internal ROM name>#<DRAM CRC>#<24bit>#<RGBA>#<PAL CRC>_ciByRGBA.png
		*/

So am I correct in assuming that two files with the same CRC but with different palette CRCs don't count as duplicates?

Yes, this is correct. Both CRCs (as far as available) as well as the format and bit-size are used to identify the correct texture. (well, concerning the palette CRC, only the first 6 bytes are used actually - but that shouldn't matter here)

See the source code snippet along with my explanations for further details:

	// crc64a = <DRAM-CRC-8bytes><FFFFFF><format-1byte><size-1byte>
	crc64a |= (0xFFFFFF00|(entry.ti.Format<<4)|entry.ti.Size);
	// crc64b = <DRAM-CRC-8bytes><palette-crc-6bytes (lowest 2 bytes are removed)><format-1byte><size-1byte>
	crc64b |= ((entry.dwPalCRC&0xFFFFFF00)|(entry.ti.Format<<4)|entry.ti.Size);

Please be aware that not all textures have a palette CRC.

As said in the first post I tried this with Internet Explorer and Firefox but Firefox won't display any image that contains '#' in the filename. This might also be an issue for other browsers.

'#' has to be replaced by '%23' in the image tags of the generated html page.

 

[taotao]

Here is the explanation I added as comment to the source code of 1964Video (in TextureFilters.cpp):

		/*
			
			(CASTLEVANIA2)#(58E2333F)#(2)#(0)#(D7A5C6D9)_ciByRGBA.png
			(------1-----)#(---2----)#(3)#(4)#(---5----)_ciByRGBA.png

			1. Internal ROM name
			2. The DRAM CRC
			3. The image pixel size (8b=0, 16b=1, 24b=2, 32b=3)
			4. The texture format (RGBA=0, YUV=1, CI=2, IA=3, I=4)
			5. The palette CRC

			<internal ROM name>#<DRAM CRC>#<24bit>#<RGBA>#<PAL CRC>_ciByRGBA.png
		*/

Thanks. That's exactly the info I couldn't find and no-one answered my thread when I asked this a while ago.

Yes, this is correct. Both CRCs (as far as available) as well as the format and bit-size are used to identify the correct texture. (well, concerning the palette CRC, only the first 6 bytes are used actually - but that shouldn't matter here)


See the source code snippet along with my explanations for further details:

	// crc64a = <DRAM-CRC-8bytes><FFFFFF><format-1byte><size-1byte>
	crc64a |= (0xFFFFFF00|(entry.ti.Format<<4)|entry.ti.Size);
	// crc64b = <DRAM-CRC-8bytes><palette-crc-6bytes (lowest 2 bytes are removed)><format-1byte><size-1byte>
	crc64b |= ((entry.dwPalCRC&0xFFFFFF00)|(entry.ti.Format<<4)|entry.ti.Size);

Please be aware that not all textures have a palette CRC.

Alright, thanks for the info. Just out of curiosity: what are the last two bytes for then?

'#' has to be replaced by '%23' in the image tags of the generated html page.

Thanks. I thought I'd tried that though.
I made a small update to the tool so that the html can be properly viewed in other browsers aside from IE.

 

[microdev]

Alright, thanks for the info. Just out of curiosity: what are the last two bytes for then?

You mean the last two bytes of the palette CRC?

Well, those are part of the CRC value. But as the key for looking up the texture is an unsigned int 64, it has only 8 bytes available for encoding information. If you would like to use the DRAM CRC(4 bytes), the palette CRC(4 bytes), the texture format(1 byte) and the bit depth(1 byte) as key, you would need 10 bytes for that. Thus, the last two bytes of the palette CRC are replaced by the texture format and bit depth information for generating the look-up key.

This is not optimal as it bears a slight risk of using the wrong alpha channel texture, if there are two files with exactly the same name besides the last two bytes of the palette CRC. But obviously, it worked fine enough till now.

 

[AkaiZagreus]

Anyone Still Have this?