Rice's Plugin Source <discussion>

[mudlord]

OK, I now have Microsoft Visual C++ 2005 redistributable installed. I have all the right files in the right place and are using the DirectX9.0c. I still cannot even see the dlls in the plugin selection. Niether the stable or the dev. in any emu, even after a restart of the computer. What am I missing here? Does no one else have this issue? Do I need to sign up and get hold of Visual C++2005 SP1 or something?

EDIT: I can select the DEBUG version just fine, though when trying to load a rom I recieve an error message:"Cannot initialize DX8, please check Direct settings".

EDIT2 - I just also installed .NET framework 3.0 and the very latest DirectX runtime components and I still cannot view either the "Stable" or the "Development" files in the plugin chooser. I also still get the same error when trying to boot the "debug".

This is by far the weirdest thing I ever seen...

I'm really sorry you are having issues with it atm, but from my own testing, the plugin loads fine (and works great) for me...I'm starting to get quite stumped where it fails for you, since you have all the dependancies for it currently. And no, you don't have to get a hold of the Visual Studio Service Pack 1, thats only useful with you have the compilers. (so it in no way effects the end-users)

In the source, I found that compiler optimizations caused the most issues. They are the ones that caused the issues with BMGLib (to do with hi-res textures). It strikes me as very strange that the C++ compiler settings are causing the plugin to not work for you..since processor specific optimizations were turned off, while still allowing for SSE. Maybe via IM I can try to fix things for you.

Again, I deeply apologise for those issues.

First the primary motivation using the database approach is too make manipulation of the data easier, which makes the end users happier. Dealing with 5000 textures is a pain. If they are all neatly packed as BLOB information and retrieved for the database program from a big database I think things will be easier all around. Less disk space, less disk fragmentation, fewer files, less mess all these benifit the programmer and the user. Downside is that SQLite is a 2.5m code chunk (yes 2.5 meg .C file), but it's not likely you need to do much with that.

Hmm, that will help with package distribution . But the plugin size will blow out...but meh, I suppose we can trim that code chunk, and up the compression on the DLLs...(but that translate to higher RAM usage, due to DLL compression loader overhead...). And I managed to get my hands on the SQLite library, btw.

At the high level end of things the database files can be created for each ROM the plugin sees and the images can be dumped. I know duke nukem zero hour had 6000 images and I only made it half way through the ROM. I suspect much of this can be eliminated by tacking an incrementing index to each image record. Merging the format to say dumping only PNG and allowing JPG and PNG for HR textures is likely a good idea. For HR textures the program just needs to look for a database in the highrez texture directory with the correct name. If an image isn't in that database then it's loaded normally. Obviously when a game is started this database will need to be spun through to find all 'hit' values for the CRC or MD5.

Hmm, those are nice ideas . Having a log of what textures are actually used definately trims down the file some somewhat. I guess it wouldnt make be much effort to make all texture dumps PNG...well, I see a logical flaw in what you said. You said to "dumping only PNG and allowing JPG and PNG for HR textures"...but that means we need JPG first too :evil: ..but blah, thats not important (which means now, we can ditch BMP). What we need is a rethinking of how to do HR, since its embedding in the graphics base, too, to allow for SQLite implementation.

 

[Doomulation]

And no, you don't have to get a hold of the Visual Studio Service Pack 1, thats only useful with you have the compilers. (so it in no way effects the end-users)

Well, from my knowledge and experience, SP1 seems to want a different runtime module to work properly. The old runtime module, when installed, didn't want to work with SP1 compiled apps.
If it helps anything, I have a link to the up-to-date module: http://www.eluni.net/downloads/vcredist_x86/vcredist_x86.exe

 

[Legend]

YOU'RE THE MAN DOOM!!! Finally it works!! Mudlord you need to post this link in the download thread. I guess the version I downloaded from Microsoft a couple days WAS an older set. When I installed this, it said that it was "removing older files and updating to the newest". So there we go.

The Stable works fine (including Brunswick Bowling). The Development one though still gives me the DirectX9 error upon trying to boot a rom.

 

[Doomulation]

Glad I could help.

 

[Cyberman]

Hmm, that will help with package distribution . But the plugin size will blow out...but meh, I suppose we can trim that code chunk, and up the compression on the DLLs...(but that translate to higher RAM usage, due to DLL compression loader overhead...). And I managed to get my hands on the SQLite library, btw.

SQLite isn't a DLL and it only adds 250K tops with careful removal of features you don't need this can shrink a fair amount. Also you can make multiple tables into a single database. SQLite is not good for HUGE databases, but really this is a small database with at most 10000 elements. We need at least 2 tables for HR TP's maybe 3. A table of valid ROM ID's, a table of HR BLOB stored textures there CRC32 and MD5 data. The advantage of SQLite not being a DLL is that it doesn't ADD as much as a DLL to the code size. At runtime that's a different matter.

The end result though is just 2 databases 1 for the TP and the other is the DUMP database. The TP database will be the only thing end users would need. I don't see the point of compressing the database information though. Namely because the largest volume data (Images) are already compressed.

Hmm, those are nice ideas . Having a log of what textures are actually used definately trims down the file some somewhat. I guess it wouldnt make be much effort to make all texture dumps PNG...well, I see a logical flaw in what you said. You said to "dumping only PNG and allowing JPG and PNG for HR textures"...but that means we need JPG first too :evil: ..but blah, thats not important (which means now, we can ditch BMP). What we need is a rethinking of how to do HR, since its embedding in the graphics base, too, to allow for SQLite implementation.

BMP is a waste of space, implementation wise and storage wise. There is no point unless people are adament on using MS Paint or something for texture enlargement.
Performing a startup listing of tags that will hit (as HR textures to load from the DB) shouldn't be hard. Then the database can be querried for the proper BLOB data on a hit. Since it only does this with things IN the database everything else should be the same. I think the changes wouldn't be over all that major.

On the dumping side of things, I can see some issues. If you check DUMP, then things get more interesting. I believe Rice has a texture cache in memory right? (not in the display card but in the computers memory). I've noticed if I've run a ROM without Texture dumping checked and then checked it, the plugin will begin spining through a huge list of file names. I believe a small adjustment to the current system (such as uniform dump of file type). A seperate tool can be made to extract / view PNG files from the dump database. This same tool can be used for creating the hirez TP's. I suppose as part of the 'processing' chain it can call external programs to edit the textures for the texture pack and add them to it. The dump format likely should have one table for ROM ID's, one table with CRC, MD5, dumped image, and one table with CRC, MD5 and image information (Height Width pixel depth). Perhaps with some tweaking on the tool merging and spliting textures will be possible (IE load screens made up of numerous 320x6 16 bit textures can be edited and enhanced as a single image then automatically split into seperate files for the TP).

Cyb

 

[mudlord]

Well, from my knowledge and experience, SP1 seems to want a different runtime module to work properly. The old runtime module, when installed, didn't want to work with SP1 compiled apps.

Oh alright, I wasnt aware of the compiler changes, if any, as I didnt bother to check the patch's changelog :sombrero: ..Least we now know it does...Thanks for the mirror of the runtimes BTW.

The Stable works fine (including Brunswick Bowling). The Development one though still gives me the DirectX9 error upon trying to boot a rom.

Thats excellent! ^^ I'm happy now that Brunswick Bowling is now working perfect for you. As I noticed, the fix from that, also fixed loads other games, and it should now be on par with the official release, I think .

Do you have FSAA enabled in the DX9 plugin? If so, turn it off. The development build uses a preliminary (and quite incomplete & buggy) implementation of hardware render to texture via DirectX render targets (hardware render to texture is one of the techniques in hardware framebuffer emulation), and this method is not compatible with FSAA, due to API limitations. So, if you have FSAA on, turn it off basically.

SQLite isn't a DLL and it only adds 250K tops with careful removal of features you don't need this can shrink a fair amount. Also you can make multiple tables into a single database. SQLite is not good for HUGE databases, but really this is a small database with at most 10000 elements. We need at least 2 tables for HR TP's maybe 3. A table of valid ROM ID's, a table of HR BLOB stored textures there CRC32 and MD5 data. The advantage of SQLite not being a DLL is that it doesn't ADD as much as a DLL to the code size. At runtime that's a different matter.

I meant the main DLLs (the actual plugins). Thats where I was refering about the compression on the DLLs, I didnt mean the SQLite code. Currently, the plugins are compressed with UPack, atm..

BMP is a waste of space, implementation wise and storage wise. There is no point unless people are adament on using MS Paint or something for texture enlargement.

Indeed, and besides, PNG supports CI indexing, so the issues of "*ci.bmp" are mute. Also, I noticed a consensus that CI BMPs dont really get dumped or need to be useable in TPs, but I guess a bit more input from the retexturers would be nice to see. In the meantime for the current pack format, maybe the texturers could just port thier CI bmps to PNG (preserving filename, except for extension and internal image format), and I could just change the BMP handling to just load from PNG files.

 

[Cyberman]

I meant the main DLLs (the actual plugins). Thats where I was refering about the compression on the DLLs, I didnt mean the SQLite code. Currently, the plugins are compressed with UPack, atm..

Duh. heheheh I'm thinking compressing the DLL is not going to change things a whole lot to be honest. I would rather it use more disk space than use more RAM when loaded. What you gain in smaller DLL space you loose in RAM. Which is more precious RAM or disk space?

Indeed, and besides, PNG supports CI indexing, so the issues of "*ci.bmp" are mute. Also, I noticed a consensus that CI BMPs dont really get dumped or need to be useable in TPs, but I guess a bit more input from the retexturers would be nice to see. In the meantime for the current pack format, maybe the texturers could just port thier CI bmps to PNG (preserving filename, except for extension and internal image format), and I could just change the BMP handling to just load from PNG files.

BMP == anachronism
As for converting, I was thinking it might be feasible to make a tool to efficiently convert dumped texture sets and TPs, or at least more efficiently than what currently is available. I suppose one can just as effectively scan for all those BMP's and convert them into PNG's. Since the CRC information is part of the file name, that helps. However I'm not sure what the deal is with the CI CRC's. Didn't Rice generate a second CRC for the palette used on those textures?
The one tool, too bind those TP's together, and RULE THEM All muhahaha

On converted TP's the only issue I can think of is lack of MD5 values (it's not a crying shame really since that is for being able to convert Jabo's TPs to the RTPF {Rice's Texture Pack Format}).

Cyb

 

[Doomulation]

Duh. heheheh I'm thinking compressing the DLL is not going to change things a whole lot to be honest. I would rather it use more disk space than use more RAM when loaded. What you gain in smaller DLL space you loose in RAM. Which is more precious RAM or disk space?

Well, correct me if I'm wrong, but as all EXEs, all DLLs get loaded into memory. The bigger the size, the more the memory. I'm not sure how these compression programs work, though... Do they add overhead code to uncompress the module in memory?

 

[Sirmatto]

Well, correct me if I'm wrong, but as all EXEs, all DLLs get loaded into memory. The bigger the size, the more the memory. I'm not sure how these compression programs work, though... Do they add overhead code to uncompress the module in memory?

From the Wikipedia page for UPX:

UPX uses a lossless compression algorithm called UCL, which is a free implementation of the proprietary NRV, Not Really Vanished, algorithm.

UCL has been designed to be simple enough that a decompressor can be implemented in just a few hundred bytes of code. UCL requires no additional memory to be allocated for decompression, a considerable advantage that means that a UPX packed executable requires no additional memory.

 

[Doomulation]

UPX uses a lossless compression algorithm called UCL, which is a free implementation of the proprietary NRV, Not Really Vanished, algorithm.

UCL has been designed to be simple enough that a decompressor can be implemented in just a few hundred bytes of code. UCL requires no additional memory to be allocated for decompression, a considerable advantage that means that a UPX packed executable requires no additional memory.

That makes it even better. Less diskspace and less memory overhead, but maybe at the cost of a little more cpu power.

 

[Legend]

Ok, yes, AA needs to be turned all the way down for the DX9 plugin to work. Hopefully that's noted somewhere that I missed.:sombrero:

It would be nice if a different reg. entry could be used for the DX9 plugin so the "stable" can use AA at the same time.

PJ1.7 now gives me that "Failed to allocate memory" message and refuses to ever boot a rom dispite attempting tens of time. God, PJ1.7 is so finicky now. But with PJ1.6, the Development plugin works fine.

 

[mudlord]

Ok, yes, AA needs to be turned all the way down for the DX9 plugin to work. Hopefully that's noted somewhere that I missed.

Its noted in the DirectX SDK docs....:evil: ..but thats cruel. I will write some unified docs for boths stable and dev branches, and a FAQ, in this this very fact will be discussed in depth.

It would be nice if a different reg. entry could be used for the DX9 plugin so the "stable" can use AA at the same time.

Sure thing dude, where do you want it? Or maybe a INI hmmm...rabiddiety already gave me some code from nrage's plugin that'll do the trick....although it needs a bit of hacking.

About the PJ64 beta problems, I personally just want to wait for things to stabilise in the core, before doing any more drastic changes. To me, theres not much point doing beta work now because PJ64 is undergoing very big changes atm with the core....as you noticed.

I'm not sure how these compression programs work, though... Do they add overhead code to uncompress the module in memory?

Yes . PE compression programs often add a compression loader stub as a extra section in the EXE/DLLs PE header. Often, these apps encrypt certain PE sections,which contain the code & resources. The EXE/DLL is then modified by the app to load that code section for the compressor first, which then loads the rest. This also holds true for copy protections as Safedisc, SecuROM and StarForce (though Starforce is a shitload more "controlling" than the rest, like in the new 4.0x branch and its insane & draconian methods to do DRM on a OS/hardware level. Tages, is different to all these, as it is embedded as C/C++ headers and libs, so it gets in the source code, which makes things hell of a lot harder to crack.)and Armadillo (used in apps like FRAPS,and the Sam and Max episodic games).

Which is more precious RAM or disk space?

That is a tricky question...I would take the RAM, but then I don't want to take up too much HD space....but then we got the argument about 160 > GB hd's these days...

 

[mudlord]

k, legend, here's a update to try....It'll not interfere with the official and stable branch plugins so your FSAA settings are safe....i also added a nice small reg key deleter to automate complete removal of the plugin's reg settings....

I might just drop the compression on the DLLs, save time for me, and plus, no fake virus warnings from shitty AV apps ^^..

edit: attachment removed, will be public soon anyway...and besides, I started to see these builds popping up on places like AEP Emulation tho...same with the unofficial wrapper updates (popping up on a French emulation site...)....

 

[Legend]

Works fine...thanks for the update! On a side note: Is Hydro Thunder compatable? It crashes for me a couple seconds after boot on both of your versions (I don't have the official anymore). The other games I tried worked fine.

 

[mudlord]

That was one game which I havent played with Rice's plugin. Its good the update worked for you, I'll do a update (with the new docs), and then I'll most likely get stuck into the database/hi-res texture work.

I usally use Glide64 and Jabo's plugins for it, so that it actually new for me..I guess I'll get the last binary Rice did (since we know things can break by compiling in a certain way....), and have a play around with that. The main games I tested with Rice's are the ones with the texture packs, so....yeah...

 

[Doomulation]

Well, you could always use a place to upload binaries that only those interested in the dev can access. No uploading attachments. That way the executables won't be snatched, if you would like it.

 

[Cyberman]

Since it's extremely immature to take dev libraries and use them as if they were the safest thing, (Like hey I got the latest and greatest.. whoops my entire computer don't work now it's all mudlord's fault Wahhhh!) I'll ask Martin about it. He might have a good suggestion.

Please Kiddies DON'T download unofficial builds and test builds, unless you WANT something to go wrong. We are not responsible for you driving off a cliff, THINK about it.

Cyb

 

[mudlord]

Cyb, I took your message and put it in the sticky, that should stop idiots from complaining to me personally about it screwing thier PC...I should put a disclaimer on PJ64 beta use too....I'll try to see about finding a safe place to put the dev alphas (like the one I gave to Legend)....since I dont want to see builds like this popping up on AEP all the time....

BTW, I started to completely overhaul texture dumping/loading. BMP loading is now a thing of the past and all CI images are now in PNG (which saves retexturers a crapload of time in weeding through images). It seemed the plugin, in TextureFilters.cpp (oddly enough), converts palleted images to RGBA on loading, which is how it handles colour indexed images. A 32-bit CRC is used for tex verification, and tex loading/dumping is done on RDRAM, while not emulating TMEM while doing it . I really got to clean up the source some more though....

 

[Doomulation]

Well, I do believe I can help with a safe place if you don't want to find it elsewhere. I can arrange a private FTP for dev builds.

 

[mudlord]

Well, I do believe I can help with a safe place if you don't want to find it elsewhere. I can arrange a private FTP for dev builds.

That would be brilliant , thanks for the offer..