What's new

Nvidia Cheats Identified!!!!

Status
Not open for further replies.

ra5555

N64 Newbie
NVIDIA SUCKS MAN, THEY SUCK BAD, 23% difference from driver cheats. 5900 Ultra's score droped from 5700 to 4700 in a new cheatproof version of 3dmark (that is readon 9700 level scores!!)

It looks like ATI could be cheating as well though. Read:

http://www.warp2search.net/article.php?sid=12390&mode=thread&order=0

EDIT: funny how nivia said 3d mark is usuless for testing graphics performance because of bla bla bla when GF FX 5800 was out, they even droped out of the beta program and wrote articals against fruture mark just because their products are inferior in it compared to the readon. They must have realized that gamers do look at the benchmark and as a result they cheated!
 
Last edited:
OP
R

ra5555

N64 Newbie
Well you probably all have witnessed the release of 3DMark03 patch 3.3.0 which disables all identified alterations Nvidia Detonator 43.51 and 44.03 introduced. For all of you interested in the details of the 'cheats' click read more.
Futuremark identified an unusual 8.2% difference with Catalyst 3.4 on a Radeon 9800 Pro. Investigation continues...




--------------------------------------------------------------------------------
Aren’t These Cheats Just Optimizations That Also Benefit General Game Play Performance?
No. There are two reasons.
Firstly, these driver cheats increase benchmark performance at the expense of image quality. Only the user and the game developer should decide how a game is meant to be experienced, and not the hardware developer. An act by hardware developer to force a different experience than the developer or the user intended, is an act that may mislead consumers, the OEMs and the media who look to our benchmark to help them make purchase decisions.

Secondly, in well-designed benchmarks like 3DMark03, all cards are instructed to do the same amount of work. Artificially reducing one card’s workload, for example, by using pre-set clip planes or using a lower precision shader against the program’s instructions, is only aimed to artificially manipulate the benchmark test result. Please note, that the cheating described here is totally different from optimization. Optimizing the driver code to increase efficiency is a technique often used to enhance game performance and carries greater legitimacy, since the rendered image is exactly what the developer intended.

What Are The Identified Cheats?

Futuremark’s audit revealed cheats in NVIDIA Detonator FX 44.03 and 43.51 WHQL drivers. Earlier GeForceFX drivers include only some of the cheats listed below.


1. The loading screen of the 3DMark03 test is detected by the driver. This is used by the driver to disregard the back buffer clear command that 3DMark03 gives. This incorrectly reduces the workload. However, if the loading screen is rendered in a different manner, the driver seems to fail to detect 3DMark03, and performs the back buffer clear command as instructed.
2. A vertex shader used in game test 2 (P_Pointsprite.vsh) is detected by the driver. In this case the driver uses instructions contained in the driver to determine when to obey the back buffer clear command and when not to. If the back buffer would not be cleared at all in game test 2, the stars in the view of outer space in some cameras would appear smeared as have been reported in the articles mentioned earlier. Back buffer clearing is turned off and on again so that the back buffer is cleared only when the default benchmark cameras show outer space. In free camera mode one can keep the camera outside the spaceship through the entire test, and see how the sky smearing is turned on and off.

3. A vertex shader used in game test 4 (M_HDRsky.vsh) is detected. In this case the driver adds two static clipping planes to reduce the workload. The clipping planes are placed so that the sky is cut out just beyond what is visible in the default camera angles. Again, using the free camera one can look at the sky to see it abruptly cut off. Screenshot of this view was also reported in the ExtremeTech and Beyond3D articles. This cheat was introduced in the 43.51 drivers as far as we know.

4. In game test 4, the water pixel shader (M_Water.psh) is detected. The driver uses this detection to artificially achieve a large performance boost - more than doubling the early frame rate on some systems. In our inspection we noticed a difference in the rendering when compared either to the DirectX reference rasterizer or to those of other hardware. It appears the water shader is being totally discarded and replaced with an alternative more efficient shader implemented in the drivers themselves. The drivers produce a similar looking rendering, but not an identical one.

5. In game test 4 there is detection of a pixel shader (m_HDRSky.psh). Again it appears the shader is being totally discarded and replaced with an alternative more efficient shader in a similar fashion to the water pixel shader above. The rendering looks similar, but it is not identical.

6. A vertex shader (G_MetalCubeLit.vsh) is detected in game test 1. Preventing this detection proved to reduce the frame rate with these drivers, but we have not yet determined the cause.

7. A vertex shader in game test 3 (G_PaintBaked.vsh) is detected, and preventing this detection drops the scores with these drivers. This cheat causes the back buffer clearing to be disregarded; we are not yet aware of any other cheats.

8. The vertex and pixel shaders used in the 3DMark03 feature tests are also detected by the driver. When we prevented this detection, the performance dropped by more than a factor of two in the 2.0 pixel shader test.

We have used various techniques to prevent NVIDIA drivers from performing the above detections. We have been extremely careful to ensure that none of the changes we have introduced causes differences in either rendering output or performance. In most case, simple alterations in the shader code – such as swapping two registers – has been sufficient to prevent the detection.

What Is the Performance Difference Due to These Cheats?

A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320.

The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks – a 24.1% drop.

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
 
OP
R

ra5555

N64 Newbie
http://www.beyond3d.com/forum/viewtopic.php?t=6032&postdays=0&postorder=asc&start=20

ATI's official statement:

The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.
 

AlphaWolf

I prey, not pray.
None of them are saints, ATI got cought on an even more blatant cheat last year, they fixed the drivers so that when quake3 was run, it would drop the texture detail.
 

vampireuk

Mr. Super Clever
NVIDIA fucked up this time around

and for the record 3dmark sucks arse as a benchmark anyway ;)

I just hope they learn from this and work their asses off for the next product cycle.
 

Tagrineth

Dragony thingy
AlphaWolf said:
None of them are saints, ATI got cought on an even more blatant cheat last year, they fixed the drivers so that when quake3 was run, it would drop the texture detail.

I'm starting do develop a half-peeve on this one.

It wasn't intentional.

The original optimisation was for the old Radeon R6 cards, and worked great.

When they carried that optimisation to R200, hoping for a quick and easy performance boost, they didn't have time to test it visually on R200 before sending out the drivers (R200 shipped with early beta drivers)... turned out the same optimisation needed tweaking on R200 - it didn't work as it should; it wound up causing the core to misinterpret the LOD requests.

We've had a pretty hefty (4 page, 65 replies) discussion on this at Beyond3D, linked here if you want to read through the bulk of it.

I'd hardly call it blatant "cheating" though anyway, since the very next driver set had none of the anomalies (EVER), and had higher performance as well. And if you think IHV's never optimise their drivers to avoid performance hiccups in popular games, you are very horribly wrong.
 
OP
R

ra5555

N64 Newbie
Nvidia response:

"Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today."
 
Last edited:

Tagrineth

Dragony thingy
ra5555 said:
Nvidia response:

"Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today."

More FUD, by the way.

nVidia is basically comparing their Pixel Shaders 2.0 and 1.4 (floating-point) performance with pre-floating point games (don't even try to tell me that DOOM3's NV3x path is using FP. Carmack never said so, and with the performance it's getting compared to the FP ARB2 path, I highly doubt it's going past PS 1.1)

Hardly an apples to apples defense. They're basically evading the NV3x's obvious problems with FP shaders altogether.


Edit: Added italics/underline to show what I'm talking about
 

AlphaWolf

I prey, not pray.
Tagrineth said:

I'd hardly call it blatant "cheating" though anyway, since the very next driver set had none of the anomalies (EVER), and had higher performance as well. And if you think IHV's never optimise their drivers to avoid performance hiccups in popular games, you are very horribly wrong.

Then how come when toms hardware renamed it to quaff3, the driver wouldn't drop the image quality? LOD whatever request or not, the driver only cared if the app that was running was named quake3.

http://www.tech-report.com/etc/2001q4/radeon-q3/index.x?pg=1

You simply can't denie that. The slider setting excuse is BS, because quaff3 wasn't effected.
 
Last edited:

Tagrineth

Dragony thingy
AlphaWolf said:
Then how come when toms hardware renamed it to quaff3, the driver wouldn't drop the image quality? LOD whatever request or not, the driver only cared if the app that was running was named quake3.

http://www.tech-report.com/etc/2001q4/radeon-q3/index.x?pg=1

You simply can't denie that. The slider setting excuse is BS, because quaff3 wasn't effected.

Actually the popular renaming was 'Quack3.exe'.

And keep in mind that at the time Quake3 was a very popular game to play (and it still is, to an extent).

Then there's also the fact that a few revisions later, other Q3-engine games had similar improvements out of nowhere. :satisfied

Back on track with a proper rebuttal:

Yes, I know it did drop the image quality though.... but as I said:

It did NOT drop any quality at all on R6-based Radeons, which was the target for the optimisation anyway. It was the same optimisation, initially, and it was thrown onto R200 on the assumption that it would have the same results - which it didn't, much to the chagrin of ATi. It was fixed immediately, and I'm sure ATi would've fixed it themselves even if nVidia themselves hadn't provided the info to review sites.

Yes, nVidia pointed out their opponent's 'cheating'. Just like ATi never pointed any fingers at nV for the current 3DMark03 hassle.
 

AlphaWolf

I prey, not pray.
Wait, why would you do this kind of "optimization" at all? E.g. why would ANYTHING in the driver target quake3 specificaly?
 

radTube

lazy bastard
AlphaWolf said:
Wait, why would you do this kind of "optimization" at all? E.g. why would ANYTHING in the driver target quake3 specificaly?

Haven't you noticed the list of games with improved performance supplied by both nVidia and ATI with every new driver release. Some games benefit more than others, because there are specific optimizations for popular games in the drivers. What's wrong with that? People don't buy games to compare graphics cards. At least I don't.
 

Eagle

aka Alshain
Moderator
What really bugs me is people that come on here and post about how 3Dmark rank some video cards lower than others and how thats such a crime.

First of all. Its business, someone has to have the better card, doesnt mean it will stay that way.

Second 3Dmark says NOTHING and its scores mean NOTHING and the software is worth NOTHING. Why use it? I have no idea.
 

vampireuk

Mr. Super Clever
Eagle said:
Second 3Dmark says NOTHING and its scores mean NOTHING and the software is worth NOTHING. Why use it? I have no idea.

Amen, I have no idea why people put so much faith into a damn synthetic benchmark.
 

Tagrineth

Dragony thingy
AlphaWolf said:
Wait, why would you do this kind of "optimization" at all? E.g. why would ANYTHING in the driver target quake3 specificaly?

DUH. What if for some reason a few textures are, curiously enough, causing stalls in the graphics pipeline, due to frequent cache misses (just as an example; stalls can be caused by any number of things)?

Or what if you've found a more efficient way to pre-cache those textures (my guess as to what the Q3A error was), which can increase performance?

Or even... what if something in the game's code is consistently causing a pipeline stall-out and subsequent crash? Obviously you'll want to special-case that game and make sure the offending commands are skipped, or replaced with a working, equivalent function?

Or how about if the game's vertex stream is specifically ordered for, say, GeForce3's vertex engine, and as a result is stalling heavily on the Radeon R300 line... of course, ATi is going to special-case it and re-order the vertices to stream better.

And then there's...

...bah, you get the point. There are a lot of reasons why you'd want to special-case a game.

Added:

Second 3Dmark says NOTHING and its scores mean NOTHING and the software is worth NOTHING. Why use it? I have no idea.

Actually, it does say something: Given a level playing field and standard DX9 rendering paths, it can give a decent (not perfect... but decent) comparison between different graphics cards' relative performance. This is why companies shouldn't optimise for 3DMark; the idea is, every card should be doing the same amount of work (barring generic optimisations like accurate Z-culling which are always-on, not special cased). It's more or less the same thing as running a Quake3 Timedemo, except that I can play Quake3, but I can't play 3DMark. =)

The other thing 3DMark is useful for is testing individual features. Generally, at least for review sites, the second half of the test suite is far more important than the "Game Tests". For example... with nVidia's "Optimisations", the Pixel Shaders 2.0 test in 3DMark03 goes twice as fast as with Futuremark's standard path.

And finally, 3DMark is very helpful for overclockers. Running 3DMark for 3 hours without a lockup = great stability. And when you're OC'ing, 3DMark scores can give you a quick indication of what kind of % performance increase you're getting. =)
 
Last edited:

vampireuk

Mr. Super Clever
3dmark is a worthless piece of crap for determining the performance of a product as it is purely synthetic and will give you no idea how the card will perform in actual games. That is why it sucks. I use actual games for benchmarking just like I am with my review :p
 

AlphaWolf

I prey, not pray.
Tagrineth said:

...bah, you get the point. There are a lot of reasons why you'd want to special-case a game.

Ok, so if its acceptable to make driver specific optimizations for any given application, e.g. quake3, then whats wrong with making a driver optimization for a benchmark?

I mean it can't be all that bad to artificialy increase the framerate in a benchmark application if its ok to artificialy increase the the framerate in a game. Doing one or the other has the same effect.

Something you people should consider, if somebody says that a company is cheating, you shouldn't denie it on their behalf because odds are its probably true. Big companies like this cheat all the time, especialy in the IT industry where competition is fierce. Anybody care to mention why the hell something as stupid as the winmodem was invented?
 
Last edited:
OP
R

ra5555

N64 Newbie
It looks like Nvidia they might have also cheated in Splinter Cell. The cheat is activated only in Benchmark mode. This is still not confirmed, but it could very well be true.

They have gone too far IMO, they blindly cheats in 3dmark and now this? If Nvidia keep this up, they are going to end up like 3dfx.

http://www.beyond3d.com/forum/viewtopic.php?t=6038
 
Status
Not open for further replies.

Top