Mac Pro 2008 with 8800 GT, hard freezes with screen savers/OpenGL. You too?

Question

I have been experiencing random intermittent hard crashes of my "early 2008" model 8-core Mac Pro. Now I have finally found an easy reproducible test case (see below), and hope others will try it and post the results. This is a video-driver related crash that I have now duplicated with two different video cards.
The symptom is that my screen randomly freezes. The mouse moves, but there's no way to use the Mac[1]. Holding the Power button to force the Mac to shutdown is the only way out.
Usually, this occurs when I get to my office in the morning, or after the gym. That's because the screen saver is running. I found that the screensaver, other OpenGL/3D type applications, are what triggers this bug.
The bug seems to be with the graphics driver. Immediately before the freeze, messages like this are printed in the system log:
Mar 26 18:34:14 Mac-Pro kernel[0]: NVChannel(GL): Graphics channel exception!
status = 0xffff info32 = 0xd = GR: SW Notify Error
Mar 26 18:34:14 Mac-Pro kernel[0]: 0000000c
Mar 26 18:34:14 Mac-Pro kernel[0]: 00200000 0000502d 00000470 00000000
Mar 26 18:34:14 Mac-Pro kernel[0]: 00000482 000002ac 00000003 00000003
Mar 26 18:34:14 Mac-Pro kernel[0]: 00000000 00000000 01be0003
Mar 26 18:34:14 Mac-Pro kernel[0]: NVChannel(GL): Graphics channel exception!
status = 0xffff info32 = 0x3 = Fifo: Unknown Method Error
Mar 26 18:34:14 Mac-Pro kernel[0]: 0000000b
Mar 26 18:34:14 Mac-Pro kernel[0]: NVChannel(GL): Graphics channel exception!
status = 0xffff info32 = 0x3 = Fifo: Unknown Method Error
Other errors like these also appear (the actual log output volume is too large for this forum):
Mar 26 21:15:15 Mac-Pro kernel[0]: NVChannel(GL): Graphics channel exception! status = 0xffff info32 = 0x6 = Fifo: Parse Error
Mar 26 18:34:14 Mac-Pro kernel[0]: NVChannel(GL): Graphics channel exception! status = 0xffff info32 = 0xd = GR: SW Notify Error
Anyway, finally I found on another thread about a similar issue[2], a recommendation to run Folding@Home program to trigger the freeze. They were talking about another Mac model, but sure enough, it triggers my problem every time. I downloaded the client[3], installed it, and then launched the app which gets put in the Applications folder. It doesn't even finish bouncing once in the dock before the Mac Pro freezes.
This is not a bug specific to Folding@Home; juding from the log messages and the effect, this is exactly the same crash that is intermittently triggered by the default Mac OS X screensavers (I use Arabesque, but I think any OpenGL screen saver does it). It's just that running Folding@Home is an easier way to trigger the bug than leaving your screensaver running for hours.
So, that is pretty unfortunate, right? I mean, not only can my new Mac Pro not sleep without losing all unsaved data (yes, I have the reboot-instead-of-wake problem[4], too, but that is another issue...) but it also crashes at random whenever the screen saver runs for a while, or somebody uses OpenGL programs. I have set my machine to never use the screen saver, but still.
So, first and foremost, my question for other 2008 Mac Pro owners with the 8800 video card is: does your Mac puke all over itself when you launch Folding@Home?
I just found it a little bit hard to believe that every Mac owner of the most expensive new Mac and the higher end video card will have these crashes. It could be, but it seemed far-fetched. So I tried some troubleshooting:
a
1.) I considered that I had a bad video card, so I got another GeForce 8800 GT and installed that. Still crashed.
2.) I disconnected my second monitor, and tried it with only one monitor. Still crashed.
3.) I booted from the pristine boot disk my Mac shipped with (10.5.1). Still crashed.
4.) I reduced the resolution on my 30" Cinema display to 800x600. Still crashed.
5.) Just to stave off the usual chorus of "try zapping your PRAM" posts, I zapped the PRAM. Of course, this did nothing (I don't think this has really fixed a problem since the 1990s...). Anyway, it still crashed.
6.) In doing all these tests, I disconnected all USB/FireWire peripherals.
7.) My Mac is totally up-to-date with Software Update as of Thu, Mar 27, 2008. Mac OS X 10.5.2.
I don't want to return this Mac to Apple; it is wicked fast and that really helps me with my work. But, the frequent crashing really is a productivity-killer.
So: does every 2008 Mac Pro owner with an 8800 GT have this problem?
(Note: there are lots of similar problems with other Macs and the new Mac OS X 10.5... I hope to specifically address this particular model and crash in this thread.)
If not, I will go to the next extreme step: pulling out some of my RAM. I have 12GB of RAM, and it all seems to work fine, no parity errors or anything. But I know sometimes problems don't happen unless there's "too much" RAM, "too high" resolution, and so on. But I would like to hear from others before I go banging on my precious DIMMs and riser cards.
Thanks for any information!
The next step will be to determine if this is just another unfortunate defect that affects ALL 2088 Mac Pros (in which case I would want to keep my fast Mac and wait for fixed drivers), or if perhaps just a subset of them have defective video cards that cause the driver to crash in this way (in which case I would want a replacement).
---------- NOTES: ----------
[1]: I can connect via SSH on the command line to the Mac. However, the GUI environment seems borked; opening apps or trying to gracefully quit them via AppleScript either does not work or really hard freezes everything, such that SSH no longer works either.
[2]: A similar (but perhaps different) issue affecting notebooks is discussed here, and there are some people with Mac Pros, too: http://discussions.apple.com/thread.jspa?messageID=6786960
[3]: The Folding @ Home client, which can be used to trigger this bug, can be gotten here: http://folding.stanford.edu/English/Download
[4]: The severe defect affecting many (most?) 2008 Mac Pro machines' ability to sleep without losing data is discussed in this other thread: http://discussions.apple.com/thread.jspa?threadID=1353551

Answer 1

Hello guys, thank you for the replies.
I do have more data at this point.
ujeezy: Those posts on MacNN and others like them did give some clues.
Jon BWFC: I think you are (partly) right that it's (partly) an issue with the driver. But I have confirmed with a fair degree of certainty in my case that it is a hardware issue too, likely with the video card and not the Mac Pro itself.
cmcom: I think it sounds like you problem is different than the one we're talking about (although who knows, the driver could be causing that too). The freeze we are seeing locks up the whole Mac, and no application crash report is produced.
I had another GeForce 8800 card, so I went through some exhaustive troubleshooting. I tested each card, installed in each of the 16x slots, booting from either my normal 10.5.2 boot disk or a clone of the pristine virgin system disk that shipped with the Mac, and tried to see if the Mac would crash.
Unfortunately this was harder than it seems, because:
a.) both my GeForce 8800 cards appear to be defective, but in different ways (see [1] and [2] below)
b.) In my testing, I thought I was seeing the crash sometimes, but later I discovered that there are different ways to make the Mac crash like this. That is, there are bugs in Mac OS X 10.5.x (probably in the NVIDIA driver) that cause similar crashes, and these bugs occur on all Mac Pro units and even other Mac models, without any hardware defects. Unfortunately, the Folding@home exercises one such bug.[3]
So the upshot is, all those times I was using Folding@home to test this issue were bogus, because Folding@home crashes any Mac Pro with the NVIDIA card, and crashes MacBook Pro machines with NVIDIA chips as well.
Argh! So I had to retest. It took a couple of days but I found another reliable way to crash my Mac Pro, more quickly than just leaving the screensaver on for a day or two: run this OpenGL virtual fish tank simulator[4] for 2-3 minutes. Unlike the Folding@home test, this did not crash any of my other Macs, and after exhaustive testing I found that it ONLY crashed the Mac Pro with one of the GeForce 880 cards installed, and not the other one. This was regardless of what slot it was installed in or what boot disk was used.
Whew! So, I concluded that indeed, there is a hardware defect in one of my GeForce 8800 cards which causes the machine to lock up. [2]
What made this problem much harder to deal with was the fact that Mac OS X (specifically the graphics driver parts of the OS, perhaps NVIDIA-specific) crashes with these same symptoms in other cases too. Like when you run Folding@home, or in some of these cases people are talking about with their MacBook Pro machines.
So, when there is a bona-fide hardware defect, it is really hard to isolate. (It took me like four days.)
The thing is, the defective NVIDIA card that crashes the Mac[2] was a REPLACEMENT card for the 8800 card that shipped with my Mac. I called Apple about the video artifacts I was seeing and they replaced the card.
I am now running the original card (and suffering the ugly video artifacts, but no crashes). At my own expense, I ordered an ATI card (the RADEON HD 2600 XT) and I will install that.
I hope someday to have my Mac work without graphics glitches or crashes. For now, operating (mostly) without crashes will have to do.
The original problem with this Mac continues, though: the video doesn't work correctly; it starts displaying visual artifacts after 1-72 hours of use[1]. That could either be because:
a.) a hardware defect in that video card
b.) a software problem with the driver/OS
c.) a hardware defect in the Mac itself
I strongly suspect A, but hopefully my buying a different card and testing with that will help further isolate what is wrong.
I am marking this thread "solved" because the specific issue I originally posted about has been resolved: the GeForce 8800 card causing the crashes is defective, and using another card fixes that issue.
[1]: The original GeForce 8800 card that shipped with my Mac Pro exhibited annoying-but-not-deadly video artifacts, which can be seen here: http://masonmark.com/wp-content/uploads/2008/04/bogusvideoartifacts.jpg
[2]: Apple replaced the video card, and that is when my deadly video-corruption-plus-frozen-Mac crash problem began. A screenshot example of that problem can be seen here: http://masonmark.com/wp-content/uploads/2008/04/crashedmac_pro_akashitball.jpg
[3]: The red herring in this case was the Mac OS X bug (seemingly specific to NVIDIA-equipped Macs including Mac Pro and MacBook Pro) that causes a very similar full system crash. This bug can easily be triggered by running the Folding@home version "6.10beta2" demo application. That is a serious bug, and since my employer is a Mac software company I reported this bug to Apple via the official channel. It is Apple bug number 5830772.
[4]: The OpenGL fish tank simulator I found, which would trigger the system crash caused by my defective 8800 video card (but would not crash any other Mac, or crash my Mac Pro with the other 8800 video card installed) can be found at: http://uri.cat/software/Fish/