Dwh's current take on SCP (Snap-Crackle-P

Question

I think that many people could solve their SCP problems mainly by looking at the problem differently. The main reason for this post is to try to help that happen. In a sense, of course, this will tend to validate my position or not, depending whether significantly more people start reporting getting rid of SCP or not. It is <i>not</i> my intent to defend my position - I think there's been plenty of that in other threads - and I intend to ignore any replies which are simply argumentati've, because experience has shown that responding makes the thread length explode, which would just bury my intended message underneath that kind of traffic. In fact, I've decided I will give all such argumentati've posts in this thread a "" rating as further encouragement to argue with me in some other thread.
OK, you've got SCP. How do I suggest you approach it's
First, do not assume your SCP has the same cause as everybody else's. I'm going to talk about the bus latency cause, because that's the one I think is behind the bulk of the complaints, but yours might be something else: you might be one of the unlucky people whose card is defecti've, for example (that's going to happen to some people, and it could be you).
Second, think of your soundcard symptoms as possibly <i>revealing</i> a problem in your system, much as canaries were used in coal mines because they are more sensiti've to toxic gases and would die before it was too late for the miners to get out. This suggests that the soundcard may be telling you to look elsewhere, and I suggest that for this particular problem, that's actually a good way to go. So, for this particular problem, ask yourself what <i>other</i> devices seem to be acti've when you hear SCP. For example, if you hear SCP when you see the disk activity LED flash, that's a big clue. Pay attention to stuff like that.
I'll get into why's later on, but for now, the kind of device most likely to be behind SCP is a relati'vely smart device geared for high performance, particularly video cards and busmastering disk controllers (especially RAID controllers), although network cards and TV cards are examples which don't seem to me to fit that mold, but sometimes seem to be behind SCP.
I think the first thing to try is to do simple things which may evoke SCP or make it worse. I suggest rapidly scrolling a large window up and down while playing audio is a good basic test for the video card. Similarly, copying a large file (or better, running a disk benchmark) while playing audio is a good basic test for disk controllers.
The next thing to try is to temporarily change settings which affect driver behavior. Examples are lowering the video acceleration and disk read ahead settings. These are things easily changed and changed back. If they stop SCP, this casts strong suspicion toward the device associated with them.
Progressing to steps which require some more effort, I am convinced these problems are fundamentally due to some design weaknesses in the PCI and PCIe buses themselves. However, there's really nothing that you can do about that. The only available true fix is having drivers for vulnerable devices take steps which avoid the situation which sets off the problem. So, this means it's a good idea to make sure all your drivers are current, including the drivers for any devices which are integrated into your motherboard. Then, if that doesn't help, see if you can find some non-current drivers to try for those devices which were already current, and try those. Sometimes you can revert to generic drivers which are built into the operating system. For example, if you remove your video card drivers, this will happen. The idea is to try a completely different driver to see if it affects the symptoms. I don't suggest running anything other than the current drivers supplied by the device manufacturer as a real solution, but it can be a very effecti've diagnostic tactic, and it can be useful information when trying to get the official driver fixed (i.e. if you can tell your video card maker that your soundcard noises stopped when you removed their video card driver...). Falling back to generic drivers may mean advanced hardware features won't be used, so if you're only experiencing SCP when there is high-intensity game action on your two-card SLI graphics rig, it may not be diagnostic; but if you get SCP when playing WMV-HD video files, it could well be enlightening.
If you're not focused on a specific non-soundcard device at this point, my next suggestion is to rule out as much as possible. This probably means opening the computer's case and removing any hardware that is not essential for your testing. You won't be able to remove integrated devices, so for those this means disabling them in the BIOS. (If you have more than one disk controller, you might be able to temporarily rearrange your system to only use one of them at a time for your tests.) Of course, you can take a slower approach like trying disabling devices in Device Manager first, but ultimately this step is to help rule out things you're not suspecting, and there's nothing like removal to make sure something isn't influencing your symptoms. If this approach proves fruitful, putting things back together piece by piece until the problems recur should cast suspicion on the last device you put back or enabled. Unfortunately, simplifying your system can make SCP go away simply by slightly improving bus latency performance without actually fixing it. So, this isolation process can only suggest which device is really behind your system's bus latency problems, and which driver you really need fixed (or device with bad drivers you might consider replacing).
If you still get SCP with your system stripped down with only essential devices left, you will at least have a smaller number of prime suspects, unless you are in a position to try substituting other devices. That's often not really practical, but if you own a computer store, or have very generous and helpful friends, it may be an option.
Ultimately, if you can figure out which device to focus on, and can find drivers for it that address the bus retry problem, you should be able to make just that one substitution, and really solve bus-latency-caused SCP, not just keep it at bay.
In some cases it may be useful to record the SCP via What-U-Hear and/or Wave, at the right sample rates, and look at it in a suitable wav editor. This can give clues as to whether the SCP is due to bus latency or something else. Within reason, I'm quite willing to try to help out in individual cases in this thread if you're trying to follow my advice. Feel free to ask questions, etc.
Now, to explain where I'm coming from a little.
There's a bottom line here. If you're talking about bus latency, which is how long a device has to wait to get the bus after making a request for it, in terms of 00's of microseconds (more than 3,000 bus-cycle times), or more, for something that operates at the speed of the PCI bus, that's all you need to know that something is seriously out of whack. It's also all you need to know to rule out the device that is waiting as being the cause of the delay. The point here is that we're not talking about sub-optimal bus latency, nor even poor bus latency, but "insanely awful" bus latency.
Any analogy is going to be a stretch here, but imagine a traffic signal in a city that does not permit a right turn on a red light. We can consider the latency a car experiences to be the time from when it arri'ves at the intersection controlled by the light to when that car gets a green light. Now, considering the normal speed of cars, etc., suppose I start talking to you about timing the light for a two-hour cycle, i.e. about one hour green, a few seconds of yellow, and about one hour red. That would be more than completely nuts, right? Well, the kinds of bus latency CL has measured is that kind of nuts. It really is.
(While I understand the arguments that if CL makes a sound card at all they should go to whatever lengths to make sure it works irregardless, there must be some point where those arguments just get unfair if the infrastructure upon which CL relies, but does not control, is sufficiently insane. That's why I personally can't bring myself to blame CL, and instead look to the design of the infrastructure first, and whatever is able to avoid causing the "insanely awful" bus latency second.)
While bus latency as high as has been reported is sufficient to explain a reasonably-designed soundcard producing the kinds of symptoms that are being observed, and that the device experiencing that kind of bus latency isn't that cause of it, it's not enough information to determine the cause. I've said there are really only two plausible things that can explain it. Either the controls to limit the length of an individual bus transaction (after the bus has been requested) fail to do so under some conditions, or after a master requests the bus, others are being given multiple turns before the bus is finally granted (i.e. you ask for the bus, but your "turn" keeps getting skipped). (The only other possibilities are either essentially even more insane bus behavior, unless your position is that CL's engineers and others are so dumb they can't use equipment like bus analyzers to measure the latency their device is exposed to, or that they are just plain lying.)
Initially my instincts had me leaning toward the first of the two cases: something wrong with the transaction latency controls. Currently, it looks like it's really the second case: one device is getting more than one transaction when others are waiting. This is after I've been able to look at the PCI specs. (Because my access is through a pcisig.org member company, I don't feel comfortable quoting from the specs., nor saying more than a minimum about their contents that is relevant.)
"Bus retry" is when a transfer is stopped by the addressed device before any data transfer has occurred. It carries a special implication that the same transaction is to be retried. It seems to be designed as a way for a slow device to delay things (a little) until it can get ready.
The failure mode I have seen described is that the host bridge (essentially the processor connection to the bus), addresses a device and then does bus retry transactions for an insanely long time. The device has some kind of small queue of things to do, which has become full, and the device signals bus retry until the full condition ends. The host bridge seems to dedicate itself to this one task, so this may block other processor I/O operations. Since the bus retry transactions do finish, albeit early, it would seem that other bus masters, like the CL soundcards at issue, should be able get the bus while this is going on, but that doesn't seem to happen. In looking over the PCI 3.0 spec, I only find 2 things which seem relevant.
There is a 0 microsecond limit on bus retry looping of a specific kind, but there is no hardware enforcement; the spec explicitly permits proper avoidance to be implemented in device drivers. If this is overlooked, failure lurks for spec-conformant hardware, and I think it's easy to overlook this.
Secondly, there are ways some masters can get an exclusi've lock of the bus, and some situations where the lock semantics can result in data overruns for video/audio/communications devices, to the extent that the 3.0 spec discourages the use of the LOCK# signal and hopes to remove completely it in the future.
I also tend to think that having 2 basic causes for such grotesque, long-standing problems is pretty unlikely, though not impossible. So far I don't see a need to speculate to the contrary, which is to say I am inclined to apply Occam's razor to this, so long as the symptoms and hard evidence seem to be covered by one basic explanation.
My current take on all this is that it's likely been standard practice for host bridges to control bus deadlocks and operation sequencing problems by some combination of bus locking and transaction ordering, with the net effect that the bus gets dedicated to bus retry loops when they occur. It's likely that these practices have been carried to the PCIe bus (because they control potentially serious problems). In PCIe-bus machines, the PCI bus hangs off the PCIe bus by means of a PCIe/PCI bridge. That means that if there is a locked bus-retry pathology on the PCIe bus, I'd expect PCI devices to see the PCI bus appear to go idle for the duration even though there are requests for the PCI bus. That seems to me to be consistent with the CL NF4 statement that it was "...the first motherboard on which these extremely long PCI service holdoffs have been observed by Creative, where another PCI device in the system was not causing the holdoff."
This suggests that chipsets might be able to do something about this. If the host bridge released the lock on the bus, at least when ending a "bus retry" transaction, it seems this would help busmastering soundcards, like the X-Fi, by letting them break in during a bus-retry loop. That should eliminate the huge bus latency measurements. However, the bridge would have to be doing something pretty complex to let other driver tasks past an unlocked bus-retry loop, so I think this could still be a big bottleneck. For this not to be a problem area, I think extended bus-retry loops have to be prevented.
Right now, extended bus-retry loops can be prevented by driver-avoidance, which IMO is a workaround, not a true solution. If you can hunt down and deal with the culprit drivers, however, at least it does fix things; it's just that that can be darn hard to accomplish. From a bus protocol standpoint, this kind of thing needs an enforced limit. A device which simply has slow circuitry doesn't need very many retry cycles, and that's what this is useful for. A device which (ab)uses it because its work queue is full leads to these problems, because the right thing to happen is do other things with the bus until the device work queue has room. In other words, this isn't a "wait for me" situation, it's a "come back later" one. Exceeding some generous "wait for me" limit should have been a reject reflected back to the driver. There is more than one reasonable way to do something like that, so I'm not going suggest how the PCI bus might have been different to help with that; the small hard limit is sufficient: the driver can look at a status bit, and request an interrupt from the device when the bit clears, as the PCI bus stands now.
-Dave
[email protected]

Answer 1

. Perhaps NVidia simply makes their 6x PCIe slots ultra high priority so their video cards can squeeze out every last frame per second in SLI mode as possible, without regard to being "nice" to other add-in cards by allowing them proper bus time?
While that's certainly not something I can rule out, I think it's much more likely the bus-retry pathology has simply spread to PCIe, having not been dealt with in PCI. But, you say, PCIe is based on serial point-to-point technology and can be doing more than one thing at at time. It's also based on the PCI programming model and I think it likely there are bottleneck points around the setting up of transfers. What that means is that the arbiter does one grant at a time, but can continue onto others unless they would conflict or it runs out of some needed resource, so when that process is locked by the PCIe Host Bridge while it tries to start up a transfer to a device that keeps responding to each attempt with bus retry, it's entirely possible the same thing happens as for the PCI bus, even though normally the PCIe bus <i>can indeed</i> do some numbers of multiple transfers in parallel.
2. Perhaps PCIe won't have this problem, once motherboards are 00% PCIe without any old PCI slots? I think this may happen, because PCIe is based on serial point-to-point technology as opposed to the older PCI based on parallel shared-bus technology.
If there was a Soundblaster X-Fi card with PCIe for connectivity (even though it doesn't need the bandwidth), then perhaps it wouldn't have any of these ongoing problems any more at all.
Hopefully, some day very soon, Creative Labs will make a sound card for PCIe. There sure is a high demand for it now, which I'm sure will only increase -- especially when PCI gets phased out completely just like ISA was a few years ago!
My current take is that this problem really appears to have spread to PCIe. Furthermore, CL has stated that PCIe has tested to be <i>even worse</i> than PCI for audio. It's better for moving more data faster, but seems to be even worse at moving small amounts of data promptly. (This is what can happen if only throughput is your God.) I'm very inclined to believe the reason for this is really that the bus-retry pathology gets worse as the bus gets faster, and I take it as some of the evidence for the problem having spread to PCIe.
-Dave
[email protected]