Dynamic Streaming - not working as expected

Question

I've been experimenting with Dynamic Streaming while in the process of writing a tutorial. The documentation is quite clear on a few points:
The keyframe interval in the various encodings should be short
The bufferlength should be at least 2x the keyframe interval
The player should sense a bandwidth change, by default, within the 4-second sampling interval and call for a switch. Then, the switch could take as long as 2x the keyframe interval after that.
What I'm finding is wildly different behavior than this.  It takes anywhere from 10-15 seconds for the player to notice the change and call for a switch, then another 20-40 seconds for the switch to happen. When switching up to a higher bitrate stream, this just means the user gets low bitrate video for longer than they ought to. But when switching down due to falling bandwidth, the buffer runs out and the user stares at the rebuffering sign for a lengthy time - long enough to give up on watching the video, for sure.
I've encoded an H.264 MP4 file at 64, 384, and 768 kbps, at 30fps and an "every 60 frames" keyframe interval.  I've streamed it rtmp via two different CDNs that use FMS 3.5, into two different Flash video players (JW Player and Flowplayer). I've restricted my bandwidth on Windows XP with Netlimiter 2.0; and on the Mac with 'ipfw'. I've set bufferlength between 4 and 10 seconds.
I've tested switching up and switching down. For up, I start with a 200kbps bandwidth limit. The video starts OK with the correct stream, then at 5 seconds I open up the bandwidth to unrestricted. For testing down, I do the opposite: start at unrestricted and then at 00:05 restrict to 200kbps.
My test page, with both players and sample code is at http://www.learningapi.com/streamingmedia-articles/dynamic-streaming-in-flash-bitrate-swit ching/  I also have a couple of screen recordings there showing the behavior of the whole process, both switching up and switching down.
I thought I've done everything right here - paid attention to every documented detail, but it works rather poorly. Can someone explain whether this is expected behavior, if the players have implemented dynamic switching poorly, or if I'm doing something wrong?
Thanks,
Larry

Answer 1

Thank you so much for your help!  To answer your questions:
In regard to switching up case - which is the stream that you start out on?
I'm starting with the bandwidth throttled to 200kbps, so the stream chosen is 64kbps. I've tried both VP6 and H.264 files, encoded at 64, 384, 768kbps.
Is RTMPT involved in this case or no tunnelling?
There is no tunneling, at least there shouldn't be. I did try a test explicitly calling for rtmpt. I get an error from the Flowplayer's bwcheck when I do this. "no live hosts to connect to"
Is FMS Edge/Origin streaming involved?
I don't know. Most of the testing I've done with Cloudfront, although I've tried Limelight as well and the results are identical.
Are you seeing the QoS bandwidth measures yourself, and if so are they adjusting slowly or suddenly after a period of time being incorrect?
JW Player does show the detected bandwidth updating every few seconds. There's a good 10 second delay (or more) before the value starts to change, and then it will show the bandwidth value gradually shifting from the old value to the new - takes about 15-30 seconds to ramp down from the unrestricted bandwith (~ 8-12Mbps) to 200kbps.  Occasionally, the bwcheck number simply does not change, as if the player is not checking at all.  I don't know if that's because the reading is averaged, or if that's really what it thinks it's measuring. But by the time it's reading about 200kbps, the stream has already been buffering for 10 seconds or more.  If I increase the bufferlength to 25 seconds or more, it just delays the point where the stream will start buffering. It's inevitable that there's always a break in playback when I restrict to 200kbps while streaming.
For Flowplayer, I can't get its bwcheck numbers as easily. Looking at the logs, it almost appears that it's not checking (or not getting a response) very often. What's interesting is that the behavior - the actual outcome for streaming - is pretty much exactly like JW Playe - which is what makes me wonder if this is a platform issue.
Thanks,
Larry