How to optimize MF Media Source for MKV playback (containing H264 and AC3)

Question

I've built a working Media Foundation Media Source for matroska (MKV files) using the sample for an MPEG Source.  I've finally gotten it to a point where I can play the file and both video and audio are visible and audible.  I followed the MPEG1Source
code quite closely actually.
However, I can't seem to make the video playback smoothly.  Audio plays fine, but video just can't seem to keep up with the playback speed. It stutters and breaks up into blocks constantly.
The only thing I have been able to try is to remove as much dynamic allocation of memory as I can.   While my parser parses the file and initially creates a bunch of objects on the heap (pertaining to track formatting, cue locations, etc)
, when I start actually parsing video and audio frames, I'm no longer dynamically allocating memory on the heap.  I even removed a std::queue for frame lengths and created a fixed length circular buffer instead.
What else do I need to try?
I am simply parsing each of the video and audio frames in the order that they appear in the file.  The frames are not Annex B compatible (for H264), so in the DeliverPayload function I replace the length section of each NAL unit within the frame
with a start code of 00 00 00 01.   Then I copy the frame into the stream's buffer.  
One thing that is very different from the MPEG1Source example is that I am not creating structs that contain all of the video "initialization" data (i.e. MPEG1StreamHeader, MPEG1SystemHeader, MPEG1PacketHeader).  Instead, I am leaving all
of it as public variables on my parser class.  So I am referring back to my parser quite a bit from the MKVSource class.  I don't know if this affects anything. 
Anyways, I'd be happy to elaborate to get this working... Thanks!
Lee McPherson

Answer 1

Just a follow-up:
I was indeed calculating the timecode / timestamp incorrectly.  MKV uses a technique called lacing to stitch frames together and save space.  While I was separating them and forwarding them correctly to the pipeline, the timecode in the header
for each laced block only applies to the first frame.  I needed to add the default duration time to each successive frame laced in the block.
While not specifying the timestamp using the SetSampleTime method caused smooth video, it was actually playing faster than the audio.  Each frame _needs_ a timestamp to sync correctly with the audio or other streams.
Lee McPherson