A/V stream out of sync with when playing audio on SourceDataLine

Question

Hi,
I'm receiving a video and an audio stream via RTP and try to play them synchronized (audio synchron with lip movement).
I'm using a DataSourceReader in order to read from a datasource and play on a JavaSound SourceDataLine. This works pretty good but unfortunately the audio is played with a delay of approx 1 second.
else if (evt instanceof NewReceiveStreamEvent) {               
     try {
          stream = ((NewReceiveStreamEvent)evt).getReceiveStream();
          DataSource ds = stream.getDataSource();
          Format format = null;
          // Find out the formats.
          RTPControl ctl = (RTPControl)ds.getControl("javax.media.rtp.RTPControl");
          if (ctl != null) {
               format = ctl.getFormat();
               log.info("Received new RTP stream: " + format);
          else
               log.info("Received new RTP stream");
          if (format != null && format instanceof VideoFormat) {
               Player video = javax.media.Manager.createRealizedPlayer(ds);
               video.start();
               this.video_player = video;                         
          else if (format != null && format instanceof AudioFormat) {
               JavaSoundDataSourceReader dsr = new JavaSoundDataSourceReader();
               Player audio = dsr.open(ds);
               audio.start();
               this.audio_player = audio;
          }          I already tried to synchronize them with
videoPlayer.addController(audio_player);
audio_player.start();but this is throwing an Exception:
javax.media.IncompatibleTimeBaseException
     at com.sun.media.multiplexer.RawBufferMux.setTimeBase(RawBufferMux.java:377)
     at com.sun.media.BasicSinkModule.setTimeBase(BasicSinkModule.java:52)
     at com.sun.media.PlaybackEngine.setTimeBase(PlaybackEngine.java:1672)
     at com.sun.media.BasicPlayer.setTimeBase(BasicPlayer.java:272)
     at com.sun.media.BasicPlayer.addController(BasicPlayer.java:1043)
     at VideoPlayback.update(VideoPlayback.java:281)Line 281 of VidePlayback.java is the "videoPlayer.addController(audio_player);" line.
My guess is that it should rather use RawSyncBufferMux instead of RawBufferMux but I don't know how to change this. Maybe directly as a processor option?
Please help as I've no clue how to improve this.

Answer 1

chummer wrote:
it should also be synchronized with the video :-)The problem with getting it synchronized is that Javasound & your sound card have internal buffering, so there is some delay between writing to them and hearing the sound come out...
Normally, JMF handles this internally somehow by aligning things to a timebase, which can be syncronized, and allowing Processors to fill up their internal buffers ahead of time during the prefetching phase so that everything that needs to play together can have an instant start time. As it goes along, presumably adjustments are made as things get out of sync.
My best advise to you would be to implement your Javasound stuff inside of a Renderer class, instead of a DataSink class. DataSinks are designed to take in data and write it out as fast as they can... whereas Renderers are given data at "real-time" intervals and are designed to be played.
When implementing your Renderer class, JMF should handle the syncronization for you. IE, it'll give you data as it should be played by calling the Process function, so you'll just need to write the data to your TDL as it comes in. I believe that "Process" may be called before "Start" to allow you to prebuffer some information, but you'll need to test that as I've never actually tried it.