Very low video framerate when sound activated

Question

Hi Guys,
hope you can help me with a problem that already cost me 2 days.
Basically I used the AVTransmit2 and AVReceive2 classes as a draft for my own implementations (see below).
                DataSource video_ds      = getVideoSource(new Dimension(640, 480), 30.0f);
          DataSource audio_ds      = getAudioSource();
                // when I use only one datasource here everything is ok (normal audio or 30fps video)
          DataSource[] sources      = new DataSource[] {video_ds, audio_ds};
          try {
               processor_ds = Manager.createMergingDataSource(sources);
          } catch (IncompatibleSourceException e) {
               return "IncompatibleSourceException creating data sources";
          processor = javax.media.Manager.createProcessor(processor_ds);
                .......The problem I have is when using both datasources, video and audio creating a merged datasource.
Then the framerate drops down to 1-2 fps. I tested this on 2 different pc's here with 3 different cameras and with JMStudio it seems to be the same.
When capturing audio and video and transmitting via RTP, the resulting frame is showing video with 1 fps (Monitor seems to be ok).
When using AVTransmit2 and AVReceive2 without any changes I've the same problem.
Is this a known problem that a webcam stream with audio can't be transferred fluently?
When needed, I can post the code of my two classes to be run standalone (It's quite big so I leave it out for the moment).
Thanks in advance for your help.
Edit:
This is the output of my capture program:
Found video capture device vfw:Microsoft WDM Image Capture (Win32):0
DataSource created for device
Supported video format: YUV Video Format: Size = java.awt.Dimension[width=640,height=480] MaxDataLength = 460800 DataType = class [B yuvType = 2 StrideY = 640 StrideUV = 320 OffsetY = 0 OffsetU = 307200 OffsetV = 384000
Supported video format: RGB, 160x120, Length=57600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=480, Flipped
Supported video format: RGB, 176x144, Length=76032, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=528, Flipped
Supported video format: RGB, 320x240, Length=230400, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=960, Flipped
Supported video format: RGB, 352x288, Length=304128, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1056, Flipped
Supported video format: RGB, 640x480, Length=921600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1920, Flipped
Setting capture format to RGB, 640x480, FrameRate=30.0
Input format for RTP conversion: RGB, 640x480, FrameRate=30.0, Length=921600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1920, Flipped
Track 0 is set to transmit as: JPEG/RTP, 640x480, FrameRate=30.0
Input format for RTP conversion: LINEAR, 44100.0 Hz, 16-bit, Stereo, LittleEndian, Signed
Track 1 is set to transmit as: gsm/rtp, 8000.0 Hz, 8-bit, Mono
- Set quality to 0.5 on com.sun.media.codec.video.jpeg.NativeEncoder$1$QCA@1ef9f1d
RTP channel opened to: 192.168.0.101 5030
RTP channel opened to: 192.168.0.101 5032and here the output of the receiver / playback program:
  - Open RTP session for: addr: 192.168.0.101 port: 5030 ttl: 1
  - Waiting for RTP data to arrive...
  - Received new RTP stream: gsm/rtp, 8000.0 Hz, Mono
      The sender of this stream had yet to be identified.
  - Received new RTP stream: JPEG/RTP
      The sender of this stream had yet to be identified.
  - A new participant had just joined: otto@meier
  - A new participant had just joined: otto@meier
  - The previously unidentified stream
      gsm/rtp, 8000.0 Hz, Mono
      had now been identified as sent by: otto@meier
  - A new participant had just joined: otto@meier
  - The previously unidentified stream
      JPEG/RTP
      had now been identified as sent by: rlamotte@chumbellAlso curious and it would be great if somebody could bring some light in there:
I thought I would need two RTPManager instances for the two tracks, so one socket for audio and one for video.
But, when I configure my receiver only for listening on port 5030 with one RTPManager instance, I get both, audio and video.
So, what's the normal approach to implement a video/audio transmission via RTPManager?
Edited by: user2293420 on 18.07.2011 08:33

Answer 1

Hi captfoss,
thanks for your reply.
"Highest possible quality" and JMF shouldn't really ever be used in the same sentence. JMF is 10 year old streaming technology designed to run on machines and networks that were medocre 12 years ago.I know, I was just referring to the 640x480 resolution. This has to be enough for the moment and maybe next year we will see how to improve that.
So, make your code work like the AVTransmit2 example works. They wrote the example the way they did for a reason.That's exactly how I have it implementing now.
private String createTransmitter() {
         DataSource ds = dataOutput;
         PushBufferDataSource pbds = (PushBufferDataSource)dataOutput;
         PushBufferStream pbss[] = pbds.getStreams();
          SendStream sendStream;
          String portNumber;
          String media = "";
          int port = 0;
         try {
              for (int i=0; i<pbss.length; ++i) {
                   SessionAddress address = null;
                   RTPManager mgr = RTPManager.newInstance();
                   if (pbss.getFormat() instanceof VideoFormat) {
                    port = localVideoPort;
                    media = "Video";
                    videoRtpManager = mgr;
               else if (pbss[i].getFormat() instanceof AudioFormat) {
                    port = localAudioPort;
                    media = "Audio";
                    audioRtpManager = mgr;
               address = new SessionAddress(InetAddress.getLocalHost(), port);
               mgr.initialize(address);                    
               mgr.addTarget(address);
                    sendStream = mgr.createSendStream(ds, i);          
                    sendStream.start();
                    portNumber = (port == SessionAddress.ANY_PORT) ? "ANY_PORT" : Integer.toString(port);
                    log.info("RTP Manager for " + media + " capture created on port " + portNumber);
     } catch (Exception e) {
          e.printStackTrace();
          return e.getMessage();
          return null;
It's working very good and stable now. When I transmit H263_RTP video with DVI_RTP audio, it works great (apart from the resolution, just the FPS which are constantly > 25). The problem is when I change to JPEG_RTP (640x480) with DVI_RTP, then the framerate drops until 1 FPS. This just the encoding part of the processor, the capture format of the datasource is always RGB and it not touched when I change the encoding format.
Also, I can transmit JPEG_RTP with 640x480 and 30 FPS but only when I disable the audio track in the processor.
Any clue why this could happen?