How can I sync with audio framerate?

Question

I hope that there is an entirely different way of doing what I'm trying to do because I've run up against what seems like an insurmountable obstacle. I have been working on a project for many months and the last piece to figure out is synchronizing application actions with the audio framerate. I have seen other apps doing this successfully (but proabably written in C++, not Java) so it can be done with standard Wintel boxes - but I'm now concerned that it cannot be done using Java Sound.
My project attempts to animate sound with visual images (not video or pictures but more abstract visualizations based on the audio itself). It works fine when simply "listeing" and responding to the audio. The final step is to be able to sync specific abstract visualizations with precise points in the audio (frames or position in microseconds). I'm now wondering if what I need is handled by the soundcard (hardware) and not available through Java. I presumed that I would be able to rely on the soundcard to handle the realtime audio stream (using its clock) and I would simply wait for increments of a certain frame count that correspond to the refresh/trigger times for my visualizations. As it turns out, after a lot of experimenting with my configuration, I can't find a way to trigger a method in my application synchronized wiht a certain period in the audio being played. I am playing audio by writing to a SourceDataLine, and listening (using loopback, I believe) on a TargetDataLine. To give an idea of what I'm looking for, if a LineListener supported notification based on reaching a certain frame position in the audio stream, that would be great. That doesn't exist (that I could find, only START/STOP/OPEN/CLOSE events). I've been trying to set the buffer size on the TargetDataLine, and the number of bytes (frames) that I read from that TDL, to achieve a synchronization with the actual rate the audio is playing through the sound card. What I've seen is that the number of frames made available through TDL is not consistent. The read() method blocks for various times and with differing incremental frame positions as the TDL buffer is replenished in real time.
My goal is to be able to detect when a number of frames corresponding to 20 milliSeconds at 48kHz or 44.1kHz sample rate have been played through the sound card. Is this possible? I've not posted any code because the code I have isn't non-functional, it's just not capable of performing the task I am trying to achieve. Suggestions as to how I can refine my existing approach, or what other direction/implementation I can pursue will be greatly appreciated.
Edit: typo.

Answer 1

ags wrote:
I think this breaks down into two related but separate issues:
First, the latency between writing audio (to a SDL) and when that audio is actually heard. If I could find a way to measure that, and it was consistent, I could adjust my processing and delivery of audio data to that SDL. As I tried to explain in my previous post, there will ALWAYS be a variability in the time when a given frame is processed in an SDL and when it sounds. Biggest variability is between the time required for the onset of the first sound (played via bytecode interpretation) and onsets of subsequent sounds (if played via code loaded into memory). But JVM switching also accounts for variability.
Have you seen this article?
http://quod.lib.umich.edu/cgi/p/pod/dod-idx?c=icmc;idno=bbp2372.2007.131
"Real Time Low Latency Audio Processing in Java"
It does a good job of listing various factors that affect real-time audio latencies.
The other problem is that even if I were able to get an accurate measurement of the delay (latency) in the audio lines, if I then set system timers to cause action in sync with the audio frame rate based on audio sample rate (frame rate for uncompressed PCM audio) there is still an error path. Again from observation not direct knowledge, it seems that there is no direct link between the sound card clock (controlling real time frame rate and what should really be the master of all actions) and the system clock (which I would use as a surrogate for the actual sound card clock to synchronize my actions with the audio being heard). Even if I were to establish perfect synchronization at the start of a song, by the time 5 or 10 minutes of audio has been processed, I suspect there will be an accumulation of errors between the sound card clock and the system clock that becomes noticeable.The sound card clock and the system clock are perfectly in synch. The problem is the granularity of the system clock. With Microsoft, the clock info is only updated once every 15 msec or so. As a consequence of this, System.currentTimeMillis() will only report the time at the last update. Also, Thread.sleep(millis) and setting a Timer are subject to this constraint. The error doesn't accumulate, though. It is just an issue of having a reduced "granularity" that creates a predictable error. (System.nanoTime will always be more accurate--it is not subject to letting the OS determine the update frequency.)
As I wrote, there is a hack that allows you to increase the granularity of the OS clock interrupt. It involves setting an independant, background thread to sleep for the maximum amount of time, e.g. Thread.sleep(Long.Max_Value). This somehow forces the OS to use a higher granularity, and with Microsoft the accuracy becomes about 1 msec (comparable to Linux and Mac systems). I learned this at the forum Java-Gaming.org.
>
I'm open to ideas, criticism or examples of how I might acheive my goal. While it might not be practical or the only way, the best I can describe what I'm looking for is to have an audio-event driven system where I can determine a frame position corresponding to the audio being rendered in real time and trigger events to keep other actions synchronized with that audio. If a constant delay (latency) was maintained and measureable, that would be OK.
This is a difficult problem.
Agreed about the difficulty!
Given all this, I think the questions are:
1) how much in advance do you know about given frames that you wish to use as cues?
2) how well can you guestimate an anchor point? (and, is there a way to improve upon the guess with feedback?)
3) how accurately can you trigger the visual cue?
Regardless of when Java decides to process a frame, it DOES manage to keep the playback steady. Thus, if you focus your efforts on finding an ANCHOR (for example, the nanoTime, or "close enough" approximation, when the first frame sounds) that you can use as a reference point for the duration of the sound, then you can use that point to calculate conversions between the frame count of the sound and the time it can be expected to play.
For example, if the frame is 44100 and you estimate frame 0 was at time 1336426670003 (System.currentTimeMillis), then set a Timer, in advance, to trigger the visual cue at 1336426671003 (I'm adding 1000 msec). The level of accuracy will remain the same regardless of the frame number and duration involved as long as you use the same anchor (and the sound playback is not interrupted).