What are the maximum sustainable streaming rates to USRP RIO

Question

Hi,
I am using a USRP-2942R.  Using one of the example VIs (host based), we created a VI which would record 2 or 4 channels simultaneously to a data file (we use two such devices for the 4 channel capture).  We would like to do this continuously for as long as possible.  By recording to our Solid State Drive, we can achieve rates of about 20MS/s per channel when capturing two channels or 10 MS/s per channel when capturing 4 channels.  However, we cannot seem to go much above this rate. Even if we setup a RAM drive for the file, we still have issues above 20 MS/s for two channels.  So, it does not seem that the file writing is the bottleneck.
I am wondering if there is any example code for doing this exact function.  Perhaps we have done something wrong in modifying the example we had.  Do you know what sample rates we should expect to achive when recording to a RAM file.  By comparison, if I run these same USRPs on this same PC but using the Ettus UHD software under Linux Ubuntu 14.04, I can achieve 100MS/s per channel on two channels when recording to a RAM file.
Thanks.
Rob Kossler

Answer 1

As a quick aside:
PCIe gen 1 is 250 MB/s per lane. The USRP RIO supports PCIe gen 1 x4, for 1 GB/s, but that 'G' is 1000, and not 1024. So in terms of bytes, it ends up being ~953 MB/s. Next, we have to remove some of the throughput for packet overhead and lastly the read response packet size. The larger the read response packet, the less packet overhead incurred. In the end, the USRP RIO can stream about 832 MB/s to the host. 800 MB/s is a nice, round number though.
Each channel has an 16-bit I and Q portion, and there are two channels. Setting the IQ rate (sample rate) to 120 MS/s results in 120 MS/s * 2 channels * 4 bytes/channel = 960 MB/s, which exceeds the total throughput of the PCIe bus. It is impossible to continuously stream both channels at 120 MS/s back to the host. However, lowering the sample rate to 100 MS/s results in 800 MB/s. Or, if you use just one channel, you'll be at 480 MB/s.
That takes care of the throughput, but there are additional issues. There are other things on the PCIe bus, and so bus hiccups may occur and the memory controller may not service the read requests from the device fast enough. If data is not pulled out of the USRP's FIFOs fast enough, the FIFOs will fill up. You can increase the size of the FPGA's DMA FIFOs in the target's settings in the project. A larger DMA FIFO will make the USRP more resilient to these hiccups.
Next, after the data is transfered from the USRP to the host PC's memory, the host PC has to do something with it. If you have complex math or graphs, the rate at which the host PC can pull data from its FIFO may be too slow. If the host's FIFO fills up, then no more data from the FPGA can be transferred. So, you can (1) increase the host's DMA FIFO buffer depth, or two, increase the rate at which the host pulls data from its FIFO (i.e. reduce CPU computation).
There are additional tweaks, such as using Zero Copy DMA, too, which can yield large improvements to streaming performance.
So, in conclusion, high throughput streaming is hard, but it's possible. The Sample Project has to cover a wide range of use-cases and care-abouts, so it cannot perfectly solve every problem. But, it can stream at full rate. By default, the FPGA's FIFOs' buffers are around 1000 elements. If you set the sample size to less than 1000, and set the sample rate to 120 MS/s, and do bursty/finite streaming, you will be able to stream at full rate. Your data will not be contiguous though. To do continuous streaming, you'll need to look into the above suggestions.
Hope this helps!