Fastest array operations on a 2D circular buffer

I'm trying to create a a type of circular buffer on large-ish set of 2d data which are samples of voltage from a data acquisition board. The boards are Measurement Computing and a lot of the nice programming features built into DAQmx for NI boards aren't easily available to me.
I'm grabbing chunks of samples, 1000 per channel * 64 channels, every 100ms. I'm calling this a 'page'. For each page of samples I take a median for noise filtering and then I publish this median for multiple threads to use. I also want to be able to string together pages of samples as if it were one longer data acquisition, for up to 30 seconds. I'm doing it this way because the threads that are expecting their data every ~100ms can't release these AI boards for long periods of time to allow other threads to use them to perform long scans. The data coming back from my boards is a 2D array.
I have enough RAM available to pre-allocate the memory to hold all these pages and I've been playing with the In Place structures for awhile now and I haven't been able to land on the magical combination that will allow me to replace any page in the buffer. It's easy enough using the subarray option of the in Place to replace either the first page or the last page but it gets more complicated to do a page somewhere in the middle without having to resort to Case statements. I tried to do it with nested In Place structures but it seems as if the subarrays that get created in the lower levels already go out of scope by the time the top level gets it assigned and I just get jibberish on the output side. Does this make sense?

SmokeMonster wrote:
I tried to do it with nested In Place structures but it seems as if the subarrays that get created in the lower levels already go out of scope by the time the top level gets it assigned and I just get jibberish on the output side. Does this make sense?
Sorry, but at least to me, this doesn't make sense. Can you post your code? I can't see how "out of scope" is a concept that applies here - LabVIEW keeps track of the memory for you and should never lose track of memory that's still in use.
I posted one approach to a 2D circular buffer; maybe it's of some use to you.

Similar Messages

How to implement multiple queues in one circular buffer?

Spoiler (Highlight to read)
Recently I have successfully shared data of 2D array among different vis in a project, using circular buffer.
However, there is only one circular buffer allowed in one project for one data type.
I want different vis, running and getting data from circular buffer in different speed, all getting data as a queue from circular buffer. Like the circular buffer's read continuous mode.
The problem is that after a vi reads in continuous mode, the read data is deleted from the circular buffer. Then other vis can not use them any more and can not keep track where they should begin to read again.
So if it is possible that there are multiple pointers or queues in circular buffer to keep track the progress of different vis, and data only update when it gets input from DAQ but not deleted when read?

LawrenceChong wrote:
Spoiler (Highlight to read)
Recently I have successfully shared data of 2D array among different vis in a project, using circular buffer.
However, there is only one circular buffer allowed in one project for one data type.
I want different vis, running and getting data from circular buffer in different speed, all getting data as a queue from circular buffer. Like the circular buffer's read continuous mode.
The problem is that after a vi reads in continuous mode, the read data is deleted from the circular buffer. Then other vis can not use them any more and can not keep track where they should begin to read again.
So if it is possible that there are multiple pointers or queues in circular buffer to keep track the progress of different vis, and data only update when it gets input from DAQ but not deleted when read?
THe last time I wrote a multi-subscriber cirular buffer was in LV 6. Since then I have found that polymorphic queues perform much better so I now use multiple queues. One queue for each subscriber of the data. The subscribers are then responsible for maintaining their own histories.
So just use multiple queues.
Ben
Ben Rayner
I am currently active on.. MainStream Preppers
Rayner's Ridge is under construction

How to handle massive array operations

Hi!,
We are working with large arrays of the order of 5000 x 100000 (rows x col).
We wish to perform typical array operations like replace subset, indexing, rotating and addition.
Our questions:
1) How do we work with this array within the same VI without creating indicators or controls. Wiring, we understand, is an option but then the wires will clutter our screen.
2) How do we pass/retrieve this array data to/from sub-VIs without creating indicators or controls.
3) What the fastest recommended methods of handling such large arrays.
4) Our array will be either Boolean or U8. Will both data types give similar speed results?
5) When using the replace subset function, can the subset be same dimension as original array?
6) Can we rotate a particular row within a 2D array? As of now, we extract the row as 1D, rotate and then replace the original row with rotated row.
Any ideas/suggestions (specific to array handling) to help speed our VI will be highly appreciated.
We are using LV 7.1 on Win2000
Thanks,
Gurdas
Gurdas Singh
PhD. Candidate | Civil Engineering | NCSU.edu

Saverio, Joe, Jason and Benoit - Thanks!
Reply to Saverio's inputs:
(1) Not sure what you mean by this. If you need to work with that data in the same VI you use wires - that's the most efficient way of doing it. Are you thinking about local variables or something?
No, we avoid locals like birdflu! What I meant was how do I carry data from one part of my block diagram (BD) to another without using wires (because wires clutter the block diagram)? Is there no variable/container which will hold data without it being a control/indicator? Some of my front panels will be displayed. In such VIs, the moment I use a control/indicator in the BD to "hold" my data, my speeds drop.
(2) OK
(3) OK
(4) LabVIEW stores Booleans as 8-bit data. You should choose the one that makes more sense for your application.
U16 makes most sense.
U8 also does the task, only catch is at the end I need to convert the array into U16 for the last operation to happen (all elements in a col are added; thus final array is 1D row array).
I would use Boolean ONLY if it gives significant speed gains over U8 (I will need to convert Boolean to U16 at the end).
Any inputs/ideas?
(5) OK
(6) You will need to be a little more specifc about this one, though I suspect the Transpose 2D Array function coupled with the Replace Array Subset is what you're looking for.
Lets say the first row of my 2D array has 5 elements, which are 3 6 12 8 1. I want this to become 12 8 1 3 6. In effect I left rotated my first row my two positions.
Rgds,
Gurdas
Gurdas Singh
PhD. Candidate | Civil Engineering | NCSU.edu

Implementing a circular buffer - which approach works best?

I'm just implementing a circular buffer (or a circular queue, to be precise). Now I wonder which is better for wrapping around when inserting:
Using the modulo operator or using some variable to store the insertionPoint and incrementing it after each insert?
Does it make any difference in performance / readability?
Greets, Oliver

Trollhorn wrote:
Uh, that article is really nice... thanks...Is that sarcasm, or is it "prob sol"?
My suggestion would simply be: Why roll your own, when you can just use a standard implementation provided by the JDK (you do know about the java.util.Queue interface and can find standard classes which implement it, right?).
So is this just for an academic exercise, or are you one of those premature optimizer kinda guys feeling the need to eek out every last millisecond of processing and/or save every last byte of memory? And maybe you fear the reaper (garbage collector) too.
I don't see a real need for a "circular queue". Circular buffers, circular lists yes (where you iterate over it in a circular fashion), but when I think of a queue I think of FIFO pushing and popping elements, not iterating over it.

Circular buffer

I am using circular buffer as provided here to store 2D array of DBL data.
Is there a way to have multiple buffer of same type? Need three circular buffer of DBL type to acquire and save data to its respective circular buffers.
Reason to store the data separate from each module is to use the respective buffer during analysis.
Any suggestions, thanks.

This is sounding more and more like what you really want is a Producer/Consumer architecture with many consumers. Use a different queue for each consumer.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

How to buil a circular buffer of message

Hi, I have a GPS that sends me their information every 30 seconds, I'm receiving this information for the serial port and I should send this information by another serial port, but every 90 seconds or more, the transfer time is variable according to the performs of the microcontroller, we need to know how buil a circular buffer of tx message so as not to miss any reading that I sent me the GPS.

One way would be to have the serial port read in one loop, placing the received messages onto a queue, with the writing done in another loop, popping the messages off the queue as needed. Of course, if you are "loading" the queue every 30 seconds, and "unloading" the same amount only every 90 seconds, you will eventually run out of storage, filling up the predefined queue size. So you will need to determine how long that would be, and how you want to handle it. This is a problem that will occur with whatever method you use, also understanding that the longer you run, the further behind the actual GPS location data you will be. In LabVIEW 8.x, click "File" and New, there is an example of a producer/consumer architecture.
Message Edited by LV_Pro on 01-27-2009 01:28 PM
Putnam
Certified LabVIEW Developer
Senior Test Engineer
Currently using LV 6.1-LabVIEW 2012, RT8.5
LabVIEW Champion

Circular Buffer implementation for LabVIEW 8.5

Greetings,
Several weeks ago I downloaded the implementation of a circular buffer for LabVIEW 8.5 from ni.com. The current webpage is:
http://zone.ni.com/devzone/cda/epd/p/id/5883
Unfortunately, this webpage has been updated since then and the available files for download (swcircularbuffer1.0.19.zip) are for LabVIEW 8.6.
So my question is: where can I find the implementation of a circular buffer for LabVIEW 8.5 ?
Thank you very much!
telmo
Solved!
Go to Solution.

Thanks Smercurio, I needed this too.
After looking at the code and seeing a note on the download page concerning a possibly more efficient method I'm wondering whether anyone has tried it yet. I'm thinking that using the in-place structure to simply overwrite a pre-defined memory space would indeed be a better implementation of a circular buffer than the somewhat kludgy bounds checking and data wrapping that the NI code provides. I plan to work on it myself of course but my boss gave me one of those "it would be great if we could have it tomorrow" talks that I'm sure most LV coders know only too well. [While the C++ guys get two weeks to write Hello Wurld. ]
Anyway, the app is quite simple: Run my USB-6251 card with a single channel as fast as possible (1.25MS/s) collecting to the circular buffer. When a trigger is pressed, store the buffer so I get a file with a few mSec of pre-trigger data.
I haven't yet tried the NI code so it may be sufficient for my needs but if anyone is working with the in-place structure and has ideas, code or suggestions I'd love to hear them.
Using LabVIEW: 7.1.1, 8.5.1 & 2013

Problem with circular buffer

Hi,
For some reason, whenever my circular buffer is filled and wraps around, right at that moment, my graph screws up for a split second by drawing a line across the graph. But for some reason, this doesn't happen all the time, only sometimes.
The picture is below:
Attachments:
graph jump.JPG ‏105 KB

Sorry, made a new thread since that one showed "Solved".
It seems the buffer works fine, something is wrong with the data being inserted in there. I don't know why it only messes up with that data.
I managed to capture some messed u p data. I will post it below.
Message Edited by Bilal_J on 04-12-2010 02:46 PM
Attachments:
Circular Buffer Debug.vi ‏17 KB
Circular Buffer.vi ‏39 KB
Circular Buffer Action Enum Typedef.ctl ‏5 KB

Global variables for circular buffer

Hallo!
I have another question:
I need to have a variable circular buffer size. So I would like to define the circular buffer size by a global varable.
Is that possible? I tried to use the global variable like {VAR_01} to insert in the desired field for circular buffer size, but dasylab only puts a zero in the field.
Thank you and have a nice day.
Hilby
Solved!
Go to Solution.

Unfortunately it is not possible to have a variable-sized buffer. Right-click into the corresponding textfield: the context menu does not show an entry to choose the global variable from a list.
M.Sc. Holger Wons | measX GmbH&Co. KG, Mönchengladbach, Germany | DASYLab, DIAdem, LabView --- Support, Projects, Training | Platinum National Instrument Alliance Partner | www.measx.com

Enumeration with Circular Buffer

Hi, im having trouble with the enumeration of Circular Buffer .
I had compilation errors with the Ring Buffer Class, it keeps pointing to public Enumeration elements () {return new RingBufferEnumeration(this);}
Any advice ? Thanks
This is the RingBufferEnumeration Class code
public class RingBufferEnumeration implements Enumeration {
private RingBufferNode firstFree, firstFilled;
     public RingBufferEnumeration(RingBufferNode a) {
          firstFree = new RingBufferNode(a);
          firstFilled = firstFree;
          firstFree.next = firstFree;
public boolean hasMoreElements(){
return firstFilled != firstFree;
public Object nextElement () {
return firstFilled= firstFilled.next;
}this is the Ring Buffer Code
public class RingBufferQueue implements Queue {
      * initialize an empty ring buffer queue
     public RingBufferQueue () {
          firstFree = new RingBufferNode(firstFilled);
          firstFilled = firstFree;
          firstFree.next = firstFree;
     private RingBufferNode firstFree, firstFilled;
      * Determines whether the collection is empty
      * @return true if the collection is empty
     public boolean isEmpty() { return firstFilled == firstFree; }
      * Determines number of elements in collection
      * @return number of elements in collection as integer
     public int size () {
          int count = 0;
          RingBufferNode p = firstFilled;
          for (; p != firstFree; p = p.next) count++;
          return count;
      * Yields enumerator for collection
      * @return an <code>Enumeration</code> that will yield the elements of the collection
      * @see java.util.Enumeration
     public Enumeration elements () {return new RingBufferEnumeration(this);}
      * add a new value to end of the collection
      * @param value element to be inserted into collection
     public synchronized void addLast (Object val) {
          if (firstFree.next == firstFilled)
               firstFree.next = new RingBufferNode(firstFree.next);
          firstFree.value = val;
          firstFree = firstFree.next;
      * access the first value in collection
      * @return element at front of collection
      * @exception java.util.NoSuchElementException no matching value
     public Object getFirst () {
          if (firstFilled == firstFree) throw new NoSuchElementException();
          return firstFilled.value; }
      * remove first value in collection
      * @exception java.util.NoSuchElementException no matching value
     public synchronized void removeFirst () {
          if (firstFilled == firstFree) throw new NoSuchElementException();
          firstFilled = firstFilled.next;
class RingBufferNode {
     public Object value;
     public RingBufferNode next;
     public RingBufferNode (RingBufferNode n)
          { next = n; }
}

im not sure is it a RingBufferNode but, im sure it is not indexed data for ring Buffer
My IndexDeque is working correctly
public Enumeration elements () { return new IndexedEnumeration(this); }IndexedEnumeration
public class IndexedEnumeration implements Enumeration {
      * initialize newly created IndexedEnumeration
      * @param d collection to enumerate over
     public IndexedEnumeration (Indexed d) { data = d; }
     private Indexed data;
     private int index = 0;
      * see if enumeration should continue
      * @return true if enumeration has at least one more element
     public boolean hasMoreElements () { return index < data.size(); }
      * get next element in enumeration
      * @return value of next element in enumeration
     public Object nextElement () { return data.elementAt(index++); }
}

Error: Non-array passed to JNI array operations

Can anyone see what I am doing wrong? I am trying to go through the array of objects example in the online JNI text book "The Java Native Interface Programmer's Guide and Specification" by Sheng Liang. The book and pertinent chapter can be found here:
http://java.sun.com/docs/books/jni/html/objtypes.html#27791
The error I received on running the program with the -Xcheck:jni switch is this:
FATAL ERROR in native method: Non-array passed to JNI array operations
     at petes.JNI.ObjectArrayTest.initInt2DArray(Native Method)
     at petes.JNI.ObjectArrayTest.main(ObjectArrayTest.java:14)Without the xcheck switch, I get a huge error log file that I can reproduce here if needed.
My c code with #ifdef _Debug statements removed for readability is listed below. The location of the error (I believe) is marked:
#include <jni.h>
#include <stdio.h>
#include "petes_JNI_ObjectArrayTest.h"
//#define _Debug
// error found running the program with the -Xcheck:jni switch
//page 38 and 39 of The Java Native Interface Guide by Sheng Liang
JNIEXPORT jobjectArray JNICALL Java_petes_JNI_ObjectArrayTest_initInt2DArray
(JNIEnv *env, jclass class, jint size)
     jobjectArray result;
     jint i;
     jclass intArrCls = (*env)->FindClass(env, "[I");
     if (intArrCls == NULL)
          return NULL;
     result = (*env)->NewObjectArray(env, size, intArrCls, NULL);
     if (result = NULL)
          return NULL;
     for (i = 0; i < size; i++)
          jint tmp[256]; // make sure it is large enough
          jint j;
          jintArray iarr = (*env)->NewIntArray(env, size);
          if (iarr == NULL)
               return NULL;
          for (j = 0; j < size; j++)
               tmp[j] = i + j;
          (*env)->SetIntArrayRegion(env, iarr, 0, size, tmp);
          (*env)->SetObjectArrayElement(env, result, i, iarr); // ***** ERROR: Non-array passed to JNI array operations
          (*env)->DeleteLocalRef(env, iarr);
     return result;
}Here is my java code
package petes.JNI;
public class ObjectArrayTest
    private static native int[][] initInt2DArray(int size);
    static
        System.loadLibrary("petes_JNI_ObjectArrayTest");
    public static void main(String[] args)
        int[][] i2arr = initInt2DArray(3);
        if (i2arr != null)
            for (int i = 0; i < i2arr.length; i++)
                for (int j = 0; j < i2arr[0].length; j++)
                    System.out.println(" " + i2arr[i][j]);
                System.out.println();
        else
            System.out.println("i2arr is null");
}Thanks in advance!
Pete

Niceguy1: Thanks for the quick reply!
So far, I mainly see the usual differences between C and C++ JNI calls. Also, in the example you gave, they use a lot of casts, but that doesn't seem to make much difference to my output. Also, in the example, they create a 0-initilized int array, and use this class to initialze the class type of the Object array. I get the class type by a call to FindClass for a class type of "[I". I'll try this their way and see what happens...
Edit: Changing the way I find the array element Object type did not help. I still get the same error as previous. Any other ideas would be greatly appreciated.
Message was edited by:
        petes1234

I'd like to know how I can acquire a waveform on a loop and do a circular buffer with this waveforms I will acquire, cause I want to reduce the noise.

I'm using trigger for this data acquisition.

deco,
If you really want a circular buffer, how much data do you want it to hold? One acquisition? Two? Three? Once this is decided, you will need to create the waveform and then continuously strip the old data out and place the new data in. This will add a bit of overhead to your VI.
There are also software noise filters in LabVIEW that you can use to reduce the noise in your signal.
Lastly, if there is a lot of noise, you may want to consider an SCXI system which will filter out most of the noise as the data is being acquired.
Randy Hoskin
Applications Engineer
National Instruments
http://www.ni.com/ask

Circular buffer error with DIO32HS

Hello,
I am using a DIO32HS in pattern buffered I/O with external clock in Labview and in Visual C.
I lose a sample when the circular buffer has wrapped the first time. It is always the 17th sample after a full buffer. If I change the buffer size, it moves along.
I tested it in Labview 6i and with Visual C. I can repeat the test and the error is always in the same spot. There are no error warnings, not in Labview or in C.
I tried it with another DIO32HS and it makes no difference.
Has anyone seen this before?
How can I solve the problem?

Hello,
I am not sure where the problem lies in your application, but here is one test you can run to see if the card is working correctly and it may be something that we are missing in the programming of the card.
You can open the Pattern Input example and the Pattern Output example VIs in labview.
Next, tie all the data lines of port 0,1 to the data lines of port 2,3. Also tie REQ1 to REQ2 (clock line). Then you can start one example with the settings of external clock, then start the other program using the internal clock. This will insure that we start the slave ports first, then the master ports will start the transfer of data after the slave ports are ready.
Hope fully you do not see the same behavior, but if you do, you may wish to contact NI technical
support. For contact information visit www.ni.com.

Is there any way to create a circular buffer using the INSERT ARRAY function?

I saw the example using the REPLACE ARRAY function, but it uses too much CPU power on my laptop. I'm using it for a very sensitive application and saw that the INSERT ARRAY function used much less CPU power.
I am also not wiring the index on the INSERT ARRAY function so that whatever is read from the buffer is added to the array.
However, since it is not indexed, I don't know how to set the "write" index back to 0 when it reaches a certain sized array (ie 1000 elements). I was looking for an array property node, but couldn't figure out if one exists for the "current write index".
Can anyone help?
Thanks

I will try to answer this question to my best understanding. I apologize if I interpreted the question wrong.
You are using the "Insert Array" vi without wiring the index. By doing that, new elements will be added (appended) to the array.
If you just want to know the current index to stop your acquisition when the array reach certain amount of elements, then you can use the "Array Size" vi to keep track of the size of the growing array. Remember that Array Size = last index + 1 because the index start at zero.
A second option is that you may want to start over the array when it hits the maximum number of elements that you will allow. If that's the case, then you may want to re-initialize the array to a NULL state by stablishing a condition (for example, when the
size of the array is 1000, re-initialize). I provided an example attached to this message showing how this can be accomplished. In the example, an 1D array grows in size using the "Insert Array" vi and inserting a random number. When the array reach 4 elements, it is re-initialized. You can expand this example by saving the array first to a file or some other holder before re-initializing the array. The example is in LabVIEW 6.
Finally, if you want to replace the current values with new values, I think you do not have other choice but to use the "Replace Array" vi.
Hope this can be of help.
Regards;
Enrique Vargas
www.vartortech.com
Attachments:
array_example.vi ‏20 KB

Operating System's I/O Buffer SIZE

Hi There,
can anybody tell me how to count Operating systems I/O Buffer size..?
i have Win 2000 Server with Ora. 9.2 on it
Thanx In Advance!!

Hi,
Run some statspack snapshots, then take some reports, you will find indication if this parameter is set as well for your application into average blocks by read into datafile section.
db_file_multiblock_read_count=16Take care, a high value encourage full table scan over index usage.
Nicolas.

Fastest array operations on a 2D circular buffer

Similar Messages

Maybe you are looking for