Producer-consumer: optimize data acquistion rate

I have a producer-consumer type of application in which one loop continuously reads data from the card and queues it and another loops reads/saves data. Additionaly one more loop plots the data. The number of channels and real-time analysis is defined by the user so the time it would take to save, display, and analyze each block (one iteration of Analog/Digital Read) of data may vary for different settings.
Currently I am setting the "number of samples to get" large enough on the Analog/Digital Read at the start so I don't get queue overflow. Now I would like to adjust it automatically while the program is running so I don't get any queue overflow and at the same time the data is not display in huge chunks at a time. Has anyone done this sort of thing before? Right now the only idea I have is to read the # of elements in the queue continuously and use some mathematical formula to increase or decrease # of samples.
Solved!
Go to Solution.

Abdel2,
Reading the number of elements in the queue continuously and using a mathematical formula to increase or decrease the number of samples would be a great method. You could also use the Decimate 1D Array Function to display a smaller grouping of your data if your issue is that you don't want to display huge chunks of data at time.
Ben Sisney
FlexRIO V&V Engineer
National Instruments

Similar Messages

How to change the transmission rate of data flow in producer/ consumer pattern

Hi All
I am new to labview and trying to design a application to transfer image data in different rates, that means I can change the transfer rate in the transmission process. In fact,I refer to the the Queue Basic.vi to achieve this function by Changing the delay value of dequeue and enqueue.The program is divided into three parts:
The first part is the sender in vc used to split and send image data.Data size is 226kb and each time we send a 1kb packet;
The second is labview rate-change part，use producer/ consumer pattern and queue to transfer data;
The third one is the receiver in vc,receive data in sequence and synthetic images.
The entire transfer process is: image data was sent from the sender to producer loop through DLL, then enqueue--dequeue in consumer loop--DLL--receiver.The sleep time in vc sender is equal to the delay value in producer loop and this makes it look like the data generated as in the loop.
Now this is where my dilemma is：
When transferring data, if the producer loop and consumer loop delay value are equal(both 20ms), I can see image synthesis at a constant speed and no data lose; if I change the consumer loop delay value and make it greater than the producer delay(one is still 20ms and another is 50ms),that means receive in a lower rate and store data in queue. I can see Elements in Queue
increase but the image data randomly lost, and the image synthesis speed is not reduced.
My question is why image data loss and I can't see the image synthesis speed reduced when change the delay value in consumer loop?
Do anyone know how to solve the problem? Are there any examples I can refer to? Any suggestions are greatly appreciated!
Attachments:
image1.JPG ‏56 KB

thisoldman,
I don't have '/usr/lib/modules/3.17.3-1-ARCH/build/Documentation/VGA-softcursor.txt' on my file system. Does it contain the info about changing the blinking speed?
P.S. The other two links I found yet before starting this thread. I only found in them the recommendations about changing "the color and shape of the cursor and turning blinking on/off" (as I mentioned in the original question). Did I miss the info about the blinking speed?
PPS: I found the vga-softcursor.txt here: https://www.kernel.org/doc/Documentatio … cursor.txt
But that file doesn't tell about changing the blink rate either.
Last edited by nbd (2014-11-23 20:55:41)

Data delay in producer consumer loop?

Hi,
I've been creating an VI using producer consumer loop. In producer loop, every loop take 1000 samples at the rate of 40000/s(or 40000s/s devided by number of channels) and the consumer loop will average the data in a way corresponding to the "sampling rate" value on the front panel. The "sampling rate"represents the data rate on the chart and in the saved data file. If it's too slow,say 1 sample per second, then the producer loop will just wait for some time.
So far the VI run smoothly, but there's one big problem. Each time I adjust the amplitude of input signal and the change doesn't show up on the chart untill several seconds later. When the "sampling rate" is really slow, the delay could be 1 minute! It's the first time I have ever used producer consumer loop and I 'm really confused of what causes this time delay.
Could someone give me a hand with this?
Thanks!
Yaqiong
Attachments:
parallel.vi ‏73 KB

You should consider further separating your tasks. For instance, the writing to file should be in its own task since file I/O can really cause delays in your processing. Also, I would separate your data processing (averaging) from the UI updates. The UI updates can be within a task that is using an event structure. It can get the data updates to update the chart but it can also quickly respond to the changes in your amplitude. I would also be very careful of the wait in your producer loop. This could starve the CPU.
One last note. Your code could really benefit from a cleanup. Your coding style is quite a mess and it makes reading (and maintaining) your code much harder. For example, your code should generally read from left to right. Running wires backwards, having outputs on the left, lots of bends, etc all lead to messy code that is difficult to read.
Mark Yedinak
"Does anyone know where the love of God goes when the waves turn the minutes to hours?"
Wreck of the Edmund Fitzgerald - Gordon Lightfoot

Timed structure for output in producer consumer data acquisition

Hello LabVIEW community,
I have a bit of a problem. I am writing a program that is primarily for data aquisition but has a few control features as well. I need the program to aquire and write several channels of data at a relitively high speed. The program does this fine ( in the top 3 loops fo the program). I need the program to also send a seires of two output signals in response to a particular condition. When a certain channel registers a value of above a specified number, I need an output channel to write an anologue signal for 225 seconds and then a signal of a differnt value to the same channel untill the condition occurs again (not for several hours).
I put this action in a timed structure inside of a case structure. The case structure and timed structure are inside of a second consumer loop. I have no idea if this is ligitimate. This event takes much longer than any of the other loops which run at 1 or 2 seconds. When I exicute the program with "highlight exexutuion" this longer (case/ timed ) loop never executes. I asssume this has to do with the mismatch in time scales of this loop and the other loops in the program. I also didn't really understand the help information on timed structures, so its possible that I just need better/ different inputs to the timed stuctures.
If anyone can see any obvious problem in the code or suggest a better way to do the output function I would really appreciate the help. My code is attached.
Thanks,
Jo

I can't figure out how to fix your code, but I can point out the things I see wrong with it.
1. You are dequeueing elements from the same queue in the same loop. In one you are not taking the element, and in parallel, you are taking an element. I see that in your bottom two loops. Why? That code won't execute until the queue gets two elements in it. And which dequeue gets it first and which gets it second is an arbitrary race condition. So there is no certainly of order (first 4 vs. last 4 currents in the one loop), or which gets kept and which gets discarded (in the other loop.)
2. You are creating and/or clearing DAQ tasks in every loop iteration. Tasks should be created before a loop, used inside the loop (read or write), then cleared once the loop is done. Anything else is wasteful, time consuming, and could lead you to run out of resources.
3. You are using the STOP function which is pretty much like hitting the abort button on the toolbar. No program should ever stop that way. All loops should exit gracefully.
4. You have a data dependncy between your bottom two loops because of the boolean wire running from one to the other. That bottom loop will only run once (if the value is True), or run forever (if the value is false).
5. Use of the dynamic datatype. You are taking waveforms, converting them to a blue wire, then coercing them into another datatype such as just displaying a scalar in an indicator. A lot of unnecessary conversions there.
6. Your thermometer indicators have a digital display you can make visible. Then you won't need the separate numeric indicator to display the value.
7. For loop that runs only 1 time because of the constant wired to the N terminal. Get rid of the For Loop.
8. Check your spelling. The word "Temperatures" on the graph, and one instance of "distillate" is missing an "l".
Until all of these problems are fixed, it is not worth talking about timing of timed structures. Some of these problems were discussed in your other thread. http://forums.ni.com/t5/LabVIEW/producer-consumer-missing-channels/m-p/3159757#M911255 But now it seems like you've made things even more complicated without fixing the basic problems.

Processing acquired data (8 simultaneous channels) in RT host (producer,consumer problem)

Hi Gents,
I have a Crio 9024 and two delta sigma modules NI 9234, I have to acquire 8 cahnnels simultaneously at 51,2 kHz from FPGA and transfert this data via DMA fifo to the RT host for proceesing (FFT for all channels and send data to a host computer via a stream network). I succeed to do this with 25,6KHz as sampling frequency but when I tried with 51,2KHz (my goal) I had problem because in RT code I have a producer loop receiveing data from fpga (DMA fifo) and sending this data to a consumer (Queue) loop in RT also for processing this data but the producer is to fast so the consumer loop can't proccess data in time and the queue is overflow. I have replaced queues with RT fifo,single process variable,... but no luck !!!
Any way how to send 8 channels data from one loop to other for processing (base band FFT N channels) at 51,2KHz ???
Please I need help !!!

DMA Fifo:
To avoid overflow you have to configure the RT side which has a default size of 10000 (Rio4.0 and above 2^14 elements) or doubled FPGA depth (fifo_config)
RT Fifo:
config example
Read data from DMA to RT Fifo
(read all elements available and write this data block to your RT fifo, which is configured to hold n-Blocks of size x, i.e. 300Blocks of 18432elements each)
--> here I suggest to read a multiple of 8, because of your channel count
Read data from RT fifo to stream
(send data blocks to the host where each block is one element in terms of streaming buffer --> configure streaming buffer)
Configuring large RT Fifo's has an impact on memory usage, so have an eye on it.
Screens are just examples (not functional code)
Hope it helps
Christian

Running one sub vi in background while the main vi uses it's generated data, producer/consumer?

Hi,
I would like to have a subrutine running at the same time as the rest of the main vi. The subrutine will be constantly producing data, and the main rutine should do different tasks depending on which data has generated the subrutine in that instant. I have read about producer/consumer design http://zone.ni.com/devzone/cda/tut/p/id/3023 but I don't know how to implement this with LV 5.1 Somebody know if it's possible, or there is something missing in my old LV version? The name of the sub vi about queues mentioned in the link are different than the one I have, and I don't achieve to connect correctly the ones I have... (that are create queue, destroy queue, insert element queue and remove element queue).
Or, does anybody know other way to do what i want?
Please, some help..
Thanks a lot.

Hi javivi,
The queues in 5.1 are capable of supporting the producer-consumer model. See the queue example under "Execution Control Examples". It's showing a simple string value being "inserted" and "removed". To insert non-string data-types, use "Flatten to String" before "Insert Queue Element" then "Unflatten from String" after "Remove Queue Element".
What "different tasks" will you do based on the data? How is the data being read by the program - are you reading an analog-input? Is it a "continuous" acquisition?
Cheers!
"Inside every large program is a small program struggling to get out." (attributed to Tony Hoare)

Can anyone tell me what's wrong with my queue or producer consumer loop?

A Labview engineer helped me with the overall structure of my program, necessitated by the fast data sample rate and post processing requirements. He recommended the producer/consumer loop and using one of the example files (Cont Acq graph voltage - int clk) to do the actual data acquistion. So I put together the loop and go my two channels of data acq going in the producer loop. However, nothing's getting picked up in the consumer loop and none of my data gets displayed over there.
Anyone care to take a look at it and let me know what I'm doing wrong?
Solved!
Go to Solution.
Attachments:
P-C Yanmar Program.vi ‏296 KB

Your Enque Element should be inside the while loop where you aquire the data. As it is now, the data has to stop being aquired before you send the last set of aquired data.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

Producer consumer timing

Back again
Thought I was getting somewhere now I am at another brick wall.
Bit of background:
I have a producer consumer set up. In the producer loop there are 2 DAQmx read VI's reading from a Pressure task and a Temperature task. These are multiple channels, 1 sample, outputtng a 1D wave form. This loop is set to run every 20ms. The two tasks were created using DAQmx Create Channel VI's and DAQmx Start Task VI's. The producer loop feeds a display on the front panel with the sensor values and this hopefully is refreshed at the 50Hz I have asked for. Secondly the loop bundles up the waveforms and puts them in a queue where they are picked up for logging by the consumer loop. In the consumer loop I have a wait 1000ms timer and a Write to Measurment File Express VI to do the logging. This is supposed to log at 1 per second.
The queue is set to enqueue at opposite end, and the max queue length is set to 1. I.E. I only want to log the value at that specific second, but I want to show the data on the front panel at 50 times a second.
Now heres my problems:
1) I replaced the DAQmx Create channel's with a MAX project task wired into my DAQmx Start Task VI. As expected the thing picks up and displays the sensor values as it did before. However there is now a problem logging. Before when I had the DAQmx Create Channel, the logging recorded my values correctly at 1 second intervals. The time stamp in the X column went up by 1 second for each set of results. But now, with the MAX project task (needed because of custom scales and differing min/max values for different sensors), The logging still seems to happen at 1 per second, but the time stamp intervals are the same as the sampling rate set in the task!!! (ie at 50Hz the time stamp goes up 20ms for each result even though the results are being recorded at 1 per second)
2) The timing just seems out of whack.
I keep getting errors that the requested data no longer exists, that it was there but then was over written. Increasing the buffer size or sampling rate is supposed to remedy this... but what sampling rate? the rate set in the task? or the loop iteration speed? I dont understand.
Secondly, despite telling my consumer loop to wait one second between picking up stuff out of the queue to log, it seems to be going round nearly the same pace as the producer loop. To top this off the producer isnt going round at the speed I asked it too either! I put indicators on the loop counters to see this. they are going round at maybe 2Hz.
I have included my VI's
What am I doing wrong, its really bugging the crap out of me!
Attachments:
EngineMon.zip ‏90 KB

The reason you would use get time and date in seconds to to ensure that you log every second. If you use the 1 second wait then you do not take into account the exicution time of your loop. If you add more to the loop in the future than you will see that you do not log every second as you thought. If you loop took 500ms to exicute and you have the 1000ms wait then the actual excution time for recoding to file is 1.5 seconds not 1000ms as you desired. The way I did it will always give you one second unless you exicution time for your loop exceeds one second.''
I did make one change to the vi. I shoulod have put the write in the true false case so that the write only happens when the 1 second has expired.
Tim
Johnson Controls
Holland Michigan
Attachments:
VDU3.vi ‏59 KB

Producer consumer error 200279

Hello,
So I’m about to go crazy and hoping someone can help me here. I’m writing a program that controls a motor and using a producer/consumer method to write the data at 10,000 intervals a second. I’ve read through multiple examples and I don’t see what I’m missing. I’m trying to record 18 different readings from a DAQ 9178, 3 readings for voltage module NI 9205, 1 reading for current module Ni 9203, 2 readings for accelerometer module Ni 9234 and 12 readings for temperature module 9213. Every time I run the program I keep getting the error message 200279 and it states that it’s trying to read samples that no longer exist. It advised me to increase the buffer rate. So I did as it suggested in multiple ways and the error messages keeps showing up. I’ve also shut down the computer, exited and restarted Labview several times and none of this worked. I have included the a picture file below. Where the problem is, is in the producer loop and the error is shown coming out of the DAQ error indicator. One idea I had but I’m unsure how to approach it is to somehow mean the data coming out of the DAQ to reduce what’s getting to the buffer. If anyone knows how to fix this problem please explain it in detail (I mean talk to me like I’m stupid) cause I’ve been stuck on this problem for almost two weeks now. Thank you in advance to anyone who can help.
SteelTiki
Solved!
Go to Solution.

The DAQ error isn't related to your choice of architecture, although your implementation of it might be hurting you. It's almost impossible to follow what's going on in your diagram because you have wires running in many directions (try to keep data flowing from left to right), there are local variables all over the place (almost definitely not needed), and there's a lot of code that's not in the image or is hidden in another frame of the sequence. Normally one would not have the event structure as the consumer loop (as you've done), and it's generally not a good idea to have multiple event structure in the same VI (with rare exceptions usually involving user events). It might not solve your problem but it would be worth taking the time to clean up your diagram. Eliminate sequence structures and local variables as much as possible, move some code into subVIs, and straighten out your wires.
As for the DAQ error: the DAQ board is reading data into a circular buffer, and you're not reading it fast enough, so samples are being overwritten. The reason you're not reading it fast enough is because you only read a single sample at a time (the DAQmx Read is configured for 1 sample on multiple channels) but you say you're acquiring 10,000 samples/second. Since your loop won't run 10,000 times per second, you're not reading all the data, which causes the error. Try configuring the DAQmx Read for multiple samples.

Producer Consumer Issues

Hi all,
I'm creating my first producer-consumer program, and I've run into a few problems. I have two producer loops (one is a pressure controller and the other is power reading loop). I've used queues to pass data into the consumer loop which records the data (at a much slower rate). The problem is that the producer loop (the pressure controller) contains the stop signal for the program. So I've had problems shutting down all 3 loops.
My questions are......Do I have to shut down the consumer loop first, then the producers? What is the best way to pass stop signals (local variables, booleans in a queue, etc)? Also when I try to use the producer loop to stop the other loops I get the attached error. Anyone have any recommendations for making an efficient stopping procedure? The wait inside the consumer loop is quite long (5 min), so I'd like the minimize the stopping time, if possible.
I've attached two versions that I've been working with. They use version 8.5.
Thanks in advance - Dan
Attachments:
Final Controller No Inputs.vi ‏274 KB
Final Controller Other.vi ‏274 KB
error.JPG ‏19 KB

I like to stop the program in the producer loop, and wire the error output of the dequeue element function to the stop of the consumer loop (when the producer loop stops the dequeue element function will generate an error, the error out will cary this to the stop and the loop will stop). Hope this helps!
CLA, CLED, CTD,CPI, LabVIEW Champion
Platinum Alliance Partner
Senior Engineer
Using LV 2013, 2012
Don't forget Kudos for Good Answers, and Mark a solution if your problem is solved.

Functinal global variable for producer - consumer architecture

Hi all,
I am using a Producer - consumer architecture for my data acquistion as in the belwo diagram.. at some time i am stuck insde the while loop continously acquiring data .. is there any way i use one stop button as a functional gloabl variable and stop inside (consumer while loop) as well as producer architecture....
Why i need this ?? so many design rules say that global variables are not a good idea...
Thanks in advance...

FGVs will work fine for your application. You can also wire your error cluster to your stop button. Your producer loop should throw an error 1 when the stop button is pressed, which will in turn stop your consumer loop. But you would need a slightly different design for this. Your consumer loop is designed to not advance until the stop button is pressed. Instead of using and enum, you can use your producer loop to control the trigger to take a measurement. No need for the while loop in the consumer loop this way. You could set up your consumer loop to take a number of measurements, and then quit or take measurements for a certain amount of time. Either way, you can stop your while loop when it times out.
If you choose to stay with this design, a simple state machine archetecure might be better based on what I see.
Reese, (former CLAD, future CLD)
Some people call me the Space Cowboy!
Some call me the gangster of love.
Some people call me MoReese!
...I'm right here baby, right here, right here, right here at home

Producer consumer of images

can someone please help me. I am totally stuck and lost. I am trying to build a procuder consumer architecture but the template from labview is not very clear. many books does not go over this concept.
i have created a vi. but i dont know what to do with the image in buffer.
can someone please guide me. thank you
Best regards,
Krispiekream
Attachments:
ProducerConsumerData 1.vi ‏115 KB

One problem is that you created a queue that
transfers Imaq references. But in your consumer loop, you are
transferring a string called elements. (The red coercion dot on the
Enqueue element gave it away.) You would need to enqueue the purple
wire.
On the other side you would need to dequeue the
purple wire (Imaq reference).
One problem is that the
purple wire is a reference to a particular memory location and is not a
particular instance of an image. As Mark was saying, you could wind up
overwriting the memory location with new image data before you had a
chance to do anything with the previous image data.
Hi Ravens Fan! First off, I want to thank you and Mark for looking over my vi.
I formatted the code exactly how example did using this.
I don't see any red coercion dot on my Enqueue element.
in order to get my program working, I added a wait timer of 200 ms and it did the trick. But I am loosing the frame rate I needed and that's the main reason for moving to producer/consumer architecture.
Can you show me what you mean? I never worked with this time of architecture before and don't know how to use it.
Thanks
Best regards,
Krispiekream
Attachments:
mltcoreimaqdx.vi ‏90 KB

How to optimize data logging in my code?

I have my program working well enough to have it run some trials in the lab. It was run under some 'extreme' conditions to make sure data acquisiton and logging is working well.
With frequency being 1000Hz, he requirements are that the relay goes ON for 20ms and and stays off for 250ms. This 250ms should equate to 250samples gathered.
After a run, the saved files range from 240 to 280 samples for a 250ms setting, but some are as little as 150 samples. I need to ensure that the correct number is logged, a +/- 5% variation is 'OK' but when the samples gathered are close to half of the desired is unacceptable.
I need the graphs wired to 'data[0]' and 'data[1]' to update in real time, to be smooth.
Any suggestions? Every time my consumer loop executes, how many samples does my DAQmx Read, at current, receive? I do not think it's an issue w DAQmx setup, but I am not 100% sure..

Is there any reason you're not using compound arithmatic to get into the logging loop? What happens in those false cases? If you're just wiring everything through, you'd be better served to only use one case structure. Feed all three boolean values into a single compound arithmatic set to AND. If all three are true, after the NOT used to invert the false value, it will enter the true case. Otherwise, it'll enter the false case. It's the same logic, but it's handled better.
You might consider taking a look at Producer/Consumer or Queued Message Handler. You can use these to send a message to an external while loop. If you have multiple cores, this makes it easy to split the task to read data on one core and leave it mostly unaffected by the logging process done on another core.

Time Stamps in Producer/Consumer Setup

I am trying to acquire signal from a Gas Analyzer, Flow Meter, and 3 RTDs. My hardware is a NI 9215 and NI 9217. I need to write the data to a file with a time stamp at which the sample was read. The program is written in a producer/consumer setup in which the data is being acquired in the prodcuer loop and sent to the consumer loop to be written. I can get the samples written to the file okay. However, I have not figured out how to put the time stamp in the data file correctly. I thought the time stamp was created with the corressponding measurement at the Data Acquisition VI. However, whenever I open the data file, there are only 0s in the time column. So, I've tried to use Format/Date String to add the time stamp. I can get it to work if I place it in the consumer loop. However, the time stamp is not accurate. It puts the same time stamp up to six digits of precision for two consecutive samples. I've tried to figure out how to add it to the producer loop, but I don't know how to add it and the measurements to the queue at the same time. Please tell me the best way to acquire signals with the exact time stamp and write them to a file using a producer/consumer design.
Example:
Date             Time       RTD1 RTD2      RTD3   CO2    Flow
5/26/07      5:20:01.25       57         56         56.5      100       0.5
5/26/07      5:20:01.30       57         56         56.5      100       0.5
5/26/07      5:20:01.35       57         56         56.5      100       0.5
Attachments:
Producer-Consumer From Scratch2.vi ‏75 KB

Okay, I've put it in a For Loop and it will connect now. However, whenever it runs the time stamp increments for a couple seconds and then it just repeats. How do I correct this?
Attachments:
Producer-Consumer From Scratch2.vi ‏88 KB
pleasework.txt ‏20 KB

Input and output on same device, producer/consumer structure

Hello interested people,
I have a question about using the same device for both digital inputs
and outputs. I have written a simple program of one while loop
that continuously polls the device, processes, and requests. I
have addressed the device using two DAQmx Asst. and I have attached
them with their error in/out cluster terminals to provide data flow and
eliminate the chance of addressing the devices at the same time (which
produces an error). Now I want to change this program structure
to a producer/consumer loop foundation with state machine.
In this design, I will have the DI in the producer loop and the DO in
the consumer loop, under one of the states. I can't simply
connect the error in/out ports in this configuration, so my question is
how to avoid the error caused by addressing the same device
simultaneously with two different tasks (input and output)? I
have attached two VI's, the "One Loop" vi is the original configuration
(simplified), and the Producer-Consumer vi is a NONSENSICAL program
that simply represents the desired configuration. (I don't need
any comments on the programming of this vi, it is only an example for
illustration of the problem).
I am thinking about bundling the input data and the error cluster, both
from the PXI 6528 DI, into one cluster, queueing that up, and
unbundling the de-queued elements for some kind of data flow between
the producer loop and the "Request" state of the consumer loop.
Is this the right approach, or am I barking up the wrong tree?
Thanks
Attachments:
One Loop DO DI.vi ‏102 KB
Producer-Consumer DI-DO.vi ‏106 KB

Hello,
It sounds to me like you really have two modes:
1. user interface actions determine execution
2. user interface is locked, and execution is automated.
I think it would make sense to use the producer consumer for an architecture. Basically you would do the following:
1. program the producer to handle the user interface as you normally would.
2. provide one additional event case in the producer which would be your "automated handling" case. In that case, you could put a state machine which could run until whatever conditions were met to put your program back in "user interface mode".
Keep in mind that you can use custom USER EVENTS to programmatically generate events ie. you can trigger the start of your "automated handling" form anywhere in your code at virtually any time.
I think this would allow you to take advantage of the producer consumer architecture in its intended spirit, while integrating an automated routine.
I hope this helps!
Best Regards,
JLS
Best,
JLS
Sixclear

Producer-consumer: optimize data acquistion rate

Similar Messages

Maybe you are looking for