Daqmx, queue, and memory

Hi all,
I have a loop that would read the analog in buffer and put the data (24x1000 2d array /element) onto three queues (one for display, one for file I/O, and one for motion control). It seems to be that this is very wasteful, but I can't think of a better way. Apparently, LV agrees with me, and it is giving me a memory error, saying that I don't have enough memory to perform the operation. I understand that I am buring a lot of memory, but it surprised me that LV can't handle that. Can you help? Thanks!
Kudos and Accepted as Solution are welcome!

Hi jyang72211,
can you upload your vi, so we can look into it?
Maybe it helps if you use an action engine for your data and only send a "new data" command with the queues.
Mike

Similar Messages

Thoughts on Stream-to-Disk Application and Memory Fragmentation

I've been working on a LabVIEW 8.2 app on Windows NT that performs high-speed streaming to disk of data acquired by PXI modules. I'm running with the PXI-8186 controller with 1GB of RAM, and a Seagate 5400.2 120GB HD. My current implementation creates a separate DAQmx task for each DAQ module in the 8-slot chassis. I was initially trying to provide semaphore-protected Write to Binary File access to a single log file to record the data from each module, but I had problems with this once I reached the upper sampling rates of my 6120's, which is 1MS/sec, 16-bit, 4-channels per board. With the higher sampling rates, I was not able to 'start off' the file streaming without causing the DaqMX input buffers to reach their limit. I think this might have to do with the larger initial memory allocations that are required. I have the distinct impression that making an initial request for a bunch of large memory blocks causes a large initial delay, which doesn't work well with a real-time streaming app.
In an effort to see if I could improve performance, I tried replacing my reentrant file writing VI with a reentrant VI that flattened each module's data record to string and added it to a named queue. In a parallel loop on the main VI, I am extracting the elements from that queue and writing the flattened strings to the binary file. This approach seems to give me better throughput than doing the semaphore-controlled write from each module's data acq task, which makes sense, because each task is able to get back to acquiring the data more quickly.
I am able to achieve a streaming rate of about 25MB/sec, running 3 6120s at 1MS/sec and two 4472s at 1KS/sec. I have the program set up where I can run multiple data collections in sequence, i.e. acquire for 5 minutes, stop, restart, acquire for 5 minutes, etc. This keeps the file sizes to a reasonable limit. When I run in this mode, I can perform a couple of runs, but at some point the memory in Task Manager starts running away. I have monitored the memory use of the VIs in the profiler, and do not see any of my VIs increasing their memory requirements. What I am seeing is that the number of elements in the queue starts creeping up, which is probably what eventually causes failure.
Because this works for multiple iterations before the memory starts to increase, I am left with only theories as to why it happens, and am looking for suggestions for improvement.
Here are my theories:
1) As the streaming process continues, the disk writes are occurring on the inner portion of the disk, resulting in less throughput. If this is what is happening, there is no solution other than a HW upgrade. But how to tell if this is the reason?
2) As the program continues to run, lots of memory is being allocated/reallocated/deallocated. The streaming queue, for instance, is shrinking and growing. Perhaps memory is being fragmented too much, and it's taking longer to handle the large block sizes. My block size is 1 second of data, which can be up to a 1Mx4x16-bit array from each 6120's DAQmx task. I tried added a Request Deallocation VI when each DAQmx VI finishes, and this seemed to help between successive collections. Before I added the VI, task manager would show about 7MB more memory usage than after the previous data collection. Now it is running about the same each time (until it starts blowing up). To complicate matters, each flattened string can be a different size, because I am able to acquire data from each DAQ board at a different rate, so I'm not sure preallocating the queue would even matter.
3) There is a memory leak in part of the system that I cannot monitor (such as DAQmx). I would think this would manifest itself from the very first collection, though.
4) There is some threading/threadlocking relationship that changes over time.
Does anyone have any other theories, or comments about one of the above theories? If memory fragmentation appears to be the culprit, how can I collect the garbage in a predictable way?

It sounds like the write is not keeping up with the read, as you suspect. Your queues can grow in an unbounded fashion, which will eventually fail. The root cause is that your disk is not keeping up. At 24MBytes/sec, you may be pushing the hardware performance line. However, you are not far off, so there are some things you can do to help.
Fastest disk performance is achieved if the size of the chunks you write to disk is 65,000 bytes. This may require you to add some double buffering code. Note that fastest performance may also mean a 300kbyte chunk size from your data acquisition devices. You will need to optimize and double buffer as necessary.
Defragment your disk free space before running. Unfortunately, the native Windows disk defragmentor only defragments the files, leaving them scattered all over the disk. Norton's disk utilities do a good job of defragmenting the free space, as well. There are probably other utilities which also do a good job for this.
Put a monitor on your queues to check the size and alarm if they get too big. Use the queue status primitive to get this information. This can tell you how the queues are growing with time.
Do you really need to flatten to string? Unless your data acquisition types are different, use the native data array as the queue element. You can also use multiple queues for multiple data types. A flatten to string causes an extra memory copy and costs processing time.
You can use a single-element queue as a semaphore. The semaphore VIs are implemented with an old technology which causes a switch to the UI thread every time they are invoked. This makes them somewhat slow. A single-element queue does not have this problem. Only use this if you need to go back to a semaphore model.
Good luck. Let us know if we can help more.
This account is no longer active. Contact ShadesOfGray for current posts and information.

List of Queues and Notifiers

Dear Sir/Madam,
1) I would like to be able to obtain a list of all queues and notifiers in memory. How to do so?
2) I would like to be able to change the behaviour of the queue. Instead of holding and waiting for an element to be read out of a full queue (when a new element in to be inserted), I would like to be able to push out the oldest element.
Any suggestions?
Thanks!
Regards
Albert

Thank you all.
Thans for the VI, but unfortunately I could not read it (version too high). However, I found "Smart Buffer" and "Recent History" VIs in the "Search Examples". Unfortunately, these are not applicable since I need to generate the queues and notifiers in undefined number of instances.
What I eventually did:
1) I would like to be able to obtain a list of all queues and notifiers in memory. How to do so?
Used a functional global variable VI which stored the names, as well as created/destroyed queues.
2) I would like to be able to change the behaviour of the queue. Instead of holding and waiting for an element to be read out of a full queue (when a new element in to be inserted), I would like to be able to push out the oldest element.
I used notifiers:
a) I call a VI that returns the status of the notifier, and I read all the available notifiers from it.
b) I append the new data to the one obtained from the notifier, and cut the result to limit the length of this "buffer".
c) I call a Send Notifier VI, which puts the new data set into the notifier.
This way is possible but uncomfortable...
DEAR NI Tech Support,
Please respond to the questions, and let me know, if there are more elegant solutions (e.g. reading a property node for the list of queues or notifiers, or writing a property node for setting the behaviour of the queue).
Thanking you in advance.
Sincerely
Albert Lysko
Message Edited by alysko on 12-10-2007 06:48 AM

Why can't I find "Data Transfer and Memory " option in channel property node?

Hi All,
I'm trying to find some properties such as "Data Transfer and Memory " in my "DAQmx Channel" property node, as below
But, in my Labview 8.5, I can't find it. I re-installed NI-DAQmx, but it didn't work. What I can get is
Did I miss something? Thank you.
Regards,
Bo
Message Edited by foolooo on 10-23-2008 07:02 AM
My blog Let's LabVIEW.
Solved!
Go to Solution.
Attachments:
temp.JPG ‏11 KB

Hi Andrey,
Thank you for your reply. But I tried this on another computer on which lv 8.6 was installed, and it worked. I just simply dropped the channel property node, and didn't connect with any channel vi, and there were available options on that one. But for this pc, it doesn't work. I also configured the hardware in Mea & AE, it didn't help.
btw, my labview is full licenced.
Best wishes,
Bo
My blog Let's LabVIEW.

Get CPU and memory usage

Hi!
I would like to know if there is any way of getting system CPU and memory usage using Java code.

I want to get the system CPU and memory usage using the performance monitor dll, the perfctrs.dll, but access this data using Java language.Then you should create wrapper dll between your java code and perfctrs.dll and convert data from format of dll to format of your java code.
So, that is next question - how to create wrapper dll, how to deal with or how perfctrs.dll works?

What is the difference between Azure RemoteApp Basic vs Standard Plans in terms of compute cores and memory?

So our customer has asked us to compare compare Amazon Workspace and Azure RemoteApp offerings for them to choose from. While looking at Amazon Workspace, it clealy defines bundles with specific CPU cores, memory and user storage. However, Azure RemoteApp
only specifies user storage and vaguely compares its basic vs. standard plans in terms of "task worker" vs. "information worker"
I tried looking up its documentation but couldn't find specific CPU cores that are dedicated per user in basic vs. standard plans. I have following questions:
Can anyone point me in the right direction or help understand how many CPU cores and memory are dedicated (or shared) per user in each plan?
Our customer would most likely need a "custom" image for their custom apps. Is it possible for us to choose specific CPU cores and memory for the users to be able to run their apps in azure remoteapp?
In case i am misunderstanding the basic difference between AWS workspace and Azure RemoteApp, i'd appreciate some help in understanding it as well.
Thanks!

Hi,
With Azure RemoteApp users see just the applications themselves, and the applications appear to be running on their local machine similar to other programs. With Workspaces users connect to a full desktop and launch applications within that.
1. Azure RemoteApp currently uses size A3 Virtual Machines, which have 4 vCPUs and 7GB RAM. Under Basic each VM can have a maximum of 16 users using it whereas under Standard each VM is limited to 10 users. The amount of CPU available
to a user depends on what the current demands are on the CPU at that moment from other users and system processes that may be on the server.
For example, say a user is logged on to a VM with 3 other users and the other users are idle (not consuming any CPU). At that moment the user could use all 4 vCPUs if a program they are running needed to. If a few moments later
the other 3 users all needed lots of CPU as well, then the first user would only have approximately 1 vCPU for their use. The process is dynamic and seeks to give each user their fair share of available CPU when there are multiple users demanding CPU.
Under the Standard plan a user will receive approximately a minimum of .4 vCPU assuming that the VM has the maximum number of users logged on and that all users are using as much CPU as possible at a given moment. Under the Basic plan the approximate
minimum would be .25 vCPU.
2. You cannot choose the specific number of cores and memory. What you can do is choose the Azure RemoteApp billing plan, which affects the user density of each VM as described above. If you need a lower density than Standard you
may contact support.
-TP

CPU usage and memory

What is FF doing when it sets at idle with nothing streaming , playing or such and uses 40-70% of CPU and memory usage keeps on going up. I just set there and watch in the task manager. Every other program uses nothing,
There has to be a programmer intelligent enough to fix this. this has been going on for years.

SImple fact is there is all "fixes" and answers are bogus. Just try using the same tabs in any other browser.
Right now Opera is 2-3% idle no memory problems. other browsers are similiar and I will post.
Mozzila is just like Microsoft . They have no desire to fix the problem

Upgrade chip and memory

Can I upgrade the chip and memory? if so what can I use. The system is a little too slow. Yes I have all the software to speed it up. Thanks
HP 2000 Notebook PC Windows 8.1 64-bit [Edited for Personal Information] AMD E-300 APU with Radeon(tm) HD Graphics microprocessor 4GB SODIMM Kinston 800MHz memory 188B KBC Version 69.18 system board F.37 system bios HGST HTS545032A7E380 SATA Disk Device Product number: E0M17UA#ABA AMD E-300 APU with Radeon(tm) HD Graphics

Here is the Service Manual:
http://h10032.www1.hp.com/ctg/Manual/c03763129.pdf
Go through Page 16-17. Each different processor is soldered to the board, so you've to replace the whole motherboard for that, you can see that their part numbers are different according to three different boards.
I suggest RAM upgrade from 4 to 8GB by adding another 4GB in the other vacant slot, with configuration DDR3 1333MHz dual channel boost.
thanks,
++Please click KUDOS / White thumb to say thanks
++Please click ACCEPT AS SOLUTION to help others, find this solution faster
**I'm a Volunteer, I do not work for HP**

HDD and memory upgrade on Satellite A350

Hi All
I have a satellite a350 laptop can anyone tell me whats the biggest harddrive I can have and memory please
Thanks

Hi
The RAM upgrade depends on the available chipset.
Not quite sure about your A350-xxx model but many of the Sat A350 notebooks were quipped with an Intel GM45 Express chipset and in such case you could upgrade the memory max up to 8GB RAM
Regarding the HDD: This notebook supports SATA HDD. This mean that you are not limited to an special size.
Theoretically it could be possible to use 500GB, 750GB 2.5 SATA HDDs

MSI 785GM-E51 and memory modules problems

Could anyone help me with one problem. I have motherboard MSI 785GM-E51 and 4 memory modules GOODRAM GR 1333D364L9/2G. But when I'm inserting ALL modules my system not starting (even POST procedure). So working variants - when I'm inserting:
- one module in first (boot) slot
or
- they inserted in first and third memory slots
Any ideas?

It is not motherboard problem, but processor itself - I have AMD Phenom II X4 965 Black Edition 3,4 Ghz Core Revision C2 and this revision (according to the article: http://www.fcenter.ru/online.shtml?articles/hardware/processors/27650#02) has problems with support memory modules DDR3-1333 Mhz more than 2 modules. Conclusion - either adjust BIOS settings concerning memory timings and memory controller voltage or replace this kind of processor with the same type but with core revision C3

Jobs stay in queue and keep printing until you cancel them--then stops the print spooler...why?

My Printer: HP Photosmart C309a
My Laptop: HP G71 series
Operating System: Windows 7 (64 bit) but runs a lot of stuff on 32 bit
Problem: I print ONE copy of a multiple page document in Adobe Reader 9. I select even pages only (after which I would reinsert the pages and tell it to print odd pages only, to get the other side). The printer makes a sound indicating the job is finished, I remove my pages to collate them, and then the printer continues to print the job again without so much as a keystroke.
Naturally, you find the printer in the devices on the control panel, click "see what's printing" and highlight the job and cancel it.
Then somehow it turns off my print spooler.
I have tried everything suggested on other forums, like
restart the laptop/restart the printer
going to c\windows\system32\spool\printers and deleting whatever is hung up in there (usually a shockwave file).
Going to Services\print spooler \properties and making sure the settings are set to automatic
Uninstall\reinstall latest driver and software
Switching from Adobe Reader 10 to Adobe Reader 9
Use System Restore to try to reset the system to a previous state.
Run Trend Micro to check for viruses and spyware--nothing shows up.
I am positive the pdf doc is NOT the problem, because I can print it on my husband's laptop multiple times over with no problems.
Does anyone have this problem as well? Has anyone found a genuine solution other than manually canceling print jobs in queue and constantly restarting the print spooler?
Thanks in advance for any suggestions.
--Holly

Hi,
I believe your printer is wirelessly connected to your computer, right?!
What's the current printer software version installed in your laptop computer?!
to see: click start>control panel>uninstall a program under Programs then it will tell you the software version like HP Photosmart c309a v13 or v14.
If the software version is v13, click here to download and install the current software.
IF all things still fail, try to use the workaround provided by HP in this link. It says:
Workaround
Your HP printing product can operate sufficiently using an alternate print driver, although there might be some limitations. For instance, some buttons on the product control panel might not function, but the product prints normally from the computer. If the solutions in this document do not solve the issue, download and install an alternate print driver.
Follow these steps to install the HP Deskjet 990C print driver.
NOTE: These steps install the new print driver using the same port that the product already uses. The product functions normally with multiple drivers on the same port.
Find the port that the product already uses.
Click the Windows icon ( ), and then click Control Panel . The Control Panel opens in a new window.
Click Hardware and Sounds .
Click Printers . The Printers folder opens.
Right-click the product icon ( ), and then click Properties . The Propertieswindow opens.
Click the Ports tab. A window opens with a list of ports. The port for the product has a checkmark or a highlight.
Note the name of the port indicated for the product.
Close the Properties window, and then continue with the next steps.
Click Add a Printer in the menu bar at the top of the Printers window. The Windows Add Printer Wizard opens.
Click Add a local printer .
Select Use an existing Port .
Click the drop-down menu next to Use an existing Port , and then select the port that you noted earlier in these steps.
Click Next .
In the Manufacturer pane, click HP .
In the Printers pane, click HP Deskjet 990c , and then click Next . (IF YOU CANT FIND DESKJET 990C, JUST CLICK WINDOWS UPDATE WITHIN THIS WINDOW. THEN FIND DESKJET 990C and then click next)
Type a name for the new printer in the Printer name box, or keep the default name.
Select Set as the default printer , and then click Next . A window opens with a progress bar as the printer installs. Then a new window opens.
Click Do not share this printer , and then click Next .
If you want to print a test page, click the Print test page button.
Click Finish to complete the driver installation.
Try printing again, but select HP Deskjet 990c from the Print dialog box.
If this solves the issue, follow these steps to use the HP Deskjet 990c print driver whenever you send print jobs to the product.
In the program you are using, select the option to print. The Print dialog box opens.
Click the Name drop-down menu, and then select HP Deskjet 990c .
Change print settings as desired in the Paper size , Quality , and Paper type drop-down menus.
Select the Print range and Copies options as desired.
Click OK . The product prints the file.
Kiko

How to delete the data in update queue and delta queue for Queued delta?

Dear BWers,
How do i delete the delta queue and update queue data before i fill the setup tables for a extraction based on Queued delta. Please help.
Thanks
Raj

Hi Raj,
I think you need some ground work for the LO extraction same as others here. Please read the 3 blogs expliciltly created for LIS by Robert Negro.
/people/sap.user72/blog/2004/12/16/logistic-cockpit-delta-mechanism--episode-one-v3-update-the-145serializer146
/people/sap.user72/blog/2004/12/23/logistic-cockpit-delta-mechanism--episode-two-v3-update-when-some-problems-can-occur
/people/sap.user72/blog/2005/01/19/logistic-cockpit-delta-mechanism--episode-three-the-new-update-methods
As well, the OSS 380078 would clear your doubts reagrding the the BW QUEUE mainatinance.
Please let me know if these material has been suffecient enough.
regarda,
raj

How to display CPU and memory utilization from ST06 in a report

Hi,
I want to display CPU Utilization and Memory utilization and File sys details from ST06 transaction in a report.
Is there any function module or any other method to do that.
Please advice.
Thanks,
Sandeep.

Hi Ranganath,
Thanks for your time.
And thank you very much for the reply.
Both the function modules are helpful.
But can u also help me in getting the data of FileSys from ST06.
Thankyou,
Sandeep.

Profile Performance and memory

Hi, guys
I am trying to analysis my Profile Data for the Profile Performance and memory.
When I check the data for VI Time, I found there are same VIs with different VI Time, what is that mean
The reason why I am doing this is because I am going to add pulse model for my VI, so I need to know the relationship between my all instruments and time, Am I in the right track
Thanks
Solved!
Go to Solution.
Attachments:
VI Time.png ‏72 KB

It looks like the timing had a resolution of 15.6 ms, which I think is common on some flavors of the Windows OS. Note that the average times on some of those VIs is zero. This suggessts that for many of the 1878 runs the time was probably recorded as zero and a few were at 15.6 ms.
This is one of the limitations of the Profiler.
Lynn

Profile Performance and Memory shows very large 'VI Time' value

When I run the Profile Performance and Memory tool on my project, I get very large numbers for VI Time (and Sub VIs Time and Total Time) for some VIs. For example 1844674407370752.5. I have selected only 'Timing statistics' and 'Timing details'. Sometimes the numbers start with reasonable values, then when updating the display with the snapshot button they might get large and stay large. Other VI Times remain reasonable.
LabVIEW 2011 Version 11.0 (32-bit). Windows 7.
What gives?
- les

les,
the number indicates some kind of overroll.... so, do you have a vi where this happens all the time? Can you share this with us?
thanks,
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

Daqmx, queue, and memory

Similar Messages

Maybe you are looking for