Passing large amount of data between steps

Hello,
in my application, I developed some step types for data acquisition and analysis.
I need to pass large arrays (>100000 points) from one step to another.
Actually I pass the sequence context, and I use Variants to retrieve data (in the second step).
This method seems however quite slow.
Has anyone any suggestion?
I use TS3.5 and CVI 8.0
Thank you
baloss

Hi,
You should be able to pass the data back directly via the parameter list of the function and like wise directly into the next function without having to use the sequence context.
Regards
Ray Farmer
Regards
Ray Farmer

Similar Messages

How to pass large amount of data between steps

Hi all，
I have some LabVIEW VIs for data acquisition。
I need to pass large amount of data(array size >5000000 each time) from one step to another.
But it is not allowed to set array size larger than 5000000.
Any suggestion?
czhen
Win 7 SP1 & LabVIEW 2012 SP1, Teststand 2012 SP1
Solved!
Go to Solution.
Attachments:
Array Size Limits.png ‏34 KB

In your LabVIEW code, put the data into a data value reference. Pass this reference between your TestStand steps. As an added bonus, you will not get an extra copy of the data at each step. You will need to use the InPlace element structure to get your data out of the data value reference.
This account is no longer active. Contact ShadesOfGray for current posts and information.

How can I copy large amount of data between two HD ?

Hello !
Which command could I user to copy large amount of data between two hard disk drives ?
How Lion identify the disk drives when you want to write some script, for example in Windows I sue
Robocopy D:\folder source\Files E:\folder destination
I just want to copy files and if the files/folders exist in destination the files should be overwritted.
Help please, I bougth my first MAC 4 days ago.
Thanks !

Select the files/folders on one HD and drag & drop onto the other HD. The copied ones will overwrite anything with the same names.
Since you're a newcomer to the Mac, see these:
Switching from Windows to Mac OS X,
Basic Tutorials on using a Mac,
Mac 101: Mac Essentials,
Mac OS X keyboard shortcuts,
Anatomy of a Mac,
MacTips,
Switching to Mac Superguide, and
Switching to the Mac: The Missing Manual,
Snow Leopard Edition.&
Additionally, *Texas Mac Man* recommends:
Quick Assist,
Welcome to the Switch To A Mac Guides,
Take Control E-books, and
A guide for switching to a Mac.

Best way to pass large amounts of data to subroutines?

I'm writing a program with a large amount of data, around 900 variables. What is the best way for me to pass parts of this data to different subroutines? I have a main loop on a PXI RT Controller that is controlling hydraulic cylinders and the loop needs to be 1ms or better. How on earth should I pass 900 variables through a loop and keep it at 1ms? One large cluster?? Several smaller clusters?? Local Variables? Global Variables?? Help me please!!!

My suggestion, similar to Altenbach and Matt above, is to use a Functional Global Variable (FGV) and use a 1D array of 900 values to store the data in the FGV. You can retrieve individual data items from the FGV by passing in the index of the desired variable and the FGV returns the value from the array. Instead of passing in an index you could also use a TypeDef'd Enum with all of your variables as element of the Enum, which will allow you to place the Enum constant on the diagram and make selecting variables, as well as reading the diagram, simpler.
My group is developing a LabVIEW component/example code with this functionality that we plan to publish on DevZone in a month or two.
The attached RTF file shows the core piece of this implementation. This VI off course is non-reentrant. The Init case could be changed to allocate the internal 1D array as part of this VI rather than passing it from another VI.
Message Edited by Christian L on 01-31-2007 12:00 PM
Christian Loew, CLA
Principal Systems Engineer, National Instruments
Please tip your answer providers with kudos.
Any attached Code is provided As Is. It has not been tested or validated as a product, for use in a deployed application or system,
or for use in hazardous environments. You assume all risks for use of the Code and use of the Code is subject
to the Sample Code License Terms which can be found at: http://ni.com/samplecodelicense
Attachments:
CVT_Double_MemBlock.rtf ‏309 KB

Looking for ideas for transferring large amounts of data between systems

Hello,
I am looking for ideas based on best practices for transferring Large Amounts of Data in and out of a Netweaver based application.
We have a new system we are developing in Netweaver that will utilize both the Java and ABAP stack, and will require integration with other SAP and 3rd Party Systems. It is a standalone product that doesn't share any form of data store with other systems.
We need to be able to support 10s of millions of records of tabular data coming in and out of our system.
Since we need to integrate with so many different systems, we are planning to use RFC for our primary interface in and out of the system. As it turns out RFC is not good at dealing with this large amount of data being pushed through a single call.
We have considered a number of possible ideas, however we are not very happy with any of them. I would like to see what the community has done in the past to solve problems like this as well as how SAP currently solves this problem in other applications like XI, BI, ERP, etc.

Primoz wrote:Do you use KDE (Dolphin) 4.6 RC or 4.5?
Also I've noticed that if i move / copy things with Dolphin they're substantially slower than if I use cp/mv. But cp/mv works fine for me...
Also run Dolphin from terminal to try and see what's the problem.
Hope that help at least a bit.
Could you explain why Dolphin should be slower? I'm not attacking you, I'm just asking.
Cause I thought that Dolphin is just a „little" wrapper around the cp/mv/cd/ls applications/commands.

What is the most efficient way of passing large amounts of data through several subVIs?

I am acquiring data at a rate of once every 30mS. This data is sorted into clusters with relevant information being grouped together. These clusters are then added to a queue. I have a cluster of queue references to keep track of all the queues. I pass this cluster around to the various sub VIs where I dequeue the data. Is this the most efficient way of moving the data around? I could also use "Obtain Queue" and the queue name to create the reference whenever I need it.
Or would it be more efficient to create one large cluster which I pass around? Then I can use unbundle by index to pick off the values I need. This large cluster can have all the values individually or it co
uld be composed of the previously mentioned clusters (ie. a large cluster of clusters).

> I am acquiring data at a rate of once every 30mS. This data is sorted
> into clusters with relevant information being grouped together. These
> clusters are then added to a queue. I have a cluster of queue
> references to keep track of all the queues. I pass this cluster
> around to the various sub VIs where I dequeue the data. Is this the
> most efficient way of moving the data around? I could also use
> "Obtain Queue" and the queue name to create the reference whenever I
> need it.
> Or would it be more efficient to create one large cluster which I pass
> around? Then I can use unbundle by index to pick off the values I
> need. This large cluster can have all the values individually or it
> could be composed of the previously mentioned clusters (i
e. a large
> cluster of clusters).
It sounds pretty good the way you have it. In general, you want to sort
these into groups that make sense to you. Then if there is a
performance problem, you can arrange them so that it is a bit better for
the computer, but lets face it, our performance counts too. Anyway,
this generally means a smallish number of groups with a reasonable
number of references or objects in them. If you need to group them into
one to pass somewhere, bundle the clusters together and unbundle them on
the other side to minimize the connectors needed. Since the references
are four bytes, you don't need to worry about the performance of moving
these around anyway.
Greg McKaskle

MessageExpired because of large amount of data

Hi Experts, i need some advice for my following scenario.
I have a 50000 data that need to be inserted into SAP R3 HR. I am using BPM when calling the RFC dan using a sync message to get the response from the RFC. However, i think because of the large amount of data, i get the error "MessageExpired" in my monitoring.
After i read some documentation, i find that a sync message has a timeout period (xiadapter.inbound.timeout.default = 180000 [ms] ) . My question is, if the 180000ms is not enough for the RFC to process the whole 50000 data, i can increase the timeout time, am i right? But then, to what maximum value will be the most appropriate for me to set the timeout value so that it will not effect the performance of XI. Anyway, does increasing the timeout value effect anything at all?
Need the advice and inputs from you experts...Thank you so much in advance...

Made,
I posted this answer to a similar request a couple of weeks ago
I had a similar issue some time back and used the following three parameters
1. Visual Administrator
SAP XI Adapter: RFC parameter syncMessageDeliveryTimeoutMsec
This parameter is explained in SAP Note 730870 Q14 and SAP Note 791379
2. Alert Configuration
SA_COMM parameter CHECK_FOR_ASYNC_RESPONSE_TIMEOUT
This parameter specifies the maximum time in seconds that can pass between the arrival of the synchronous request message on the Integration Server and the asynchronous response message. If this time period is exceeded, a system error is returned to the caller.
3. Integration Process TimeoutControlStep
My design is such that timeout parameter 3 which is set in the Integration Process will be triggered first. This will give me control to handle the timeout via an SMTP email prior to either of the other timeouts taking effect
Regards,
Mike

DSS problems when publishing large amount of data fast

Has anyone experienced problems when sending large amounts of data using the DSS. I have approximately 130 to 150 items that I send through the DSS to communicate between different parts of my application.
There are several loops publishing data. One publishes approximately 50 items in a rate of 50ms, another about 40 items with 100ms publishing rate.
I send a command to a subprogram (125ms) that reads and publishes the answer on a DSS URL (app 125 ms). So that is one item on DSS for about 250ms. But this data is not seen on my man GUI window that reads the DSS URL.
My questions are
1. Is there any limit in speed (frequency) for data publishing in DSS?
2. Can DSS be unstable if loaded to much?
3. Can I lose/miss data in any situation?
4. In the DSS Manager I have doubled the MaxItems and MaxConnections. How will this affect my system?
5. When I run my full application I have experienced the following error Fatal Internal Error : ”memory.ccp” , line 638. Can this be a result of my large application and the heavy load on DSS? (se attached picture)
Regards
Idriz Zogaj
Idriz "Minnet" Zogaj, M.Sc. Engineering Physics
Memory Profesional
direct: +46 (0) - 734 32 00 10
http://www.zogaj.se

LuI wrote:
>
> Hi all,
>
> I am frustrated on VISA serial comm. It looks so neat and its
> fantastic what it supposes to do for a develloper, but sometimes one
> runs into trouble very deep.
> I have an app where I have to read large amounts of data streamed by
> 13 µCs at 230kBaud. (They do not necessarily need to stream all at the
> same time.)
> I use either a Moxa multiport adapter C320 with 16 serial ports or -
> for test purposes - a Keyspan serial-2-USB adapter with 4 serial
> ports.
Does it work better if you use the serial port(s) on your motherboard?
If so, then get a better serial adapter. If not, look more closely at
VISA.
Some programs have some issues on serial adapters but run fine on a
regular serial port. We've had that problem recent
ly.
Best, Mark

Query about clustering unrelated large amounts of data together vs. keeping it separate.

I would like to ask the talented enthusiasts who frequent the devolper network to tell me if I have understood how Labview deals with clusters. A generic description of a situation involving clusters and what I believe Labview does is shown below. An example of this type of situation is shown for generating the Fibonacci sequence is attached to illustrate what I am saying.
A description of the general situation:
A cluster containing several different variables (mostly unrelated) has one or two of these variables unbundled for immediate use and then the modified values bundled back into the cluster for later use.
What I think Labview does:
As the original cluster is going into the unbundle (to get original variable values) and the bundle (to update stored variable values) a duplicate of the entire cluster is made before picking out the individual values chosen to be unbundled. This means that if the cluster also contains a large amount of unrelated data then processor time is wasted duplicating this data.
If on the other hand this large amount of data is kept separate then this would not happen and no processor time is wasted.
In the attached file the good method does have the array (large amount of unrelated data) within the cluster and does not use the array in more than one place, so it is not duplicated. If tunnels were used instead, I believe at least one duplicate is made.
Am I correct in thinking that this is the behaviour Labview uses with clusters? (I expected Labview only to duplicate the variable values chosen in the unbundle code object only. As this choice is fixed at compile time it would seem to me that the compiler should be able to recognise that the other cluster variables are never used.)
Is there a way of keeping the efficiency of using many separate variables (potentialy ~50) whilst keeping the ease of using a single cluster variable over using separate variables?
The attachment:
A vi that generates the Fibonacci sequence (the I32 used wraps at ~44th value, so values at that point and later are wrong) is attached. The calculation is itterative using a for loop. 2 variables are needed to perform the iteration which are stored in a cluster (and passed from iteration to iteration within the cluster). To provide the large amount of unrelated data, a large array of reasonably sized strings is provided.
The bad way is to have the array stored within the cluster (causing massive overhead). The good way is to have the array separate from the other pieces of data, even if it passes through the for loop (no massive overhead).
Try replacing the array shift registers with tunnels in the good case and see if you can repeat my observation that using tunnels causes overhead in comparison to shift registers whenever there is no other reason to duplicate the array.
I am running Labview 7 on windows 2000 with sufficient memory so that the page file is not used in this example.
Thank you all very much for your time and for sharing your Labview experience,
Richard Dwan
Attachments:
Fibonacci_test.vi ‏71 KB

> That is an interesting observation you have made and seems to me to be
> quite inexplicable. The trick is interesting but not practical for me
> to use in developing a large piece of software. Thanks for your input
> - I think I'll be contacting technical support for an explaination
> along with some other anomolies involving large arrays that I have
> spottted.
>
The deal here is that the bundle and unbundle nodes must be very careful
when they are swapping elements around. This used to make copies in the
normal cases, but that has been improved. The reason that the sequence
affects it is that it affects the algorithm so that it orders the
element movement so that the algorithm succeeds in avoiding a copy.
Another, more obvious way
is to use a regular bundle and unbundle, not
the named variety. These tend to have an easier time in the algorithm also.
Technically, I'd report the diagram to tech support to see if the named
bundle/unbundle case can be handled as well. In the meantime, you can
leave the data unbundled, as in the faster version.
Greg McKaskle

Streaming large amounts of data of socket causes corruption?

I'm wrinting an app to transfer large amounts of data via a simple client/server architecture between two machines.
Problem: If I send the data too 'fast', the data arrives corrupted:
- Calls to read() returns wrong data (wrong 'crc')
- Subsequent calls to read() do not return -1 but allow me to read e.g. another 60 or 80 KBytes.
- available() returns always '0'; but I'll get rid of that method anyway (as recommended in other forum entries).
The behaviour is somewhat difficult to repeat, but it fails for me reliably when transferring the data between two separate machines and when setting the number of packets (Sender.TM) to 1000 or larger.
Workaround: Reduce number of packages send to e.g. 1; or intruduce the 'sleep' on the Sender side. Another workaround: Changing alone to java.nio.* did not help, but when I got rid of the Streams and used solely ByteBuffers, the problem disappeared. Unfortunately the Streams are required by other parts of my application.
I'm running the code on two dual-CPU machines connected via
Below are the code of the Sender and the Listener. Please excuse the style as this is only to demonstrate the problem.
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.channels.Channels;
import java.nio.channels.SocketChannel;
import java.util.Arrays;
public class SenderBugStreams {
    public static void main(String[] args) throws IOException {
        InetSocketAddress targetAdr = new InetSocketAddress(args[0], Listener.DEFAULT_PORT);
        System.out.println("connecting to: "+targetAdr);
        SocketChannel socket = SocketChannel.open(targetAdr);
        sendData(socket);
        socket.close();
        System.out.println("Finished.");
    static final int TM = 10000;
    static final int TM_SIZE = 1000;
    static final int CRC = 2;
    static int k = 5;
    private static void sendData(SocketChannel socket) throws IOException {
        OutputStream out = Channels.newOutputStream(socket);
        byte[] ba = new byte[TM_SIZE];
        Arrays.fill(ba, (byte)(k++ % 127));
        System.out.println("Sending..."+k);
        for (int i = 0; i < TM; i++) {
            out.write(ba);
//            try {
//                Thread.sleep(10);
//            } catch (InterruptedException e) {
//                // TODO Auto-generated catch block
//                e.printStackTrace();
//                throw new RuntimeException(e);
        out.write(CRC);
        out.flush();
        out.close();
import java.io.IOException;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.nio.channels.Channels;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
public class ListenerBugStreams {
    static int DEFAULT_PORT = 44521;
     * @param args
     * @throws IOException
    public static void main(String[] args) throws IOException {
        ServerSocketChannel serverChannel = ServerSocketChannel.open();
        serverChannel.socket().bind(new InetSocketAddress(DEFAULT_PORT));
        System.out.print("Waiting...");
        SocketChannel clientSocket = serverChannel.accept();
        System.out.println(" starting, IP=" + clientSocket.socket().getInetAddress() +
            ", Port="+clientSocket.socket().getLocalPort());
        //read data from socket
        readData(clientSocket);
        clientSocket.close();
        serverChannel.close();
        System.out.println("Closed.");
    private static void readData(SocketChannel clientSocket) throws IOException {
        InputStream in = Channels.newInputStream(clientSocket);
        //read and ingest objects
        byte[] ba = null;
        for (int i = 0; i < SenderBugStreams.TM; i++) {
            ba = new byte[SenderBugStreams.TM_SIZE];
            in.read(ba);
            System.out.print("*");
        //verify checksum
        int crcIn = in.read();
        if (SenderBugStreams.CRC != crcIn) {
            System.out.println("ERROR: Invalid checksum: "+SenderBugStreams.CRC+"/"+crcIn);
        System.out.println(ba[0]);
        int x = in.read();
        int remaining = 0;
        while (x != -1) {
            remaining++;
            x = in.read();
        System.out.println("Remaining:"+in.available()+"/"+remaining);
        System.out.println(" "+SenderBug.TM+" objects ingested.");
        in.close();
}

Here is your trouble:
in.read(ba);read(byte[]) does not read N bytes, it reads up to N bytes. If one byte has arrived then it reads and returns that one byte. You always need to check the return value of read(byte[]) to see how much you got (also check for EOF). TCP chops up the written data to whatever packets it feels like and that makes read(byte[]) pretty random.
You can use DataInputStream which has a readFully() method; it loops calling read() until it gets the full buffer's worth. Or you can write a little static utility readFully() like so:
    // Returns false if hits EOF immediately. Otherwise reads the full buffer's
    // worth. If encounters EOF in mid-packet throws an IOException.
    public static boolean readFully(InputStream in, byte buf[])
        throws IOException
        return readFully(in, buf, 0, buf.length);
    public static boolean readFully(InputStream in, byte buf[], int pos, int len)
        throws IOException
        int got_total = 0;
        while (got_total < len) {
            int got = in.read(buf, pos + got_total, len - got_total);
            if (got == -1) {
                if (got_total == 0)
                    return false;
                throw new EOFException("readFully: end of file; expected " +
                                       len + " bytes, got only " + got_total);
            got_total += got;
        return true;
    }

General practices - managing large amounts of data

I have a basic question regarding data management in Java. Right now, I'm working on a program that deals with hundreds of "pages" of content, each with images, HTML, and some state data. I maintain info on each page in a ContentItem object. Similarly, there can be many categories of content, each of which I maintain in a ContentCategory object.
At this point, I am controlling access to data using manager classes. For example, I have a ContentManager class that contains a global, static list of content and has the methods you'd expect such a class to have (getContent, addContent, deleteContent, etc.). Similarly, I have a CategoryManager class.
Basically, any time my program has to deal with large amounts of data or objects, I sort of fall back on these manager classes. I'm wondering if this is a reasonable practice or not. If not, perhaps some of the more experienced developers here could recommend a design pattern that fits these situations.

Thanks for the reply. I do have a sort of ad-hoc database that I'm using. I have a class called ContentReference which holds just the basic state information about a ContentItem. The actual HTML and image data are stored on disk and retrieved as needed. The files for each content item are just serialized copies of ContentItem objects. Each content item has a unique ID value which is passed to the ContentManager's getContent method. The getContent method retrieves the object from disk and returns the ContentItem object. The filenames for all the content items are based on the ID value, so the getContent method doesn't have to search through all the files until it finds the right one.
In doing this, I don't have to keep all the HTML and image data in memory. Only the ContentReference objects are kept in memory. ContentItems are loaded as needed. It seems a little messy to have two objects that refer to the same thing, but I didn't see any other way of doing it.

Storing large amounts of data

Hello,
I'd like to use Berkeley DB for logging large amounts of data - i.e. structures that are ~400KB in size and I need to store them ~10 times per second for up to several hours, but I get into quite big performance issues the more records I insert into the database. I've set the pagesize to its maximum (64KB - I split my data into several packages so it doesn't get stored on an overflow page) and experimented with several chache sizes (8MB, 64MB, 2GB, 4GB), but I haven't managed to get rid of the performance issues, independent of which access method I use (although I got the "best" results when using DB_QUEUE, but that varies heavily from day to day).
To get to the point: Performance starts at "0" seconds per insert (where 1 "insert" = 7 real inserts because of splitting up the data), between the 16750. and 17000. insertion it takes ~0.00352 per insert and when reaching the 36000. insertion it already takes about 0.0074 seconds per insert, and so on ...
Does anyone have an idea on how I can increase my performance? Because when the time needed for each insertion keeps increasing over time, it's not possible to keep the program running at its intended speed at some point.
Thanks,
Thomas

Hello,
A good starting point are the suggestions in the Berkeley DB Reference Guide at:
http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/am_misc_tune.html
http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/transapp_tune.html
http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/transapp_throughput.html
Thanks,
Sandra

Power BI performance issue when load large amount of data from database

I need to load data set from my database, which have large amount of data, it will take so many time to initialize data before I can build report, is there any good way to process large amount of data for PowerBI? As I know many people analysis data based
on PowerBI, is there any suggestion for loading large amount of data from database?
Thanks a lot for help

Hi Ruixue,
We have made significant performance improvements to Data Load in the February update for the Power BI Designer:
http://blogs.msdn.com/b/powerbi/archive/2015/02/19/6-new-updates-for-the-power-bi-preview-february-2015.aspx
Would you be able to try again and let us know if it's still slow? With the latest improvements, it should take between half and one third of the time that it used to.
Thanks,
M.

Large Amount of Data in JSF

Hello,
I am using the Table Group component for displaying data in my application designed in Java Studio Creator.
I have enabled paging on the component. I use CachedRowSet on the bean for the page for getting the data. This works very well at the moment in my development environment. At the moment I am testing on small amount of data.
I was wondering how does this component perform with very large amounts of data (>75,000 rows). I noticed that there is a button available for users to retrieve all the rows. So I was wondering apart from that instance, when viewing in a paged mode does the component get all the results from the database everytime ?
Which component would be best suited for displaying large amounts of data in a table format?
Thanks In Advance!!

Thanks for your reply. The table control that I use does have paging as a feature and I have enabled it. It still takes time to load the data initially.
I wonder if it is got to do with the logic of paging. How do you specify which set of 20 records to extract from SQL.
Thanks for your help!!

Open Large amount of data

Hi
I have a file on application server in .dat format, it contains large amount of data may be 2 million of records or more, I need to open the file to check the record count, is there any software or any option to open the file, I have tried opening with Notepad, excel .... it gives error..
please let me know
Thanks

Hi,
Try this..
Go to AL11..
Go to the file directory..Then in the file there will be field called length..which is the total length of the file in characters..
If you know the length of a single line..
Divide the length of the file by the length of single line..I believe you will get the number of records..
Thanks,
Naren

Passing large amount of data between steps

Similar Messages

Maybe you are looking for