Memory problem with loading a csv file and displaying 2 xy graphs

Hi there, i'm having some memory issues with this little program.
What i'm trying to do is reading a .csv file of 215 mb (6 million lines more or less), extracting the x-y values as 1d array and displaying them in 2 xy graphs (vi attacked).
I've noticed that this process eats from 1.6 to 2 gb of ram and the 2 x-y graphs, as soon as they are loaded (2 minutes more or less) are really realy slow to move with the scrollbar.
My question is: Is there a way for use less memory resources and make the graphs move smoother ?
Thanks in advance,
Ierman Gert
Attachments:
read from file test.vi ‏106 KB

Hi lerman,
how many datapoints do you need to handle? How many do you display on the graphs?
Some notes:
- Each graph has its own data buffer. So all data wired to the graph will be buffered again in memory. When wiring a (big) 1d array to the graph a copy will be made in memory. And you mentioned 2 graphs...
- load the array in parts: read a number of lines, parse them to arrays as before (maybe using "spreadsheet string to array"?), finally append the parts to build the big array (may lead to memory problems too).
- avoid datacopies when handling big arrays. You can show buffer creation using menu->tools->advanced->show buffer allocation
- use SGL instead of DBL when possible...
Message Edited by GerdW on 05-12-2009 10:02 PM
Best regards,
GerdW
CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
Kudos are welcome

Similar Messages

Loading a CSV file and accessing the variables

Hi guys,
I'm new to AS3 and dealt with AS2 before (just getting the grasp when the change it).
Is it possible in AS3 to load an excel .csv file into Flash using the URLLoader (or ???) and the data as variables?
I can get the .csv to load and trace the values (cell1,cell2,cell3....) but I'm not sure how to collect the data and place it into variables.
Can I just create an array and access it like so.... myArray[0], myArray[1]? If so, I'm not sure why it's not working.
I must be on the completely wrong path. Here's what I have so far....
var loader:URLLoader = new URLLoader();
loader.dataFormat = URLLoaderDataFormat.VARIABLES;
loader.addEventListener(Event.COMPLETE, dataLoaded);
var request:URLRequest = new URLRequest("population.csv");
loader.load(request);
function dataLoaded(evt:Event):void {
var myData:Array = new Array(loader.data);
trace(myData[i]);
Thanks for any help,
Sky

just load your csv file and use the flash string methods to allocate those values to an array:
var myDate:Array = loader.data.split(",");

// Code Help need .. in Reading CSV file and display the Output.

Hi All,
I am a new Bee in code and started learning code, I have stared with Console application and need your advice and suggestion.
I want to write a code which read the input from the CSV file and display the output in console application combination of first name and lastname append with the name of the collage in village
The example of CSV file is
Firstname,LastName
Happy,Coding
Learn,C#
I want to display the output as
HappyCodingXYZCollage
LearnC#XYXCollage
The below is the code I have tried so far.
// .Reading a CSV
var reader = new StreamReader(File.OpenRead(@"D:\Users\RajaVill\Desktop\C#\input.csv"));
List<string> listA = new List<string>();
while (!reader.EndOfStream)
var line = reader.ReadLine();
string[] values = line.Split(',');
listA.Add(values[0]);
listA.Add(values[1]);
listA.Add(values[2]);
// listB.Add(values[1]);
foreach (string str in listA)
//StreamWriter writer = new StreamWriter(File.OpenWrite(@"D:\\suman.txt"));
Console.WriteLine("the value is {0}", str);
Console.ReadLine();
Kindly advice and let me know, How to read the column header of the CSV file. so I can apply my logic the display combination of firstname,lastname and name of the collage
Best Regards,
Raja Village Sync
Beginer Coder

Very simple example:
var column1 = new List<string>();
var column2 = new List<string>();
using (var rd = new StreamReader("filename.csv"))
while (!rd.EndOfStream)
var splits = rd.ReadLine().Split(';');
column1.Add(splits[0]);
column2.Add(splits[1]);
// print column1
Console.WriteLine("Column 1:");
foreach (var element in column1)
Console.WriteLine(element);
// print column2
Console.WriteLine("Column 2:");
foreach (var element in column2)
Console.WriteLine(element);
Mark as answer or vote as helpful if you find it useful | Ammar Zaied [MCP]

Problem with loading a Sequence File - Limited Windows Account

Dear Sirs,
I have a problem and a very less time to fix it; so I'm asking for your kind help.
I have a production PC, with a Debug Deploy License of TS 3.1 and LV 7.1; the operating system is Win XP SP2.
No problem when I login the session with my Administrator Windows Account; on the contrary, when I try to login with a Limited Windows Account, I'm not able to load my Sequence File, and the text appearing is attached to this message.
Someone can advise me a solution to quickly solve my problem?
Thank you very much and Best Regards.
Stefano
Attachments:
TS_error_winXP_login_limited_account.jpg ‏224 KB

Stefano,
if you open files in TestStand, TestStand tracks this in different configuration files (for example for the "most recently used" list). Therefore, TestStand tries to save these changes in the ini-files. It seems to me, that the installationdirectory of TestStand is not permitted to be used for writing for your user login in Windows.
So you have different solutions here, the simpliest being:
Give your user the right to write files into the < TestStand >\-folder.
hope this helps,
Norbert B.Message Edited by Norbert B on 11-16-2006 07:28 AM
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

Problem with loading Danish dictionaries files.

Hi,
I want use Danish spell check for my project and when my app start load
spell check files I get exception. Files are here http://www.stavekontrolden.dk/main/top/index.php.
Please tell me is thare any way to use Danish spell check in flex application?
And alos I can't find Finnish spell check that I could use with flex.
Is flex support Finnish spell check ?? If no will flex support it?
Error decription:
Main Thread (Suspended: Error: null cannot be parsed to a squiggly dictionary)
Thanks,
Vladimir

As Ned said the problem might be with the file structure on the server. Other problems may be with spelling. You computer may not be case sensitive to the names of the .swf files that you are loading, but your server might. So, on your computer when you are testing, second.swf is the same as Second.swf, or even second.SWF. But, on the server each of these variations is seen as a different file. Another problem is that when you are loading files from the same computer in testing, those files are close and so they load very quickly from your computer. When the files have to travel from the server, there will likely be a measurable period of time before the file is ready to be shown. This is particularly problematic with video that is embedded into an .swf. The whole file will need to be downloaded and uncompressed before it is available to play. If you stream the video as an external file, you can shorten the wait considerably.

Problems with load DSO from file

I am working with BI 7.0.
I am loading data from file to DSO and my problem is that data is not active in my DSO, the data are loaded how New Data and I haven´t the data in query and DSO.
The settings of my dso are:
Type: standard
SIDs Generation upon Activation: X
Set Quality Status to 'OK' Automatically: X
Activate Data Automatically: X
I have created one transformation from datasource to DSO and one DTP that extract from datasource ( No PSA).
When I execute the DTP there aren´t errors, the status is green and ok but the data is not active.
If I try activate the data manually the job is launched but the data no active.
Thanks.

hai
Isabel Bautista
u r saying the data is not active in dso and also u r loading the data from file directly( no psa).
i think u know in B.I the data should be in psa then only u can load the data using DTP .so how can u load the data with out data in psa .i thinks may be because of this u r data is not activating in dso
if it helps
assign some points ok

Problem with loading data from file in ../demo/file/test

hi there,
I am using ODI to load text files located on a UNIX server where ODI is installed.
I am using the FILE_GENERIC data server and physical schema that came with ODI and I can reverse and load the demo files located in ../demo/file
I created a sub-directory 'test' under ../demo/file and placed a file there.
When I try to reverse this file (in directory ../demo/file/test) I get the error 'The directory ../demo/file/test specified in your schema does not exist'
I am sure the userid has equal rights to both directories ../demo/file and ../demo/file/test
What could I be missing?

In the topology manager
1. in the logical architecture : If it doesn't already exists (it should exist if your can use it on your local PC), you import the file technology
2. In the physical architecture, you create your data server for files .In your case, it shoud already exist for same reason as above. Is has a blue icon under the file technology
2. Select the Data Server, right click and select Insert Physical Schema . The Physical Schema window appears. On Definition tab, enter a name and then the path of your directory in the data schema field. No need to enyter a work schema for files.
4. Go to the Context tab, Select a Context and an existing Logical Schema for this new Physical Schema.
Select an existing Context in the left column, the type the name of a Logical Schema in the right column. This Logical Schema is automatically created and associated to this physical schema in this context.
It should work but i have never experienced it on a Linux system...

ITunes / iPod having problems with lyrics in MP3 files and not with M4A ?

Over the last two month (without this forum) I believed sometimes to be the only one having a weird problems with iTunes lyric format structure.
Now after more than 50h of investigations interesting results appear.
The field of trouble:
a) People are not able to delete lyrics in iTunes
b) People discover that iTunes and their iPod/iPhone do not show the same lyrics
c) it seems that the lyrics of certain songs can't synchronised with the iPod
d) it seems that the problem in iTunes can appear more than one times that itunes corrupts the contents of lyrics in a way that:
e) the text structure is damaged, no more line breaks, no empty lines between text sections.
After all my suspicion is: All of this trouble can be caused by one problem:
iTunes (at least latest version 9.0.3 (15) has a -unkonwn- problem with the lyric-text structure in MP3-files. It seems that iTunes does not have the same problems if the file is in M4A-file format, created by apple-store files being downloaded, or CDs being converten in iTunes using the build in AAC-converter!
Converting "infected" files (via AAC) to M4A are clean again (on iTunes and iPod) showing no more one of the trouble ahead. The ghost is eliminated. Everything synchronizes as it should be.
And I for myself have to convert more than 3000 songs now... to ged rid of this nightmare.
Now I'm very curious of You experiences. Can You confirm my suspicion? Or have You had similar problems with Your M4A-files too?
- Best Regards ThomasK_DJ, south Germany

Try this -> http://mp3val.sourceforge.net/ to verify and fix MP3 files.
Also, it could be the version of the ID3 tags.
Select a song in iTunes then go to iTunes menu Advanced -> Convert ID3 tags and select a different version and see what happens.

Having problems with launchd a plist file and launch on demand.

First, my plist file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Debug</key>
<true/>
<key>KeepAlive</key>
<false/>
<key>Label</key>
<string>TestApp</string>
<key>OnDemand</key>
<true/>
<key>Program</key>
<string>/Applications/Test/TestApp</string>
<key>ProgramArguments</key>
<array>
<string>daemon</string>
</array>
<key>Sockets</key>
<dict>
<key>SockFamily</key>
<string>IPv4</string>
<key>SockPassive</key>
<true/>
<key>SockServiceName</key>
<string>8585</string>
<key>SockType</key>
<string>stream</string>
</dict>
<key>WorkingDirectory</key>
<string>/Applications/Test/</string>
</dict>
</plist>
The idea is, if another app which I have tries to connect to 127.0.0.1:8585 that my test app will launch and do its stuff. The test app is the server and the guy making the call is the client.
The plist file is located in "/Library/LaunchDaemons" and is owned by root and only readable by anyone but root. Similarly TestApp is only readable/executable by root and no one else can do anything. Finally TestApp is a C++ program not an actual application ie:TestApp.app/Contents/etc...
If it were windows, it would be TestApp.exe.
Anyway, when I invoke my other program, TestApp isn't launched. (I rebooted after I put the plist file in so it would be read). I know TestApp and TestClient can talk, if I run them both myself everything is fine. I just want to get launchd to do it on demand. There isn't a clear way for me to debug this because if there is an error, I do not know where to look to see it, if any such errors are reported. Syslog doesn't show anything of interest in this reguard.
Any help is greatly appricated.

Hello Mark:
You are seeing a "normal" behavior, if this is the case on the load of a first report. Subsequent reports should be significantly faster. Only way to make the first load appear faster is to load the CR assemblies and dlls on app start. E.g.; load a fake report on app start. Unfortunately, those 20+ seconds will now move to the initialization / load of the app...
Ludek
BTW., downloading the latest Service packs is never a bad idea and may help a bit, but not 10+ seconds for sure.

[SOLVED Thanks ]// problem with loading XML in projectors and Firefox

Hi !
I've been working and testing my movie in flash only. when I
start the file in IE it works but when I try to load the file in
firefox or make a standalone exe player it won´t read the
XMLfile. I've tried to allow all access and set the root in the
tilestructor to a trusted location in the flashplayers settings.
/Micke

HI Micke,
What is the code you are using to load the XML file ?
Regards
FlashJester Support Team
e. - [email protected]
w. -
http://www.flashjester.com
There is a very fine line between "hobby" and
"mental illness."

Problem with loading a sound file

Hello, I am trying to play a sound file (located in same dir as the class files) from a swing app using this line.
AudioClip clip = Applet.newAudioClip(new URL("sound.au"));When I run the program, I always get a MalformedURL Exception.
Could someone please tell me the correct way to load a sound file in swing.
Thanks

"sound.au" is no valid URL, hence the exception.
How about "file:///c/myfiles/sound.au" or something?

Saving HTML with DIV to DOC file and display correctly

I'm generating a DOC file of HTML. I happen to use DIV and
CSS to make it look the way I want it to, but MS WORD is ignoring
the formating.
MS Word 2000

You might want to provide some more details if you expect a
useful response. I've generated Word files using ColdFusion and
haven't had any problems with the styles. You might want to try
creating a document in Word, saving it as HTML, and then seeing
what the styles look like. Keep in mind that you have different
units of measurement for print (points and inches instead of
pixels) that could affect a document.

Reading a Blob (CSV file) and displaying the contents

Hello Experts,
I’m currently working on a system that allows the users to upload an Excel spreadsheet (.xls) in the system. The upload page is a PL/SQL cartridge. Then I’ve written a Java servlet (using Oracle Clean Content) to convert the XLS into a CSV and store it back in the database. (it is stored in the “uploaded_files” table as a blob). I’m trying to create another procedure to read the contents of the blob and display a preview of the data on the screen (using an html table (will be done using cartridge)). After the preview, the user can choose to submit the data into the database into the “detail_records” table or simply discard everything.
Can anyone provide me any guidelines and/or code samples for reading the contents of a blob and displaying it as an html table? My data contains about 10 rows and 20 columns in the spreadsheet. I’ve been looking into using associative arrays or nested tables but I’m getting confused about how to implement them in my situation.
Any help is greatly appreciated.

BluShadow,
Thanks for your response. The reason that it is a blob is that we use the same table for uploaded files for several applications. Some files are text, some excel, some other formats. Hence the table has been set up in a generic sense.
However, in my procedure, I'm using
DBMS_LOB.createtemporary and
DBMS_LOB.converttoclob
Is there any way you could possibly provide an example of how to read the contents of the file?

Problem with command line argument files and quoted strings

I'm trying to use the following custom tag:
-tag com.me.Test.testTag:f:"My Test Tag"
This works fine when sent straight to the command line, but if I put it in an argument file (@argfile) along with the rest of my command, I get the following error:
javadoc: Illegal package name: "My Test Tag"
Any idea why this is happening? If I use a one-word tag header and omit the quotes, javadoc runs with no problems. I'm using Sun's SDK 1.4.1.
Thanks for your help!
Linda Howard

White space is treated a bit differently in an @argfile,
to enable users to put packages or options on separate lines.
You might try escaping the spaces:
"My\ Test\ Tag"
or using a non-breaking space:
"My Test Tag"
What operating system are you on?
-Doug Kramer
javadoc team

Problem with CS3 InCopy text files and InDesign CS5

I have a large annual magazine that I am developing. There is one section that doesn't change much from year to year and I was hoping to grandfather in the InCopy text files from last year's edition. I've successfully used this strategy the past few years without issue. However, now when I try to check out those text files from last year, I receive the following message "This story needs to be coverted to "InCopy Document" format in order to edit it."
I am running both InDesign CS5 and InCopy CS5 in SL 10.6.4. The previous year's magazine document was in CS3 InDesign format and the text files were in InCopy CS3. Is there some sort of compatibility issue? Is there a specific method for converting them so I can check them out in InDesign? I haven't attempted to check them out in InCopy CS5 yet but I will try that next. But I really need the ability to check out the content via InDesign too.
Thanks in advance.
Robert

AFAIK, you're going to actually need to open the INCX files in InCopy and then resave them as ICML file. You can then relink them all in InDesign.
Other choice is killing the InCopy links and then relinking everything.
Best practice would be to finish any job started in CS3 with CS3 and then move to CS5 with new projects.
Bob

Memory problem with loading a csv file and displaying 2 xy graphs

Similar Messages

Maybe you are looking for