Memory problem reading CSV to array

I've written a CSV reader in order to read training and test data into an array of arrays. Now the way I'm doing it I use stringTokenizer to return a string array of the csv file then convert that to a float array. When I convert the string to a float I make the string null, I hoped that would keep the memory down but when I get to files of about 65000 rows and 20 columns it runs out of memory. I'm sure this can be solved by increasing the heap size but I just found the whole float to string to float a little inefficient. My questions is this; is there are better way to convert a csv of floats into an array of floats, ie without first converting it to a string and then back to a float?
Error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     at InputTokenizer.csvReader(InputTokenizer.java:33)
     at run.main(run.java:49)
import java.io.*;
import java.util.*;
public class InputTokenizer {
     public String[][] numbers;
     float[][] floatInputs;
     public float[][] csvReader(String fileLocation, int numRows, int numCols){
          File file = new File(fileLocation);
          numbers = new String[numRows][numCols];
          String line = null;
          int row = 0;
          int col = 0;
          try{
               BufferedReader bufRdr = new BufferedReader(new FileReader(file));
               while((line = bufRdr.readLine()) != null)
                    StringTokenizer st = new StringTokenizer(line,";");
                    while (st.hasMoreTokens())
                         //get next token and store it in the array
                         numbers[row][col] = st.nextToken();
                         col++;
               row++;
               col=0;
               float[][] floatInputs = new float[numRows][numCols];
               for(int i=0; i<numCols;i++){
                  for(int j=0;j<numRows;j++){
                       Float buffer = new Float(numbers[i][j]);
                       numbers[i][j] =null;
                       floatInputs[i][j] = buffer;
          catch(Exception e)
               System.out.println("Exception while reading csv file: " + e);
          return floatInputs;
}

Stupid questions, changed the code to this and it seems to work = )
import java.io.*;
import java.util.*;
public class InputTokenizer {
     float[][] floatInputs;
     public float[][] csvReader(String fileLocation, int numRows, int numCols){
          File file = new File(fileLocation);
          floatInputs = new float[numRows][numCols];
          String line = null;
          int row = 0;
          int col = 0;
          try{
               BufferedReader bufRdr = new BufferedReader(new FileReader(file));
               while((line = bufRdr.readLine()) != null)
                    StringTokenizer st = new StringTokenizer(line,";");
                    while (st.hasMoreTokens())
                         //get next token and store it in the array
                         floatInputs[row][col] = new Float(st.nextToken());
                         col++;
               row++;
               col=0;
          catch(Exception e)
               System.out.println("Exception while reading csv file: " + e);
          return floatInputs;
}

Similar Messages

  • Problem reading csv file with special character

    Hai all,
    i have the following problem reading a csv file.
    442050-Operations Tilburg algemeen     Huis in  t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
    when reading this line with FM GUI_UPLOAD this line is split up in two lines in data_tab of the FM,
    it is split up at this character 
    Line 1
    442050-Operations Tilburg algemeen     Huis in
    Line 2
    t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
    Anyone have a idea how to get this in one line in my interbal table??
    Greetz Richard

    Hi Greetz Richard
      Problably character  contains same binary code as line feed + carriage return.
      You can use statement below as workaround.
    OPEN DATASET file FOR INPUT IN TEXT MODE ENCODING UNICODE
    In this case your system must support Unicode encoding
    Kind regards

  • Memory problem with loading a csv file and displaying 2 xy graphs

    Hi there, i'm having some memory issues with this little program.
    What i'm trying to do is reading a .csv file of 215 mb (6 million lines more or less), extracting the x-y values as 1d array and displaying them in 2 xy graphs (vi attacked).
    I've noticed that this process eats from 1.6 to 2 gb of ram and the 2 x-y graphs, as soon as they are loaded (2 minutes more or less) are really realy slow to move with the scrollbar.
    My question is: Is there a way for use less memory resources and make the graphs move smoother ?
    Thanks in advance,
    Ierman Gert
    Attachments:
    read from file test.vi ‏106 KB

    Hi lerman,
    how many datapoints do you need to handle? How many do you display on the graphs?
    Some notes:
    - Each graph has its own data buffer. So all data wired to the graph will be buffered again in memory. When wiring a (big) 1d array to the graph a copy will be made in memory. And you mentioned 2 graphs...
    - load the array in parts: read a number of lines, parse them to arrays as before (maybe using "spreadsheet string to array"?), finally append the parts to build the big array (may lead to memory problems too).
    - avoid datacopies when handling big arrays. You can show buffer creation using menu->tools->advanced->show buffer allocation
    - use SGL instead of DBL when possible...
    Message Edited by GerdW on 05-12-2009 10:02 PM
    Best regards,
    GerdW
    CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
    Kudos are welcome

  • USB/memory card reader problem,  any help would be great

    Hello all, I bought a new memory card reader today and our computer will not recoginize it's there. The reader is a Targus universal 32 in 1 memory card reader/writer and I think the model is TGR-CRD25, it also says USB 2.0. I didn't have any problems with the old reader as far as the machine "seeing" the reader. This new reader does show up in the system profiler but the little power light will not come on and I get a "low power for device" type message when I plug it in. I unplugged the keyboard and plugged in the new reader to the USB port where the keyboard was connected and the light came on on the reader but iphoto was still saying no storage device when I tried to import pictures? Any ideas?

    There is very little extra power available once the mouse is hooked to the keyboard. The "real" solution is to get a USB Hub with its own power supply.

  • Memory Card Reader problems

    I have seen this problem for a long time on Arch. This was occuring when I used an external SCSI memory card reader, and now I notice the same thing with my new linux box which has an internal 15-in-1 card reader connect to the USB card. It goes like this ...
    When I insert the card, my KDE desktop fails to notice it. When I open a terminal and run dmesg, nothing relevant shows up. When I attempt to mount the device, I'm told it doesn't exist. After some passage of time, it will still fail, unless I do a fdisk on that device to see if fdisk sees it. Sometimes it takes a few fdisks to find it, but eventually it seems to find it. At that point I can mount the device. But one thing that has been true in the past as well as the present ... I must issue a fdisk command for that device (and then exit fdisk) before Arch Linux ever sees the device. This has been true since 0.6x Arch, and now true of 0.72 Arch, on two different boxes and with two different card readers.

    i had this prob ages ago. was even convinced it was some physical slots that were broke, (rear slots worked fronts didnt )or one particular cardreader (gtech 21-in-1) as it glitched under windows too.
    at some point tho, in Arch anyways, the problem went away, and i think now it was more to do with modules and confs and it got solved more by tinkering than knowing what was what definitively. we live and learn
    maybe try as su - root  lsmod |grep usb
    my reply is
    [root@localhost ~]# lsmod |grep usb
    usb_storage            78656  0
    usbhid                 47392  0
    usbserial              29260  0
    usbcore               116256  7 ohci_hcd,usb_storage,usbhid,ehci_hcd,uhci_hcd,usbserial
    ide_core              116940  29
    usb_storage,ide_scsi,ide_floppy,ide_cd,ide_disk,via82cxxx,trm290,
    triflex,slc90e66,sis5513,siimage,serverworks,
    sc1200,rz1000,piix,pdc202xx_old,pdc202xx_new,ns87415,
    it821x,hpt366,hpt34x,generic,cy82c693,cs5530,cmd64x,atiixp,
    amd74xx,alim15x3,aec62xx
    (ide_core line reformatted so page isnt a mile wide.)
    also, check in rc.conf MODULES for usbserial and if theres a ! in front of it, remove it.

  • Use Performance Tool to check memory problem

    Hi, I am new to mac programming.
    How can I use the performance to check memory problem such as "uninitialized memory read", "memory access out of array boundary" etc?
    Thanks,
    fm

    Don't know about source code analyzer, but dbx, the Sun Studio debugger, is capable of detecting memory leaks at run time. You can use bcheck script on your application or start dbx, issue "check -leaks" and then run your app under dbx.

  • Memory Problem with SEt and GET parameter

    hi,
    I m doing exits. I have one exit for importing and another one for changing parameter.
    SET PARAMETER exit code is ....
    *data:v_nba like eban-bsart,
           v_nbc like eban-bsart,
           v_nbo like eban-bsart.
           v_nbc = 'CAPX'.
           v_nbo = 'OPEX'.
           v_nba = 'OVH'.
    if im_data_new-werks is initial.
      if im_data_new-knttp is initial.
        if im_data_new-bsart = 'NBC' or im_data_new-bsart = 'SERC' or im_data_new-bsart = 'SERI'
           or im_data_new-bsart = 'SER' or im_data_new-bsart = 'SERM' or im_data_new-bsart = 'NBI'.
          set parameter id 'ZC1' field v_nbc.
        elseif im_data_new-bsart = 'NBO' or im_data_new-bsart = 'NBM' or im_data_new-bsart = 'SERO'.
          set parameter id 'ZC2' field v_nbo.
        elseif im_data_new-bsart = 'NBA' or im_data_new-bsart = 'SERA'.
          set parameter id 'ZC3' field  v_nba.
        endif.
      endif.
    endif. *
    and GET PARAMETER CODE IS....
      get parameter id 'ZC1' field c_fmderive-fund.
      get parameter id 'ZC2' field c_fmderive-fund.
      get parameter id 'ZC3' field c_fmderive-fund.
    FREE MEMORY ID 'ZC1'.
      FREE MEMORY ID 'ZC2'.
       FREE MEMORY ID 'ZC3'.
    In this code i m facing memory problem.
    It is not refreshing the memory every time.
    So plz give me proper solution.
    Its urgent.
    Thanks
    Ranveer

    Hi,
       I suppose you are trying to store some particular value in memory in one program and then retieve it in another.
    If so try using EXPORT data TO MEMORY ID 'ZC1'. and IMPORT data FROM MEMORY ID 'ZC1'.
    To use SET PARAMETER/GET PARAMETER the specified parameter name should be in table TPARA. Which I don't think is there in your case.
    Sample Code :
    Data declarations for the function codes to be transferred
    DATA : v_first  TYPE syucomm,
           v_second TYPE syucomm.
    CONSTANTS : c_memid TYPE char10 VALUE 'ZCCBPR1'.
    Move the function codes to the program varaibles
      v_first  = gv_bdt_fcode.
      v_second = sy-ucomm.
    Export the function codes to Memory ID
    EXPORT v_first
           v_second TO MEMORY ID c_memid.        "ZCCBPR1  --- Here you are sending the values to memory
    Then retrieve it.
    Retrieve the function codes from the Memory ID
      IMPORT v_first  TO v_fcode_1
             v_second TO v_fcode_2
      FROM MEMORY ID c_memid.                                   "ZCCBPR1
      FREE MEMORY ID c_memid.                                   "ZCCBPR1
    After reading the values from memory ID free them your problem should be solved.
    Thanks
    Barada
    Edited by: Baradakanta Swain on May 27, 2008 10:20 AM

  • Memory problems with PreparedStatements

    Driver: 9.0.1 JDBC Thin
    I am having memory problems using "PreparedStatement" via jdbc.
    After profiling our application, we found that a large number oracle.jdbc.ttc7.TTCItem objects were being created, but not released, even though we were "closing" the ResultSets of a prepared statements.
    Tracing through the application, it appears that most of these TTCItem objects are created when the statement is executed (not when prepared), therefore I would have assumed that they would be released when the ResultSet is close, but this does not seem to be the case.
    We tend to have a large number of PreparedStatement objects in use (over 100, most with closed ResultSets) and find that our application is using huge amounts of memory when compared to using the same code, but closing the PreparedStatement at the same time as closing the ResultSet.
    Has anyone else found similar problems? If so, does anyone have a work-around or know if this is something that Oracle is looking at fixing?
    Thanks
    Bruce Crosgrove

    From your mail, it is not very clear:
    a) whether your session is an HTTPSession or an application defined
    session.
    b) What is meant by saying: JSP/Servlet is growing.
    However, some pointers:
    a) Are there any timeouts associated with session.
    b) Try to profile your code to see what is causing the memory leak.
    c) Are there references to stale data in your application code.
    Marilla Bax wrote:
    hi,
    we have some memory - problems with the WebLogic Application Server
    4.5.1 on Sun Solaris
    In our Customer Projects we are working with EJB's. for each customer
    transaction we create a session to the weblogic application server.
    now there are some urgent problems with the java process on the server.
    for each session there were allocated 200 - 500 kb memory, within a day
    the JSP process on our server is growing for each session and don't
    reallocate the reserved memory for the old session. as a work around we
    now restart the server every night.
    How can we solve this problem ?? Is it a problem with the operating
    system or the application server or the EJB's ?? Do you have problems
    like this before ?
    greetings from germany,

  • Dual Channel memory problems on Neo 875P

    I am having considerable trouble getting the NEO 875P to work with dual channel DDR400 DIMMs. I have two Kingston KVR400X64C25 DDR400 256MB memory modules, each of which works correctly by itself. However, once I move one
    of the modules to channel B, I cannot boot my operating system (currently Windows XP.) The problem also occurs in Windows 2000, which I also have on the system. I receive blue screen errors that indicate paging/memory problems. I have upgraded my BIOS to the latest version (1.3)* in the hope that it would solve my problems, but it has not. The POST screen detects that there is 512MB of memory working in Dual Channel mode, but even when it does get through the O/S booting process (rarely), it crashes within minutes. Putting just one of the chips in makes the system perfectly stable. If anyone has any information, please let me know!
    Thanks,
    Jim Keller
    http://www.centerfuse.net
    * My board was dead after trying to flash the 1.3 BIOS, but after about 10 attempts of using CTRL+HOME method to recover the BIOS, it finally worked with version 1.0. I then re-flashed with 1.3, and it rebooted fine. Are there any updates about all of the problems with v1.3 ?

    >I can only answer based on what I've read on this forum
    I figured as much, I was just wondering if you knew whether MSI themselves frequents the forums or answers emails sufficiently.
    I've had an interesting development though. I stumbled upon Tom's Hardware Guide's article about 875P motherboards at http://www17.tomshardware.com/motherboard/20030519/i875p-01.html
    They mentioned that they needed to increase the memory voltage from 2.5 to 2.6 in the BIOS to get some memory modules to work. I did this, and for the first time, I am running in Dual channel DDR mode. However, the article also mentions that "the system does not run completely stably", so I guess I'll have to wait and see if it crashes again. However, I'm far enough beyond the XP splash screen to post this message, so I guess we're getting somewhere!
    -Jim Keller
    http://www.centerfuse.net

  • HP Envy Touchsmart 15-J104sa Memory Card Reader not working

    I took delivery of an HP Envy Touchsmart 15-J104sa running Windows 8.1 this week and the internal memory card reader isn't working. It doesn't recognise any SD cards (I've tried more than one) and continues as if nothing has been inserted; showing no error messages. All of the cards work on my previous HP laptop. The only way to get the laptop to show an inserted SD card is to run the driver update software (http://ftp.hp.com/pub/softpaq/sp63001-63500/sp63415.exe) every time an SD card is inserted; this must be done with the card in the reader. If the software is run and then the card is inserted nothing happens. If the card is ejected and then re-inserted (or a new card inserted) it isn't recognised and the process has to be restarted and the driver reinstalled even though nothing has changed.
    Can somebody please help me with this issue or I will have to return the laptop to HP. I do a lot of photography so being able to read an SD card quickly and reliably is extremely important for me; it's not a problem I expected from a brand new laptop straight out of the box.
    Thank you all.
    This question was solved.
    View Solution.

    Unfortunately non of the driver updates solved the problem but a BIOS update through the HP Support application seems to have remedied the issue. Whilst I appreciate there were no specific drivers available I find it hard to accept that HP should be able to release a model where they know the hardware will not be fully supported by the OS and in this case not function.

  • Memory problem with 5330 XpressMusic

    Hi,
    I seem to have a memory problem with my 5330, I have a 1Gb micro disc installed. My son sent me a photo from his phone, when it arrived I got a warning on the scren that 'there is not enough memory to receive messages', I have tried to delete as much as I can, a lot tells me I can't delete it, but I still get the same warning. This warnig shows up each time I switch on the phone. How can I get myself more space? Can I move things from the phone memory to the memory card?
    Gerald
    Message Edited by warmbells on 01-Nov-2009 06:19 PM

    Are you sure you're using a 5330 XpressMusic? Or is it a 5320 XpressMusic? The former is yet to be released, AFAIK.
    Assuming that you are using a 5320, try these steps:
    -> Move all photos, music, videos stored in the phone memory to the memory card. You can use the built-in File Manager to accomplish this.
    -> Delete all files that you would've received via bluetooth, present in your Inbox. You can save them to the mass storage.
    -> Clear the browser cache.
    -> Clear the Sent Items folder.
    -> Make sure you don't have too many messages in your inbox.
    Hope this helps
    Cheers,
    DeepestBlue
    5800 XpressMusic (Rock Stable) | N73 Music Edition (Never Say Die) | 1108 (Old and faithful)
    If you find any post useful, click on the Green "Kudos" Button on the left to say Thank You...

  • Memory problem with ITS

    Hello friends,
    we are having memory problem with integrated ITS.. on one application server all memory of ITS is getting exhausted. We think few users take lots of memory and it never gets released.
    I checked Note 742048 - Integrated ITS, memory requirement in application server but parameters looks ok. Sometime, when i kill a user session in SM04, some part of memory was released. However i am not able to find which sessions are taking max memory as i do not see any workproess active in SM50.
    in SITSPMON :
    Memory Consumption: Overview
    Sessions:     27      24,710,431 Bytes       915,201 Bytes/Session       2,433,8
    Templates:            14,793,306 Bytes
        Sess. & Templ.    39,503,737 Bytes     Currently available to ITS: 81.92MB o
    ITS Session     memory type     Peak          Memory     Total     Current
    USER 001 2463     Session Memory     1,955,757     938,597     7,574     723
    USER 001 2503     Session Memory     2,008,621     965,245     9,181     930
    USER 001 2523     Session Memory     2,412,477     856,925     11,327     82
    thanks
    ashish

    and what is 2463 is line : USER(user ID) 001(Client) 2463(??) Session Memory 1,955,757 938,597 7,574 723 as this is not a work process ID.]
    Basically i am trying to correlate ITS session with SM04 Session of user.

  • Memory problem with my Nokia 3220

    Hi,I am new here and I have a memory problem with my Nokia 3220.I deleted all of my stuff in galery exept the BlueSquare theme and Nokia Tune but it says that i only have 483kb free memory and galery has 1,7mb memory taken. What should i do ?...I hope you understood my problem and i hope you can help me!
    Shibuy
    Thank You!!

    I have the same problem recently. I bought my nokia 3220, 3 years ago and all was ok but now I download some free themes from a web site and suddenly I realized that my cellphone has less memory. In the phone options says that I have 2,2 MB in my Gallery when actually I have only 800 KB in my Gallery. I don't know what happen!! Maybe it's a virus or what is the solution?? Can I  restart the cellphone ?

  • Memory Problem With 4gb Crucial Ballistix and Asus M4A785TD-V EVO

    motherBoard: ASUS M4A785TD-V EVO
    bios Version :2005
    Video: amd hd5750
    Processor: AMD PHENON II X2 550 3.100MHZ
    Memory: Crucial
    Modele Memory: blt4g3d1608dt1tx0
    Capacity: 4GB
    greetings to all
    i have asus m4a785td-v evo mobo. i ve bought 4 gb crucial ballistix ram today. it says on product its 1600 mhz and cl8 but it shows on my pc 1333 mhz and cl9.  when i was searching for this problem i came across this topic: http://forum.crucial.com/t5/Crucial-Ballistix-gaming-memory/Memory-Probleme-With-8GO-Crucial-Ballistix-and-Asus-M4A785TD-V/td-p/9464 can i apply the same settings? how can i fix that?  is it possible via bios settings? thx in advance and sry for my bad english. here is some screenshots: 

    I am not good at RAM setting but the 'Timings Table' shows there is XMP-1600 profile available so I believe it would be enough to turn on that memory profile in BIOS.

  • LOTS of Memory problems on new 8 core

    I just moved from my totally stable G5  running Logic Pro8 on Tiger to my new Mac Pro 8 core. I am having nothing but problems with
    low memory issues. Everything is up to date and it doesn't seem to matter if I'm in 32 or 64 bit, running reason in Rewire or not. I am having
    trouble narrowing down the problem. It seems to be random. I have seen a lot of posts about this but no real solutions. I'm thinking I got
    a bad computer. it's 2 months old. This computer has 6 gig of RAM. I have talked with lots of friends running the same systems with
    no problems. What is the latest? I have been on the phone with Apple and Sweetwater and it's all vague. A couple examples and as you can see I don't have
    that much going yet. WAY less than I could run on my G5. I have also been unable to get a second monitor to work with the mini display to DVI connector
    but it did work with mini display to VGA on the same monitor which is weird, and it seemed slow.
    (Mac Pro 8 Core Snow Leopard 10.6.8 - Logic Pro9 9.1.5 - Reason5 (tried 6 for 64 bit as well) - APOGEE Ensemble - Omnisphere, Stylus, KONTACT4,
    East West Play, Superior Drums, UAD 2 and others)

    OK. Here is a perfect example of what I am dealing with. I have a song open with a movie because I am scoring today (I never had trouble on my G5 importing and workin with movies and dialogue tracks). I have Reason open
    with one of the Reason5 Bass sounds, which takes alot of memory but again never an issue on my G5. I have TWO audio tracks with no plug-ins. Superior Drums, one kit. 1 ESX with a simple cymbal swell sound. I try and load a pad in Omnisphere and I get the below message. If I turn the bass into audio and quit Reason5 it will let
    me load the omnisphere sound, so I'm wondering how much of this is a Reason/ Rewire issue. Again never had
    trouble with this on G5, Tiger, Reason4 and Logic8. I am running at 32 bit because Reason5 is not 64 bit. Reason6 is 64 bit and even though I don't like the new interface I will pruchase it and run at 64 bit if it will
    work. Again I keep hearing different things about 64 bit and 32 bit. I'm trying both and I don't see any
    consistent change, still weird and random memory issues.

Maybe you are looking for

  • Please can someone tell me how to save a MP3 file from my email on my iPad so I can put in on Facebook? Thankyou

    Please can someone tell me how to save a MP3 file from my email on my iPad so I can put it on Faceboo? I use hotmail and please tell me step by step as I am not a tech guru lol. Thankyou

  • Problem with Disk Premission?

    Hello all ! I can't repair my disk premission, everytime I try I get this message... "2012-07-27 17:55:51 +0100: Warning: SUID file "System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/MacOS/ARDAg ent" has been modified and will not be

  • ADMT - Minimum required attributes?

    I plan on using ADMT to migrate some accounts from a recently acquired company. I've linked their AD accounts to mailboxes on our Exchange 2010 server and have populated their mailbox AD accounts with all the census information. I want to "merge" the

  • Setting target for the handler in CPO

    Hi, I would like to know how to set the target while invoking a handler at CPO end from CPSC. For testing at CPO end, i usually set the target at start time or as a default target. But i need to select the target based on the user who invokes the pro

  • Checking for status of a long-running BPEL process

    Hi experts, I have one BPEL process for originating customers' loan which usually involves various steps which takes around 15-20 mins to complete per instance. I would like to implement a user interface that can check the process's progress and disp