Large File Reading and Processing

Hi:
Suppose I have a file much larger than my computer's memory e.g. say on a Windows XP system, the memory is 256 MB RAM and the file size is 1 GB on disk. Now if I want to create a File object and read from this file, is this possible to do? If yes, how about the performance?
I understand that this is more of an Operating System question but nevertheless it affects the Java process.
Thanks!
Rahul.

If you mean, you plan to read the whole file into memory, then:
Theoretically possible, sure. The process would be much bigger (in terms of memory used) than the amount of memory available, which means that it would start swapping out to disk, and that would reduce performance, although to what extent depends on XP's virtual memory implementation, which I know nothing about.
But if all you mean is that you want to create a File object and read bits of the file, it won't make any difference. File represents a name or description of a file, not the file itself (you can create File objects for files that don't exist). So it doesn't need to read the whole into memory just to create a File object. You can easily create a File object for the huge file, and then read it in line by line, throwing away old lines as you go, and you won't use up very much memory at all. (Unless, of course, there's some horrible weird artifact of XP that I'm not aware of.)

Similar Messages

  • Reading and processing file(s) in a filesystem

    I'm fairly new to java programming and need help.
    I need to write code which will and read and process 1 or many files from a filesystem. If a file is found, read and process it. Then do same for all files found.
    Please help.
    Thanks in advance!

    Some classes of the package java.io will be used .
    http://java.sun.com/j2se/1.4.1/docs/api/java/io/package-tree.html
    (The java.io.File has the methods list() and listFiles(), of which the former returns an array String[] and the latter returns an array java.io.File[]. )

  • Programmatically reading and processing a spool file

    Hello,
    I would like to read and process a spool file generated by a background job in my ABAP program.
    Can anyone advise if this is possible in SAP and if so, the best way to do this?
    The only information I will have in my program is the job name and the date on which the job was run. I then need to find the corresponding spool and read this into an internal table.
    Any ideas on how to do this would be appreciated.
    Thanks,
    Ruby

    DATA : it_spool      TYPE STANDARD TABLE OF rsporq    .
    *&      Form  find_spool_request_id
    FORM find_spool_request_id.
      CALL FUNCTION 'RSPO_FIND_SPOOL_REQUESTS'
        EXPORTING
          allclients          = '320'
          datatype            = '*'
          has_output_requests = '*'
          rq0name             = nast-dsnam
          rq1name             = '*'
          rq2name             = '*'
          rqdest              = 'LOCL'
          rqowner             = sy-uname
        TABLES
          spoolrequests       = it_spool
        EXCEPTIONS
          no_permission       = 1
          OTHERS              = 2.
      IF sy-subrc <> 0.
        MESSAGE i000 DISPLAY LIKE 'E' WITH text-002.
        LEAVE LIST-PROCESSING.
      ENDIF.
    ENDFORM.                    " find_spool_request_id
    READ TABLE it_spool INTO wa_spool INDEX 1.
      v_spoolno = wa_spool-rqident.
    *Get Spool request attributes
      SELECT SINGLE *
        FROM tsp01
        INTO tsp01
        WHERE rqident EQ v_spoolno.
    Edited by: K.Manas on Jan 7, 2011 6:22 AM

  • Small NCP packets for large file read

    I am running the lastest 4.91sp4 client and am getting really slow
    performance through a application called bimap which reads and processes
    large map files. When these files are on a Netware server (OES or 6.5 NSS
    volume), Ethereal shows that NCP read requests are for only 512bytes and
    then an NCP read OK is returned with 512bytes worth of data. This takes
    the program several minutes to read the file and process. When the same
    files sits on a Micrsoft Server Ethereal shows bursts of TCP packets all
    with 1500bytes of data, resulting in a much faster load.
    LIP is turned on and seems to work OK when copying files around NetWare
    server, I see the bursts of 1500bytes of data, but doesn't seem to work
    when using this bimap application. Any ideas?

    Hi,
    Sasha wrote:
    >
    > Their not on the same segments, the routers are all correct according to
    > our Networks team. Ethereal definateley shows the NCP read request as
    > 512Bytes. Other applications such as Word opening large files are OK from
    > same PC, Ethereal shows NCP request of 4096. Once again using the Bimap
    > app and reading the same files stored on a Microsoft Server works OK as
    > well. My understanding with LIP is that the large packet size will try and
    > be negotiated, if it fails it reverts to it's default of 512. I assume it
    > is failing and I don't know why?
    It sounds like the application is specifically reading 512 byte
    fragments on purpose. The fact you see bigger reads on Windows is likely
    because on windows oplocking (caching) is enabled, but isn't on Netware.
    CU,
    Massimo Rosen
    Novell Product Support Forum Sysop
    No emails please!
    http://www.cfc-it.de

  • Transfer of large files to and from Egnyte failing...

    One of my clients uses Egnyte for file management. For a typical job I will usually be required to download 5GB of files and upload 1.5GB.
    However, when at home, transfer of large files to and from Egnyte will often fail. (On download, Chrome gives the error message: "connection failed". Uploading, Egnyte's error message is: "HTTP error").
    I have three machines at home. Two Macs (running Yosemite and Lion) and a PC running Windows 7. I've had no luck with any of them on any browser but when using other people's broadband I have no problem at all (using my MacBook).
    I have no firewalls running. Yes, I've turned everything on-and-off-again. So that leaves me to think that the problem lies with my BT Homehub 4 router. But why would my router be botching the transfer of large files? I've switched the router's firewall off, tried adding my Mac to DMZ (whatever that is) but that seems to be the most I can do. Ethernet is no different to wireless.
    I've not noticed this porblem when using other file transfer sites (like WeTransfer).
    What's going on?
    Please help!

    From my own experience (I admin a few gaming servers and often get disconnections from them in the middle of monitoring operations) and based on other users experiences here on the forums I suspect BT have been having some core infrastructure issues which can lead to A) intermittent packet loss B) extended packet delay - both of which can cause servers to assume a 'failure' and disconnect or suspend upload/download.
    I dont know what package you are on from BT (I'm Infinity 2) << and as its Hogmany im the one that drawn the short straw to keep cheaters off out servers << so I'm a bit intoxicated and may not make total sense atm.
    https://community.bt.com/t5/BT-Infinity-Speed-Connection/BT-Infinity-issues-for-the-last-few-days/td...
    ^^ this thread illustrates issues that people have been having over the last few weeks.
    This probably wont help - but it might make you aware that you arent alone in ONGOING issues.
    Happy New Year !

  • How to read and process oData service's ATOM XML output?

    Hi,
    I have a RFC created in system A which will call oData services of system B. I am calling this in a report using class methods,  cl_http_client. I will get the response in a ATOM XML format. Is there a standard way to read and process such ATOM XML in abap report without using further addons?
    Regards,
    Vinay

    You need to pass your filename String as a parameter to your functionality. It depends how you're currently set up though. We can't really see the top of your call so it's difficult to determine what you are calling and we don't really know from where you're calling either.
    If you're running standalone, then on launch of the application, you can feed in a file name as an argument that you can read in in String args[] in the main function and pass down to your XML splitter.
    If you're a method in a class that's part of a bigger pile, you can feed the file name as a String to the method from wherever you call from if it makes sense architecturally.
    You might also want to pass down a File object if that makes sense in your current code (i.e. if you're using your file for other purposes prior to the split, to avoid recreating closing/opening for no reason).
    Depends what you're trying to do. If I put together a piece like this, I would probably create an <yourcurrentrootpackage>.xml.splitter package.
    Also, on a side note, you're problem isn't really reading and writing XML in java, but seems more to be making your functionality generic so that any XML file can be split with your code.
    Regards
    JFM

  • File read and write operations

    how do use file read and write operations?
    can anyone give simple program?

    http://www.tutorialspoint.com/cplusplus/cpp_files_streams.htm
    http://www.cplusplus.com/doc/tutorial/files/
    check this
    and with mfc
    http://www.functionx.com/visualc/fileprocessing/serialization.htm
    https://msdn.microsoft.com/en-us/library/6337eske.aspx
    http://www.informit.com/library/content.aspx?b=Visual_C_PlusPlus&seqNum=90

  • IPhone4 how through the USB mobile phone file reading and writing?

    iPhone4 how through the USB mobile phone file reading and writing?

    No idea what you are asking.
    Please explain

  • File reader and comparing words

    hi, im currently trying to create a project and i need a little help to get it started!
    What i need to do is create an application which can read a text file. I also need to have a list of words already in the application, and if a word in the list is also in the text file, id like this to be printed in the terminal.
    can anybody help me out with how to start this? ive been looking for relevant posts, and there seems to be alot of things on file reading and comparing words, but they r pretty difficult to follow and not really the same as what im looking for.
    thanks for your help!
    Torre

    You can place a for loop within a for loop to to compare every word in the temp array with every word in the other array.
    try {
    String[] array = new String[] {"wow", "cool"};
    BufferedReader in = new BufferedReader(new FileReader("text.txt"));
    String str;
    while ((str = in.readLine()) != null) {
    String [] temp = null;
    temp = str.split(" ");
    for (int i = 0 ; i < temp.length ; i++) {
      for(int j = 0; j < array.length; j++) {
        if (array[j].equals(temp)) {
    System.out.println(temp);
    } catch (IOException e) {
    } finally {
    //Always close your resources in your finally block... good coding practice.
    in.close();

  • Read and Process Microsoft Word's Form

    Hi,
    Is there any WDP API or open source Java API to read and process Ms Word's Form, such as input field, pull down list etc.? I know of POI's HWPF, but it can only do some basic Word processing.
    Thanks.
    - julius

    JAWIN or NEWJAWIN!!!!
    newjawin consists of a java-com-codegenerator, which generates Java-Proxy-classes for COM-components like MS Word, Excel, ...
    1. download newjawin
    2,. start codegenerator (exe)
    3. choose COM-Component (f..e. winword.exe)
    4. press "generate"-Button
    5. Use Javaproxyclasses to access full MS Word
    Or you can use a commercial software like J-Integra that provides ready-to-use Java API to access Word. The benefit of using this tool  is that it exposes all the formatting tools Word provides like adding header, footer, page numbers, tables, etc. Also, if you require user interaction to manipulate the Word document, you can listen to Word events from your Java program and implement event handlers accordingly.

  • File Reader and BuffredReader

    Hi All,
    Is it that one buffred reader object works with one file reader.???
    I am writing a code for reading the text files with fixed format,
    So i used one file reader and operated two buffered reader object with the same file reader,
    but its not working.
    Please help

    Is it that one buffred reader object works with one
    file reader.???what?
    I am writing a code for reading the text files with
    fixed format,
    So i used one file reader and operated two buffered
    reader object with the same file reader,
    but its not working.what do you mean by "not working"?
    post your code so that people know what you are trying to do.

  • File from application server -Read and process and delete the file .

    Hi All,
             I writing a ZEE program which will read the file from application server(file will be in text delimat format) and moving the data to internal table and uploadind data base.The part which iam facing problem is -
    > I hve read the file from application server like below ,
    OPEN DATASET ld_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
    IF sy-subrc NE 0.
    EXIT.
    ELSE.
      DO.
        CLEAR: wa_string, wa_uploadtxt.
        READ DATASET ld_file INTO wa_string.
        IF sy-subrc NE 0.
          EXIT.
        ELSE.
          SPLIT wa_string AT con_tab INTO wa_uploadtxt-name1
                                          wa_uploadtxt-name2
                                          wa_uploadtxt-age.
    Spliting part i having problem .i need to seperate each field which is seperate from tab delimite and pass it into respective fields of internal table .
    Second if another file come to the appicaltion server , wether my first file will be there ? or should i need to delete the first file after redaing how should i proceed further ?.
    Regards
    SRIRAM...
    THANKS In ADVANCE

    HI,
    1.
    OPEN DATASET ld_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
    IF sy-subrc NE 0.
    EXIT.
    ELSE.
    DO.
    CLEAR: wa_string, wa_uploadtxt.
    READ DATASET ld_file INTO wa_string.
    IF sy-subrc NE 0.
    EXIT.
    ELSE.
    SPLIT wa_string AT cl_abap_char_utilities=>horizontal_tab INTO wa_uploadtxt-name1
    wa_uploadtxt-name2
    wa_uploadtxt-age.
    APPEND  wa_uploadtxt TO itab.
    ENDDO.
    CLOSE DATASET ld_file.
    ENDIF.
    2. If the another file get's generated in the application server with same file name then the data in old file get's over written with the new data. You are not required to delete the file.

  • When will Elements 13 be updated to read and process Lummix FZ1000 RAW files?

    Elements 13 will not read and proceess RAW fileds from Lummix FZ 1000.  How soon can we expect an update to address this?

    Since pse 13 ships with camera raw 8.6 and that camera only requires camera raw 8.5, pse 13 should already open the Lumix DMC-FZ1000 files without any needed updates.
    Which operating system are you using?
    If in the pse 13 editor you go to Help>About Plug-in and click on the words Camera Raw, what version does it say?
    (on a mac Photoshop Elements Editor>About Plug-in)
    Have you tried Help>Updates from within the pse 13 editor to get camera raw 8.7.1 and pse 13.1 update?

  • Large file size and fuzzy type

    I'm new at using FCE2 and composed my first short 4 minute video. It includes a still image, a PSD layered file which I discovered was quite handy, a few clips and type using the type generator. I exported it to QuickTime movie. Three observations: I was floored by the nearly 1GB size, fuzzy type, and the file on the hard drive says it's a FCE movie inclusive of the little FCE icon by the file name. I was expecting to see a QuickTime icon with file name and the type of file is a QuickTime movie. Is all this normal? I'm very disappointed in the fuzzy type. Oh, also, the still image became blurry. Why?? Just so you know, the still image was a special file 640x480 with a pixel aspect ratio of D4/D16 Anamorphic
    I saved the project under a new name and redid it taking out the Photoshop image, removing it from the bin also, and the new movie exported even larger, over a gig in size. Huh??

    Thanks for the reply Tom. After I posted the first time I went to the Finder, Get Info, and I saw that I could change the 'Opens with' to QuickTime and therefore it became a QuickTime file. About that Anamorphic business I read a 'how to' on dealing with images before bringing them into video. The tip says in the 'New' file dialogue box to choose 640x480 size and in the pull down menu at the bottom where you can choose the 'Pixel Aspect Ratio' it was suggest to use that Anamorphic setting. I did it but it certainly didn't look right but I went with it.
    Again after I posted I looked at the Format of one of the clips and saw the size to be 720x480, Compressor is DV/DVCPRO-NTSC, Pixel Aspect is NTSC-CCIR 601, Anamorphic field is blank. I'm running Photoshop CS2. So I went back there and created a new blank file to use a template for dealing with stills but this time I used the Preset pull down menu and chose NTSC DV 720x480 with guides and the Pixel Aspect Ratio automatically loaded D1/DV NTSC (0.9), clicked Ok and viola, the blank file looks exactly like the Canvas in FCE. I haven't played with a still with this new setting but I will try it on the little project I'm working on.
    As for viewing it, I am looking at it on my Mac flat screen. I went into QuickTime Preferences and checked the box for high quality, thank you. Thanks for reassuring me on the file size.
    I also don't know what "D4/D16 Anamorphic" means.
    I don't understand the fuzzy type. I'm aware these are 72 ppi files and video is not resolution dependent but rather pixel dependent. Computer monitors display at 72 ppi, televisions are higher. I have yet to complete the process of burning a DVD and playing it back on a TV. Maybe that's where I'll see the type showing up sharper.
    At any rate, just dealing with this itty bitty project tells me I have a lot to learn about video, never mind learning about how to use FCE as well.

  • File Read and Write using File Adapter in Bpel

    In Bpel Process i am using File Adapter ( Schema is Opaque) for read and write the file contents. i am able do successful deployment and read, write function in first time deployment, after that again i tired to run the application, its not going to write the content of file, its only writing the file with out data's or content in that file.
    Please help me...
    Saravanan

    Hi Eric
    In my domain.log file having the following details. In this file im unable to find out what the exact problem. Please look at this and help me.
    <2008-01-22 18:25:42,024> <INFO> <default.collaxa.cube.compiler> validating "C:\product\10.1.3.1\OracleAS_1\bpel\domains\default\tmp\.bpel_BPELProcess2_1.1_298e83988d77b6640c33dfeec11ed31b.tmp\BPELProcess2.bpel" ...
    <2008-01-22 18:25:49,850> <INFO> <default.collaxa.cube.engine.deployment> <CubeProcessFactory::generateProcessClass>
    Process "BPELProcess2" (revision "1.1") successfully compiled.
    <2008-01-22 18:25:49,914> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> Loading JCAActivationAgent for {portType=Read_ptt}
    <2008-01-22 18:25:49,914> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> JCAActivationAgent::load - Locating Adapter Framework instance: OraBPEL
    <2008-01-22 18:25:49,930> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> JCAActivationAgent::load - Done loading JCAActivationAgent for processId='bpel://localhost/default/BPELProcess2~1.1/
    <2008-01-22 18:25:49,930> <INFO> <default.collaxa.cube.engine.deployment> Process "BPELProcess2" (revision "1.1") successfully loaded.
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> JCAActivationAgent::uninit Shutting down the JCA activation agent, processId='bpel://localhost/default/BPELProcess2~1.0/', activation properties={portType=Read_ptt}
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> Adapter Framework instance: OraBPEL - performing endpointDeactivation for portType=Read_ptt, operation=Read
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.ws> <File Adapter::Outbound> Endpoint De-activation called in adapter for endpoint : D:\MAXIMUS_Project_Softwares\jdevstudiobase10132\jdev\mywork\MyLabs\BPELProcess2\in
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> JCAActivationAgent::init - Initializing the JCA activation agent, processId='bpel://localhost/default/BPELProcess2~1.1/
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> JCAActivationAgent::initiateInboundJcaEndpoint - Creating and initializing inbound JCA endpoint for:
    process='bpel://localhost/default/BPELProcess2~1.1/'
    domain='default'
    WSDL location='rd.wsdl'
    portType='Read_ptt'
    operation='Read'
    activation properties={portType=Read_ptt}
    <2008-01-22 18:26:02,698> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> Adapter Framework instance: OraBPEL - endpointActivation for portType=Read_ptt, operation=Read
    <2008-01-22 18:26:02,730> <INFO> <default.collaxa.cube.activation> <File Adapter::Inbound> Endpoint Activation called in File Adapter for endpoint: D:\MAXIMUS_Project_Softwares\jdevstudiobase10132\jdev\mywork\MyLabs\BPELProcess2\in
    <2008-01-22 18:26:02,730> <INFO> <default.collaxa.cube.activation> <AdapterFramework::Inbound> Adapter Framework instance: OraBPEL - successfully completed endpointActivation for portType=Read_ptt, operation=Read
    <2008-01-22 18:26:02,890> <WARN> <default.collaxa.cube.activation> <File Adapter::Inbound> PollWork::run exiting, Worker thread will die
    <2008-01-22 18:26:04,171> <INFO> <default.collaxa.cube.ws> <File Adapter::Outbound> Managed Connection Created
    <2008-01-22 18:26:04,171> <INFO> <default.collaxa.cube.ws> <File Adapter::Outbound> Connection Created
    <2008-01-22 18:26:04,171> <INFO> <default.collaxa.cube.ws> <File Adapter::Outbound> FileInteraction Created

Maybe you are looking for