Remote huge file load problem

---Following is the code to load a huge image file (50-100M)
InputStream in = new FileInputStream(readFile);     
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];     
while((i = in.read(buffer)) != -1){
out.write(buffer, 0, i);
in.close();     
byte[] result = out.toByteArray();
---It will crash (or hang) on toByteArray() when file size exceed 18mb.
Is there any way to load large files?
Thanks

1)I am new to the java forum. May I know what is
documentation for java BUT?http://java.sun.com/docs/index.html
2)Also may I know where I can find topics about
processing the file without loading the whole of it
into memory?
3) Actuall we are loading this file into bytearray in
java at client side(PC or MAC) and use rpc to
transfer to server(UNIX) - the C code in server side
will take this byte buffer and use fwrite to create
this huge file. Is there a better way than this?a) Break the file into smaller chunks and send the chunks together with a chunk identifer to the server. The server will then re-construct the file from the chunks.
b) Install an ftp server on the server machine and user FTP to sent the file.

Similar Messages

  • Flat-File Loading problem

    Hi Friends,
    I am struggling with flat-file loading problem. I am trying to load a .csv file into data target. I took all pre-cautions while loading data. I look into preview and simulate the data. Everything is ok, but when i schedule the data, i found 0 records in the monitor. The following is the STATUS message of the above problem:
       No data available
    Diagnosis
    The data request was a full update.
    In this case, the corresponding table in the source system does not
    contain any data.
    System response
    Info IDoc received with status 8.
    Procedure
    Check the data basis in the source system.
             Can anybody help me what is the problem and procdure to resolve it?
    Regards,
    Mahesh

    Hi Eugene,
    Thanks for the quick reply. The following screen-shot tells you the messages of detail tab;
    OVER ALL STATUS MISSING WITH MESSAGES OR WARNINGS
    REQUEST: MISSING MESSAGES
    EXTRACTION
    EVERYTHING IS OK
    DATA REQUEST RECEIVED
    NO DATA AVAILABLE DATA ELECTION ENDED.
    PROCESSING
    NO DATA
               The above message was shown in details tab. Pls guide me to locate the problem.
    Regards,
    Mahesh

  • Unicode huge file conversion problem

    I use the command in Unix "<b>sapiconv -f 4110 -t 8600 -o <source file name> <destination file name></b>" calling by ABAP command  "CALL 'SYSTEM' ID 'COMMAND'  "   to convert non-unicode source file to unicode destination file in code page 8600 (Thai language). For the small size source file, It has no problem, but for the huge size source file (27MB - the file was extracted and created by OPEN DATASET from asset master data about 95000 records), it has problem.  There are only about 1300 Records (about 200k) in destination file and <b>the remain records are missing!?!?</b> And every time I run the program, It still has certain result.
    What can I do for this case? or the command 'sapiconv' has the size limitation for unicode file conversion? or which additional parameter should I use?
    Thank you in advance.
    David

    Hello,
    You  can use this function module
    CALL FUNCTION 'ARCHIVOBJECT_GET_CLIENT'
              EXPORTING
                ARCHIV_DOC_ID               =   V_ARCDOCID
                ARCHIV_ID                        =   V_ARCHIVID
                PATH                                 =   V_PATH
    *            DOC_TYPE                       =   'TXT '
    *         IMPORTING
    *            REPATH                           =
    *            COUNT                             =
    *            FILE_NAMES                    =
             EXCEPTIONS
               ERROR_ARCHIV                = 1
               ERROR_COMMUNICATIONTABLE       = 2
               ERROR_KERNEL                = 3
               ERROR_HTTP                     = 4
               OTHERS                             = 5.       
    I think it will help to you,
    Anil.

  • No Data Available (Flat File Loading Problem))

    Hi Friends,
    I loaded the data from FlatFile(Application Server) I am able to see in the Preview
    but the data is not laoding into the Cube. Request status is green but with zero records.
    Following is the message
    No data available
    Diagnosis
    The data request was a full update.
    In this case, the corresponding table in the source system does not
    contain any data.
    System response
    Info IDoc received with status 8.
    Procedure
    Check the data basis in the source system
    Can anybody help
    Regards,
    CV

    Hi chakri,
    first u have cto check  ur flat file in the Presentation server OR Application server
    2. Ur flat file fields sequence same In the Trasfer structure .
    U should maintain the same sequence in Transferstructure  what u have maintained in the flat file.
    3. when ur loading date from flatfile source System u have to use FULLUPDATE  mode .
    check the above .
    Thanks,
    kiran

  • Flat File Load Problem

    Hi All,
    Has anyone come across where flat file uploads cuts out the last 1 or 2 columns from the upload.
    This happens in the preview and in the actual load.
    The data separators(,) and the Escape sign (") are OK
    This are sample lines below:
    "60    ","105906      ","A","253215      ","2005052700000000","516007      ",".TonyA        ","1.17                 ","0.20         ","2005050600000000","7     ","1   "
    "102   ","42577       ","A","560194      ","2005070400000000","825293      ","A.HARCOURT     ","23.95                ","4.19         ","2005062300000000","8     ","56  "
    "149   ","103179      ","A","254010      ","2005020900000000","208864      ","Paul Glew      ","2.16                 ","0.38         ","2004121300000000","9     ","1   "
    It only loads up to the column with 0.20,4.19,0.38 and cuts it out from the date onwards (3 last columns).
    Anybody seen this???

    Hi Guys,
    I really don't have much of a leeway in manipulating the file for upload. It is generated by a legacy system and the upload will be automated.
    However, I am using a subset of the data for testing which I have been changing to troubleshoot the errors. What confuses me in this situation is that I have similar data structures with the same spaces, date formats etc. that is working.  To answer your questions:
    In the preview the data is displayed OK and first date in the line is displayed without the suffix zeros up to the last 3 columns which is blank completely, the following column shows 4 zeros and the last column is blank as well.
    I have tested the data by physically removing the last 3 columns from the file. When I preview that it also gives me zeros in the 2nd last column even though there is no data to be read.
    I have also tested it by removing unwanted spaces and it did not make a difference.
    I have also tested via a logical path as well as pc file.
    I am thinking other ways to test this......any ideas still......
    K

  • External SWF file load problem

    I'm loading external SWF file which has motion tween.After
    the loading progress bar finished to load than calls SWF file.If I
    test the movie locally works fine but when simulate the download it
    seems it jumps end of the timeline.Is anyone could tell me why is
    this??
    Thanks

    k, never mind..... finally figured it all out.

  • 3D walkthrough - W3D file load problem

    Hi,
    I found one interesting source file:
    here
    The file name is:
    wallCollision_V-Ray-FPS_mouseMove.dir
    I downloaded and started to study this file, everything works
    good, except that I am not able to separate the W3D file to act as
    external file, because currently it is attached inside this
    director file, i tried to delete the current w3d scene and
    re-import the same different w3d scene but it doesn't work,
    Can anyone help me with this?

    This problem can be rectified by using '/' before the filename.
    "/filename.m3g".
    Regards
    Dhaval

  • Nib file loading problem

    Newbie working from a Utility Application template provided by Xcode--
    ResultsViewController *viewController = [[ResultsViewController alloc] initWithNibName:@"ResultsView" bundle:nil];
    self.resultsViewController = viewController;
    [viewController release];
    UIView *foo = self.resultsViewController.view; // workaround
    self.infoButton = self.resultsViewController.iButton;
    iButton is an IBOutlet property in ResultsViewController. The associated button is a subview of the view contained in the nib file owned by ResultsViewController.
    If the workaround statement is removed, iButton == 0 when the last statement is reached. Why? Or, what should I be doing in place of the workaround to initialize the IBOutlet property?

    The point of IBOutlet is to hook them up in Interface Builder. Cntrl-click on the view controller and drag to the button. Select it in the list.
    Now when you load the xib it will be filled in.
    Note that the view won't be loaded until it is accessed, that's why you're workaround is required or an access to the view if it's not set to autoload.

  • File load problem

    Hi,
    I tried a few things but no success yet.
    I have class Item -> productcode
    class StockItem extends Item -> pr, qty
    class Book1 extends StockItem -> auth, pub, yr, cat
    class Reference1 extends Book1 -> ref
    I think I am getting something wrong with the code below.
    The complete code is at:
    http://www.multiline.com.au/~wje/java/question.html
    This is the error message I get:
    D:\Assignment2.java:992: cannot resolve symbol
    symbol : constructor Reference1 (java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String)
    location: class Reference1
    item = new Reference1(code,pr,qty,auth,pub,yr,cat,ref);
    ^
    1 error
    public void load(String fileName) {
        try {
          java.util.List lineBuffers = readBookStoreFile(fileName);
          this.fileName = fileName;
          Map map = new HashMap();
          Item item;
          items = new Vector();
          for (int i = 0; i < lineBuffers.size(); i++) {     
            String buffer = (String)lineBuffers.get(i);
            StringTokenizer tokens = new StringTokenizer(buffer, ",");
            if (tokens.countTokens() != 8)
              throw new RuntimeException("Line with incorrect token count");
            String code = tokens.nextToken();
            String pr = tokens.nextToken();
            String qty = tokens.nextToken(); 
              String auth = tokens.nextToken();       
            String pub = tokens.nextToken();
            String yr = tokens.nextToken();
            String cat = tokens.nextToken();
            String ref = tokens.nextToken();        
            item = (Reference1)map.get(code);
            if (item == null) {
              item = new Reference1(code,pr,qty,auth,pub,yr,cat,ref);
              items.addElement(item);
            map.put(code, item);
        catch (Exception e) {
          e.printStackTrace();
          System.exit(1);
      }Thank you in advance for your help
    Andonny

    The class Reference1 does not have a Constructor that accepts 8 strings as argument.

  • Splitting a huge file  in chunks.

    We have a requirement to split file containing 23 million records in chunks of  5000 records.I have used "Recordsets per message" option but it splits just 2.25 lakh records and from that records there will not be any idocs to be posted.Does anyone has idea on this?

    Hi Sravya
    Have a look at these threads,
    Processing huge file loads through XI
    Handling Large Files in XI
    cheers
    Sameer

  • File Adapter-Problem Reading Huge Files

    Hi,
    Here is the issue that i am facing
    When reading huge file(csv file upto 6MB-8MB) the communication channel configured as File Adapter with a polling interval of 7 min(420 sec) is inconsistent in reading the complete file.Sometimes it reads the the complete file of 6 MB and sometimes it reads a part of the file say 3MB/6MB.Can this inconsistent behaviour be resolved.??
    Your suggestions highly appreciated.
    Regards
    Pradeep

    Hi Pradeep !
    8mb is not a huge file for XI, I think it is a small one. Maybe your problem is not the size..please check if XI is not starting to read the file before it is completely written to the source folder. If you are creating that csv file from another application directly to the poll source directory of the XI scenario specified in the file adapter, and your poll interval is small, XI could start reading the file while you are still writing it. If this is the case, try to put the file with a different extension or filename than the specified in file adapter comm channel and when the file is completely written, rename it to its final filename and check if you are still having that misbehavior.
    You can write the file to a temp directory and the move it to the XI directory once finished.
    Regards,
    Matias.

  • CSS Image loading problem by dynamically loading from fxml file?

    h1. Introduction
    The application I am developing loads an FXML file from a Controller. The FXML uses CSS and images for buttons are set in the CSS.
    CSS directory structure
    <package>.fxml
    Images
    <package>.fxml.resources.<subdir>
    Example CSS Code
    .buttonImage {
        -fx-background-image: url("resources/subdir/image.png");
    }Example loading fxml from controller code
             URL location = getClass().getResource("/package/fxml/UI.fxml");
             FXMLLoader fxmlLoader = new FXMLLoader(location);
             fxmlLoader.setLocation(location);
             fxmlLoader.setBuilderFactory(new JavaFXBuilderFactory());
             Parent root = (Parent)fxmlLoader.load(location.openStream());
             Controller = (Controller) fxmlLoader.getController();
             newPane.getChildren().add(root);h1. Problem
    The fxml file does not load and causes the following error:
    javafx.fxml.LoadException: Page language not specified.Note, the fxml file loaded correctly before images were added.
    Any ideas of what might be going wrong?
    h1. Attempted
    I have attempted the following: tried changing the url of the image as an absolute path.

    No Problem: Note, this is the actual FXML, my previous examples were simplifications of package names and class names in order to directly illustrate the problem. Note, I am using SceneBuilder to develop the FXML file.
    <?xml version="1.0" encoding="UTF-8"?>
    <?import java.lang.*?>
    <?import java.net.*?>
    <?import java.util.*?>
    <?import javafx.collections.*?>
    <?import javafx.scene.*?>
    <?import javafx.scene.control.*?>
    <?import javafx.scene.layout.*?>
    <?import javafx.scene.text.*?>
    <AnchorPane id="AnchorPane" prefHeight="396.0000999999975" prefWidth="350.0" styleClass="mainFxmlClass" xmlns:fx="http://javafx.com/fxml" fx:controller="com.monkygames.kbmaster.controller.ProfileUIController">
      <children>
        <Label layoutX="14.0" layoutY="14.0" text="Profiles">
          <font>
            <Font size="18.0" />
          </font>
        </Label>
        <GridPane layoutX="15.0" layoutY="44.0" prefHeight="99.0" prefWidth="335.0">
          <children>
            <Label text="Type: " GridPane.columnIndex="0" GridPane.rowIndex="0" />
            <ComboBox fx:id="typeCB" prefWidth="249.0" GridPane.columnIndex="1" GridPane.rowIndex="0">
              <items>
                <FXCollections fx:factory="observableArrayList">
                  <String fx:value="Item 1" />
                  <String fx:value="Item 2" />
                  <String fx:value="Item 3" />
                </FXCollections>
              </items>
            </ComboBox>
            <Label text="Program: " GridPane.columnIndex="0" GridPane.rowIndex="1" />
            <ComboBox fx:id="programCB" prefWidth="249.0" GridPane.columnIndex="1" GridPane.rowIndex="1">
              <items>
                <FXCollections fx:factory="observableArrayList">
                  <String fx:value="Item 1" />
                  <String fx:value="Item 2" />
                  <String fx:value="Item 3" />
                </FXCollections>
              </items>
            </ComboBox>
            <Label text="Profile: " GridPane.columnIndex="0" GridPane.rowIndex="2" />
            <ComboBox prefWidth="249.0" GridPane.columnIndex="1" GridPane.rowIndex="2">
              <items>
                <FXCollections fx:factory="observableArrayList">
                  <String fx:value="Item 1" />
                  <String fx:value="Item 2" />
                  <String fx:value="Item 3" />
                </FXCollections>
              </items>
            </ComboBox>
          </children>
          <columnConstraints>
            <ColumnConstraints fillWidth="false" halignment="RIGHT" hgrow="NEVER" maxWidth="-Infinity" minWidth="10.0" prefWidth="80.0" />
            <ColumnConstraints hgrow="SOMETIMES" minWidth="10.0" prefWidth="100.0" />
          </columnConstraints>
          <rowConstraints>
            <RowConstraints minHeight="10.0" prefHeight="30.0" vgrow="SOMETIMES" />
            <RowConstraints minHeight="10.0" prefHeight="30.0" vgrow="SOMETIMES" />
            <RowConstraints minHeight="10.0" prefHeight="30.0" vgrow="SOMETIMES" />
          </rowConstraints>
        </GridPane>
        <HBox layoutX="14.0" layoutY="150.0" prefHeight="159.0" prefWidth="335.0">
          <children>
            <Label text="Description: " />
            <TextArea prefHeight="172.0" prefWidth="247.0" wrapText="true" />
          </children>
        </HBox>
        <HBox alignment="CENTER" layoutX="11.0" layoutY="334.0" prefHeight="48.0" prefWidth="329.0">
          <children>
            <Button fx:id="newProfileB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="newProfile" text="New Profile" />
            <Button id="newProfileB" fx:id="cloneProfileB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="cloneProfile" text="Clone Profile" />
            <Button id="newProfileB" fx:id="importProfileB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="importProfile" text="Import Profile" />
            <Button id="newProfileB" fx:id="exportProfileB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="exportProfile" text="Export Profile" />
            <Button id="newProfileB" fx:id="printPDFB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="pdfProfile" text="Print to PDF" />
            <Button id="newProfileB" fx:id="deleteProfileB" contentDisplay="GRAPHIC_ONLY" minHeight="48.0" minWidth="48.0" mnemonicParsing="false" onAction="profileEventFired" prefHeight="48.0" prefWidth="48.0" styleClass="deleteProfile" text="Delete Profile" />
          </children>
        </HBox>
      </children>
      <stylesheets>
        <URL value="@master.css" />
        <URL value="@profile.css" />
      </stylesheets>
    </AnchorPane>Edited by: 960799 on Jan 4, 2013 8:58 PM
    Edited by: 960799 on Jan 4, 2013 8:59 PM

  • Sorting file content of a huge file and less RAM

    Hi,
    I want to sort the content of a big file. In a file each line ended by \n is one record. I want to sort the lines alphabetically.
    for eg.
    The file test.txt contains the following data
    pqr xyz \n
    abc xyz \n
    The sorted file should contain
    abc xyz \n
    pqr xyz \nand also the file size is very big around 20GB. and RAM is very less so it is not even possible to load such a huge file in memory.
    How to solve the above problem? Can anyone suggest me the good solution?
    Thanks

    Getting rid of the file in favor of a database sounds pretty good to me.Have you any clue how much time it will take to insert 20 GB into a relation database?
    I find szgy's solution sutaible for me
    but I think before merging (step 5) the data in
    separate files (compare set of two files) should be
    compared and should be written to a new file. Then
    these resultant files should be compared again tili
    it becomes a single file.you can merge all the files at once to using a heap, hence avoiding unnecessary file-writes.
    >
    The above step is required since I'm reading data
    from a file. I'll be taking the first few lines and
    sorting them and writing into new file. Here each
    file is sorted. but if you merge all the sorted files
    as it is the resultant file will not have sorted data
    since there was no comparision.
    Now I'll have to think upon sorting algorithms and
    merging of files after comparing data between two
    filesYou can email me at [email protected] to get a full working code that does all this for integers, including merging the results with help of a heap.
    For 10000000 (400 MB) it takes (in ms):
    other     read     write sort
    2579     16655     35407     37422
    Total is 80 sek, which is fairly ok for 400 MB of data I think (5 MB/sek) including reads and writes.
    Gil

  • Upload huge file size: "The page cannot be displayed" browser

    What are precautions to be take for uploading a huge file
    I have an upload file operation in the web application. The web server is OC4J.
    //UploadForm.jsp
    <FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data>
    <input type="file" name="fileName">
    </FORM>
    After I deploy the application to the web server, if I upload a small file size, it is fine.
    But if i upload huge data ,data is uploading but if any action button is clicked
    After 30 seconds, it has error "The page cannot be displayed" shown on the web browser.

    Hi All,
    I have a similar time out problem, it ll be great if some one provide a solution.
    The problem I'm facing is becasue of the time out setting in the proxy server which is 1.5 mins. The app server is sending Response properly but after some 2 mins or so because the processing is a bit complicated (3 API calls done intermittently for different validations).
    I cant request for change in proxy server time out setting. So I have to handle this with in my application.
    It will be greatful if anyone helps us out of this.
    Thanks
    Noufal

  • Huge file size reduced by unlocking

    I ended up with a 2.7GB file that consisted of 8 locked tracks that had been locked and unlocked lots of times. I knew that was about ten times the real size. The only way to get rid of whatever was causing the huge file size was to unlock the tracks and Save As Archive which reduced the size to 400MB.
    Why did this happen?
    Also, is there a detailed manual available that tells you all the tricks about GB? Things like: exactly what does locking a track do (Help is not very informative); and how do Save, Save As and Save As Archive differ in their treatment of purple, orange and other regions. I have had problems saving everything when I backup to an external drive, so I'd like to find out exactly how these Saves work.

    Christoph,
    Yes I just did that. I saved a copy of the song and then trashed the freeze files that should not have been there. The song played fine, no problem. THe file went from 1.6G to 176M.
    I have found this situation in a few other files also.
    These are my saving habits when I'm working on a song. I work for a while using "Save". Then after a while I start to save many major changes as "Save as Archive". I always end a session this way. Usually I'm not concerned about disc space. I use "Save as archive" all the time because I would rather have a file with everything I'm using contained within that file. Last year I accidently trashed 10,000 loops and I have since found that "save as archive" is a helpful backup when I'm acting stupid around the trash.
    I may be one of the few folks who uses "Save as archive " so often and somehow it seems that every once in a while a hiccup occurs and the freeze files do not erase when the tracks are unlocked.
    Since this discovery I have gone thru many other files and found this to be the case every now and then. Some of the files didnt hint at being over stuffed but had a few errant freeze files. Normally I would never go and check a files contents but that one file at 1.6G grabbed my attention
    Just thought I would post to see if anyone has this problem.

Maybe you are looking for

  • My images is exporting bigger then my art board? CS6

    Hi i have a artboart set to 241 × 283 but when i export my image it exports it to 506 × 590 so im confused on all of this please help? Thanks Cody

  • Java DC Build Error

    Hi all. I want to have an utility Java DC, I mean a DC with some Java classes having methods and constants to use in some others DC's. I have created the Java DC and add into the Java build path, some Webdynpro libraries, when i use those API's  the

  • Select customer name from a drop down Select List or be able to type it in

    Hi, Is there a way to allow my users to have an option to either select a customer name from a drop down Select List or be able to type it in... Thanks in advance

  • Physical to Virtual Copy of 11i EBS and 10gR2 DB instances for Upgrade Test

    We recently completed a P2V copy of our 11.5.10.2 EBS instance and its corresponding database. We then ran the adcfgclone.pl script on each tier to rename the SID and basically create a new instance. When I navigate to the newly assigned web address

  • Reason codes

    Hi , We have a contact center UCCE 8x, in which few of the Agents are migrated to Siebel CRM CTI Application from CAD. We are using Cisco CRM adapters for the CTI communications between CRM and ICM. So the call is maturing to the agent properly, but