Import very large csv files into SAP

Hi
We're not using PI, but have middleware called Trading Networks. Our design is fixed (not my decision) to not upload files into Application Server and import it from there. Our design dictates that we must write RFCs and Trading Networks will call the RFC per interface with very large file sent as table of strings. This takes 14 minutes to import into SAP plain Z-table from where we'll integrate. As a test we uploaded the file to Application Server and integrated into Z-table from there. This took 4 minutes. However our architect is not impressed that we'll stretch available Application Server to it's limits.
I want to propose that the large file be split in e.g. 4 parts at Trading Networks level and call 4 threads of the RFC which could reduce integration time to e.g. 3 minutes 30 seconds. Is there someone that has suggestions in this regard especially about a proposed, working, elegant solution for integrating large files with our current environment? This will form the foundation of our project.
Thank you and best regards,
Adrian

Zip compression can be tried. The RFC will receive zip stream which will be decompressed using CL_ABAP_ZIP.

Similar Messages

  • How do i import my local csv files into HANA database as temporary tables  using java program?

    I want to import my local csv file into database for further apply Join with other tables programmatic in JAVA

    Hi Vivek,
    Please go through the following blogs and video to resolve your issue.
    Loading large CSV files to SAP HANA
    HANA Academy - Importing Data using CTL Method - YouTube
    Hope it helps.
    Regards
    Kumar

  • Best data Structor for dealing with very large CSV files

    hi im writeing an object that stores data from a very large CSV file. The idea been that you initlize the object with the CSV file, then it has lots of methods to make manipulating and working with the CSV file simpler. Operations like copy colum, eliminate rows, perform some equations on all values in a certain colum, etc. Also a method for prining back to a file.
    however the CSV files will probly be in the 10mb range maby larger so simply loading into an array isn't posable. as it produces a outofmemory error.
    does anyone have a data structor they could recomend that can store the large amounts of data require and are easly writeable. i've currently been useing a randomaccessfile but it is aquard to write to as well as needing an external file which would need to been cleaned up after the object is removed (something very hard to guarentee occurs).
    any suggestions would be greatly apprechiated.
    Message was edited by:
    ninjarob

    How much internal storage ("RAM") is in the computer where your program should run? I think I have 640 Mb in mine, and I can't believe loading 10 Mb of data would be prohibitive, not even if the size doubles when the data comes into Java variables.
    If the data size turns out to be prohibitive of loading into memory, how about a relational database?
    Another thing you may want to consider is more object-oriented (in the sense of domain-oriented) analysis and design. If the data is concerned with real-life things (persons, projects, monsters, whatever), row and column operations may be fine for now, but future requirements could easily make you prefer something else (for example, a requirement to sort projects by budget or monsters by proximity to the hero).

  • Script to import contacts from CSV file into Address Book

    Hi,
    I am wondering if it would be possible to import contacts from a CSV file into a group in Address Book using applescript, and if so, if anyone is aware of a script that does this or something similar.
    I have a CSV file with three columns 'First Name', 'Last Name' and 'Email Address'.  I would like these to correspond with  'First', 'Last' and (Work) 'Email' in Address Book.
    Ideally the script would also check to see if entries already exist for the people being imported, but this is less crucial.
    Thanks in advance for any assistance,
    Nick

    Hi,
    Test this on a very small csv file (four or five contacts) to see if it works for you. It will not de-duplicate. You don't give the name of the group, so you'll need to replace Name of group in the second line with your group name, enclosed in double quotes.
    --begin script
    set the_people to read (choose file with prompt "Choose your CSV file")
    set the_group to "Name of group"
    set old_tids to AppleScript's text item delimiters
    set AppleScript's text item delimiters to ","
    set par_count to (count paragraphs in the_people)
    repeat with x from 1 to par_count
              set next_par to paragraph x of the_people
              set first_name to text item 1 of next_par
              set last_name to text item 2 of next_par
              set e_mail to text item 3 of next_par
              tell application "Address Book"
                        set nu_person to make new person with properties {first name:first_name, last name:last_name}
      make new email at end of emails of nu_person with properties {label:"Work", value:e_mail}
                        add nu_person to group the_group
      save
              end tell
    end repeat
    set AppleScript's text item delimiters to old_tids
    --end script
    Watch out for bad line breaks in the lines beginning "set nu_person..." and "make new email..."

  • Import and view CSV Files into OTM

    Hi All,
    Can you please help me in importing CSV files into Oracle Test Manager and to view the imported files in OTM.

    Hi,
    I have just googled and find the link :)
    http://oracleats.com/blog/?p=785

  • How do I split a Large CSV file into Multiple CSV's Using Powershell

    I am a novice at powershell but this looks to be the best tool to do this task. have a csv file that looks like this:
    Date,Policy,Application
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    Is it possible to split this CSV into multiple CSV's based on "Application".
    Lets say the output might look like:
    None.csv
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    AppBiz.csv
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    PeopleBiz.csv
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    Any help would be greatly appreciated

    I think this might be what you want:
    Import-Csv applications.csv |
    Group Application |
    foreach {
    $_.Group | Export-Csv "$($_.Name).csv" -NoTypeInformation
    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    Very nice! 4x faster..
    I doubt the OP will get what you just did there..
    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________
    Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

  • How can I import an existing csv. file into Outlook on a Mac with OS10.7

    Tried to import a csv. file on my Mac (OS 10.7.2) into Microsoft Outlook 2011 (for Mac) on the same computer, ended up with a useless Excel page - it opened the csv file in Excel with only a few contacts and uselessly formatted, at that!. Also, tried to save the csv file as a text file, then importing it into Word 2011 and Excel, with the same results. Anyone able to help? Would like to have the contained address/contact info look concise and neat, without duplicates or garbled/missing information.
    [email protected]

    DPS can use H.264 files, but I have no idea about the details since I've never done any of it myself. Ask in the ID/ DPS forums.
    Mylenium

  • I wish to import a Bank csv file into my numbers spreadsheet but all of it goes into one column which I can't separate

    I wish to import a csv file from my Bank into a numbers spreadsheet but I can't expand it from the one column it's in . Can anybody help  Thankyou

    This Automator Service (Dropbox download) should do the job, eliminating the need for a trip to TextEdit, etc.
    The one-time installation is simple: just doubleclick the .workflow package. You may have to click "Download Anyway" in System Preferences > Privacy & Security.
    To use, in Finder select the csv file, rightclick, and in Services in the contextual menu select CSV to Tabs on Clipboard. Choose the delimiter when prompted (usually it's a comma but there are other choices). When the "ready to paste" notification appears a few seconds later, click once in a Numbers cell and command-v to paste the data. (If needed Numbers will automatically expand the table to accommodate the data.)
    To remove or rename the Service, in Finder hold down the option key and in the menu Go > Library > Services.
    SG

  • How to import data from CSV file into a table by using oracle forms

    Hi,
    I have a CSV file and i want to insert in oracle database in a table by using a button in oracle forms.
    the user can select CSV file by using open dialog .
    can any one help me to find method to make import and select file from the client machine ?
    thx.

    1. create table blob
    CREATE TABLE IB_LOVE
      DOC          BLOB,
      CONTRACT_NO  VARCHAR2(20 BYTE)                NOT NULL
    )2. use the code below to insert:
           INSERT INTO ordmgmt.ib_love
                       (contract_no, doc
                VALUES (:control.contract_no_input, NULL
           lb$result :=
             webutil_file_transfer.client_to_db (:b2.file_name, v_file_blob_name, v_col_blob_name,
                                                 'CONTRACT_NO = ' || :control.contract_no_input);
           :SYSTEM.message_level := 25;
           COMMIT;
           :SYSTEM.message_level := 0;3. use the code below to download
         if :control.CONTRACT_NO_INPUT is not null then
              vboolean :=   webutil_file_transfer.DB_To_Client_With_Progress(
                               vfilename,  --filename                       
                              'IB_LOVE', ---table of Blob item                       
                              'DOC',  --Blob column name                       
                              'CONTRACT_NO = ' || :CONTROL.CONTRACT_NO_INPUT, ---where clause to retrieve the record                       
                              'Downloading from Database', --Progress Bar title                       
                              'Wait to Complete'); --Progress bar subtitle  client_host('cmd /c start '||vfilename);
              client_host('cmd /c start '||vfilename);                 
         else
              errmsg('Please choose contract no');
         end if;4. use the code below to open file dialog
    x:= WEBUTIL_FILE.FILE_OPEN_DIALOG ( 'c:\', '*.gif', '|*.gif|*.gif|', 'My Open Window' ) ;
    :b2.FILE_NAME:=X;

  • Trying to import from different csv files into multiple JTables

    This might seem like something that isn't commonly done, but I'll see if anyone has done this before anyways.
    I currently have a JTable (tab #1) and it successfully reads in the data from a CSV file to populate the cells. Now, I am looking into making 3 more JTables. I can tab between the four different tables, but I can't seem to figure out how to get the data to go to the correct JTable. Tab #1 = table, Tab #2 = table2, Tab #3 = table3, Tab #4 = table4.
    Thanks.
            try
                FileInputStream fileInput = new FileInputStream ("upcoming.csv");
                BufferedReader InputCSV = new BufferedReader (new InputStreamReader (fileInput));
                line = InputCSV.readLine(); //start reading into the records.
                    //Create data for the table.
                    //the row variable = starting row
                int row = 0;
                while(line != null)
                    tmp = line.split(",");
                    for (int col = 0; col < columnNames.length; ++col)
                        data[row][col] = tmp[col]; //store cells' data to the 2-dimensional array.
                    row++;
                    line = InputCSV.readLine();
                }//end of WHILE-statement.
            } //END of TRY-statement.
            catch(Exception e)
                System.out.println(e);
            }

    I have been successful in my quest to get 4 different datasets into 4 different JTables. The previous posters suggestion to use the Table model led me to this page:
    http://java.sun.com/j2se/1.5.0/docs/api/javax/swing/table/TableModel.html
            TableModel myData = new MyTableModel();
            JTable table = new JTable(myData);I had some very similar code in my program. I use the DefaultTableModel and the SortFilterModel before having 4 different JTables.
            DefaultTableModel model = new DefaultTableModel(data, columnNames);
            DefaultTableModel model2 = new DefaultTableModel(data2, columnNames);
            DefaultTableModel model3 = new DefaultTableModel(data3, columnNames);
            DefaultTableModel model4 = new DefaultTableModel(data4, columnNames);I use the code in the first post 4 times, and only change the filename and the data#.
    I also have some code that counts the number of lines that will be needed.
    Thanks.

  • HTMLDB_tools, processing a very large csv file in a collection

    Hi Guys,
    I'm new to APEX and trying to load approx 1,000,000 rows into a table from a csv file, does anyone have a better way of doing it than htmldb_tools.parse_file (which is very slow)
    Cheers

    It's not Apex, but you could use SQL*Loader. It's really very fast!
    greets, Dik

  • Read very very large excel file into 2d array (strings or varient)

    Hi all,
    Long time user, first time poster on these boards.
    Looking at the copius amounts of great info related to reading Excel data from .xls files into labview, i've found that every one i've found from various people use the ActiveX method (WorkSheet.Range) which two strings are passed, namely excel's LetterNumber format to specify start and end.
    However, this function does not work when trying to query huge amounts of information. The error returned is "Type Mismatch, -2147352571" I have a very large excel sheet i need to read data from and then close the excel file (original file remains unchanged). However this file is gigantic (don't ask me, I didn't make it and I can't convince them to use something more appropriate) with over 165 columns at 1000 rows of data.I can read a large number of columns, but only a handful of rows, or vice versa.
    Aside from creating a loop to open and close the excel file over and
    over reading pieces of it at a time, is there a better way to read more
    data using ActiveX? Attached is code uploaded by others (with very minor modification) as an example.
    Thanks,
    Attachments:
    Excel Get Data Specified Field (1-46col).vi ‏23 KB

    Hi Maddox731,
    I've only had a very quick glance through your thread, and I must admit I haven't really thought it through properly yet. Sounds like you've come up with your own solution anyway. That said I thought I'd take a bit of a scatter gun approach and attach some stuff for you regradless. Please forgive my bluntness.
    You'll find my ActiveX Excel worksheet reader, which may or may not contain the problem you've come across. I've never tried it with the data size you are dealing with, so no promises. I've also attached my ADO/SQL approach to the problem. This was something I moved onto when I realised the limitations of AX. One thing I have noticed is that ADO/SQL is much faster than AX, so there may be some gains for you there with large data sets if you can implement it.
    I should add that I'm a novice to all this and my efforts are down to bits I've gleamed from MSDN and others' LV examples. I hope it's of some use, if only to spark discussion. Your ctiticism is more than welcome, good or bad.
    Regards.
    Attachments:
    Database Table Reading Stuff.zip ‏119 KB

  • How to import old iPhone SMS.CSV file into new iPhone?

    Hey everyone,
    So I've spent a couple days on this as I just upgraded to the iPhone5S from my 4S.  Long story short, my backup restore didn't work after multiple tries (gave me the file is corrupt or not compatible with the iPhone error). 
    I tried:
    -reinstalling itunes
    -updating itunes
    -combining the 5S backup with the old 4S backup
    -tinyumbrella
    and a few of other things as I've forgot now.
    The most important things I needed was the contact list and SMS history.  I've got the contact list from extracting is successfully from the iTunes backup folder and using iCloud, but cannot figure out how to import the SMS.CSV file into my iPhone5S.
    Any suggestions on how I can get my SMS messages from my iPhone4s into my 5S?
    If anyone has any other suggestions on getting my Backup Restore to work that would be awesome too!
    Thanks!

    A word on SMS text messages. It is pracitcally impossible to import an SMS text-messages file to a new iPhone and have the Messages app on the new device read the messages.
    You can transfer, however contacts and other individual data (such as calendars and notes) from one iphone to the next. I had my old Nokia which whose contacts I exported to a CSV file. I then imported the file to my iphone which converted the file into multiple contacts in the Contacts app on the iphone. You can use a contact management app such as this one in order to import the contact file to the new iPhone.

  • Importing a csv file into addressbook

    So I did a search & found this question has come up quite often, but with the person having a slightly different problem, so here goes.
    I am trying to import a csv file into address book, when I go import -> text tile it comes up like it should do and I pick the file, if I select ok at this point, it will quite happily import all of the data no trouble (but in the wrong categories). So I make the changes to what I want ie. Phone other --> Phone (work). Then when I click ok, the button goes from blue to white but nothing happens.
    When I open the csv file in textedit I can see all the values separated by commas.
    Any Ideas?

    I too just had this problem. To clarify all fields work and you can map them. I found that the Outlook CSV file had two problems which were tripping up the AddressBook import function. Note these were both in street address fields so I suspect the poster who said that addresses don't work may have been having that problem.
    The CSV load program does not work if you have commas "," in your data. In my case this was very common for some European addresses as well as some company names. And then the second problem was if there was a carriage return or new line character in a field. This later one was a problem with the address fields. I suspect an AddressBook export to CSV might have this same issue. Specifically, when there were multiple lines to the street address, they had been entered as multiple lines in street address one, as opposed to entering them in multiple fields such as "street 1", "street 2", etc.
    To solve the problem I opened the CSV file using Numbers and did a global find and replace on: commas, substituting the string --- and on the newline character "¶" (ALT+Rreturn) with *.
    Note to reduce some of the remapping I also changed all of the labels with "business" in the header row of the CSV file to "work". This made it a little easier for the AddressBook import function to map fields more correctly.
    After these set of changes rather than the alternating blue then white with nothing behavior it cleanly loaded all of my names. I could then search for "---" and "*" in addressbook's search field to identify all addresses which I had to clean up.
    I hope that is helpful to someone else. The import function really does work. However, it is not robust enough to deal with certain oddities in one's data.

  • How to Import XML file into SAP B1

    Dear All,
    I have a scenario like,
    I am receiving a XML file from a 3rd party application for the daily Creation,Update of Item Master,BP Master, Marketing Documents. I want to import this file into SAP B1 through its approp objects. I understand DTW has limitation in its file types (Semicolo,Tab,Comma,ODBC). How do i do this ? Please guide me.
    Thanks,
    Thanga Raj K

    Hy folks,
    I´m frim Brasil and I've been studying the tool EFM (Eletronic File Manager) to learn more about it!
    There I saw that we can extract to XML "any" infomation from the database we want, mainly through the GEP.
    However, as I've seen, this Add-On can not import any XML file into SBO, unless for the BFP wich can be imported in conjunction with the BTHF Add-on.
    So I ask: how is it possible to import XML data into SBO database? Is it possible to be done through the EFM? or  it´s really necessary to write a code specifically to do that?
    Besides, I know that de B1iSN fit to this necessity... but when I tried to use it, by the custom "object" for BP, for example, there are some data wich the mapping conteined in this custom "process" that can not be imported... I tried to understand how to map those other fields not imported by the custom but this has been dificult to me as I am a implementation consultant focused in administrative process not on development...
    Could you please help me with this subject!
    Thanks a lot,
    Denis

Maybe you are looking for