Import very large csv files into SAP
Hi
We're not using PI, but have middleware called Trading Networks. Our design is fixed (not my decision) to not upload files into Application Server and import it from there. Our design dictates that we must write RFCs and Trading Networks will call the RFC per interface with very large file sent as table of strings. This takes 14 minutes to import into SAP plain Z-table from where we'll integrate. As a test we uploaded the file to Application Server and integrated into Z-table from there. This took 4 minutes. However our architect is not impressed that we'll stretch available Application Server to it's limits.
I want to propose that the large file be split in e.g. 4 parts at Trading Networks level and call 4 threads of the RFC which could reduce integration time to e.g. 3 minutes 30 seconds. Is there someone that has suggestions in this regard especially about a proposed, working, elegant solution for integrating large files with our current environment? This will form the foundation of our project.
Thank you and best regards,
Adrian
Zip compression can be tried. The RFC will receive zip stream which will be decompressed using CL_ABAP_ZIP.
Similar Messages
-
How do i import my local csv files into HANA database as temporary tables using java program?
I want to import my local csv file into database for further apply Join with other tables programmatic in JAVA
Hi Vivek,
Please go through the following blogs and video to resolve your issue.
Loading large CSV files to SAP HANA
HANA Academy - Importing Data using CTL Method - YouTube
Hope it helps.
Regards
Kumar -
Best data Structor for dealing with very large CSV files
hi im writeing an object that stores data from a very large CSV file. The idea been that you initlize the object with the CSV file, then it has lots of methods to make manipulating and working with the CSV file simpler. Operations like copy colum, eliminate rows, perform some equations on all values in a certain colum, etc. Also a method for prining back to a file.
however the CSV files will probly be in the 10mb range maby larger so simply loading into an array isn't posable. as it produces a outofmemory error.
does anyone have a data structor they could recomend that can store the large amounts of data require and are easly writeable. i've currently been useing a randomaccessfile but it is aquard to write to as well as needing an external file which would need to been cleaned up after the object is removed (something very hard to guarentee occurs).
any suggestions would be greatly apprechiated.
Message was edited by:
ninjarobHow much internal storage ("RAM") is in the computer where your program should run? I think I have 640 Mb in mine, and I can't believe loading 10 Mb of data would be prohibitive, not even if the size doubles when the data comes into Java variables.
If the data size turns out to be prohibitive of loading into memory, how about a relational database?
Another thing you may want to consider is more object-oriented (in the sense of domain-oriented) analysis and design. If the data is concerned with real-life things (persons, projects, monsters, whatever), row and column operations may be fine for now, but future requirements could easily make you prefer something else (for example, a requirement to sort projects by budget or monsters by proximity to the hero). -
Script to import contacts from CSV file into Address Book
Hi,
I am wondering if it would be possible to import contacts from a CSV file into a group in Address Book using applescript, and if so, if anyone is aware of a script that does this or something similar.
I have a CSV file with three columns 'First Name', 'Last Name' and 'Email Address'. I would like these to correspond with 'First', 'Last' and (Work) 'Email' in Address Book.
Ideally the script would also check to see if entries already exist for the people being imported, but this is less crucial.
Thanks in advance for any assistance,
NickHi,
Test this on a very small csv file (four or five contacts) to see if it works for you. It will not de-duplicate. You don't give the name of the group, so you'll need to replace Name of group in the second line with your group name, enclosed in double quotes.
--begin script
set the_people to read (choose file with prompt "Choose your CSV file")
set the_group to "Name of group"
set old_tids to AppleScript's text item delimiters
set AppleScript's text item delimiters to ","
set par_count to (count paragraphs in the_people)
repeat with x from 1 to par_count
set next_par to paragraph x of the_people
set first_name to text item 1 of next_par
set last_name to text item 2 of next_par
set e_mail to text item 3 of next_par
tell application "Address Book"
set nu_person to make new person with properties {first name:first_name, last name:last_name}
make new email at end of emails of nu_person with properties {label:"Work", value:e_mail}
add nu_person to group the_group
save
end tell
end repeat
set AppleScript's text item delimiters to old_tids
--end script
Watch out for bad line breaks in the lines beginning "set nu_person..." and "make new email..." -
Import and view CSV Files into OTM
Hi All,
Can you please help me in importing CSV files into Oracle Test Manager and to view the imported files in OTM.Hi,
I have just googled and find the link :)
http://oracleats.com/blog/?p=785 -
How do I split a Large CSV file into Multiple CSV's Using Powershell
I am a novice at powershell but this looks to be the best tool to do this task. have a csv file that looks like this:
Date,Policy,Application
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
Is it possible to split this CSV into multiple CSV's based on "Application".
Lets say the output might look like:
None.csv
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
AppBiz.csv
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
PeopleBiz.csv
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
Any help would be greatly appreciatedI think this might be what you want:
Import-Csv applications.csv |
Group Application |
foreach {
$_.Group | Export-Csv "$($_.Name).csv" -NoTypeInformation
[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
Very nice! 4x faster..
I doubt the OP will get what you just did there..
Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________
Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx -
How can I import an existing csv. file into Outlook on a Mac with OS10.7
Tried to import a csv. file on my Mac (OS 10.7.2) into Microsoft Outlook 2011 (for Mac) on the same computer, ended up with a useless Excel page - it opened the csv file in Excel with only a few contacts and uselessly formatted, at that!. Also, tried to save the csv file as a text file, then importing it into Word 2011 and Excel, with the same results. Anyone able to help? Would like to have the contained address/contact info look concise and neat, without duplicates or garbled/missing information.
[email protected]DPS can use H.264 files, but I have no idea about the details since I've never done any of it myself. Ask in the ID/ DPS forums.
Mylenium -
I wish to import a csv file from my Bank into a numbers spreadsheet but I can't expand it from the one column it's in . Can anybody help Thankyou
This Automator Service (Dropbox download) should do the job, eliminating the need for a trip to TextEdit, etc.
The one-time installation is simple: just doubleclick the .workflow package. You may have to click "Download Anyway" in System Preferences > Privacy & Security.
To use, in Finder select the csv file, rightclick, and in Services in the contextual menu select CSV to Tabs on Clipboard. Choose the delimiter when prompted (usually it's a comma but there are other choices). When the "ready to paste" notification appears a few seconds later, click once in a Numbers cell and command-v to paste the data. (If needed Numbers will automatically expand the table to accommodate the data.)
To remove or rename the Service, in Finder hold down the option key and in the menu Go > Library > Services.
SG -
How to import data from CSV file into a table by using oracle forms
Hi,
I have a CSV file and i want to insert in oracle database in a table by using a button in oracle forms.
the user can select CSV file by using open dialog .
can any one help me to find method to make import and select file from the client machine ?
thx.1. create table blob
CREATE TABLE IB_LOVE
DOC BLOB,
CONTRACT_NO VARCHAR2(20 BYTE) NOT NULL
)2. use the code below to insert:
INSERT INTO ordmgmt.ib_love
(contract_no, doc
VALUES (:control.contract_no_input, NULL
lb$result :=
webutil_file_transfer.client_to_db (:b2.file_name, v_file_blob_name, v_col_blob_name,
'CONTRACT_NO = ' || :control.contract_no_input);
:SYSTEM.message_level := 25;
COMMIT;
:SYSTEM.message_level := 0;3. use the code below to download
if :control.CONTRACT_NO_INPUT is not null then
vboolean := webutil_file_transfer.DB_To_Client_With_Progress(
vfilename, --filename
'IB_LOVE', ---table of Blob item
'DOC', --Blob column name
'CONTRACT_NO = ' || :CONTROL.CONTRACT_NO_INPUT, ---where clause to retrieve the record
'Downloading from Database', --Progress Bar title
'Wait to Complete'); --Progress bar subtitle client_host('cmd /c start '||vfilename);
client_host('cmd /c start '||vfilename);
else
errmsg('Please choose contract no');
end if;4. use the code below to open file dialog
x:= WEBUTIL_FILE.FILE_OPEN_DIALOG ( 'c:\', '*.gif', '|*.gif|*.gif|', 'My Open Window' ) ;
:b2.FILE_NAME:=X; -
Trying to import from different csv files into multiple JTables
This might seem like something that isn't commonly done, but I'll see if anyone has done this before anyways.
I currently have a JTable (tab #1) and it successfully reads in the data from a CSV file to populate the cells. Now, I am looking into making 3 more JTables. I can tab between the four different tables, but I can't seem to figure out how to get the data to go to the correct JTable. Tab #1 = table, Tab #2 = table2, Tab #3 = table3, Tab #4 = table4.
Thanks.
try
FileInputStream fileInput = new FileInputStream ("upcoming.csv");
BufferedReader InputCSV = new BufferedReader (new InputStreamReader (fileInput));
line = InputCSV.readLine(); //start reading into the records.
//Create data for the table.
//the row variable = starting row
int row = 0;
while(line != null)
tmp = line.split(",");
for (int col = 0; col < columnNames.length; ++col)
data[row][col] = tmp[col]; //store cells' data to the 2-dimensional array.
row++;
line = InputCSV.readLine();
}//end of WHILE-statement.
} //END of TRY-statement.
catch(Exception e)
System.out.println(e);
}I have been successful in my quest to get 4 different datasets into 4 different JTables. The previous posters suggestion to use the Table model led me to this page:
http://java.sun.com/j2se/1.5.0/docs/api/javax/swing/table/TableModel.html
TableModel myData = new MyTableModel();
JTable table = new JTable(myData);I had some very similar code in my program. I use the DefaultTableModel and the SortFilterModel before having 4 different JTables.
DefaultTableModel model = new DefaultTableModel(data, columnNames);
DefaultTableModel model2 = new DefaultTableModel(data2, columnNames);
DefaultTableModel model3 = new DefaultTableModel(data3, columnNames);
DefaultTableModel model4 = new DefaultTableModel(data4, columnNames);I use the code in the first post 4 times, and only change the filename and the data#.
I also have some code that counts the number of lines that will be needed.
Thanks. -
HTMLDB_tools, processing a very large csv file in a collection
Hi Guys,
I'm new to APEX and trying to load approx 1,000,000 rows into a table from a csv file, does anyone have a better way of doing it than htmldb_tools.parse_file (which is very slow)
CheersIt's not Apex, but you could use SQL*Loader. It's really very fast!
greets, Dik -
Read very very large excel file into 2d array (strings or varient)
Hi all,
Long time user, first time poster on these boards.
Looking at the copius amounts of great info related to reading Excel data from .xls files into labview, i've found that every one i've found from various people use the ActiveX method (WorkSheet.Range) which two strings are passed, namely excel's LetterNumber format to specify start and end.
However, this function does not work when trying to query huge amounts of information. The error returned is "Type Mismatch, -2147352571" I have a very large excel sheet i need to read data from and then close the excel file (original file remains unchanged). However this file is gigantic (don't ask me, I didn't make it and I can't convince them to use something more appropriate) with over 165 columns at 1000 rows of data.I can read a large number of columns, but only a handful of rows, or vice versa.
Aside from creating a loop to open and close the excel file over and
over reading pieces of it at a time, is there a better way to read more
data using ActiveX? Attached is code uploaded by others (with very minor modification) as an example.
Thanks,
Attachments:
Excel Get Data Specified Field (1-46col).vi 23 KBHi Maddox731,
I've only had a very quick glance through your thread, and I must admit I haven't really thought it through properly yet. Sounds like you've come up with your own solution anyway. That said I thought I'd take a bit of a scatter gun approach and attach some stuff for you regradless. Please forgive my bluntness.
You'll find my ActiveX Excel worksheet reader, which may or may not contain the problem you've come across. I've never tried it with the data size you are dealing with, so no promises. I've also attached my ADO/SQL approach to the problem. This was something I moved onto when I realised the limitations of AX. One thing I have noticed is that ADO/SQL is much faster than AX, so there may be some gains for you there with large data sets if you can implement it.
I should add that I'm a novice to all this and my efforts are down to bits I've gleamed from MSDN and others' LV examples. I hope it's of some use, if only to spark discussion. Your ctiticism is more than welcome, good or bad.
Regards.
Attachments:
Database Table Reading Stuff.zip 119 KB -
How to import old iPhone SMS.CSV file into new iPhone?
Hey everyone,
So I've spent a couple days on this as I just upgraded to the iPhone5S from my 4S. Long story short, my backup restore didn't work after multiple tries (gave me the file is corrupt or not compatible with the iPhone error).
I tried:
-reinstalling itunes
-updating itunes
-combining the 5S backup with the old 4S backup
-tinyumbrella
and a few of other things as I've forgot now.
The most important things I needed was the contact list and SMS history. I've got the contact list from extracting is successfully from the iTunes backup folder and using iCloud, but cannot figure out how to import the SMS.CSV file into my iPhone5S.
Any suggestions on how I can get my SMS messages from my iPhone4s into my 5S?
If anyone has any other suggestions on getting my Backup Restore to work that would be awesome too!
Thanks!A word on SMS text messages. It is pracitcally impossible to import an SMS text-messages file to a new iPhone and have the Messages app on the new device read the messages.
You can transfer, however contacts and other individual data (such as calendars and notes) from one iphone to the next. I had my old Nokia which whose contacts I exported to a CSV file. I then imported the file to my iphone which converted the file into multiple contacts in the Contacts app on the iphone. You can use a contact management app such as this one in order to import the contact file to the new iPhone. -
Importing a csv file into addressbook
So I did a search & found this question has come up quite often, but with the person having a slightly different problem, so here goes.
I am trying to import a csv file into address book, when I go import -> text tile it comes up like it should do and I pick the file, if I select ok at this point, it will quite happily import all of the data no trouble (but in the wrong categories). So I make the changes to what I want ie. Phone other --> Phone (work). Then when I click ok, the button goes from blue to white but nothing happens.
When I open the csv file in textedit I can see all the values separated by commas.
Any Ideas?I too just had this problem. To clarify all fields work and you can map them. I found that the Outlook CSV file had two problems which were tripping up the AddressBook import function. Note these were both in street address fields so I suspect the poster who said that addresses don't work may have been having that problem.
The CSV load program does not work if you have commas "," in your data. In my case this was very common for some European addresses as well as some company names. And then the second problem was if there was a carriage return or new line character in a field. This later one was a problem with the address fields. I suspect an AddressBook export to CSV might have this same issue. Specifically, when there were multiple lines to the street address, they had been entered as multiple lines in street address one, as opposed to entering them in multiple fields such as "street 1", "street 2", etc.
To solve the problem I opened the CSV file using Numbers and did a global find and replace on: commas, substituting the string --- and on the newline character "¶" (ALT+Rreturn) with *.
Note to reduce some of the remapping I also changed all of the labels with "business" in the header row of the CSV file to "work". This made it a little easier for the AddressBook import function to map fields more correctly.
After these set of changes rather than the alternating blue then white with nothing behavior it cleanly loaded all of my names. I could then search for "---" and "*" in addressbook's search field to identify all addresses which I had to clean up.
I hope that is helpful to someone else. The import function really does work. However, it is not robust enough to deal with certain oddities in one's data. -
How to Import XML file into SAP B1
Dear All,
I have a scenario like,
I am receiving a XML file from a 3rd party application for the daily Creation,Update of Item Master,BP Master, Marketing Documents. I want to import this file into SAP B1 through its approp objects. I understand DTW has limitation in its file types (Semicolo,Tab,Comma,ODBC). How do i do this ? Please guide me.
Thanks,
Thanga Raj KHy folks,
I´m frim Brasil and I've been studying the tool EFM (Eletronic File Manager) to learn more about it!
There I saw that we can extract to XML "any" infomation from the database we want, mainly through the GEP.
However, as I've seen, this Add-On can not import any XML file into SBO, unless for the BFP wich can be imported in conjunction with the BTHF Add-on.
So I ask: how is it possible to import XML data into SBO database? Is it possible to be done through the EFM? or it´s really necessary to write a code specifically to do that?
Besides, I know that de B1iSN fit to this necessity... but when I tried to use it, by the custom "object" for BP, for example, there are some data wich the mapping conteined in this custom "process" that can not be imported... I tried to understand how to map those other fields not imported by the custom but this has been dificult to me as I am a implementation consultant focused in administrative process not on development...
Could you please help me with this subject!
Thanks a lot,
Denis
Maybe you are looking for
-
How do i share my itunes library across accounts on my Mac mini server?
got the new server for the holidays and I migrated my entire itunes library from my old PC. It was my expectation that the accounts on the server would all have access to the same library, however, that does not appear to be the case. When my son l
-
I'm trying to update my application and it is asking me to change my billing account it is not accepting the security code. I am not trying to buy an application just to update and install free apps
-
No themes available when creating a new book in iPhoto 08
I cannot create a new Book in iPhoto 08 version 7.1.5 (378). When the dialog opens, the Book Type ComboBox is empty. If I press the Choose button, a new error dialog appears with the message: "No Available Themes".
-
Hello Guys, me again I'm seeing an issue when the client tries to install the NAC agent on his PC. The client reports that an update is available for which I click OK, then it appears to download the new agent (really fast btw) and then it starts ins
-
Saved an image as library and now all in iphoto images disappeared
I was saving an image from email to iphoto, and somehow it was saved as my iphoto library. All of my photos are now gone, there were hundreds, and the one image I just saved is just there. Where are my photos??? Don't they get saved somewhere on my h