Massive Import from Local Flat Excel Files

Hi there,
I have to import to HANA a large amount of Data from local files (Excel Files).
I have a lot of tables with 100+ columns and like 10kk, 20kk or 30kk of rows.
At the moment i do this things:
1) Download a table from SAP in a .txt format, with like 800k rows per file;
2) I use a program to split each file into many files with 65k rows each;
3) Import the smaller file to excel. Here i keep all columns in Text type, except for Decimal Numbers for wich i use General Type;
4) I use the import feature of Hana Studio for each excel file.
So, for example, i have the table AFVC with 34kk rows. I downloaded 44 .txt files with like 500k ~ 800k rows each.
After that i split each file into like 16 files with 65k rows each one. I open excel and import each file and i have to change some coloumn types.
Now i can import each xls file into HANA with hana Studio.
So i have to import to hana like 700 files. After i imported to excel each file..
This is what i'm doing, and it's not so fast, i'm getting crazy.
What can i do to improve the load?
I don't have the FTP user yet (i requested that) and i don't have SLT avaiable.
My real problem is that with so many coloumns i can't import to hana files with more than 65535 rows, or it give me a "out-of-memory" error.
Thanks
Alessandro

Hi Alessandro,
I would try to use the import-statement.
I don't think that there's any limit, and also the error log which it creates is much more usefull then the errors shown in the import data wizard.
Greets,
ben

Similar Messages

I bought Numbers because my Excel files would not work with Mavericks. How can I import a non-functioning Excel file into numbers? (As of now, the file NAME transfers but data is not.) but

I bought Numbers because my Excel files would not work with Mavericks. How can I import a non-functioning Excel file into numbers? (As of now, the file NAME transfers but data does not.)

HI Dave,
I'm surprised you get no message when you attempt opening the Excel document in Numbers.
I'm assuming Numbers 3.1, since you are running Mavericks.
Does the Numbers file that opens show any content?
Does it contain a table?
How large?
Does it contain more than one tab (indicating more than one sheet)?
Do those other tabs contain a table?
Have you checked those tables for data content?
There have been cases in the past of imported documents opening with white text on a white background, giving the appearance of containing nothing. Check for this by selecting a block of cells, then applying a Fill colour or a Text colour using the Format button (paintbrush).
I'd also suggest attempting to open the Excel files using LibreOffice, which can be downloaded from the linked website.
Regards,
Barry

What is the best way to insert massive data into an existing excel file?

dear gurus,
i am wondering that what is the best way to insert massive data into an existing excel file, more performance perspective.
the file is read from BDS , and we want to insert data into it .
the way i can think of is
1. OLE AUTOMATION
   i think performance will be a big problem
2. Office integration
    i am not sure it's facing the same performance issue ?
3 . XXL_SIMPLE_API/FULL_API
    I am not sure whether they can insert data into an existing excel file?
could you please give me some advices?
br.
jun

Hi,
If you want to APPEND data( add data to an existing excel file) from SAP, then use GUI_DOWNLOAD fm with APPEND = 'X' paramter.
Best regards,
Prashant

Firefox can't read any Bookmark that was imported from my PC with file extension .url. Safari reads them fine. Is there a fix, so I can use Firefox instead of Safari? Many thanks if so. I have the latest version of Firefox

Firefox can't read any Bookmark on my Mac that was imported from my PC with file extension .url. Safari reads them all fine. Is there a fix, so I can use Firefox instead of Safari? Many thanks if so. I have the latest version of Firefox
== URL of affected sites ==
http://anysite.url
== User Agent ==
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7

Hello JF.
I don't think that extension is supported. I believe Firefox can only read .json and .html.
You may want to read this though:
[http://support.mozilla.com/en-US/kb/Importing+bookmarks+and other data from Safari Importing bookmarks and other data from Safari]

How to download data from spool to excel file

Hi,
I have requirement like....i need to download data from spool to excel file.
Please let me know the process how to download...
<removed by moderator>
<removed by moderator>
Thanks,
Khasimsa
Moderator message: please (re)search yourself before asking, do not assign priorities, do not offer re-ward
locked by: Thomas Zloch on Sep 8, 2010 1:18 PM

Hi try this way..
*FM called that returns the Spool Request Number data into and internal table
CALL FUNCTION 'RSPO_RETURN_ABAP_SPOOLJOB'
    EXPORTING
      rqident              = p_spool                     "Spool Request Number
      first_line           = 1
    TABLES
      buffer               = it_spool_xls                            "Internal table that will have the Spool Request No data
    EXCEPTIONS
      no_such_job          = 1
      not_abap_list        = 2
      job_contains_no_data = 3
      selection_empty      = 4
      no_permission        = 5
      can_not_access       = 6
      read_error           = 7
      OTHERS               = 8.
*To convert the spool data into excel format
CALL FUNCTION 'SO_RAW_TO_RTF'
    TABLES
      objcont_old = it_spool_xls               "Internal table having spool data
      objcont_new = it_xls_spool.           "Int table having Excel format data converted from Spool data
"call GUI down Load by passing it_xls_spool
Prabhudas

Using Firefox 3.6.20 and Windows XP Home SP3. Firefox sporatically deletes bookmarks from the bookmarks toolbar. Additionally, when I try to import from a saved HTML file, Firefox does not import the complete file. Results are inconsistent.

Using Firefox 3.6.20 and Windows XP Home SP3. Firefox sporatically deletes bookmarks from the bookmarks toolbar. Additionally, when I try to import from a saved HTML file, Firefox does not import the complete file. Results are inconsistent.

A possible cause is a problem with the file places.sqlite that stores the bookmarks and the history.
*http://kb.mozillazine.org/Bookmarks_history_and_toolbar_buttons_not_working_-_Firefox

When importing from my camera or file in Lightroom 5 my pictures are over-exposed by 1-2 stops, while they are well exposed on the camera-screen. How to get the 'right-exposed pictures on the screen?

When importing from my camera or file in Lightroom 5 my pictures are over-exposed by 1-2 stops, while they are well exposed on the camera-screen. How to get the 'right-exposed pictures on the screen?

There is an option in your Lightroom preferences on the General tab, "Treat JPEG files next to raw files as separate images". If you have that option checked then Lightroom will import and display both your raw and JPEG files. Are you using active D-lighting on your camera? If you are then you need to turn off that feature.

Extract data from Oracle in excel file

Hi,
I have a requirement where in I need to extract data from Oracle in excel file and the excel worksheet name should be "Data".
for eg. excel file name "AR Data_DDMMYY" and excel worksheet name "Data"
I have used the UTL_FILE API to extract the tab delimited data which can be opened in excel but it is not exactly an excel file as the worksheet name is same as the file name.
I tried using utl_file.fcopy and frename.
Is there any way to do this using PLSQL?
select * from v$version;
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
"CORE 10.2.0.5.0 Production"
TNS for HPUX: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - ProductionSample Code:
declare
cursor c is
select * from scott.emp;
v varchar2(100);
f utl_file.file_type;
file_name varchar2(100) := 'AR Data.xls';
dir varchar2(50) := 'CESDIR191710';
--select * from dba_directories
begin
f := utl_file.fopen(dir, file_name, 'W');
v := 'EMPNO'||chr(9)||'ENAME'||chr(9)||'JOB'||chr(9)||'SAL'||chr(9)||'HIREDATE'||chr(9)||'DEPTNO';
utl_file.put_line(f, v);
for i in c
loop
v := i.empno||chr(9)||i.ename||chr(9)||i.job||chr(9)||i.sal||chr(9)||i.hiredate||chr(9)||i.deptno;
utl_file.put_line(f, v);
end loop;
utl_file.fclose(f);
--utl_file.frename(dir, file_name, dir, replace(file_name, '.xls', '_')||to_char(sysdate, 'MMDDYY')||'.xls', false);
utl_file.fcopy(dir, file_name, dir, replace(file_name, '.xls', '_')||to_char(sysdate, 'MMDDYY')||'.xls');
end;Thanks
Imran

Imran Soudagar wrote:
Hi,
I was able to generate the excel 2007 file with the data using the package from below link;
http://technology.amis.nl/2011/02/19/create-an-excel-file-with-plsql/
but the requirement is to generate excel 2003 file.
I tried changing the .xlsx to .xls and it gives a note while opening the file "The file you are trying to open, abc.xls, is in a different format than specified by the file extension. Verify that the file is not corrupted and is from a trusted source before opening the file. Do you want to open the file now?"Then you have three options:
1) stop using anton's package and find another one that supports the old and deprecated version of Excel 2003
2) write your own package to produce an Excel file.
3) Upgrade your version of Excel to a recent version
I tried the programs from other links on the forum but I am still getting this message. The client does not want this message to be displayed as the excel file works as an input to another system.
Can anyone help me with the issue?
Also, is it true that the programatically generated excel file is actually an xml file which is renamed to .xls and hence it shows the message while opening such files?Yes, Excel supports several different formats for it's files. By default, if you save an XLS files from Excel, it writes it out in a Microsoft proprietary binary format, which you would be hard pushed to replicate easily from PL/SQL. Excel also has the ability to save it's files as XML format, which is more readable and easier to produce programatically, whilst still allowing you to have multiple sheets, formulas and formatting included in it. That's the format that most people who need formatting and multiple sheets opt for when programatically generating their data as excel workbooks. (There's also an SLYK format that people used to use before that, but it's not as flexible)
If you want to write your own, the easiest thing to do is to start with a blank workbook in Excel, and put in your basic requirements e.g. a couple of named sheets, and some data in different formats (number, date, text etc.) and different formatting options etc. Save that file in XML format from Excel and then open up the file using notepad/wordpad to look at the structure. There'll be a whole load of redundant rubbish Microsoft put in there, but you should be able to figure out the basic structure of XML required to give you what you want.

Transfer excel files from iMac to excel file on ipad

transfer excel files from iMac to excel file on iPad. Using iTunes, I transfered files on my mac using Numbers and it worked fine, but trying the same method for an Excel file didn't work (i.e., the Excel file never showed up on my iPad.)
Thanks for any help you can provide.
LJ

http://www.wondershare.com/pdf/transfer-pdf-to-ipad.html
http://www.bythom.com/pdffaq.htm
How to Transfer PDFs to an iPad
http://www.dummies.com/how-to/content/how-to-transfer-pdfs-to-an-ipad.html
 Cheers, Tom

Importing from XML to Excel - need to know how to append to existing Excel file

Hello,
I have several XML files that I will need to import into one Excel spreadsheet. I have been able to figure out how to import an XML file to an Excel file but when I import a second XML file, the contents of the second XML overwrite the Excel spreadsheet.
I am looking for an option to append. Thanks in advance
private void XmlToExcel(string xmlFilePath, string excelFilePath)
object misValue = System.Reflection.Missing.Value;
DataSet Trans_ds = new System.Data.DataSet();
Trans_ds.ReadXml(xmlFilePath);
Excel.Application excel = new Excel.Application();
excel.Application.Workbooks.Add(true);
System.Data.DataTable table = Trans_ds.Tables[0];
Worksheet worksheet = (Worksheet)excel.ActiveSheet;
Range excelRange = (Excel.Range)worksheet.get_Range("A" + worksheet.Rows.Count, Type.Missing);
int lastRow = excelRange.get_End(XlDirection.xlUp).Row;
int ColumnIndex = 0;
int rowIndex = lastRow++;
foreach (System.Data.DataColumn col in table.Columns)
ColumnIndex++;
excel.Cells[lastRow, ColumnIndex] = col.ColumnName;
foreach (DataRow row in table.Rows)
rowIndex++;
ColumnIndex = 0;
foreach (DataColumn col in table.Columns)
ColumnIndex++;
excel.Cells[rowIndex + 1, ColumnIndex] = row[col.ColumnName];
worksheet.StandardWidth = 20.0;
worksheet.SaveAs(excelFilePath, misValue, misValue, misValue, misValue, misValue, misValue, misValue, misValue, misValue);
excel.Quit();
releaseObject(worksheet);
releaseObject(excel);

See if this helps. I opened the old excel file and added the new rows into the existing workbook.
private void XmlToExcel(string xmlFilePath, string excelFilePath, Boolean append)
object misValue = System.Reflection.Missing.Value;
DataSet Trans_ds = new System.Data.DataSet();
Trans_ds.ReadXml(xmlFilePath);
Excel.Application excel = new Excel.Application();
Excel.Workbook Bk = excel.Workbooks.Add(true);
System.Data.DataTable table = Trans_ds.Tables[0];
Excel.Worksheet worksheet = (Excel.Worksheet)excel.Worksheets[1];
Excel.Range excelRange = (Excel.Range)worksheet.get_Range("A" + worksheet.Rows.Count, Type.Missing);
int lastRow = excelRange.get_End(Excel.XlDirection.xlUp).Row;
int ColumnIndex = 0;
int rowIndex = lastRow++;
foreach (System.Data.DataColumn col in table.Columns)
ColumnIndex++;
excel.Cells[lastRow, ColumnIndex] = col.ColumnName;
foreach (DataRow row in table.Rows)
rowIndex++;
ColumnIndex = 0;
foreach (DataColumn col in table.Columns)
ColumnIndex++;
excel.Cells[rowIndex + 1, ColumnIndex] = row[col.ColumnName];
worksheet.StandardWidth = 20.0;
if (append == false)
worksheet.SaveAs(excelFilePath, misValue, misValue, misValue, misValue, misValue, misValue, misValue, misValue, misValue);
else
//open old workbook
Excel.Workbook oldworkbook = (Excel.Workbook)excel.Workbooks._Open(
excelFilePath,
Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing,
Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing,
Type.Missing, Type.Missing);
//get first sheet in workbook
Excel.Worksheet oldxlSht = (Excel.Worksheet)oldworkbook.Worksheets[1];
//get column A of old worksheet to find last row in sheet
Excel.Range xlRange = (Excel.Range)oldxlSht.get_Range(
"A" + oldxlSht.Rows.Count, Type.Missing);
int LastRow = xlRange.get_End(Microsoft.Office.Interop.Excel.XlDirection.xlUp).Row;
int newrow = LastRow + 1;
//get last row of new worksheet
xlRange = (Excel.Range)worksheet.get_Range(
"A" + oldxlSht.Rows.Count, Type.Missing);
LastRow = xlRange.get_End(Microsoft.Office.Interop.Excel.XlDirection.xlUp).Row;
// copy rows of new worksheet to old worksheet
xlRange = (Excel.Range)worksheet.Rows["1:" + LastRow, Type.Missing];
xlRange.Copy(oldxlSht.Rows[newrow, Type.Missing]);
oldworkbook.Save();
excel.Quit();
releaseObject(oldxlSht);
releaseObject(worksheet);
releaseObject(excel);
jdweng

How to view data in Xcelsius from an external Excel file

Hi Everyone,
i wonder if you know How to view data in Xcelsius from external Excel file and not depend only on the emebded excel sheet.

Hi,
As per my understanding, excel has to be imported in Xcelsius to draw up the dashboard. Your aggregated data sits in the xlf file. Using an external excel file will not be possible (I guess).
One of the ways to connect to an external excel, is the use XML Maps but then your data has to be in the form of XML which will be pulled into excel first and then needs to be brought in Xcelsius using Data Manager.
You can explore XML Maps in xcelsius in SDN.
Regards
Nikhil

Downloading from alv to excel file

hai
while downloading from alv grid to excel file
i am not getting values properly. i have taken 110 fields. i am not getting prper way. can u plz tel me way....

Hi ....
I have the similar problem. My coumns are not in any sequence when I download it to excel sheet.
How do I fix it.
Neha

How to get data from an existing Excel file using report generation toolkit

I'm trying to use the Excel Get Data vi from the report generation toolkit, but i can't figure out how to turn a browsable path into the right type for the VI. I've tried using the New Report.vi, but that doesn't work unless you use a template. It won't open an existing excel file and make it an open report to pull data from.
Essentially, i have a bunch of excel files that have data in them and i want to use a VI to analyze the data. So, i'm trying to pull in all the data directly from the excel file so i don't have to reproscess them all into text so i can use the more standard spreadsheet VIs. But to even convert the excel file programtically in labview won't i still need to be able to open the excel file and get the data?
Solved!
Go to Solution.

I C.
Got it,
the reason why I can view it using my Excel file because its in .CSV.
I open Microsoft Excel to open .CSV file.
But you can also open .csv file with notepad.
So, you cannot open .xls with Read Spreadsheet File
I learn something everyday
Best regards,
Krispiekream

Import from a sample.DMP file ????? hlp???

when import from a dump file an error occurs>>>>>>
Export file created by export:v10.01.00 via conventional path
IMP-00013: only a DBA can import a file exported by another DBA
IMP-00000: Import terminated unsuccessfully.
What can i do .... help meee??????
When i log on to sys with password as sysdba ....

Hi.
Which user did take the export?
And how is the USERID input set to imp.exe?
Rgds
Kjell

Import from a sample.DMP file

when import from a dump file an error occurs>>>>>>
Export file created by export:v10.01.00 via conventional path
IMP-00013: only a DBA can import a file exported by another DBA
IMP-00000: Import terminated unsuccessfully.
What can i do .... help meee??????
\

Login with a user that has the DBA role...(or SYS or SYSTEM)

Massive Import from Local Flat Excel Files

Similar Messages

Maybe you are looking for