File to SAP with huge ammount of data

Hi,
what is the best way to store data, coming from file adapter, to a SAP Z-Table through XI/PI ?
Sender = File Adapter (one file with aprox. 1,5 million records)
Structure = simple structure with 5-10 fields
Receiver = SAP Z-Table (Mapping 1:1)
Thx,
Gordon

Hi Gordon,
> 1. do you have some info about the proxy configuration for that scenario ?
You need to do some settings for proxy communication. Please check this blog for the same:
/people/vijaya.kumari2/blog/2006/01/26/how-do-you-activate-abap-proxies
>2. what about using idoc instead of rfc / proxy ?
Why do you want to create an idoc for updating one table. Its very simple logic. Just you need to load the data whatever you are getting. So it will be a simple proxy or rfc to create / update the table. It is the easiest route.
Regards,
---Satish

Similar Messages

Build daily Big flat file using SAP R/3 material master data

Hi all,
I must build an interface between SAP R/3 4.7 and an external application
Daily a flat file must be generated contaning Material data in full exchange mode
(volume 100 000 materials, <10 fields => ~ File of 30Mo)
No complex mapping, only format mapping (Ex : Price quantity...)
I don't know yet if I will use Idocs or BAPI to extract data. However, I'm sure I must use multiple inbound messages because of volume
After reading documentation, I consider the both following solutions to manage it :
1) A simple asynchronous interface with a file adapter in construction mode APPEND to build flat file progressively with arrival of SAP messages
=>
viable solution ?
Is there problem of concurrent access to file in such a solution ?
(idocs or bapi data will be processed ones following the others )
2) BPM Collecting and Bundling Messages - One Interface
(Unlimited Loop with Deadline trigger correspondint to time I must build flat file)
=>
Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Which solution do you advise ?
Is there another ?
Last question : (more general one)
In such scenario, I can't see XI-value added comparing a creation of flat file directly with R/3.
I will generate a big number of idocs, use a big part of XI memory ...
Can you give me your opinion about it ?
Thanks a lot
Best regards
Jean-Charles
Message was edited by:
Jean-Charles
Message was edited by:
Jean-Charles

Hi Jean-Charles,
>>>volume 100 000 materials, <10 fields => ~ File of 30Mo
You have a performance task...
>>>Is there problem of concurrent access to file in such a solution ?
No, the adapter framework has its own queueing, the message would be stored and executed later. If sequence is important, you can use EOIO as well.
>>>BPM Collecting and Bundling Messages - One Interface
BPM is not required in your case, you should avoid becoz of performance task. You can append at file adapter, no problem. Bundling and splitting is even possible at Integration Server without BPM.
>>>Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Avoid using BAD words
>>>Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Use the <a href="https://websmp110.sap-ag.de/quicksizing">Quick Sizer</a> for sizing the hardware.
Regards,
Udo

Data with huge volume of data with DTP

Hi Experts,
I have this problem with upload of huge volume of data with DTPs.
I have my initialisation done as I am doing reloads, Now I have this data from fiscal year period 000.2010 to 016.9999.
I have huge volume of data.
I have tried uploading this data in chunks by dividing 3 months for each DTP and had made full load.
But when I processed the DTP the data packages are decided at source and I have about 2000 data packages.
Now my request is turning to red after processing about 1000 datapackages, batch processes allocated to this also stopped.
I have tried dividing DTP only by month and processed the DTP I have same problem. I have deleted the indexes before uplaoding to the cube, Changed the setting battch processing from 3 to 5.
Please can any one advise what could be problem.I am uplaoding this reloads in quality system.
How can upload this data which are in millions.
Thanks,
Tati

Hi Galban,
I have made the parallel processing from 3 to 5 even and the datapakcage size
Can you please advise in this area how can I increase the data package size as the data package size for my upload is the package size corresponds to package size in source it is determined dynammically at runtime.
Please advise.
Thanks
Tati

How to save an excel file as CSV with semi-colon as data separator?

We are creating a flat data file to load the flat file data to our BW system with Excel. When we save the file, there are three kinds of CSV types for selecting from the "Save as type" list box:
CSV (Comma delimited)
CSV (Macintosh)
CSV (MS-DOS)
There is no CSV (semi colon delimited) type. If we pick CSV (Comma delimted) as the type, then when clicking the "Preview ..." picture icon under the "DataSource/Trans. Structure" tab of the InfoSource for this flat file (with the CSV radio button checked the Data separator default is ";"), the File Upload Preview window shows the data is messed up that some columns of the excel flat file are combined into one column.
Since the save type we picked is "CSV (Comma delimited)", then we try to change the default Data separator from ";" to "," in the preview selection screen, but still not helpful!
How to resolve the above problem?
Thanks

Hi Kevin,
This "," is defined in your windows setting. If you want to have ";" as separator then go to control Panel, select Regional Options, go to Numbers Tab and define the List separator as ";". After that when you will save your excel file as CSV(MS-DOS) it will have ";" as separator. This will make sure that the amounts are not broken in to two different fields while loading.
Else if you keep "," as separator then you can also go into the Excel and define all number fields as Number without thousand separator.
Let me know if you have any doubts in this.
Regards,
Rohit

Misalignement in Excel sheet,while downloading reports with huge data

Hi,
While downloading reports in excel with huge volume of data ,we are getting misalignments in the excel sheet(some data comes in wrong rows). Can you please suggest what could be the reason and how to correct it.
Regards,
Roopak

Hi,
If you are using Office 2003, then you cannot download morethan 65000+. However if you use Office 2007, I think it gives you somewhere around 400000 records. (I didnt scroll below that!!!). Also the columns are morethan IV.
However assuming that, even that version can have limitation, you can always download your data in a wordpad/notepad/textpad etc which are basically text editing tools. They dont have any such row or column restrictions.
Once downloaded, you can randomly split them into some x number of files, based on the number of records and then open the same documents using xls sheet using tab value as delimiter. Now the alignment will be perfect in the excel sheet.

Working with large amount of data

Hi! I am writing an peer-to-peer video player and I need to operate with huge amounts of data. The downloaded data should be stored in memory for sharing with other peers. Some video files can be up to 2 GB and more. So keep all this data in RAM - not the best solution i think =)
Since the flash player does not have access to the file system I can not save this data in temporary files.
Is there a solution to this problem?

No ideas? very sad ((

Pls Help me with steps to add data from xml file to SAP B1 through B1iSN.

Pls Help me with steps to add data from xml file to SAP B1 through B1iSN. I am getting stuck in xsl transformation. not able to understand where the mapping code needs to be added.
Pls explain me the steps for adding data from xml to B1 quotation step by step.
thanks and regards
Priya

Hi,
Have you checked this: https://sap.na.pgiconnect.com/p45508295/?launcher=false&fcsContent=true&pbMode=normal ?
Thanks,
Gordon

I have a scenario, ECC-PI-Message broker. ECC sending IDOC to PI, PI execute mapping and sending data to Message borker.(with almost one to one mapping)., IDOC(AAE)-PI-JMS. Now my requirement is., from PI after mapping we need to save file in SAP fold

I have a scenario, ECC-PI-Message broker. ECC sending IDOC to PI, PI execute mapping and sends data to Message borker(thru JMS channel).(with almost one to one mapping)., IDOC(AAE)-PI-JMS. Now my requirement is., from PI after mapping we need to create file with same data what ever send to Message broker and put the file in SAP folder without touching mapping. Is it possible? Please advise with the steps. We are using the ICO for this senario. Quick response is appriciated.

Hi Pratik,
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/502991a2-45d9-2910-d99f-8aba5d79fb42?quicklink=index&overridelayout=true
This link might help.
regards
Anupam

Upload data from excel (not the excle file) into SAP

Guys,
how can we upload data from excel sheet into SAP? I mean just the data not the entire file,
I have a requirement where user press a button in excel sheet and the data in the sheet will get uploaded into SAP.
I am sure we have to use BAPI and some VB programming for macros, I will really appriciate if anyone can help how to achive this.
some sample code exampe will help.
Cheers!

I think u r writing BDC for Uploading the data from excel flile to sap. for this is the code I am sending u can use then for Uploading data from excel to sap.
DATA: lv_filename TYPE rlgrap-filename.
FIELD-SYMBOLS : <fs>.
DATA : l_intern TYPE alsmex_tabline OCCURS 0 WITH HEADER LINE.
DATA : l_index TYPE i.
PARAMETERS : startcol TYPE i ,
      startrow TYPE i ,
      endcol TYPE i ,
      endrow TYPE i .
PARAMETERS: p_flnam LIKE rlgrap-filename.
AT SELECTION-SCREEN ON VALUE-REQUEST FOR p_flnam.
CALL FUNCTION 'F4_FILENAME'
EXPORTING
   program_name        = sy-repid
FIELD_NAME          = ' '
   IMPORTING
     file_name           = p_flnam .
MOVE p_flnam TO lv_filename.
Uploading the flat file from the desktop
START-OF-SELECTION.
CALL FUNCTION 'ALSM_EXCEL_TO_INTERNAL_TABLE'
    EXPORTING
      filename                      = lv_filename
      i_begin_col                   = startcol
      i_begin_row                   = startrow
      i_end_col                     = endcol
      i_end_row                     = endrow
    TABLES
      intern                        = l_intern
EXCEPTIONS
INCONSISTENT_PARAMETERS       = 1
UPLOAD_OLE                    = 2
OTHERS                        = 3
IF sy-subrc <> 0.
MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
        WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
SORT l_intern BY row col.
LOOP AT l_intern.
    MOVE l_intern-col TO l_index.
    ASSIGN COMPONENT l_index OF STRUCTURE itab TO <fs>.
    MOVE l_intern-value TO <fs>.
    AT END OF row.
      APPEND itab.
      CLEAR itab.
    ENDAT.
ENDLOOP.
I hope it will help u.
Regards
Nayan

Data Services XI3.1 function module files for SAP R/3 is not working

Hi guys,
Thank you for the quick response so far. I am very grateful to you all.
I got an issue and will try to explain as detail as I could and
hope ur guys won't mind. :P
I pass the functions module to SAP tech guy to install to SAP using CTS
method, and using 900086.R63 file type because my chinese client is
running SAP on unicode environment.
My DS installed on my laptop client; job server plus db2 is on HP
unix. SAP is on another HP Unix server. The function module I used is
supplied from Data Services XI3.1.
Anyway, the installation is successfully done with GUI wizard windows of
SAP Workbench instead of tp command line method.
1)However, I checked the ZAW0 function group, I opened and can't see any functions
listed under it, although the table structures ZACTA, ZTAB2048 and other items
were created.
2)So I tried to create SAP Datastore, Transport Target, etc and tried to
extract file. It failed and log said "can't open file ---
/db2/temp/curcode.txt". I checked and SAP working directory /db2/temp didn't have
curcode.txt, because the result file supposed to be extracted and stored there.
The SAP tech guy checked SAP and cant find generated ABAP program of
ZCURCD is running anyway, although my local CURCD ABAP program is
created.
3)Is it the 900086.R63 not complete? Is my function installation correct? I could view data
on datastore, but just cant run job to extract. Another non-unidcode 900200.S08 group
has larger file sizes. Should I try to install also this 900200.S08 files? Will it corrupt SAP
and cause system errors?
4)After that, I had tried to install the functions manually into ZAW0
one by one using cut and paste, however after that, I cant view the data view anymore with
error. Besides, I tried to run Check on the functions and they all returned
syntax errors, so I cant Activate any of them with SAP. Maybe dependency is missing??
Then, I deleted the manually installed functions and can view data again, but just cant extract
data by job.
5)Another side issue is all these SAP datastore creation and viewing is
through DI 11.5 old installation. The new Data Services X3.1 unable to
create datastore with database error, although the function module
installed is came from Data Services. Does anyone know the issue?
Thank you very much to read it patiently. :P

Hi,
Thank you for the informative link. It did help a lot in solving the problem.
(1) to (4) --- The Sap tech guy didnt install the function programs correctly, and he redo the transport again using CTS. Now I can see all the functions inside the function group ZAW0. While for the problem of cannot generate extracted file to SAP working directory for downloading, it is due to unix directory access permission setting. And also have to include the ftp username into SAPSYS user group as the help link suggested.
(5) For this problem, I reinstalled again the DS with all required components like Server Manager. The network technical guy helped me with the first installation, and the installation was not complete. So now the issue is solved, I can create R/3 Datastore with DS.

How to upload the data from excel(3 tabs) file to sap environment

Hi all,
This is Mahesh.
how to upload the data from excel(3 tabs) file to sap environment (internal tables) while doing bdc.

Hi,
The FM 'ALSM_EXCEL_TO_INTERNAL_TABLE' makes it possible to load a worksheet into an internal table in ABAP.
However, if you want to get the data from several worksheets, I think you are stuck with OLE access to your Excel Workbook...
You can find a solution for 2 worksheets in this post :
TO UPLOAD DATA FROM 2 EXCEL SHEETS INTO TWO INTERNAL TABLES
I think you can easily modify it to handle any number of worksheets.
Hope it helps !
Best regards,
Guillaume

Does /sapmnt need in cluster file system(SAP ECC 6.0 with oracle RAC)

We are going to be installing SAP with Oracle 10.2.0.4 RAC on Linux SuSE 10 and OCFS2. The Oracle RAC documentation states:
You must store the following components in the cluster file system when you use RAC
in the SAP environment:
- Oracle Clusterware (CRS) Home
- Oracle RDBMS Home
- SAP Home (also /sapmnt)
- Voting Disks
- OCR
- Database
What I want to ask is if I really need put SAP Home(also /sapmnt) on cluster file system? I will build a two nodes oracel 10g RAC and I also have another two nodes to install SAP CI and DI. My orginial think is sapmnt is a NFS share, and mount to all four nodes(RAC node and CI/DI), and all oracle stuff was on OCFS2(only two rac nodes are OCFS), anybody can tell me if SAP Home(also /sapmnt) can be NFS mount not OCFS2, thanks.
Best regards,
Peter

Hi Peter,
I don't think you need to keep /sapmnt in ocfs2 . Reason any file system need to be in cluster is,in RAC environment, data stored in the cache of one Oracle instance to be accessed by any other instance by transferring it across the private network and preserves data integrity and cache coherency by transmitting locking and other synchronization information across cluster nodes.
AS this applies to redo files, datafiles and control files only , you should be fine with nfs mount of /sapmnt sharing across and not having ocfs2.
-SV

How do I create rules files in Sap Data Services

Hello Guys,
I'm new with Bo Sap Development Data Services, and am migrating SSIS packages 2008 for the bods.
I am studying the process of cleanup and word processing via bods.
But I noticed that my installation does not have Rules Files.
My question is:
How do I create rules files in Sap Data Services library with support for Portuguese?

Hello Ramana,
So with version 4.2 of Sap Data Services do not currently have the rules files?
How to mount files for text comparison Cleanse facing Addresses, Names, Clients, Companies? If you can give me some help link?
Thanks again ...

Report in Excel format fails for huge amount of data with headers!!

Hi All,
I have developed an oracle report which fetches upto 5000 records.
The requirements is to fetch upto 100000 records.
This report fetches data if the headers are removed. If headers are given its not able to fetch the data.
Have anyone faced this issue??
Any idea to fetch huge amount of data by oracle report in excel format.
Thanks & Regards,
KP.

Hi Manikant,
According to your description, the performance is slow when display huge amount of data with more than 3 measures into powerpivot, so you need the hardware requirements for build a PowerPivot to display huge amount of data with more than 3 measures, right?
PowerPivot benefits from multi-core processors, large memory and storage capacities, and a 64-bit operating system on the client computer.
Based on my experience, large memory, multiprocessor and even
solid state drives are benefit PowerPivot performance. Here is a blog about Memory Considerations about PowerPivot for Excel for you reference.
http://sqlblog.com/blogs/marco_russo/archive/2010/01/26/memory-considerations-about-powerpivot-for-excel.aspx
Besides, you can identify which query was taking the time by using the tracing, please refer to the link below.
http://blogs.msdn.com/b/jtarquino/archive/2013/12/27/troubleshooting-slow-queries-in-excel-powerpivot.aspx
Regards,
Charlie Liao
TechNet Community Support

Can data in a Project 2010 file be synced with PWA timesheet data submitted against an un-published task?

I recently learned that best practice when you're mid-project and have a task that should no longer be charged to is to use the "Close Task to Update" option in Project Server 2010 instead of setting the "Publish" field to "No"
to remove the task from team members' timesheets because removal of task entries can lead to data inconsistencies.
Prior to learning this, one of these data inconsistencies was introduced into one of my project files. Specifically, I set the "Publish" status to "No" on completed task without realizing that a team member had submitted a correction
to a previous time period's timesheet that had not yet been approved. These corrected data are now shown in the Reporting Database, but they did not get carried through to the project file.
Is there a way to reconcile the data shown in the project file with the data in the Reporting Database?

Prasanna,
Here's what I think happened:
User submitted incorrect actuals against Task A on his timesheet. (In addition to the expected hours against the current period [TS1], he had accidentally submitted actuals against a future time period [TS2] that was not part of my current review window.)
Not realizing he had incorrectly reported his time, I accepted the actuals and published them to the project.
I updated the project, changing the Publish flag for Task A to No since no further work was expected against the task.
The following week, the user resubmitted a correct version of TS2.
The reporting database got updated with the new TS2 data.
Everything in my approval queue that was submitted by this user has been approved and published; however, the data in my project file does not match what I am seeing in the reporting database.
Do I just need to change the Publish flag for Task A back to Yes to get my project file to capture the changes that were made during the resubmission of TS2?

File to SAP with huge ammount of data

Similar Messages

Maybe you are looking for