How to seperate a big flat file

Hi Everyone,
I have a performance issue with loading a very big flat file. I have a 200 million flat file and I am trying to load it via Oracle Data Integrator to Sql server 2005. Oracle data integrator is trying to execute a bcp tool like below.
bcp CDWH_DW.dbo.Campaign_History_Outbound in "\\server\CDWH\FlatFile2Send \FILES\ailing_active.txt" -f"\\Server\Campaign\Campaign_Response_Outbound.fmt" -o"\\server\CDWH\FlatFile2Send\ \LOG\sony_mailing.log" -T -S"sqlserver " -C"RAW" -m 1
If I have a problem with data in this huge file it is very difficult to find what is wrong. So I am trying to find out a solution for this issue. Do you have a suggestion? I am thinking to separate this huge file a lot of small piece and loading them in a loop but I am thinking about how to separate this big file to a small piece.
Thank You for your Helps
Kind Regards
Caner Sahan

Hi Caner...
I believe that is possible.
There is any column, from text file, that can be used as PK?
What is the type of this text, delimited? positional?
Do you want this only for development or will keep it for production?
Cezar Santos
http://odiexperts.com

Similar Messages

How to convert IDOC to flat file in XI

Hello SDNers
I have a scenario IDOC --> XI --> Flat file. My question is how to conver IDOC to flat file. At least there are two ways to do that:
1) Define a flat structure in PI that reflects the IDOC structure, then using content conversion to convert the flat structure to flat file in hte receiver communication channel.
2) There is a [document |https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/46759682-0401-0010-1791-bd1972bc0b8a] talking about using a generic ABAP mapping to map any IDOC to flat file.
For the option 1, I don't want that way, because I would need to define a flat structure in PI for each IDOC which is too much effort. (I would have dozens of IDOC scenarios)
The second one is very attractive. However the document is not complete to follow. Basically it does not explain how the inbound flat structure looks like. I guess the generic ABAP mapping is used to map any IDOC_XML structure to a flat structure, then in the file receiver communication channel to convert the flat structure to flat file format. But the document does not mention how the flat structure looks like. Has somebody tried this scenario? If yes, can you please let me know the steps to do in ESR (IR) and ID? Basically what structure the IDOC_XML should be mapped to using the generic ABAP mapping class?
Or somebody has other ideas on how to convert IDOC to flat file?
Thank you so much
Eric

> 1) You are saying that for Java mapping and abap mapping, the target data type (messag type) is just a dummy one, the real (XML) message the receiver will recieve is the one (stream) produced by the java or abap mapping. Is that correct?
Definitely yes!
>
> 2) For the abap mapping option, I would define a dummy target data type. Do I need to do any content conversion in the receiver adapter (communication channel)?
None wat so ever.The output of your Java Mapping will be the Native Idoc Format.
>
> 3) I tried with a dummy target data type as mentioned in my second post, can you see any clue in the error message what is wrong?
Dats one thing I cant help much. Just make sure that the ABAP report is valid and take the help of a ABAP'er to debug what is going wrong. Not much of a ABAP guy, can read ABAP code, but cant write one myself
Try to test the program standalone to see how it works. Maybe take some Idoc XML file as input and then dump the output to a .txt file and so on.
Regards
Bhavesh
PS : All this is from what I have read of this guide. I have not convert Idoc XML to Native Idoc but have used this guide to convert Native Idoc to Idoc XML .

How To Convert IDOC to FLAT File ?

Dear Expert,
        My requirement is to convert the IDOC to FLAT File using XI. How can i do this. I have gone thru the Guide How to convert IDOC to Flat file using ABAP mapping but it does not talk about what are all the stpes i need to do in IR & ID.
I am ready to use ABAP mapping since i am an ABAPER.
         Is it recommended to use ABAP mapping since we expect daily 500 IDOCs to be converted to a flat file & transfer this flat file to some FTP location.
         Can anybody tell me the stpes how to do this in XI. The steps i am expecting like how to define the Data Type / Message type / Mapping etc & how to configure the channel ?
         The Flat file i want to convert should look like the way IDOC file gets created in FILE FTP port in SAP.
Regards,
Umesh

I think there is some confusion... Let me explian my problem once again.
I want to convert the Shipment IDOC as it is the moment the shipment documents gets created in R/3. I dont want to collect many IDOCs & make as single file. For each IDOC it should create a Flat File.
Secondly if i have to MAP the whole IDOC the graphical mapping is too difficult.
So instead of that can i use ABAP/XSLT/JAVA whichever is easy. ?
Since i am ABAPER i can use ABAP mapping.
The Output file should look like as below.
EDI_DC40 9000000000011785526620 3012 OILSHI01                                                    OILSH1                                           SAPP01    LS SAPCLNT900                                                                                A000000018LS WBIMQSI                                                                                20071031161506                                                                                20071031161506
E2OILSH001                    90000000000117855260000010000000250 1011537344 1101                1 300X1 TRKGMKG 0202KG KG X        ZTLF X XX 000   BLX X        X60002200710310000002007103100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000        E2OILSA                       9000000000011785526000002000001031011537344000001SP          0000201496000000000000000000                                           00                                        00 V                                                                                E2OILSA                       9000000000011785526000003000001031011537344000002SP          0000201496000000000000000000                                           00                                        00 S
E2OILSV                       900000000001178552600000400000103MH04CG6052        LPBW50D62401033       000020149601KG KGM   35200.00000             17200.00000             18000.00000                               0.000          0.000          0.000          DLLPGB220071031000000 00000000000000000000000000000000000                    0000XX    6000AAAA0.00000                 0.00000                 17200.00000             35200.00000             18000.00000             MH04CG6052 new bulk tender approved rs                                          LPG Bulk Lorry - WR                     Refinery TPP                                                                                BABA TRANSLINES                     kg        kg        0.000             0.00           0.00                                                                                E2OILSC                       9000000000011785526000005000004041 001                                                 18000.000000           18000.000000           35200.000000           0.000000               0.000000                  KG                                                                                E2OILSQ001                    9000000000011785526000006000005050011 MH04CG6052MH04CG60521J          05041089055000                        00000118000.0000000 KG 18000.000000           18000.000000           6000V04 18000.000      KGM18000.000      KGM00000000 V        0.000000                  0.000000               0.000000                  0.000000                           0.000000               0.000000               0.000000               0.000000                          0001                                                                                E2OILS2                       9000000000011785526000007000004040001000001000200020000.000000               0.000000               0.000000               0.000000               0.000000               0.000000               0.000000               0.000000               0.000000               XX0000000000000000000000000000 XX0504108905J5000              PUNE LPG BOTTLING PLANT            START     0001 1END       0001 10.000000               0.000000               0.000000               0.000000               0.000000               0.000000
Pl. Help me its Urgent.
Regards,
Umesh

How to load a long flat file field into BW

Hi all,
I have a field from flat file with 256 characters to be loaded into BW. As we all know the max length an info object can take is only 60. How can I split the flat file field so that I can load that data into BW and how can I then concatenate them in the BEx.
Can anyone suggest a way?
Thanks in advance.
Vinay

Hi Vinay,
we already had this discussion in this forum. In short:
1. create a number range object
2. create a table with a number field a key and a text field
3. while loading the data check your text against the table. If the text already exist, get the number out of the table. If the text doesn't exist, get a new number out of your number range object and post this number + the text into your table.
4. in your loaded data always work with the number you assigned to the text
5. create a infoobject for the number-field
6. create a variable that takes it's value out of the table.
regards
Siggi

How to find that which flat file is belongs to which application server?

How to find that which flat file is belongs to which application server?

Hi,
Search im www.sap-img.com, you will get most of answers.
Jogdand M B

Build daily Big flat file using SAP R/3 material master data

Hi all,
I must build an interface between SAP R/3 4.7 and an external application
Daily a flat file must be generated contaning Material data in full exchange mode
(volume 100 000 materials, <10 fields => ~ File of 30Mo)
No complex mapping, only format mapping (Ex : Price quantity...)
I don't know yet if I will use Idocs or BAPI to extract data. However, I'm sure I must use multiple inbound messages because of volume
After reading documentation, I consider the both following solutions to manage it :
1) A simple asynchronous interface with a file adapter in construction mode APPEND to build flat file progressively with arrival of SAP messages
=>
viable solution ?
Is there problem of concurrent access to file in such a solution ?
(idocs or bapi data will be processed ones following the others )
2) BPM Collecting and Bundling Messages - One Interface
(Unlimited Loop with Deadline trigger correspondint to time I must build flat file)
=>
Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Which solution do you advise ?
Is there another ?
Last question : (more general one)
In such scenario, I can't see XI-value added comparing a creation of flat file directly with R/3.
I will generate a big number of idocs, use a big part of XI memory ...
Can you give me your opinion about it ?
Thanks a lot
Best regards
Jean-Charles
Message was edited by:
Jean-Charles
Message was edited by:
Jean-Charles

Hi Jean-Charles,
>>>volume 100 000 materials, <10 fields => ~ File of 30Mo
You have a performance task...
>>>Is there problem of concurrent access to file in such a solution ?
No, the adapter framework has its own queueing, the message would be stored and executed later. If sequence is important, you can use EOIO as well.
>>>BPM Collecting and Bundling Messages - One Interface
BPM is not required in your case, you should avoid becoz of performance task. You can append at file adapter, no problem. Bundling and splitting is even possible at Integration Server without BPM.
>>>Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Avoid using BAD words
>>>Each SAP message is a small message but the outbound XML will represent a 30 Mo flat file => Is it a problem ??
Use the <a href="https://websmp110.sap-ag.de/quicksizing">Quick Sizer</a> for sizing the hardware.
Regards,
Udo

How can we load a flat file with very, very long lines into a table?

Hello:
We have to load a flat file with OWB. The problem is that each of the lines in the file might be up to 30.000 characters long (up to 1.000 units of information in each line, 30 characters long each)
Of course, our mapping should insert these units of information as independent rows in a table (1.000 rows, in our example).
We do not know how to go about it. We usually load flat files using table functions, but we am not sure that they will be able to cope with these huge lines. And how should we pivot those lines? Will the Pivot operator do the trick? Or maybe we should pivot those lines outside the database before loading them?
We are a bit lost. Any suggestion would be appreciated.
Regards
Edited by: [email protected] on Oct 29, 2008 8:43 AM
Edited by: [email protected] on Oct 29, 2008 8:44 AM

Yes, well, we could define a 1.000 column external table, and then map those 1.000 columns to the Pivot operator… perhaps it would work. But we have been investigating a little bit, and we think that we have found a better solution: there is a unix utility called “fold”. This utility can split our 30.000 character lines in 1.000 lines, 30 characters long each: just what we needed. Then we can load the resulting file using an external table.
We think this is a much better solution that handling 1.000 columns in the external table and in the Pivot operator.
Thanks for your help.
Regards
Edited by: [email protected] on Oct 29, 2008 10:35 AM

How to create security for flat files under a Webserver Directory?

Hello,
I have an Application where I need to show a Flat Excel File (with open/save option) from a URL from Portal.
This works fine.
But a user can view this file without login to Portal.
Though we restricted the directory, file access is uncontrolled.
What should we do?
Thanks
Madhav

I mean, how to enforce Login to Portal on flat files in the Web Server Directory(if the users are accessing directly from URL), they should be accessible as long as they logged into Oracle Portal. But right now, they can access by typing the URL for this flat file directly into browser.
Thanks for your time
Madhav

How to Delta Update in Flat Files

Hi all,
I would like to know is delta update is supported in flat files ? If yes how does this work ..... is there a SAP white paper or document on this
Thanks
Nathan

Hi......
Yes Delta upload is supported by Flat files..........In this case........when the delta data records are loaded by flat file......... This delta type is only used is 'F'.......for DataSources for flat file source systems. Flat files are imported to the PSA when you execute the InfoPackage. The staging process then begins.........The delta queue is not used here.
As Siggi already told in the Following thread......post your flatfile data to an ods object with update mode overwrite. The ods object will do the delta for you then.........
Delta Upload for Flat File
Also Check this.......
SAP NetWeaver 7.0 BI: Data Transfer Process with Only get Delta Once
SAP NetWeaver 7.0 BI: Data Transfer Process (DTP) / The Normal DTP ...
Hope this helps........
Regards,
Debjani........
Edited by: Debjani Mukherjee on Nov 12, 2008 9:40 AM

How to remove CR from flat file in Windows

One of the columns in my table is varchar2(2000) and contains multi-line data.
New lines are identified by the linefeed terminator (chr(10)) in the table.
I am writing all records from this table to a flat file. My machine is running Win XP.
I am using UTL_FILE.PUT procedure to write to the file.
Problem is, for multi-line data each line gets appended with a <CR> in addition to the linefeed.
So each line in the file ends with <CR><LF> as opposed to only <LF> in the table.
So my question is, how do I get rid of this extra <CR> or chr(13) in the file ?
Is there anything I can set in the UTL_FILE package or alternatively is there any
DOS utility which can strip the file of the chr(13) ?
I understand this is Oracle default behaviour while writing to file on Win platform, i.e.
using a <CR><LF> as a line terminator. How can I change this behaviour.
Please help, very urgent !!
Regards,
Azhar.

Hi Kishore,
Thanx for the response.
What am trying to do is dump the table in a flat file.
Then create an External table (exactly same as original table) pointing to the flat file.
Due to occurence of the extra <CR> column length increases. For eg. multi-line column data of varchar2(2000) in original table having say 43 lines becomes 2042 characters now. As a result I will have to make external table with column length varchar2(2042) to read the column !! that's the problem.
I want external table definition to be exactly the same as original table.
This is part of a generic utility which am developing.
The only option I have right now is to use unix2dos utility and strip the <CR> characters, but I am looking for something which can eliminate <CR> at file creation itself.

How to load data thru flat file in demantra?

Hi,
I am using seeded data model for loading data for DM module in demantra.
As per data model there is 3 staging tables for item, location and sales_data i.e. t_src_item_tmpl,t_src_loc_tmpl,t_src_sales_tmpl respectively.
I have seen the final table for sales data i.e. sales_data table. ther is only one item_id and location_id column against which sales_date and quantity is loaded.
I have same sales_data and quantity to be loaded for different location hierarchy and item hierarchy .
How the data file is created ? if you have any sample data files , can you please share with me ? If any changes needs to be done in data model for loading datardsa in custom hierarchies ?
Regards

Welcome user10979220 to the wonderful forum.
"How many rows will be there in item flat file ?
How many rows will be there in location flat file ?
How many rows will be there in sales data flat file ?"
Answer to these questions are based on the number of sales that has occured as well as the item hierarchy and the product hierarchy.
From the example
Item_name | organization | ship to site | Sales date | quantity
item 1 org1 site1 19-Jan-2008 80
item1 org1 site2 20-Jan-2008 100
I can say that the number of rows in sales data is 2 while that in item file is 1 and location file is 2.
The answer to how the flat file will look is
Item file can have
Item_Id|Item_Name
01     Item1
Location file can have
Org_Id|Org_Name|Site_Id|Site_name
O_01     Organization 1     S_01     Site 1
O_01     Organization 1     S_02     Site 2
Sales file can have
Item_Id|Item_Name|Org_Id|Org_Name|Site_Id|Site_name|sales_date|Quantity
01     Item1     O_01     Organization 1     S_01     Site 1     19-Jan-2008 80
01     Item1     O_01     Organization 1     S_02     Site 2     20-Jan-2008 100
Once you define the data model and load the data it is advisable not to change the data model since the next time when you load the incremental data it will error out due to the mismatching columns. So its better that you define the data model first and then create the files accordingly.
Hence after loading data if you want to view sales data at the product_category1 level then you should already have the product_category1 level defined in the data model.Else You have to create the product_category1 level and associate it in the item hierarchy.Also the sales file should have columns for product_category1.
Then the item file will look like
Item_Id|Item_Name|Product_category1_Id|Product_category1_desc
01     Item1     PC_01     Product_category 1
and the sales file will look like
Item_Id|Item_Name|Org_Id|Org_Name|Site_Id|Site_name|sales_date|Quantity
01     Item1     O_01     Organization 1     S_01     Site 1     19-Jan-2008 80
01     Item1     O_01     Organization 1     S_02     Site 2     20-Jan-2008 100
This is just a basic overview about loading the data into the staging tables.In short the item table should have the item hierarchy in it, the location file should have the location hierarchy in it and the sales file must contain the details about the sales at the lowest item-location level.
Hope I answered your query.
Thanks and Regards,
Shekhar

How to create Fixed Length Flat File from Open Hub in BI 7.0

My requirement is to produce a Fixed length Flat file by Open Hub destination. My Open Hub has four fields. Now the requirement is to create another extra field in Open Hub which will contain all of the four fields value. In addition to that the fields should be fixed length that means if for any field no value is there and the field length/type is CHAR4 then 4 spaces should be there as the field value. SO, basically the Open Hub output will be single field which will contain information of four fields.
How to get this using End Routine of Transformation (from DSO to Open Hub) ?

Hi,
You can map the four input fields to the new field in the Open Hub, and change rule type to "Routine".
For example, if your source fields are called "first", "second", "third" and "forth", the ABAP routine could be similar to:
DATA: l_t_1 TYPE C LENGTH 4,
      l_t_2 TYPE C LENGTH 4,
      l_t_3 TYPE C LENGTH 4,
      l_t_4 TYPE C LENGTH 4.
IF source_fields-first IS INITIAL.
l_t_1 = '    '.
else.
MOVE source_fields-first TO l_t_1.
endif.
IF source_fields-second IS INITIAL.
l_t_2 = '    '.
else.
MOVE source_fields-second TO l_t_2.
endif.
IF source_fields-third IS INITIAL.
l_t_3 = '    '.
else.
MOVE source_fields-third TO l_t_3.
endif.
IF source_fields-forth IS INITIAL.
l_t_4 = '    '.
else.
MOVE source_fields-forth TO l_t_4.
endif.
CONCATENATE l_t_1 l_t_2 l_t_3 l_t_4 into result.
In the example, the program uses four blank spaces if any of the fields has no value.
Additionally, if non-initial values in input fields could be shorter than 4 characters (if the input fields have no fixed length), you could use STRLEN to evaluate if it is necessary to add blank spaces to complete the fixed length of 4.
I hope this helps you.
Regards,
Maximiliano

How to Create Hierarchy From Flat file Structure

Hi Gurus,
There is a scenario for me regarding the Hierarchy.
Required Hierarchy structure - Region>Director>Manager-->Sales id
I have flat file which gives the info like user id , sales id , manager id, director id.
But the transaction data Flat file has structure with sales id, region id, sales amt, sales qty.
Note : Region id is another Master Data.
How i can create hierarchy from the first flat file which doesnot have region info in that but it is available in the transaction data Flat file.
Is there anyway we can create hierarchy based on the first Flat file structure which contains more that 1,00,000 records.
Try to Suggest me in this regard .
This is urgent.
Regards,
Mano

Hi Mano,
                Defining the source system from which to load data
Choose the source system tree File ® Create.
       2.      Defining the InfoSource for which you want to load data
Optional: choose InfoSource Tree ® Root (InfoSources) ® Create Application Components.
Choose InfoSource Tree ® Your Application Component ® Other Functions ® Create InfoSource 3.x ® Direct Update.
Choose an InfoObject from the proposal list, and specify a name and a description.
       3.      Assigning the source system to the InfoSource
Choose InfoSource Tree ® Your Application Component ® Your InfoSource ® Assign Source System. The transfer structure maintenance screen appears.
The system automatically generates DataSources for the three different data types to which you can load data.
○       Attributes
○       Texts
○       Hierarchies (if the InfoObject has access to hierarchies)
The system automatically generates the transfer structure, the transfer rules, and the communication structure (for attributes and texts).
       4.      Maintaining the transfer structure / transfer rules
Select the DataSource for uploading hierarchies.
IDoc transfer method: The system automatically generates a proposal for the DataSource and the transfer structure. This consists of an entry for the InfoObject for which hierarchies are loaded. With this transfer method, the structure is converted to the structure of the PSA during loading, which affects performance.
PSA transfer method: The transfer methods and the communication structure are also generated here.
       5.      Maintaining the hierarchy
Choose Hierarchy Maintenance, and specify a technical name and a description of the hierarchy
Hope this helps
Regards
Karthik
Assign points if Helpful

How to Upload the excel flat file in LSMW

Hi all,
Can any one plz Help me,
How to Upload a excel file format by suing the lsmw .
Regadrs.
[email protected]
Upender..
Edited by: Upender Reddy on Sep 12, 2008 8:37 AM

Hi,
In 7 th step of LSMW we need to specify the file ie flat file.
In this step you describe all files to be used in the following steps:
· Your legacy data on the PC and/or R/3 server
· The file for the read data
· The file for the converted data.
As your legacy data is on the PC: ie your excel file that is tab delimited or comma delimited.
here give ur file name, description , click data from one src structure, then delimiter is tabulator if so, in file structure click both check boxes if in ur flat file ur first row is column, file type is record end indicator, code page ASCII and conitnue that's it.
you can go through this link very useful :
****************/Tutorials/LSMW/LSMWMainPage.htm.
thanx.

How to read an IDOC flat file and send it to XI

Hi,
I'm looking for a way to read a complicated Idoc structured flat file sent to XI by a legacy system and then pushing that data to and R/3 system. Can anyone recommend a good technique?
Thanks,
Steeve

hi steven....
this shud help u...
for FileXIIDoc
File->XI->IDOC (how to start?)
also check this blog for specifying adapter specific identifiers in this scenario
These are steps to follow:
STEPS IN DESIGN
1. Import idoc from the SAP System (SAP02) through imported objects.
2. Create Data Type, Message Type, Message Interface for the Outbound File.
3. Create Message Mapping with file Message and idoc (Matmas04)
5.Create Interface mapping between File Interface and the idoc.
Steps to be followed in Creation of Technical and Business System
Technical System
In SLD, click on technical landscape->New Technical System->WebAS ABAP.
Business System.
1.Goto Business Landscape->New Business System->Name the business system.
2. Choose the technical system that needs to be attached->WebAS ABAP->Technical System Name.
Preconfiguration steps for transfer of data to IDOCS
1.Create a RFC Destination in SM59 of XI Server of type.
2.In transaction IDX1 ,create a port for the client of XI server.
3. The port has to be displayed in the idx2 transaction and we should be able to view the idocs which we imported through integration repository.
Steps in Configuration
1. Goto service without party, right click->Assign a business system.
2. Choose the business system to be added.
3. Create a File Sender Service with communication channel for the same.
4. Create an idoc receiver channel under the business system. Mention the RFC Destination and the port created for the same.
5. Mention the logical Name in the Service->Adapter Specific Parameters of both the File Sender Service as well as the business system.
6. Make the Sender Agreement for theFile Sender with communication channel.
7.Create the Receiver Determination and Interface Determination for the business system.
8.Create the Receiver Agreement for the Business system.
TCODES for File>XI>IDoc
SM59-rfc destination on recving side
IDX1-port on recving side
IDX2-load metadata
Bus. service adapter specific Identifiers-->Logical sys name (check this blog:/people/michal.krawczyk2/blog/2005/03/29/xi-error--unable-to-convert-the-sender-service-to-an-ale-logical-system )
we05-IDoc status records
BD54-Create Partner Number
We20-Create Partner Profile
and also go thru this blog...
/people/anish.abraham2/blog/2005/12/22/file-to-multiple-idocs-xslt-mapping
hope it helps....
regards...
vishal
/people/michal.krawczyk2/blog/2005/03/29/xi-error--unable-to-convert-the-sender-service-to-an-ale-logical-system

How to seperate a big flat file

Similar Messages

Maybe you are looking for