Process Flat Files

Hi everyone,
I currently need to process about 400 txt files, equivalent to about 1TB of data in maximum 5-6 hours. I need to run a very simple scrip, where I would capture about 0.10% of the data. Something like a select function with some where clause. Originally I
was thinking to import this data in a SQL server however I'm not sure my computer could handle the workload in a reasonable time frame.
What would you suggest? Is there any other products that Microsoft offers to process extremely large flat files?
Thank you in advance.
Cheers,

Hi AlexB0865,
The solution you have mentioned should be the best one in my opinion. The performance bottleneck in this case are:
Load data from 400 text files, which is about 1 TB large. Each file should be about 2.5GB
Bulk insert data to SQL Server
Filter data from the 1TB data
For the first chanllange, it won't be an issue in SSIS per my testing. I have a file with 23504761 rows, which is 2.5GB large. This file can be imported into a SQL Server table within 1 minute.
For the second chanllange, we can split the bulk insert to improve the performance.
For the third chanllange, with proper index(es) created, the performance won't be a problem.
Henk details the testing result regarding this same scenario in blog:
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
If you have any more question, please feel free to ask.
Thanks,
Jinchun Chen

Similar Messages

How to Process flat File in Oracle Apps through Concurrent Program

Hello Everyone,
My client has a request, to process a bank file (Lockbox) which is a flat file that will be copied on UNIX box and I will have to create a new concurrent request that will process this flat file and will update receipt information in Oracle Apps database tables.
Could you please suggest, if there are any other standard Oracle Apps functions (Example FND) available which can be used through Concurrent program that can be used to open a file from a particular directory and can be read from the flat file and after processing this file can be closed.
Please let me know, if you have a small example, that would help me a lot.
Thanks

There are base concurrent programs in Accts Receivable that do consume lockbox flat files. Pl see the AR Setup/User Guides at
http://download.oracle.com/docs/cd/B40089_10/current/html/docset.html
Srini

Unable to process flat-files without delimiters in ODI 11.1.1.7

Hi,
We have a requirement to process a flat file using ODI 11.1.1.7 (installed on Windows and mounted on Oracle 11g Database). The flat-file is a normal ASCII file coming from mainframes machine and has no delimiters (column as well as rows). The first 48 characters is ROW 1, from character 49 to 96 is ROW 2 and so on. I am unable to create a DataStore for this file in ODI. If I reverse engineer using the file RKM, it is creating a datastore with over 2000 columns which is not the case (actual file layout given below). The absence of a row delimiter is making ODI think that the entire file has only one single row.
I am also unable to create a DataStore manually. If I am not providing any value for the "row delimiter" it is throwing me an error. However, I am able to process this file using SQL*Loader (there is a FIX option available where we can specify the length of a row).
Is there a way to do it from within ODI? I tried to modify a KM to create a control file for SQL*Loader and execute it but the control file is not getting generated as expected.
File Layout:
Column 1 - Warrant Number (Position 1 - 9) - Number
Column 2 - Type (Position 10-10) - Number
Column 3 - Warrant Amount (Position 11-18) - Packed Decimal Signed
Column 4 - Issue Fund (Position 19-22) - String
Column 5 - Issue Sub Fund (Position 23-24) - String
Column 6 - Filler (Position 25-48) - String
Thanks
Srivatsan P

Hi LuizFilipe,
I tried also with your method but i am getting below error when i am trying to view data.
See com.borland.dx.dataset.DataSetException error code: BASE+62
com.borland.dx.dataset.DataSetException: Execution of query failed.
Chained exception:
java.sql.SQLException: ODI-40439: Could not read heading rows from file
at com.sunopsis.jdbc.driver.file.FileResultSet.<init>(FileResultSet.java:164)
at com.sunopsis.jdbc.driver.file.impl.commands.CommandSelect.execute(CommandSelect.java:57)
at com.sunopsis.jdbc.driver.file.CommandExecutor.executeCommand(CommandExecutor.java:33)
at com.sunopsis.jdbc.driver.file.FilePreparedStatement.executeQuery(FilePreparedStatement.java:135)
at com.borland.dx.sql.dataset.o.f(Unknown Source)
at com.borland.dx.sql.dataset.QueryProvider.e(Unknown Source)
at com.borland.dx.sql.dataset.JdbcProvider.provideData(Unknown Source)
at com.borland.dx.dataset.StorageDataSet.refresh(Unknown Source)
at com.borland.dx.sql.dataset.QueryDataSet.refresh(Unknown Source)
at com.sunopsis.graphical.frame.edit.AbstractEditFrameGridBorland.initialize(AbstractEditFrameGridBorland.java:628)
at com.sunopsis.graphical.frame.edit.AbstractEditFrameGridBorland.<init>(AbstractEditFrameGridBorland.java:869)
at com.sunopsis.graphical.frame.edit.EditFrameTableData.<init>(EditFrameTableData.java:50)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at oracle.odi.ui.editor.AbstractOdiEditor$1.run(AbstractOdiEditor.java:176)
at oracle.ide.dialogs.ProgressBar.run(ProgressBar.java:656)
at java.lang.Thread.run(Thread.java:662)
Well i created file datastore with selected below things
File format : fixed
Heading(number of lines) : 0
field separator : null
record separator : by default --> MS-DOS
text delimiter: (blank)
decimal separator : (blank)
then when i did reverse engineering, able to see the row of data and then i separate the data with c1,c2,c3,c4
please let me know did i miss something.
Thanks
Himanshu

Can SAP XI AS2 Adapter process flat files? Pl. tell me how to configure it

I have a requirement to read and write delimited ASCII flat files using the AS2 adapter in SAP XI. The AS2 adapter should be able to read from delimited ASCII Flat files and also write to delimited ASCII flat files. Kindly let me know how to configure the AS2 adapter so that it can read and write delimited flat files. I badly need to know this configuration for a client requirement which requires file processing by the AS2 adapter in XI 7.0. Thanks in advance

Hi,
Is the requirement is to just read the flat file and process it ? if so, u can use File Adapter provided , ur source system provides u the data in the file format in the some directory
following thread discussions may help u -AS2 adapter-
Seebuger AS2 adapter for XI as sender for multiple messages
Seeburger AS2 Adapter
Hope this helps,
Regards,
Morthy

Processing flat file within ODI

Friends,
I need to read a flat file consisting of 'detail or main record' and 'trailier Record
1) Detail Record consist of each transaction details including amount etc.
2) Trailer Record consist of total value on detail records ( number of records and total amount)
How can I verify the total amount in trailer against sum of all detail records?
Thanks
KS

hehehehe
Was a "bad way" to say "execute the calculations".
An way to do it is:
1) create a ODI variable
2) at refresh tab put something like
select t.col_trailier_sum_amount - sum(d.col_amount)
from loaded_tabledetail d, loaded_table_trailier t
where d.record_type = 'detail' and t.record_type = 'trailer"
group by t.col_trailier_sum_amount
* I'm using 2 tables at this example but is the same with just one...
3) create a package and drag and drop:
a) the interface that load the temp table
b) the variable in Refresh mode
c) the variable in evaluate mode ( set the evaluate operator to "=" and value to "0")
d) choose the OK (true) and KO (false) path that you need to each case
Does it make sense to you?

Idoc data in flat file structure

Dear Experts,
We have idoc data in flat file structure. We need to fetch it using ftp and map it to an idoc.
please tell me how to proceed .
Thanks,
Aju

Hi,
For flat file you need to use the File content conversion parameters.
Refer the blog,
SAP Network Blog: How to process flat files with multiple documents like POs, SOs etc. in a File to IDoc scenario
How to process flat files with multiple documents like POs, SOs etc. in a File to IDoc scenario
Thanks
Swarup

Error while creating process chain for flat file loading

Hi All,
I had Created a process Chain to load Transactiion data(Full load) form flat file which is in my computer.
start>Load>DTP>DeleteIndex>DTP loading CUBE--> Create Index
but the system is throwing an error as "An upload from the client workstation in the background is not possible."
I dont know why this error is coming?
Can some one help me
Regards
Mamta

Hi Mamta,
Basically if you want to load the DS through FF using process chain, the FF has to be placed in Application server. We cant load the FF when it is located in the client local workstation(FF On your PC).
So better you remove the Infopackage step from the PC. Load the IP manually. Once it is completed you can start the process chain with the following steps:
start>DTP>DeleteIndex>DTP loading CUBE> Create Index
Hope it is clear & helpful!
Regards,
Pavan

Flat File automation process - limitations

Hello Everyone,
I would really appreciate any insight on the process improvement suggestions.
Background:
Currently we have around 12 territories providing a new flat file with new data on a daily basis depending on the business activity. Which would also mean that, on a given day if there is no activity would mean no flat file provided to BI for loading process.
The flat files provided need to be loaded into the BI system (PSA - DSO - InfoCube).
The flat file loading process has been automated for the daily file by implementing the logical file name for each territory.
1. The process variant in the process chain is to ensure if the flat file is available on the App server (Custom ABAP program).
2. 12 InfoPackages have been created to pick the data from the flat file on the app server and load the data over into the PSA.
3. All the InfoPackages merge into an "AND" event in the process chain before the DTP load into the DSO kicks off.
4. DSO Activation
5. Recon between the flat file and the DSO to ensure all the data from flat file has been loaded into the DSO.
6. DTP loads into the InfoCube.
7. Recon between the InfoCube and the DSO itself.
8. Moving the flat file from one folder into another.
All the above processes are automatically performed without any issues if the flat file is available on the server.
Problem / Issue:
As one of the major limitations of the above design is the flat file for sure needs to be made available on the app server in order for the whole data flow in the process chain to continue without any breakpoints.
Current workaround / process improvement in place:
Based on the above limitation and upon further research, I was able to apply the OSS Note to give us the option of maintaining multiple DTPs for the same data target with different filter values.
So, even if have individual data stream for each territory with a different DTP the issue still remains where the process variant (ABAP program to check if file exists) or the InfoPackage load if the ABAP program is removed will fail.
Which means due to the above fact, the support team is alerted about the process chain failure.
Question / Suggestions required:
The main questions or any suggestions would be welcome, if one of you can let us know an approach where the flat file check program doesn't have to give a hard failure in the process chain for the rest of the process chain to continue with the loading process. (As in order for the rest of the process chain to continue the only options we have are Error, Success, Always).
I have also looked into the Decision process variant available in the process chain, but based on the options available within I cannot utilize it for the loading process.
Error can be caused by generating an error message in the ABAP program which in turn is causing the issue of alert being sent over even if the rest of the process chain finishes.
Success would mean the flat file needs to be available. Always cannot be used in this case as it will cause a failure at the InfoPackage level.
If the InfoPackage load can be avoided without a hard error to be generated, the process chain does not have to remain in the failed state which in turn will not trigger any alert to the support team.
Please do let me know if you need more details about the above process improvement question.
Thanks
Dharma.

The main issue with this as you mentioned is that the file has to be available for sure.
We had a similar issue - we had a very critical data load that had to happen everyday , failure of the file getting loaded would mean that the reports for the day would be delayed.
We were running on UNIX and we implemented a simple UNIX command that would not complete till the file was available in the app server.
Something like
while ( the file does not exist )
delay of 15 seconds
you will come out of the while only after the while completes which means that the file becomes available.
You can write a similar ABAp program to check file availability if required and put it into your program.
we also had a failover process where we created a zero byte file with the same name if the file did not come beyond a certain number of tries and the PSA would load zero records and the data load will continue.
Edited by: Arun Varadarajan on Jan 26, 2009 10:18 AM

Error while trying to process multiple Recordsets in Flat file.

Hi All,
I am working on Flat File to Flat File scenario and my structure is as follows.
Recordset
     Record1
          Field1
          Field2
     Record2
          Field3
          Field4
     |
     Record9
          Field5
          Field6
     I am going to receiver multiple Recordsets from my input and need to pass them as output flat file after doing some manipulations in mapping(I am using Java mapping).
     In Moni I am able to see multiple Recordset XMLs created but the message is failing in receiver communication channel with error
"Failed to process message content. Reason: Exception in XML Parser (format problem?):'java.lang.Exception: Message processing failed in XML parser: 'Conversion configuration error: Unknown structure '' found in document', probably configuration error in file adapter (XML parser error)' (Software version: 3.0.1)"
     When I am trying to pass Single Recordset I am able to see the output, but when I am trying with multiple Recordsets it is throwing error.
     Can anybody help me in finding the root cause to this problem.
     My Receiver channel Content conversion is as follows.
     RecordsetStructure: Record1,Record2, -- - - ,Record9
          Record1.fieldFixedLengths
          Record1.fieldNames
          Record1.endSeparator so on till Record9
Regards,
Jayaram.G

You might want to check the following things
Are u specifying field names,separators for Record1,Record2..Record9.
Is you occurence repeats after record1..record9 again?
Change your structure occurence as per the runtime data you provide..
See whether your java mapping modifies the structure that does not match with fcc configuration. You might want to pay attention over there too.

Idoc to flat file using Integration process does not show up at all

Hello All,
I created a scenario IDOC to Flat File using Integration Process and one without IntegrationProcess. I am able to execute the one without Integration Process easily.
However when I send the IDOC from SAP then the Integration Process scenario just does not work.
I Checked SXMB_MONI There is no entry for the Integration Process.I can see entry for other without Integration Process.
SXI_CACHE : I can see my Integration Process active
SWEQADM : I have activated the Event Queue and Background job
SWF_XI_ADM_BPE : ALL BPE is showing green status.
All syntax check on IP are fine.
There is no Basis Help for me so there could be some additional basis steps which need to be configured. Has anyone faced such an issue with Integration Process?
Regards
Prashanta

Hope this might help you. when u configure an IDoc as Sender with BPM this is how you proceed.
1. let us split the scenario into two.
a. Sender to BPM ( Integration Process).
b. BPM to Receiver.
2. First let us configure the first Sender to BPM.
a. Sender Agreement( Not required ).
b. Receiver Agreement( NOT Required).
c. Receiver Determination( Sender Service, Outbound Interface, Receiver Service(Integration Process).
d. Interface Determination( Inbound Interface(Sender Abstract Interface), Message Mapping(NOT Required if your using transformation stepin Integration Process)).
3. Now let us configure the Second BPM to Receiver.
a. Sender Agreement( NOT Required)
b. Receiver Agreement( Sender Service(Integration process), Receiver Abstarct Interface,Receiver service, Inbound Interface).
c. Receiver Determination( Sender Service(Integration Process), Receiver Abstarct Interface,Receiver Service).
d. Interface Determination( Inbound Interface, Message Mapping(NOT Required)).
So Finally we have One Receiver Agreement, Two Receiver Determinations, Two Interface Determinations.
Please let me know if you have any more queries.
Regards
Gopi

Error Processing a line in Flat File ActiveSync

We are using IDM 5.5 running on Websphere 6.0.2.3.
We are feeding a caret seperated flat file to IDM which in turn writes to Sun One Directory Server.
A sample format of the caret seperated file is
ssocorrespondlanguage^ssouid^mail^cn^sn^givenname^appaccess^uid^ssostatus
^A1C1C423-71F0-138F-CB8C-BCC3BAAF484F^[email protected]^Luigi Marra^Marra^Luigi^portal^[email protected]^A
^3DCC1E95-4E7D-113A-BFC0-AAA3BA195331^[email protected]^adhi asokan^asokan^adhi^portal^aadhees^A
We are facing an issue where IDM is not able to process a line.
I suppose its because of the "@" or "." character in the uid field as higlighted. Not sure though.
But am not able to figure it out how to aviod this.
The error which we are getting is
2005-12-02T15:31:11.152-0500: Error Processing Line: {gessouid=A1C1C423-71F0-138F-CB8C-BCC3BAAF484F, sn=Marra, cn=Luigi Marra, uid=[email protected], gessostatus=A, gegcfappaccess=portal, mail=[email protected], givenname=Luigi, gessocorrespondlanguage=, diffAction=create}
com.waveset.adapter.iapi.IAPIException: Item Resource:uniteam.it@GECF_B2B_FF(id=null) was not found.
Have someone seen this issue.
Or someone have any idea on this.
Thanks.

Thanks for the reply.
Sorry! I was out of pocket for couple of days........
I just double checked. I am using unique Employee ID as the "Unique key for diff" and also I have 'process diff. only' flag to set to true.
In my tests, I modified only contractor expiry date in the incoming sap flat file and then tried processing the file. It gave me the above error. Please note that the same employee id was available in the FFA* file also.
Interestingly, I tried the same test for another account and it worked for that account. That's what actually totally threw me off.
Thanks once again.........
- Lalit

Multiple flat files appended, is file locked until all processed?

Hi All,
Have a scenario in place where we are collecting multiple flat files on ECC and using append mode on the receiver file adapter to create a single file for final output. We are only interested in submitting the output to it's target after all files sent to PI have been appended.
Does PI 7.0 lock the appended file until all files have been appended so I am sure I end up with a complete single file on output? If not, what are my options to ensure a completely appended file is output? Would it be a viable option to pass the appending file through another sender file adapter using 'msecs to wait before modification check'?
Thanks!
Don
Edited by: donald.meredith on Dec 1, 2011 2:31 PM

How many files are you collecting from ECC and appending at receiver side and how much time it is taking to complete the process?
if its possible to schedule your business to read data from FTP after completing append then no worries, else sometimes it creates trouble.
The option you mentioned is reliable one , but you ended up creating one more interface.
Regards,
Raj

Flat File Active Sync - Notify admin incase of data processing errors

Dear Friends,
We have couple of Requirements to use OOTB flat file active sync adapter
1. To read data from a flat file and update the records in Sun Identity Manager system
2. Notify admin if there are any data processing errors while reading data from a flat file. The data processing errors can occur if there is an invalid data. for example, lets say the input flat file has 3 columns defined, but the file conatins records which has four values.
firstname,lastname,email
testfirst,testlast,[email protected],12345
Req#1 is working fine. There are no issues with that.
Req#2: if the file contains invalid data, i noticed that the active sync adapter throws an Array Index out of bound exception. so, we need to send an email notification to the admin whenever data processing errors occurs.
I noticed that whenever the data processing exception occurs, the active sync adapter stops processing records and the active sync input form was not triggered. Unless active sync form was triggered, it's very difficult to determine whether the data was read successfully or not.
Please let me know if there are any configurations/customizations to me made on OOTB flat file active sync adapter to handle data processing errors and send email notifications to administrators.
Appreciate your help
Thanks
Vijay

Hi,
We have same requirement that
"Notify admin if there are any data processing errors from a flat file.
The data processing errors can occur if there is an invalid data or account is locked etc..."
In short notify admin if any error logged in Active sync Log file while active sync runs.
Yes,I noticed same that whenever the data processing exception occurs, the active sync adapter stops processing records and the active sync input form was not triggered. Unless active sync form was triggered, it's very difficult to go ahead to meet the requirement.
Please let me know if there are any configurations/customizations to me made on flat file active sync adapter to send email notifications to administrators.
Thanks,
Sudheer

Flat file upload in Process Chain error!

Dear Expert:
We wirte routine in infopackage and use ftp upload flat file from BW Application server(AL11--> can open the file), I manually upload, it can sucessful upload, but in Process chain, it always deal with the process and display yellow light. I must manually stop the job.
any sugession for me? thank you very much!
Best Regards
Bessie

Dear all:
I found the way to slove this problem. in the Routine, i create a TCP/IP connection. I adjust it manually in SM59.
At "Technical Setting" Tab:
Set Activation Type: Start on Application Server.
Set Start type of External Program: Remote Execution
That's OK!
Thank you for your feedback!
Bessie

Beginner Process chains on flat files

Hi,
I have 6 ODSes which I load from 5 different flat files located on my desktop on a daily basis and a 6th file which I load once per week. All 7 files are then pushed to a cube before reporting can be done.
Being the first timer on creating process chain, do you have any straight forward steps to guide me?
Is the fact that the files are on a local machine a problem?
Any hints on making it perform well will be appreciated.
Thanks

Hi Amanda Baah
In general in AL11 system wont allow you to place thats way u need to place it in the servel location Via TELNET ..
Ask your BASIS guy they will help you...
Hope itz clear a little atleast...!
Thanks & Regards
R M K
***Assigning pointz is the only way of saying thanx in SDN ***
**Learning the thingz is never end process if u stop it will Be a devil if u continue it will be a divine***
> Hi,
>
> it did not work as I thought.. I have a few questions
> on this:
>
> 1.
> Regarding AL11, I see the existing folders but there
> is no option to upload or download files. Any guide?
>
> 2.
> Regarding the comment Instead you use the function
> module ARCHIVFILE_CLIENT_TO_SERVER to upload the
> local files to Application server. Can you direct me
> on how to execute this and download and upload files?
> I thought it was a tcode initially.
>
> Thanks

Process Flat Files

Similar Messages

Maybe you are looking for