Too many flat files with different formats.

Hi gurus,
i have a headache problems. that is follows:
We will upload data from flat files,but you know there are about 100 .xls files and with different formats.
and as you know the datasource of the flat file must correspond to every fields of the flat file.
but you know there are 100 flat files and if we donot do any optimise,we will create about 100 transformation and 100 datasource to meet our requirements.
but that seems impossible.
is there any good idea to decrease the number of transfomation ?

Hi AK1987,
To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
the detail steps, please walk through the following blog:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support

Similar Messages

SSIS project - read multiple flat files with different formats

hi all,
i need to import multiple flat files with different formats into different tables of the sql server database and not able to figure out the best way out in ssis to do so...
please advise the possible methods in ssis to do so and if possible the process which can be dynamic as file names or columns might change in future.

Hi AK1987,
To import flat files with dynamic columns, we can use Script Task inside a Foreach Loop Container to parse the first row of the flat file to get the columns names and save them into a .NET variable, then, we can create “Create Table” script based on this
variable, and then store the script into a SSIS package variable. After that, we create a staging table based on the package variable, load the flat file data to the staging table. Eventually, we load data from the staging table to the destination table. For
the detail steps, please walk through the following blog:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support

How to load a flat file with utf8 format in odi as source file?

Hi All,
Anybody knows how we can load a flat file with utf8 format in odi as source file.Please everybody knows to guide me.
Regards,
Sahar

Could you explain which problem are you facing?
Francesco

SSIS - Import Multiple flat files with different metadata

Hi ,
I have set of flat files with different metadata structure, I would like to load them into staging tables.
1. ) Can we load the flatfiles into the staging tables with out having multiple data flow task.
2.) If possible , can we programmatically select the staging table based on the metadata of the flatfile and load them.
Please advise.
Thanks
Thiya

Nope in SSIS a data flow task needs to have a fixed metadata. So if your file metadata varies then best option would be use OPENROWSET syntax to pull the data and populate into your staging table. You may also use
SELECT .. INTO StagingTable ... FROM OPENROWSET (...)
syntax to create staging table at runtime based on the file metadata
http://sqlmate.wordpress.com/2012/08/09/use-your-text-csv-files-in-your-queries-via-openrowset/
If you want to do this in SSIS you need to create data flow dynamically using script task and build the metadata
see
http://www.selectsifiso.net/?p=288
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

XML file with different format

Hello
 I have a requirement where the customer needs XML file with different format. How do I achieve the second format mentioned below instead of first format.
Normally we create the XML file in the following format:
- <Entries>
 - <Organization>
 <MemberName>0000000002</MemberName>
 <FullName>BAY GYNECOLOGICAL ASSOCIATES</FullName>
 <OrgId>0000000002</OrgId>
 <OrgType>CUST</OrgType>
 - <AssociatedToOrg>
 <Name>0002</Name>
 - </AssociatedToOrg>
</Organization>
Now the customer wants like the following format:
<Organization MemberName="522_Community_Customer" FullName="522 Community Customer" OrgId="DEA123456" OrgIdType="DEA" >
 <AssociatedToOrg Name="MN"/>
 </Organization>
Thanks
Naga

you can solve it with a XSL mapping. refer to the SDN to know how develop it
/people/aleksey.popov2/blog/2010/01/26/consuming-webservices-with-tag-in-wsdl-using-xslt
XSLT Editor for creating xlst mappings
http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/190eb190-0201-0010-0ab3-e69f70b6c257
http://wiki.sdn.sap.com/wiki/display/XI/XSLTMappingSteps
http://wiki.sdn.sap.com/wiki/display/XI/FileTOFile-UsingXSLTmapping%28forBeginners%29
http://wiki.sdn.sap.com/wiki/display/Snippets/ConvertflatXMLfiletonestedusing+XSLT
http://wiki.sdn.sap.com/wiki/display/Java/MultiMappingwithJavaandXSLTmappings
Edited by: Rodrigo Alejandro Pertierra on Oct 1, 2010 5:26 PM

Using ssis import a multiple flat files with different mappings

I have an scenario for import a file
20 different flat file source and having XML mapping document in a table.
get files from that path(i have path and file name extension in table)
i have map this file based on the file name extension in dynamically
create a 20 different staging table based on mapping in dynamically
kindly help for this

Hi Karthick,
As Arthur said, if you don't want to hard code the data flow, you need to read XML from the SQL Server table and use Script Task to implement the dynamic columns mapping. To read XML from a SQL table, you can refer to:
http://blog.sqlauthority.com/2009/02/13/sql-server-simple-example-of-reading-xml-file-using-t-sql/
In the Script Task, we need to parse the first row of the flat file to get the columns information and save that to a .NET variable. Then, the variable will be used to create the query that will be used to create SQL table. For more information, please see:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support

Flat file with different layouts in a single file

Hello Friends,
Good Morning.
I am new to ODI - Oralce Data Integrator 11G.
I have a flat file where I had to load the flat file data to a oracle DB ( 11G ).
The flat file has 5 different layouts ( Data of 5 different suppliers coming as a single file and each layout is different ) and each layout corrresponds to one supplier information.
I had to load each layout data into a different table using a single source file ( That has all supplier information).
Please advice.
Thanks/Kumar

Ok below are the layouts.
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
Few more question to you.
Are they always come in sequential like below ?
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
BASE record
DS01 record
RM01 record
CR01 record
NW01 record
Do you want only insert or insert/update as well ?

What have "Too many open Files" to do with FIFOs?

Hi folks.
I've just finished a middleware service for my company, that receives files via a TCP/IP connection and stores them into some cache-directory. An external program gets called, consumes the files from the cache directory and puts a result-file there, which itself gets sent back to the client over TCP/IP.
After that's done, the cache file (and everything leftover) gets deleted.
The middleware-server is multithreaded and creates a new thread for each request connection.
These threads are supposed to die when the request is done.
All works fine, cache files get deleted, threads die when they should, the files get consumed by the external program as expected and so on.
BUT (there's always a butt;) to migrate from an older solution, the old data gets fed into the new system, creating about 5 to 8 requests a second.
After a time of about 20-30 minutes, the service drops out with "IOException: Too many open files" on the very line where the external program gets called.
I sweeped through my code, seeking to close even the most unlikely stream, that gets opened (even the outputstreams of the external process ;) but the problem stays.
Things I thought about:
- It's the external program: unlikely since the lsof-command (shows the "list of open files" on Linux) says that the open files belong to java processes. Having a closer look at the list, I see a large amount of "FIFO" entries that gets bigger, plus an (almost) constant amount of "normal" open file handles.
So perhaps the handles get opened (and not closed) somehwere else and the external program is just the drop that makes the cask flood over.
- Must be a file handle that's not closed: I find only the "FIFO" entries to grow. Yet I don't really know what that means. I just think it's something different than a "normal" file handle, but maybe I'm wrong.
- Must be a socket connection that's not closed: at least the client that sends requests to the middleware service closes the connection properly, and I am, well, quite sure that my code does it as well, but who knows? How can I be sure?
That was a long description, most of which will be skipped by you. To boil it down to some questions:
1.) What do the "FIFO" entries of the lsof-command under Linux really mean ?
2.) How can I make damn sure that every socket, stream, filehandle etc. pp. is closed when the worker thread dies?
Answers will be thanked a lot.
Tom

Thanks for the quick replies.
@BIJ001:
ls -l /proc/<PID>/fdGives the same information as lsof does, namely a slowly but steadily growing amount of pipes
fuserDoesn't output anything at all
Do you make exec calls? Are you really sure stdout and stderr are consumed/closed?Well, the external program is called by
Process p = Runtime.getRuntime().exec(commandLine);and the stdout and stderr are consumed by two classes that subclass Thread (named showOutput) that do nothing but prepending the corresponding outputs with "OUT:" and "ERR" and putting them into a log.
Are they closed? I hope so: I call the showOutput's halt method, that should eventually close the handles.
@sjasja:
Sounds like a pipe.Thought so, too ;)
Do you have the waitFor() in there?Mentioning the waitFor():
my code looks more like:
try {
 p = Runtime.getRuntime.exec(...);
 outshow = new showOutput(p.getInputStream(), "OUT").start;
 errshow = new showOutput(p.getErrorStream(), "ERR").start;
 p.waitFor();
} catch (InterruptedException e) {
 //can't wait for process?
 //better go to sleep some.
 log.info("Can't wait for process! Going to sleep 10sec.");
 try{ Thread.sleep(10000); } catch (InterruptedException ignoreMe) {}
} finally {
 if (outShow!=null) outShow.halt();
 if (errShow!=null) errShow.halt();
/**within the class showOutput:*/
/**This method gets called by showOutput's halt:*/
public void notifyOfHalt() {
 log.debug("Registered a notification to halt");
 try {
 myReader.close(); //is initialized to read from the given InputStream
 } catch (IOException ignoreMe) {}
}Seems as if the both of you are quite sure that the pipes are actually created by the exec command and not closed afterwards.
Would you deem it unlikely that most of the handles are opened somewhere else and the exec command is just the final one that crashes the prog?
That's what I thought.
Thanks for your time
Tom

Loading Flat file in ASCII format

Hi All,
    We are loading data from flat file in ASCII Format . Kindly let me know what are the prerequisites we need to maintain when we load data in ASCII Format .
Cheers!
Raghu

Hi Raghu,
I always load in delimited ASCII. True ASCII format has the advantage of being the fastest way to enter data to BW. But you have to make sure, that the values you import are perfect aligned. So you have to fill in blanks to get the values and characteristics aligned. Delimited ASCII separates by a separator (usually a tabulator or a ";").
Example:
ASCII
10200865678983 10PRODA     2011
10200865678983 20PRODB     4012
Delimited ASCII:
1020;08;8656789383;10;PRODA;20;11
1020;08;8656789383;20;PRODB;40;12
Depending on system you may also be careful how you name your files. In Unix you have to be careful in using Upper/Lower case. Win and Unix systems use different ways to access directories. You have to keep this in mind, especially if you wish to use a function to find the filenames you wish to import.
Also because of Unix or Windows systems you should remember that Unix just need a CR to separate lines, while Windows needs CR/LF. So if you try to import Windows files in Unix systems you will get interesting results. Look at the files you wish to import with transaction AL11. If you see some interesting black cubes at the end of line, you have a windows file you try to load in unix.
Kind regards,
Jürgen

Java.io.IOException: Too many open files while deploying in soa 11g

hi all,
I am getting a strange error while deploying any composite .. it's a hello world kinda composite but while i am trying to deploy it i am getting "java.io.IOException: Too many open files" while deployment.. i have tried to deploy it in 2-3 ways but all of them resulted in the same error..bouncing the soa server might be an option but can someone give an insight as why it is happening and can it be resolved without restarting the server..
Thanks

yes..so this problem is with unix only ..coz i previously worked in Windows ..never got this problem..

Flat file with fixed lengths to XI 3.0 using a Central File Adapter---Error

Hi
According to the following link
/people/michal.krawczyk2/blog/2004/12/15/how-to-send-a-flat-file-with-fixed-lengths-to-xi-30-using-a-central-file-adapter
In Adapter Monitor I got the following error,
In sender Adapter,
Last message processing started 23:47:35 2008-10-25, Error: Conversion of complete file content to XML format failed around position 0 with java.lang.Exception: ERROR converting document line no. 1 according to structure 'Substr':java.lang.Exception: Consistency error: field(s) missing - specify 'lastFieldsOptional' parameter to allow this
last retry interval started 23:47:35 2008-10-25
length 15,000 secs
some one help me out ?
Thanks
Ram

from the blog you referenced -
 /people/michal.krawczyk2/blog/2004/12/15/how-to-send-a-flat-file-with-fixed-lengths-to-xi-30-using-a-central-file-adapter
goto step 4
additional parameters
add as the last entry
<recordset structure>.lastFieldsOptional Yes
e.g.,
Substr.lastFieldsOptional Yes

How to handle flat file with variable delimiters in the file sender adapter

Hi friends,
I have some flat files in the ftp server and hope to poll them into XI, but before processing in XI, I hope to do some content conversion in the file sender adapter, according to the general solution, I just need to specify the field names, field seperator, end seperator, etc. But the questions is:
The fileds in the test data may have different amount of delimiters (,), for example:
ORD01,,,Z4XS,6100001746,,,,,2,1
OBJ01,,,,,,,,,,4,3
Some fileds only have 1 ',' as the delimiter, but some of them have multiple ','.
How can I handle it in the content conversion?
Regards,
Bean

Hi Bing,
Plz do refer the following blogs u will get an idea:
File content conversion Blogs
/people/venkat.donela/blog/2005/03/02/introduction-to-simplefile-xi-filescenario-and-complete-walk-through-for-starterspart1
/people/venkat.donela/blog/2005/03/03/introduction-to-simple-file-xi-filescenario-and-complete-walk-through-for-starterspart2
/people/arpit.seth/blog/2005/06/02/file-receiver-with-content-conversion
/people/anish.abraham2/blog/2005/06/08/content-conversion-patternrandom-content-in-input-file
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
/people/venkat.donela/blog/2005/03/02/introduction-to-simplefile-xi-filescenario-and-complete-walk-through-for-starterspart1
/people/venkat.donela/blog/2005/03/03/introduction-to-simple-file-xi-filescenario-and-complete-walk-through-for-starterspart2
/people/venkat.donela/blog/2005/06/08/how-to-send-a-flat-file-with-various-field-lengths-and-variable-substructures-to-xi-30
/people/anish.abraham2/blog/2005/06/08/content-conversion-patternrandom-content-in-input-file
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
/people/jeyakumar.muthu2/blog/2005/11/29/file-content-conversion-for-unequal-number-of-columns
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/michal.krawczyk2/blog/2004/12/15/how-to-send-a-flat-file-with-fixed-lengths-to-xi-30-using-a-central-file-adapter
/people/arpit.seth/blog/2005/06/02/file-receiver-with-content-conversion
http://help.sap.com/saphelp_nw04/helpdata/en/d2/bab440c97f3716e10000000a155106/content.htm
Regards,
Vinod.

Save one file in different formats at once.

Hello,
I am a illustrator who needs to save files in different formats. Like 1 with a watermark, one small one for the website, another one for shop, a pdf file for the prints, etc. Is there a way to do all those things at once? Cause it really takes a lot of time.
I make my illustrations in photoshop, but if it's nessecary to only do those thing in another adobe programm, than that's no problem.
Please help, thank a lot!
Renske

Or try Image Processor Pro:
http://blogs.adobe.com/jnack/2011/05/new-image-processor-pro-script-for-cs5.html

Send a flat file with fixed lengths to XI 3.0 using a Central File Adapter?

Hello,
I'm wondering if someone have experience setting up conversion for different record structures. The example shown,
/people/michal.krawczyk2/blog/2004/12/15/how-to-send-a-flat-file-with-fixed-lengths-to-xi-30-using-a-central-file-adapter,
(in a greate way) only picture one kind of structure.
How should it be done if the file would contain
10Mat1
20100PCS
The first record structure has columns
ID(2),Material(10)
The second redcord structure has columns
ID(2),Quantity(3), Unit of messure (3)
Brgds
Kalle
Message was edited by: Karl Bergstrom

The configuration would be like follows:
Content Conversion Parameters:
Document Name: <your message type name>
Document Namespace: <your message type namespace>
Document Offset: <leave empty>
Recordset Name: <any name>
Recordset Namespace: <leave empty>
Recordset Structure: row1,,row2,
Recordset Sequence: any
Recordsets per Message: *
Key Field Name: ID
Key Field Type: String
Parameters for Recordset Structures:
row1.fieldNames ID,Material
row1.fieldFixedLengths 2,10
row1.keyFieldValue 10
row2.fieldNames ID,Quantity,UOM
row2.fieldFixedLengths 2,3,3
row2.keyFieldValue 20
Instead of row1 and row2 you can choose any name.
Regards
Stefan

Convert the flat file to xml format.

hi,
I need to write a interface program in the R/3 to pull the flat file data from the unix application server and do some manipulation and place back into the unix application server in XML format, From the unix box XML file taken by the XI server.
pls give me some idea to convert the flat file to XML format, through any function module or any other logic is there...
with regards,
Thambee.

Hi Thambe
in addition to the above posts
Program to convert flat file to XML file.
please download tool from this link:
http://www.download.com/Stylus-Studio-2008-XML-Enterprise-Suite/3000-7241_4-10399885.html?part=dl-StylusStu&subj=dl&tag=button&cdlpid=10399885
how to use:
http://www.stylusstudio.com/learn_convert_to_xml.html
http://www.sap-img.com/abap/sample-xml-source-code-for-sap.htm
Flat file to XML
CONVERTION OF FLAT FILE TO XML : NO OUT PUT FOUND
Converting Idoc flat file representation to XML
how to convert flat file into IDOC-XML
Thanks
sandeep sharma
PS ; if helpful kindly reward points

Too many flat files with different formats.

Similar Messages

Maybe you are looking for