Regarding ssis - Filter first 5 rows from flat file

Hi,
i have a requirement like this every day we receive 10-15 Flat files , we need to load these data into SQL Table ,here nothing is complex for this we need to use foreachloop Container,Dataflowtask..etc. here the issue is each flat file have top 5 rows has
Client information like ( We need to delete First 5 rows) and then insert into SQL Table . please suggest me how to develope the package..

You need to use a script task to read file one by one and remove the first 5 rows from the file. The file name/path can be stored as a variable and read it from the script code. Insert the script task inside the foreach loop container just before importing
the file.
Refer the link to remove a line in .net
http://stackoverflow.com/questions/7008542/removing-the-first-line-of-a-text-file-in-c-sharp
Regards, RSingh

Similar Messages

Ssis - import data from flat file to table (sql server 2012)

i have create a ssis for importing data from flat file to table in sql server 2012.
but i got the below error for some column in data flow task.
error: cannot processed because than one code page (950 and 1252)
anyone helps~

Hi,
The issue occurs because the source flat file uses ANSI/OEM – Tranditional Chinese Big5 encoding. When processing the source file, the flat file connection manager uses code page 950 for the columns. Because SQL Server uses code page to perform conversions
between non-Unicode data and Unicode data, the data in the code page 950 based input columns cannot be loaded to code page 1252 based destination columns. To resolve the issue, you need to load the data into a SQL Server destination table that includes Unicode
columns (nchar or nvarchar), and convert the input columns to Unicode columns via Data Conversion or the Advanced Editor for the Flat File Source at the same time.
Another option that may not be that practical is to create a new database based on the Chinese_Taiwan_Stroke_BIN collation, and load the data to non-Unicode columns directly.
Reference:
http://social.technet.microsoft.com/Forums/windows/en-US/f939e3ba-a47e-43b9-88c3-c94bdfb7da58/forum-faq-how-to-fix-the-error-the-column-xx-cannot-be-processed-because-more-than-one-code-page?forum=sqlintegrationservices
Regards,
Mike Yin
TechNet Community Support

Regarding master data ,transactional data loading from flat file

Hi friends,
Please tell me how to load master data and transactional data from flat file ....
Thanks in advance ,
Regards,
ramnaresh.

Hi,
Please use the 'search forum' functionality and search the BI Forum with say 'flat file loading'. You would get plenty of links of previous threads.
BR/
Mathew.

Importing From Flat File with Dynamic Columns

HI
I am using ssis 2008,i have folder in which I have Four(4) “.txt” files each file will have 2 columns(ID, NAME). I loaded 4
files in one destination, but today I receive one more “.txt” file here we have 3 columns (ID, NAME, JOB) how can I get a message new column will receive in source. And how can I create in extra column in my destination table dynamically …please help me

Hi Sasidhar,
You need a Script Task to read the names and number of columns in the first row of the flat file each time and store it in a variable, then create a staging table dynamically based on this variable and modify the destination table definition if one ore more
new columns need to be added, and then use the staging table to load the destination table. I am afraid there is no available working script for your scenario, and you need some .NET coding experience to achieve your goal. Here is an example you can refer
to:
http://www.citagus.com/citagus/blog/importing-from-flat-file-with-dynamic-columns/
Regards,
Mike Yin
TechNet Community Support

Reading data from flat file Using TEXT_IO

Dear Gurus
I already posted this question but this time i need some other changes .....Sorry for that ..
I am using 10G forms and using TEXT_IO for reading data from flat file ..
My data is like this :-
0|BP-V1|20100928|01|1|2430962.89|27|2430962.89|MUR|20100928120106
9|2430962.89|000111111111|
1|61304.88|000014104113|
1|41961.73|000022096086|
1|38475.65|000023640081|
1|49749.34|000032133154|
1|35572.46|000033093377|
1|246671.01|000042148111|
Here each column is separated by | . I want to read all the columns and want to do some validation .
How can i do ?
Initially my requirement was to read only 2 or 3 columns so i did like this ...
Procedure Pay_Simulator(lfile_type varchar2,lac_no varchar2,lcur varchar2,lno_item number,ltotal number,ldate date,lpay_purp varchar2,lfile_name varchar2)
IS
v_handle utl_file.file_type;
v_filebuffer varchar2(500);
line_0_date VARCHAR2 (10);
line_0_Purp VARCHAR2 (10);
line_0_count Number;
line_0_sum number(12,2);
line_0_ccy Varchar2(3);
line_9_sum Number(12,2);
line_9_Acc_no Varchar2(12);
Line_1_Sum Number(12,2);
Line_1_tot Number(15,2) := 0;
Line_1_flag Number := 0;
lval number;
lacno varchar2(16);
v_file varchar2(20);
v_path varchar2(50);
Begin
v_file := mcb_simulator_pkg.GET_FILENAME(lfile_name); -- For the file name
v_path :=rtrim(regexp_substr( lfile_name , '.*\\' ),'\'); For the Path
v_path := SUBSTR (lfile_name,0, INSTR (lfile_name, '\', -1));
v_handle := UTL_FILE.fopen (v_path, v_file, 'r');
LOOP
UTL_FILE.get_line (v_handle, v_filebuffer);
IF SUBSTR (v_filebuffer, 0, 1) = '0' THEN
SELECT line_0 INTO line_0_date
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_0, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 3;
SELECT line_0 INTO line_0_Purp
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_0, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 4;
SELECT line_0 INTO line_0_count
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_0, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 7;
SELECT line_0 INTO line_0_sum
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_0, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 8;
SELECT line_0 INTO line_0_ccy
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_0, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 9;
ELSIF SUBSTR (v_filebuffer, 0, 1) = '9' THEN
SELECT line_9 INTO line_9_Acc_no
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_9, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 3;
SELECT line_9 INTO line_9_sum
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_9, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 2;
ELSIF SUBSTR (v_filebuffer, 0, 1) = '1' THEN
line_1_flag := line_1_flag+1;
SELECT line_1 INTO line_1_sum
FROM (SELECT LTRIM (REGEXP_SUBSTR (v_filebuffer, '[^|]+{1}', 1, LEVEL)) line_1, ROWNUM rn
FROM DUAL
CONNECT BY LEVEL <= LENGTH (REGEXP_REPLACE (v_filebuffer, '[^|]*')) + 1)
WHERE rn = 3;
Line_1_tot := Line_1_tot + line_1_sum;
END IF;
END LOOP;
DBMS_OUTPUT.put_line (Line_1_tot);
DBMS_OUTPUT.PUT_LINE (Line_1_flag);
UTL_FILE.fclose (v_handle);
END;
But now how can i do ? Shall i use like this select Statement for all the columns ?

Sorry for that ..
As per our requirement ...
I need to read the flat file and it looks like like this .
*0|BP-V1|20100928|01|1|2430962.89|9|2430962.89|MUR|20100928120106*
*9|2430962.89|000111111111|*
*1|61304.88|000014104113|*
*1|41961.73|000022096086|*
*1|38475.65|000023640081|*
*1|49749.34|000032133154|*
*1|35572.46|000033093377|*
*1|246671.01|000042148111|*
*1|120737.25|000053101979|*
*1|151898.79|000082139768|*
*1|84182.34|000082485593|*
I have to check the file :-
Validation are 1st line should start from 0 else it should raise an error and insert that error into one table .
The for 2nd line also same thing ..it should start from 9 else it should raise an error and insert that error into one table .
Then the 3rd line should start from 1 else it should raise an error and insert that error into one table .
After that i have to do a validation like i will read the 1st line 2nd column .. It should be like this BP-V1 else raise an error and insert that error to a table . Then i will check the 3rd column which is 20100928 , it should be YYYYMMDD format else same thing ERROR.
Then like this for all columns i have different validation .......
Then it will check for the 2nd line 3rd column . this is an account no .1st i will check it should be 12 char else ERROR .Then I will check that what user has imputed in the form.Like for example User putted 111111111 then i will check with this 000111111111 which is there in the 2nd line . I have to add 000 before that Account no which user imputed .
Then the lines which is starting from 1 , i have to take all the 2nd column for all the lines which is starting from 1 and i have to do a sum . After that i have to compare that sum with the value in the 1st lines ( Starting from 0) 6th column . It should be same else ERROR ...
Then same way i have to count all the lines which is starting from 1 . Then i have to compare with the 7th column of 1st line . It should be same . Here in this file it should be 9.
MY CODE IS :-
Procedure Pay_Simulator(lfile_type varchar2,lac_no varchar2,lcur varchar2,lno_item number,ltotal number,ldate date,lpay_purp varchar2,lfile_name varchar2)
IS
v_handle TEXT_IO.file_type;
v_filebuffer varchar2(500);
line_0_date VARCHAR2 (10);
line_0_Purp VARCHAR2 (10);
line_0_count Number;
line_0_sum number(12,2);
line_0_ccy Varchar2(3);
line_9_sum Number(12,2);
line_9_Acc_no Varchar2(12);
Line_1_Sum Number(12,2);
Line_1_tot Number(15,2) := 0;
Line_1_flag Number := 0;
lval number;
lacno varchar2(16);
v_file varchar2(20);
v_path varchar2(50);
LC$String VARCHAR2(50) ;--:= 'one|two|three|four|five|six|seven' ;
LC$Token VARCHAR2(100) ;
i PLS_INTEGER := 2 ;
lfirst_char number;
lvalue Varchar2(100) ;
Begin
v_file := mcb_simulator_pkg.GET_FILENAME(lfile_name); For the file name
v_path :=rtrim(regexp_substr( lfile_name , '.*\\' ),'\'); For the Path
--v_path := SUBSTR (lfile_name,0, INSTR (lfile_name, '\', -1));
Message(lfile_name);
v_handle := TEXT_IO.fopen(lfile_name, 'r');
 BEGIN
 LOOP
 TEXT_IO.get_line (v_handle, v_filebuffer);
 lfirst_char := Substr(v_filebuffer,0,1);
 --Message('First Char '||lfirst_char);
 IF lfirst_char = '0' Then
 Loop
 LC$Token := mcb_simulator_pkg.Split( v_filebuffer, i , '|') ;
 Message('VAL - '||LC$Token);
 lvalue := LC$Token;
 EXIT WHEN LC$Token IS NULL ;
i := i + 1 ;
End Loop;
 Else
 Insert into MU_SIMULATOR_output_ERR (load_no,ERR_CODE,ERR_DESC) values (9999,'0002','First line should always start with 0');
 Forms_DDL('Commit');
 raise form_Trigger_failure;
 End if ;
 TEXT_IO.get_line (v_handle, v_filebuffer);
 lfirst_char := Substr(v_filebuffer,0,1);
 LC$Token := mcb_simulator_pkg.Split( v_filebuffer, i , '|') ;
 --Message('Row '||LC$Token);
 IF lfirst_char = '9' Then
 Null;
 Else
 Insert into MU_SIMULATOR_output_ERR (load_no,ERR_CODE,ERR_DESC) values (8888,'0016','Second line should start with 9');
 Forms_DDL('Commit');
 raise form_Trigger_failure;
 End IF;
 LOOP
 TEXT_IO.get_line (v_handle, v_filebuffer);
 lfirst_char := Substr(v_filebuffer,0,1);
 LC$Token := mcb_simulator_pkg.Split( v_filebuffer, i , '|') ;
 --Message('Row '||LC$Token);
 IF lfirst_char = '1' Then
 Null;
 Else
 Insert into MU_SIMULATOR_output_ERR (load_no,ERR_CODE,ERR_DESC) values (7777,'0022','The third line onward should start with 1');
 Forms_DDL('Commit');
 raise form_Trigger_failure;
 End if;
 END LOOP;
 --END IF;
 END LOOP;
 EXCEPTION
 When No_Data_Found Then
 TEXT_IO.fclose (v_handle);
 END;
Exception
 When Others Then
 Message('Other error');
END;
I am calling the FUNCTION which you gave SPLIT as mcb_simulator_pkg.Split.

How to import a logo from flat file to SAP in SMARTFORM.

how to import a logo from flat file to SAP in SMARTFORM. i need to know whether TIFF format is used and also the full steps.
Thanks,

Hi,
You can not directly use a logo from a file into a Smart form. First you will have to upload it to SAP. This has to be done only once. Once it is imported into SAP then you can use it in smart form. To import a logo to SAP, goto SE78. Open the tree under GRAPHICS. Select the file type and click on the Import (first button on the toolbar, just before delete button). Select the logo file and give it some appropriate name along with description. Select appropriate type of the image and click Continue. Your logo will be uploaded into SAP from a file. Now in your smartform, Create a Graphic Node, give the name of the logo along with other properties. In this way you can import a logo from a flat file into SAP and consequently in a smart form.
Reward points if the answer is helpful.
Regards,
Mukul

Can i able to put filter for my source flat file?

Hi all,
Please help me with the best practise of ODI.
My source is flat file and i want to put filter.
can i able to put filter for my source flat file? If yes, please help me with the best practise of applying filter.
Regards
Suresh

Hi ,
If you are trying to create at the Model -->Datastore ---> Filter ---> Insert condition
then it will not work for File technology . You will get " Invalid Format Description "
But you can specify a filter in the interface .
Just drop the column(s) from your flat file data store into the canvas and then specify the filter condition .
Thanks,
Sutirtha

Error in loading data from flat file....

Hi,
While loading the data from flat file , only one column is getting populated in the ODS and after that the sixth column is getting populated with the null values. Also when I save the changes in CSV file .. its not getting saved.
Could you please tell me what could be the probable reason for this.
Also what are the points we should keep in mind while loading from flat files.
Regards,
Jeetu

Hi,
You need to take care of -
1. you flat file structure ( left to right columns) should match with the datasource ( top to down) column to column. if you don't wish to laod some column, leave it as blank column in flat file.
2. your file should not be open when loading data
3. make sure your transfer rules & update rules are defined properly.
4. do the simulation in infopakage first & that will give you fair idea.
5. to start with load data first in PSA Only, check & then take it forward.
hope it helps
regards
Vikash

Data loading from flat file into bw

Hi Experts,
When we load data from flat file into BW the sequence of fields in flat file should be matched with the transfer structure or communication structure?
wid thanX,
arijit

Arijit,
The sequence of fields in a flat file must be the same as this one in the transfer structure.
The comm and tran structures are the same after the first creation of datasource from flat file source system. After that someone could change the sequence in the CS -- if the mapping of fields in transfer rules remains unchanged, then the load is to be succesful.
In case of changing the sequence in TS you'll get either error or wrong load (values to be loaded into some infobjects will go to another ones.)
Best regards,
Eugene

Loading Cube From Flat File

Hi All,
Ex: If cube is loaded from Flat file , say with 10 records and during next load only 1 record is changed.
Is it possible to load cube with all 10 records without deleting previous request which should not result in duplicate records in cube?

Hi.......
Infocube is always additive...........so there will be Duplicate records.........Only DSO has the overwritten option.......So for this u hav to go for DSO.........
But if u need to load in infocube only..........then u can do one thing..........
Start process
Delete Traget Content(Process : Complete Deletion of Data Target Contents )
Load infocube
Then first content of the cube will get deleted..........then cube will be loaded.........this will solve ur problem......
Hope this helps.....
Regards,
Debjani.......

How to upload schedule line from flat files to sap file

dear all,
i want to upload the schedule lines from flat files to sap schedulle lines
but the flat files have 15 schedule lines and the data is as per date
so how to upload that and the fields available in flat files are more than the sap screen
we are having more than 6 items
and 15scedule lines its abt 90data to be upload
for one customer in every 15 day
so how to do this
is there any direct use in functional side
with out the help of any abap
but my user will do it
so he need a permanent solution
with regards
subrat

Hi Subrat ,
u can upload the data either ( Master /Transaction) data with the help of lsmw. for that all u need to do is go through the lsmw and do it. in that u can go Batch input recording/ BAPI/ IDOC any of that. here i am sending the LSMW Notes go through it and do the work.
once u create the LSMW project then u can ask the data from user or u can explain the user about the program and can run the flat file to upload the data.
if u require LSMW material Just send me blank mail from u. my mail id is [email protected]
Reward if Helpful.
Regards,
Praveen Kumar.D

Error while loading table from flat file (.csv)

I have a flat file which i am loading into a Target Table in Oracle Warehouse Builder. It uses SQL Loader Internally to load the data from flat file, I am facing an issue. Please find the following error ( This is an extract from the error log generated)
SQL*Loader-500: Unable to open file (D:\MY CURRENT PROJECTS\GEIP-IHSS-Santa Clara\CDI-OWB\Source_Systems\Acquisition.csv)
SQL*Loader-552: insufficient privilege to open file
SQL*Loader-509: System error: The data is invalid.
SQL*Loader-2026: the load was aborted because SQL Loader cannot continue.
I believe that this is related to SQL * Loader error.
ACtually the flat file resides in my system ( D:\MY CURRENT PROJECTS\GEIP-IHSS-Santa Clara\CDI-OWB\Source_Systems\Acquisition.csv). I am connecting to a oracle server.
Please suggest
Is it required that i need to place the flat file in Oracle Server System ??
Regards,
Ashoka BL

Hi
I am getting an error as well which is similar to that described above except that I get
SQL*Loader-500: Unable to open file (/u21/oracle/owb_staging/WHITEST/source_depot/Durham_Inventory_Labels.csv)
SQL*Loader-553: file not found
SQL*Loader-509: System error: The system cannot find the file specified.
SQL*Loader-2026: the load was aborted because SQL Loader cannot continue.
The difference is that Ashoka was getting
SQL*Loader-552: insufficient privilege to open file
and I get
SQL*Loader-553: file not found
The initial thought is that the file does not exist in the directory specified or I have spelt the filename incorrectly it but this has been checked and double checked. The unix directory also has permission to read and write.
Also in the error message is
Control File: C:\u21\oracle\owb_staging\WHITEST\source_depot\INV_LOAD_LABEL_INVENTORY.ctl
Character Set WE8MSWIN1252 specified for all input.
Data File: /u21/oracle/owb_staging/WHITEST/source_depot/Durham_Inventory_Labels.csv
Bad File: C:\u21\oracle\owb_staging\WHITEST\source_depot\Durham_Inventory_Labels.bad
As can be seen from the above it seems to be trying to create the ctl and bad file on my c drive instead of on the server in the same directory as the .csv file. The location is registered to the server directory /u21/oracle/owb_staging/WHITEST/source_depot
I am at a lost as this works fine in development and I have just promoted all the development work to a systest environment using OMBPlus.
The directory structure in development is the same as systest except that the data file is /u21/oracle/owb_staging/WHITED/source_depot/Durham_Inventory_Labels.csv and everything works fine - .ctl and .bad created in the same directory and the data sucessfully loads into a oracle table.
Have I missed a setting in OWB during the promotion to systest or is there something wrong in the way the repository in the systest database is setup?
The systest and development databases are on the same box.
Any help would be much appreciated
Thanks
Edwin

Data is not updated to Infocube (BI 7.0) when i load from flat file

Hi Experts,
               when I try to load data from flat file to Infocube (BI 7.0) with full or Delta, I am gettin following errors.
1) Error while updating to data target, message No: RSBK241
2) Processed with Errors, Message No: RSBK 257.
Till transformations everthing is successful but updating data to cube is not successful...
what is the message no. indicates? can i find solution anywhere with respective to Message NO.?
Anybody please help me..
Regards
Anil

Hi Bhaskar,
                   There is no problem with cube activation.But i try to activate the cube by using
the function module " RSDG_CUBE_ACTIVATE". It goes to short dump.Anyhow i want to know
that in what scenario we use this function module to activate the cube.
         There is no special characters in the file. I delete the requests in psa, cube and Re run the
Infopackage & DTP.But I am getting same error.
The following errors are found in updating menu
of details tab (display messages option)
1) Error while updating to data target, message No: RSBK241
2) Processed with Errors, Message No: RSBK 257.
                 Thank you
Regards
Anil
Edited by: anilkumar.k on Aug 20, 2010 8:35 PM

Data Source creation for Master Data from Flat File to BW

Hi,
I need to upload Master Data from Flat File. Can anybody tell step by step right from begining of creation of DataSource up to Loading into Master Data InfoObject.
can any body have Document.
Regards,
Chakri.

Hi,
This is the procedure.
1. Create m-data with or without attributes.
2. Create infosource .
    a) with flexible update
         or
    b) with direct update
3. Create transfer rules and assign tyhe names of m-data and attribute in "Transfer rules" tab and transfer them to communication structure.
4. Create the flat-file with same stucture as communication structure.
5. if chosen direct update then create infopackage and assign the name of flat-file and schedule it.
6. if chosen flexible update the create update rule with assigning name of the infosource and the schedule it by creating infopackage.
Hope this helps. If still have prob then let me know.
Follow this link also.
http://help.sap.com/saphelp_nw2004s/helpdata/en/b2/e50138fede083de10000009b38f8cf/frameset.htm
Assign points if helpful.
Vinod.

How often in the real time projects extract data from flat files n process

I am going thru teh BODS data integrator, and trying to understand the demand of ETL services extract data from a flat file, is that really impt in teh real time jobs.
Thank you very much for the helpful info.

Hi,
As per the inputs given by you guys i started loading data from flat file.
I try to load 28 files from i which i was able to load 24 files succesfully.For the other 4 i got this error messages
1) Error 'Enter period in the format __.YYYY...' at conversion exit CONVERSION_EXIT_PERI6_INPUT (field CALMONTH record 1, value DUMYTRA)
Message no. RSDS012
2) a) Error 'The argument '1,008.00' cannot be interpreted as anumber' on assignment field QUANT_B record 11714 value 1,008.00
Message no. RSDS013
b) Error 'The argument '1,110.00' cannot be interpreted as anumber' on assignment field QUANT_B record 15374 value 1,110.00
Message no. RSDS013
3) a) Error 'The argument '1,140.00' cannot be interpreted as anumber' on assignment field QUANT_B record 1647 value 1,140.00
Message no. RSDS013
b) Error 'The argument '2,028.00' cannot be interpreted as anumber' on assignment field QUANT_B record 4625 value 2,028.00
Message no. RSDS013
4) Error 'The argument '1,151.00' cannot be interpreted as anumber' on assignment field QUANT_B record 7808 value 1,151.00
Message no. RSDS013
I'am unable to trace out what is the error exactly.
I checked this values in files they are perfect.
can anybody please guide me on this issue.
With Regards,
Pradeep.B

Regarding ssis - Filter first 5 rows from flat file

Similar Messages

Maybe you are looking for