Beginner Process chains on flat files

Hi,
I have 6 ODSes which I load from 5 different flat files located on my desktop on a daily basis and a 6th file which I load once per week. All 7 files are then pushed to a cube before reporting can be done.
Being the first timer on creating process chain, do you have any straight forward steps to guide me?
Is the fact that the files are on a local machine a problem?
Any hints on making it perform well will be appreciated.
Thanks

Hi Amanda Baah
In general in AL11 system wont allow you to place thats way u need to place it in the servel location Via TELNET ..
Ask your BASIS guy they will help you...
Hope itz clear a little atleast...!
Thanks & Regards
R M K
***Assigning pointz is the only way of saying thanx in SDN ***
**Learning the thingz is never end process if u stop it will Be a devil if u continue it will be a divine***
> Hi,
>
> it did not work as I thought.. I have a few questions
> on this:
>
> 1.
> Regarding AL11, I see the existing folders but there
> is no option to upload or download files. Any guide?
>
> 2.
> Regarding the comment Instead you use the function
> module ARCHIVFILE_CLIENT_TO_SERVER to upload the
> local files to Application server. Can you direct me
> on how to execute this and download and upload files?
> I thought it was a tcode initially.
>
> Thanks

Similar Messages

Error while creating process chain for flat file loading

Hi All,
I had Created a process Chain to load Transactiion data(Full load) form flat file which is in my computer.
start>Load>DTP>DeleteIndex>DTP loading CUBE--> Create Index
but the system is throwing an error as "An upload from the client workstation in the background is not possible."
I dont know why this error is coming?
Can some one help me
Regards
Mamta

Hi Mamta,
Basically if you want to load the DS through FF using process chain, the FF has to be placed in Application server. We cant load the FF when it is located in the client local workstation(FF On your PC).
So better you remove the Infopackage step from the PC. Load the IP manually. Once it is completed you can start the process chain with the following steps:
start>DTP>DeleteIndex>DTP loading CUBE> Create Index
Hope it is clear & helpful!
Regards,
Pavan

PROCESS CHAIN and flat file

hi xperts,
I tried to load the flat file to ODS in the process chain.
I got this error:
An upload from the client workstation in the background is not possible
,can anybody tell me how to solve this error.What are the proper settings for loading a flat file in process chains.

Hi,
in the process chain at the background, u cannot load the data using the option Client Workstation instead you would need to select the option Application Server and then save it , go to the schedule tab of the IP scheduler and then click on the start button and then go to monitor screen, u willt hen see the records being extracted.
1. change the option to Application Server
2. Save
3. under schedule tab in Infopackage scheduler , select "Start to load data immediately" and then click on Start button.
4. after u see "Data was requested" at the status bar of the Infopackage scheduler then you click on Monitor icon and see the records being extracted.
Do let me know if it was successfull or not.

How to Trigger the process chain using Excel Files?

Hi All,
We have 9 excel files(4 master data attributes and text files and 1 transaction file).
Everyday new records will be added into the excel sheets and we need to trigger all these files into the bw server through process chain automatically . As we have all these Excel files in the desktop.
Please suggest how to achieve this task?
Thanks
Gowtham

Hi,
You cannot load flat files to BW automatically using process chains.
You have to upload them to Apllication server and point your Infopackage to the directory path of the flat file in application server. then process chains can load it.
How to guide for loading flat file to application server.
Program to load CSV file from local desktop to Application server(For flat file Data extraction)
-Sriram

Processing records using Flat file ActiveSync resource

Hi Folks,
I was working on flat file activeSync resource adapter. Documentation says the following..
The Flat File Active Sync adapter can track the timestamp of a flat file. In addition, the adapter can archive the last file processed and then compare it to the most recent version. Identity Manager will then act on the accounts that are different in the two files.
If these features are enabled, the first time Identity Manager polls the source flat file, the system copies the file and places it in the same directory. The copied (archived) file is named FFAS_timestamp.FFAS, with the timestamp indicating the last time the original file was changed. The format of the timestamp is determined by the operating system on which the source file resides.
On each subsequent poll, Identity Manager compares the timestamp on the original file with the most recent timestamp. If the new timestamp value is the same as the previous value, then the file has not changed, and no further processing is performed until the next poll. If the timestamp values are different, Identity Manager checks for the presence of the FFAS file. If the file does not exist, Identity Manager processes the updated source file as if it were a new file.
If the timestamps are different and the archived FFAS file exists, Identity Manager compares the source file with the archived file. The comparison will filter any users that have not changed. If a user has changed, then it will be sent through the adapter in the normal manner, and the configured process, correlation and delete rules determine what to do with the user.
I have set the following attributes in synchronization policy as
Track Last Processed Timestamp : true
Process Differences Only : true
Unique Key for Diff : <some attribute>
I have the following flat file..
PERSONID,accountID,FIRSTNAME,LASTNAME,NATID,BIRTHDATE,EXTERNALID,TYPE,COUNTRY
10004,,test,4,123444,12/01/1804,204,A,USA
10005,,test,5,,12/01/1805,205,M,USA
There is no user account in IDM repository. After processing this file, my code creates the above accounts in IDM.
I deleted the above two accounts from IDM using admin interface and the accounts are not created again. So far good.
Now I opened the above flat file and just saved again. It creates the account in IDM.
According to the above documentation, it should not create the account as the flat file is already processed based on diff operation. Then Why it is creating the account again in IDM? Any logic??
Your repsonse will be appreciated.

Hi,
I am pretty sure this has been working for me but with IAPI.Cancel. Please try to capitalize the C.
Regards,
Patrick

Process chains from the flat file by using filezilla client version in BI

Hi experts,
please let me know how to create the process chains from flat file by using filezilla client version.
so far, I didn't work with file zilla FTP. can anybody give detailed step by step procedure to find the flat files and download them and creating process chains from that flat file.
Thanks & Regards,
Babu..

Hi,
Check these:----
Process chain configuration for Flat file loading
http://wiki.sdn.sap.com/wiki/display/BI/Howtowriteroutinetofetchcurrentday%27sfilename
Regards,
Suman

Error while activating Process Chains

Hi all,
while activating the Process Chains, i am getting the following error "Job BI_PROCESS_PSAPROCESS could not be scheduled. Termination with returncode 8"
when i double click on the error msg, i got the following help msg: "
Message no. RSPC065
Diagnosis
Program RSPROCESS is to be scheduled as job BI_PROCESS_PSAPROCESS under user ALEREMOTE.
Can any one please show some way to solve this problem? please do this favor, i have been suffering with error for a long time.
Points will be given
Thanks
Ganesh

Hi,
Just analyze the error message that you get while activating the PC, don't give any server name.If you are trying to run process chain using Flat file, it won't work,
and you should have source system R/3 or you have own datasources in BW system itself at that toime you can use PC to extract data. If your source system is Flat ile, it won't work. other wise you should place your flat file in application server, using AL11 tcode.
OSS : 511475
Symptom
You cannot schedule or perform any batch jobs with the BW or source system background user.
The error RSPC 065 occurs in the process chains:"Job could not be scheduled, termination with return code 8"
Other terms
RSPC065
Reason and Prerequisites
The user type is
"CPIC" up to 4.6B
"Communication" as of 4.6C
This user type may not execute or start any batch jobs, irrespective of the user authorizations.
Solution
Set the type of background user to
"Background" up to 4.6B
"System" as of 4.6C
This user type corresponds to the "Communication" type and may also perform background functions.
Through the Customizing, the BW user is automatically created by mistake as a communication user.Depending on your BW system release, you can solve this problem as follows:
BW 2.0B
 Import Support Package 24 for 2.0B (BW2.0B patch24 or SAPKW20B24) into your BW system. The Support Package is available once note 456551 with the short text "SAPBWNews BW 2.0B Support Package 24", which describes this Support Package in more detail, has been released for customers.
BW 2.1C
 Import Support Package 16 for 2.1C (BW2.1C patch16 or SAPKW21C16) into your BW system. The Support Package is available once note 456566 with the short text "SAPBWNews BW 2.1C Support Package 16" has been released for customers.
BW 3.0A
 Import Support Package 8 for 3.0A (BW3.0A patch08 or SAPKW30A08) into your BW system. The Support Package is available once note 452632 with the short text "SAPBWNews BW 3.0A Support Package 08" has been released for customers.

Error handling when loading flate files

Hi
We have a Process Chain loading flate files from a server into a ODS. We want to create a routine to avoid errors in the ODS when no files is found.
The error we want to handle is
Error 1 when loading external data
Message no. RSAR234
We want to handle this error message as OK to avoid red request in the ODS
Please help
Kind regards
Ola Einar Langen

Hi, and thanks for your hints.
We ended up with the following codes. It's will mark the prosess chain red (but this is as wanted) but we have implemented that the prosess chain will proceed even with error in the loading process
The code for the routine implemented in the infopackage in "External data" tab.
p_filename = 'filename.txt.
OPEN DATASET p_filename FOR INPUT IN BINARY MODE.
If file could not be open
IF SY-SUBRC NE 0.
p_subrc = SY-SUBRC.
delete the current request
CALL FUNCTION 'RSSM_DELETE_REQUEST'.
ENDIF.
close the file if OK
CLOSE DATASET p_filename.
p_subrc = 0.
The solution is no OK for us
Kind regards
Ola Einar Langen

Process chain creation in BI7

Hi,
I am creating a process chain for flat file upload..in this i have created the following processes..
1.generate index
2.load data
I have manually ran the infopackge, the data is loaded upto PSA.
Could anyone let me know what is the other process type that can be added to load the data from PSA to Data target and please give the steps to do....
Thanks in advance
-Manjula

It is always advisable to transport the process chain from DEV to Production. You can just do this by simply gather all the necessary objects for process chain into your transport.
During the support phase, you have to monitor the data load through process chain. That can be done via transaction code rspcm. In case there is any error in the process chain ( various errors can be there, data loading error, delta error). In that case you have to go to the step of the process chain when the error occurs (it will be marked as red) , rectify the error and process the step manually at the point so that the following process finishes.
Hope this is clear now.

Flat File automation process - limitations

Hello Everyone,
I would really appreciate any insight on the process improvement suggestions.
Background:
Currently we have around 12 territories providing a new flat file with new data on a daily basis depending on the business activity. Which would also mean that, on a given day if there is no activity would mean no flat file provided to BI for loading process.
The flat files provided need to be loaded into the BI system (PSA - DSO - InfoCube).
The flat file loading process has been automated for the daily file by implementing the logical file name for each territory.
1. The process variant in the process chain is to ensure if the flat file is available on the App server (Custom ABAP program).
2. 12 InfoPackages have been created to pick the data from the flat file on the app server and load the data over into the PSA.
3. All the InfoPackages merge into an "AND" event in the process chain before the DTP load into the DSO kicks off.
4. DSO Activation
5. Recon between the flat file and the DSO to ensure all the data from flat file has been loaded into the DSO.
6. DTP loads into the InfoCube.
7. Recon between the InfoCube and the DSO itself.
8. Moving the flat file from one folder into another.
All the above processes are automatically performed without any issues if the flat file is available on the server.
Problem / Issue:
As one of the major limitations of the above design is the flat file for sure needs to be made available on the app server in order for the whole data flow in the process chain to continue without any breakpoints.
Current workaround / process improvement in place:
Based on the above limitation and upon further research, I was able to apply the OSS Note to give us the option of maintaining multiple DTPs for the same data target with different filter values.
So, even if have individual data stream for each territory with a different DTP the issue still remains where the process variant (ABAP program to check if file exists) or the InfoPackage load if the ABAP program is removed will fail.
Which means due to the above fact, the support team is alerted about the process chain failure.
Question / Suggestions required:
The main questions or any suggestions would be welcome, if one of you can let us know an approach where the flat file check program doesn't have to give a hard failure in the process chain for the rest of the process chain to continue with the loading process. (As in order for the rest of the process chain to continue the only options we have are Error, Success, Always).
I have also looked into the Decision process variant available in the process chain, but based on the options available within I cannot utilize it for the loading process.
Error can be caused by generating an error message in the ABAP program which in turn is causing the issue of alert being sent over even if the rest of the process chain finishes.
Success would mean the flat file needs to be available. Always cannot be used in this case as it will cause a failure at the InfoPackage level.
If the InfoPackage load can be avoided without a hard error to be generated, the process chain does not have to remain in the failed state which in turn will not trigger any alert to the support team.
Please do let me know if you need more details about the above process improvement question.
Thanks
Dharma.

The main issue with this as you mentioned is that the file has to be available for sure.
We had a similar issue - we had a very critical data load that had to happen everyday , failure of the file getting loaded would mean that the reports for the day would be delayed.
We were running on UNIX and we implemented a simple UNIX command that would not complete till the file was available in the app server.
Something like
while ( the file does not exist )
delay of 15 seconds
you will come out of the while only after the while completes which means that the file becomes available.
You can write a similar ABAp program to check file availability if required and put it into your program.
we also had a failover process where we created a zero byte file with the same name if the file did not come beyond a certain number of tries and the PSA would load zero records and the data load will continue.
Edited by: Arun Varadarajan on Jan 26, 2009 10:18 AM

Download file from process chain

Hello experts,
i have just created a data model and report from flat file(CSV) in BW 7.0.
I understand that if I want to include the extraction from the DataSource into the InfoPackage in the process chain, the CSV file needs to be on the Application Server.
The external database is being updated with new data daily. There is a URL - www.example.com/export.php - that when accessed, deliveres the updated flatfile from the database.
My question is: can i integrate this download in a process chain to be automatically triggered, so that every morning I have the updated data in the report?
Thanks.

Hi Andrei,
Am not sure how to execute the url using the process chain. But still we can check whether the file is available in the application server or not.
We have a process variant called OS Command, using this variant type we can check whether the file is available or not for some tim intervals. In case file is availabe then the next process will trigger else it will keep on searching for the file until it finds.
Regards
Chaithanya

Process Flat Files

Hi everyone,
I currently need to process about 400 txt files, equivalent to about 1TB of data in maximum 5-6 hours. I need to run a very simple scrip, where I would capture about 0.10% of the data. Something like a select function with some where clause. Originally I
was thinking to import this data in a SQL server however I'm not sure my computer could handle the workload in a reasonable time frame.
What would you suggest? Is there any other products that Microsoft offers to process extremely large flat files?
Thank you in advance.
Cheers,

Hi AlexB0865,
The solution you have mentioned should be the best one in my opinion. The performance bottleneck in this case are:
Load data from 400 text files, which is about 1 TB large. Each file should be about 2.5GB
Bulk insert data to SQL Server
Filter data from the 1TB data
For the first chanllange, it won't be an issue in SSIS per my testing. I have a file with 23504761 rows, which is 2.5GB large. This file can be imported into a SQL Server table within 1 minute.
For the second chanllange, we can split the bulk insert to improve the performance.
For the third chanllange, with proper index(es) created, the performance won't be a problem.
Henk details the testing result regarding this same scenario in blog:
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
If you have any more question, please feel free to ask.
Thanks,
Jinchun Chen

Process chain 'OS COMMAND'

Hi
I would like to use the OS_COMMAND in process chain to rename file to another filename after data loaded from file into the targeted cube. Eg: I would like to delete brand.txt from folder named /interface/.
I would like to create the command name via tcode SM49. However, I did not know the command to enter in fields 'Operating system command' and 'Parameters for operating system command'. Can anyone please guide me?
Thanks and regards
Kang Ring

Hi,
You can find System OS command process type in Other Processes section of RSPC.here you can add the same in your process chains and fulfill your requirement.
hope this is clear for you.
Regards
Ramsunder

Heterogeneous systems flat file correlation

Hey guys,
I have a question regarding the following scenario:
Portal(WebDynpro) XML file->XI>ftp server->legacy system(outside the organization LAN)>ftp server(response file with errors )->correlation XI->portal database\portal iView.
I need to create a synchronous scenario with error handling and rollback of the process.
The flat file is created per entity (user data entered on the portal,Employee data,etc..)
I'm talking about 5000 files created per hour from the portal (there are also file processes needed to be correlated sending from the legacy system).
My question is : is it reasonable to use correlation in ordered to create this synchronous scenario?there are gonna be many bpm process open and waiting to be continued...
What do you recon????
Nimrod

Nimrod
I dont see an issue unless you have duplicate correlations..We have tested it with similar loads , it works perfectly fine!!

Upload data from flat file

Is there a way to upload data from a flat file and check of existing records and perform updates and inserts respectively on a table using MERGE as available in Oracle 9i version.
I don't want to use SQLLOADER utility for this purpose as I will be having existing data which might give exceptions for primary key, etc.
Is it possible to do this using UTL_FILE utility and use of MERGE statement.
Any inputs will be highly appreciated.
Thanks

This is what i am doing as of today :-
This process reads the flat file and uploads and updates data into 2 different tables.
create or replace procedure EPS.GET_DATA(filelocation varchar2,filename varchar2)
IS
--declare
my_file utl_file.file_type;
-- variables used to parse the input data line ...
line char(45);
l_EPCI_CID_ID char(25);
l_cid number :=0;
l_EPCI_BILLING_ACCOUNT_ID char(15);
l_account number :=0;
l_EPCI_BILLING_COMPANY_CD char(2);
l_EPCI_SOURCE_SYSTEM_CD char(3);
-- timing variables ...
l_start_time date;
l_end_time date;
l_difference number;
l_run_id varchar2(40) := '';
counter number := 0;
mrow number :=0;
oldCustomerCid number :=0 ;
rowsFound number :=0;
rowsFoundForCustomer number :=0;
rowsFound2 number :=0;
rowsFound3 number :=0;
mUpdate number :=0;
cUpdate number :=0;
cInsert number :=0;
cSkipped number :=0 ;
mrowupd number :=0;
rowsSkipped number :=0;
begin
-- Open the input file ....
begin
dbms_output.put_line('opening file :'||filename||' from location :'||filelocation);
my_file := utl_file.fopen(filelocation,filename,'r');
exception
when utl_file.invalid_filehandle then
dbms_output.put_line('invalid_filehandle');
raise_application_error(-20100,'file error');
when no_data_found then
dbms_output.put_line('no data found exception');
raise_application_error(-20100,'file error');
when others then
null;
end;
-- Read data and insert it into the table ...
dbms_output.put_line('program started: now = '|| to_char(sysdate,'HH:MI:SS'));
l_start_time := sysdate;
begin
dbms_output.put_line('looping cursor to search existing groups');
loop
utl_file.get_line(my_file,line);
l_EPCI_CID_ID := substr(line,1,25);
l_cid := to_number(l_EPCI_CID_ID);
l_EPCI_BILLING_ACCOUNT_ID := substr(line,26,15);
l_account :=to_number(l_EPCI_BILLING_ACCOUNT_ID);
l_EPCI_BILLING_COMPANY_CD := substr(line,41,2);
l_EPCI_SOURCE_SYSTEM_CD := substr(line,43,3);
-- Populating Mad.Mad_Customer_Profile table with EPS CID, If CID exists then updating MCP_LAST_INVOICE_LOAD_DT with system date
if(oldCustomerCid <> l_cid) then
Select count(*) into rowsFoundForCustomer from MAD.MAD_CUSTOMER_PROFILE WHERE MCP_CID_ID = l_cid ;
if rowsFoundForCustomer=0 then
Insert into MAD.MAD_CUSTOMER_PROFILE values(l_cid,sysdate,sysdate,'EBILL');
cInsert := cInsert+1;
else
Update MAD.MAD_CUSTOMER_PROFILE set MCP_LAST_INVOICE_LOAD_DT=SYSDATE,
MCP_UPDATE_DATE=SYSDATE, MCP_USER_ID='EBILL' where MCP_CID_ID=l_cid;
cUpdate := cUpdate+1;
end if;
else
cSkipped := cSkipped+1;
end if ;
oldCustomerCid := l_cid;
-- Populating EPS.EPS_CBS_INVOICE_CUSTOMER table with the flat file data.
select count(*) into rowsFound
from EPS.EPS_CBS_INVOICE_CUSTOMER
WHERE EPCI_CID_ID=l_cid and
to_number(EPCI_BILLING_ACCOUNT_ID)=l_account
AND rtrim(EPCI_BILLING_COMPANY_CD)=rtrim(l_EPCI_BILLING_COMPANY_CD)
AND rtrim(EPCI_SOURCE_SYSTEM_CD)=rtrim(l_EPCI_SOURCE_SYSTEM_CD)
and rtrim(EPCI_ACTIVE_IND)='Y';
if rowsFound=0 then
Declare
Cursor c1 is
select * from EPS.EPS_CBS_INVOICE_CUSTOMER
WHERE
to_number(EPCI_BILLING_ACCOUNT_ID)=l_account
AND rtrim(EPCI_BILLING_COMPANY_CD)=rtrim(l_EPCI_BILLING_COMPANY_CD)
AND rtrim(EPCI_SOURCE_SYSTEM_CD)=rtrim(l_EPCI_SOURCE_SYSTEM_CD)
and EPCI_ACTIVE_IND='Y';
myRec eps.eps_cbs_invoice_customer%ROWTYPE ;
Begin
if not c1%isopen then
open c1 ;
end if;
loop
fetch c1 into myRec;
exit When c1%notfound;
UPDATE EPS.EPS_CBS_INVOICE_CUSTOMER
SET EPCI_ACTIVE_IND='N',EPCI_UPDATE_DT=SYSDATE,EPCI_UPDATE_USER_ID='EBILL'
WHERE
--EPCI_CID_ID=L_CID and
TO_NUMBER(EPCI_BILLING_ACCOUNT_ID)=l_account
AND rtrim(EPCI_BILLING_COMPANY_CD)=rtrim(l_EPCI_BILLING_COMPANY_CD)
AND rtrim(EPCI_SOURCE_SYSTEM_CD)=rtrim(l_EPCI_SOURCE_SYSTEM_CD);
mrowupd := mrowupd+1;
end loop;
close c1;
insert into EPS.EPS_CBS_INVOICE_CUSTOMER(EPCI_CID_ID,EPCI_BILLING_ACCOUNT_ID,
EPCI_BILLING_COMPANY_CD,EPCI_SOURCE_SYSTEM_CD,EPCI_ACTIVE_IND,EPCI_UPDATE_DT,EPCI_UPDATE_USER_ID)
Values(l_cid,l_EPCI_BILLING_ACCOUNT_ID,l_EPCI_BILLING_COMPANY_CD,l_EPCI_SOURCE_SYSTEM_CD,'Y',SYSDATE,'EBILL');
mrow := mrow+1;
end ;
Else
rowsSkipped:=rowsSkipped+1;
END IF;
end loop;
Exception
when no_data_found then
utl_file.fclose(my_file);
dbms_output.put_line('Number of lines parsed =');
when value_error then
dbms_output.put_line('value error');
raise_application_error(-20100,'file error');
when utl_file.invalid_path then
dbms_output.put_line('invalid path');
raise_application_error(-20100,'file error');
when utl_file.invalid_mode then
dbms_output.put_line('invalid_mode');
raise_application_error(-20100,'file error');
when utl_file.invalid_filehandle then
dbms_output.put_line('invalid_filehandle');
raise_application_error(-20100,'file error');
when utl_file.invalid_operation then
dbms_output.put_line('invalid_operation');
raise_application_error(-20100,'file error');
when utl_file.read_error then
dbms_output.put_line('read_error');
raise_application_error(-20100,'file error');
when utl_file.write_error then
dbms_output.put_line('write_error');
raise_application_error(-20100,'file error');
when utl_file.internal_error then
dbms_output.put_line('internal_error');
raise_application_error(-20100,'file error');
WHEN DUP_VAL_ON_INDEX THEN
dbms_output.put_line('DUPLICATE RECORD FOUND WHILE INSERTING RECORD');
when others then
dbms_output.put_line('un-handled');
raise_application_error(-20100,'file error');
null;
end;
-- Now commit and close the file ...
COMMIT;
dbms_output.put_line('program finished: now = '||
to_char(sysdate,'HH:MI:SS'));
dbms_output.put_line('Rows Inserted into EPS table :'|| mrow);
dbms_output.put_line('Rows Updates in EPS table:'|| mrowupd);
dbms_output.put_line('Rows Skipped :'|| rowsSkipped);
dbms_output.put_line('Rows Inserted into Customer table :'|| cInsert);
dbms_output.put_line('Rows Updates in Customer table:'|| cUpdate);
dbms_output.put_line('Duplicate CID''s Skipped :'|| cSkipped);
l_end_time := sysdate;
l_difference := l_end_time - l_start_time;
utl_file.fclose(my_file);
end;

Beginner Process chains on flat files

Similar Messages

Maybe you are looking for