Flat File automation process - limitations

Question

Hello Everyone,
I would really appreciate any insight on the process improvement suggestions.
Background:
Currently we have around 12 territories providing a new flat file with new data on a daily basis depending on the business activity. Which would also mean that, on a given day if there is no activity would mean no flat file provided to BI for loading process.
The flat files provided need to be loaded into the BI system (PSA - DSO - InfoCube).
The flat file loading process has been automated for the daily file by implementing the logical file name for each territory.
1. The process variant in the process chain is to ensure if the flat file is available on the App server (Custom ABAP program).
2. 12 InfoPackages have been created to pick the data from the flat file on the app server and load the data over into the PSA.
3. All the InfoPackages merge into an "AND" event in the process chain before the DTP load into the DSO kicks off.
4. DSO Activation
5. Recon between the flat file and the DSO to ensure all the data from flat file has been loaded into the DSO.
6. DTP loads into the InfoCube.
7. Recon between the InfoCube and the DSO itself.
8. Moving the flat file from one folder into another.
All the above processes are automatically performed without any issues if the flat file is available on the server.
Problem / Issue:
As one of the major limitations of the above design is the flat file for sure needs to be made available on the app server in order for the whole data flow in the process chain to continue without any breakpoints.
Current workaround / process improvement in place:
Based on the above limitation and upon further research, I was able to apply the OSS Note to give us the option of maintaining multiple DTPs for the same data target with different filter values.
So, even if have individual data stream for each territory with a different DTP the issue still remains where the process variant (ABAP program to check if file exists) or the InfoPackage load if the ABAP program is removed will fail.
Which means due to the above fact, the support team is alerted about the process chain failure.
Question / Suggestions required:
The main questions or any suggestions would be welcome, if one of you can let us know an approach where the flat file check program doesn't have to give a hard failure in the process chain for the rest of the process chain to continue with the loading process. (As in order for the rest of the process chain to continue the only options we have are Error, Success, Always).
I have also looked into the Decision process variant available in the process chain, but based on the options available within I cannot utilize it for the loading process.
Error can be caused by generating an error message in the ABAP program which in turn is causing the issue of alert being sent over even if the rest of the process chain finishes.
Success would mean the flat file needs to be available. Always cannot be used in this case as it will cause a failure at the InfoPackage level.
If the InfoPackage load can be avoided without a hard error to be generated, the process chain does not have to remain in the failed state which in turn will not trigger any alert to the support team.
Please do let me know if you need more details about the above process improvement question.
Thanks
Dharma.

Answer 1

The main issue with this as you mentioned is that the file has to be available for sure.
We had a similar issue - we had a very critical data load that had to happen everyday , failure of the file getting loaded would mean that the reports for the day would be delayed.
We were running on UNIX and we implemented a simple UNIX command that would not complete till the file was available in the app server.
Something like
while ( the file does not exist )
delay of 15 seconds
you will come out of the while only after the while completes which means that the file becomes available.
You can write a similar ABAp program to check file availability if required and put it into your program.
we also had a failover process where we created a zero byte file with the same name if the file did not come beyond a certain number of tries and the PSA would load zero records and the data load will continue.
Edited by: Arun Varadarajan on Jan 26, 2009 10:18 AM