Processing file in chunks

I have a file structure like Header-Transaction Records(100K)-Trailer with Transaction Records of 100k.
I want to process the this file through file-xi-file scenario. I am using the "Recordsets per message"
(5000 records per each chunk)parameter in the sender file communicaiton channel to process in chunks. But I didn't observed anything differce in processing by XI. The file is processed same way as it before without any "Recordsets per message" parameter set. How can I monitor that this parameter is took into effect and the system performance is improved.
If I got to write a operating sytstem command/script to split , how to write it for the file structure that i am
talking abt above ??
thank
kumar

Hi Palnati
This works for FCC and NFS else its not having any effect.
I replied your question in other thread.
handling huge files
Thanks
Gaurav

Similar Messages

Processing Large Files using Chunk Mode with ICO

Hi All,
I am trying to process Large files using ICO. I am on PI 7.3 and I am using new feature of PI 7.3, to split the input file into chunks.
And I know that we can not use mapping while using Chunk Mode.
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
Thanks in Advance,
- Pooja.

Hello,
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
According to this blog:
File/FTP Adapter - Large File Transfer (Chunk Mode)
The following limitations apply to the chunk mode in File Adapter
As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following limitations would apply
Only for File Sender to File Receiver
No Mapping
No Content Based Routing
No Content Conversion
No Custom Modules
Probably you are doing content conversion that is why it is not working.
Hope this helps,
Mark
Edited by: Mark Dihiansan on Mar 5, 2012 12:58 PM

Read file in chunks

hello,
as many of you know i'm trying to solve a problem memory with labview 7.1
many of you told me to read file in chunks and i'm trying to do it...
but i have some questions...
in attachment you find see my VI (it's incomplete): i would read my files 1 Mb for each step (now my file is 80 Mb)...i would ask you what i have to link to count input of the readfile vi in the true case (the last for-cycle-step, where i have to save the last part of my file).
Should it be correct done in this way?
Have i done the save-process in the correct way? (initialize the array knowing the array size and the using the arrayreplacesubset vi in the for loop)
thanks all
Using LabVIEW 7.1
Attachments:
ReadFileInChunks.vi ‏58 KB

EOF function give me the size of the file and not the offset of the EOF?
so in which way could i stop the while cycle?
yes,,,i'm reading the file line by line and the i'll concatenate strings (1000 in the example) to obtain a chunk of data
if i put in the count input of the read file function (with the line mode True) i don't obtain all lines in the output...i don't know why
you can see the image in attachment
in fact was to solve my problem that i tryed to concatanate strings...if i could receive in the output 1000 lines in a automatic way should be better...
and yes...in my case i'll make my operation in the case structure, when i obtaine 1000 lines
Using LabVIEW 7.1
Attachments:
examplee.png ‏43 KB

Creating database problem with ORA-01519: error while processing file '' ne

Dear all,
I am having rough time with creating database manually. can anyone help me with the following errors.
ALERT LOGFILE:-
Sun May 31 12:00:39 2009
Errors in file d:\oracle\product\10.1.0\admin\oracle5\udump\oracle5_ora_3444.trc:
ORA-01501: CREATE DATABASE failed
ORA-01519: error while processing file '%ORACLE_HOME%\RDBMS\ADMIN\SQL.BSQ' near line 5744
ORA-00604: error occurred at recursive SQL level 1
ORA-01119: error in creating database file 'D:\oracle\product\10.1.0\oradata\ORACLE5 \datafile\undotbs1.dbf'
ORA-27040: file create error, unable to create file
OSD-04002: unable to open file
O/S-Error: (OS 3) The system cannot find the path specified.
Error 1519 happened during db open, shutting down database
USER: terminating instance due to error 1519
Sun May 31 12:00:39 2009
Errors in file d:\oracle\product\10.1.0\admin\oracle5\bdump\oracle5_reco_3416.trc:
ORA-01519: error while processing file '' near line
CREATE DATABASE SCRIPT:-
create database ORACLE5
user sys identified by bala
user system identified by bala
logfile
group 1 ('D:\oracle\product\10.1.0\oradata\ORACLE5\logfile\redo01.log') SIZE 50M,
group 2 ('D:\oracle\product\10.1.0\oradata\ORACLE5\logfile\redo02.log') SIZE 50M,
group 3 ('D:\oracle\product\10.1.0\oradata\ORACLE5\logfile\redo03.log') SIZE 50M
maxlogfiles 5
maxlogmembers 5
maxloghistory 1
maxdatafiles 50
maxinstances 1
datafile 'D:\oracle\product\10.1.0\oradata\ORACLE5\datafile\system01.dbf' size 200m autoextend on maxsize unlimited extent management local
undo tablespace UNDOTBS1 datafile 'D:\oracle\product\10.1.0\oradata\ORACLE5 \datafile\undotbs1.dbf' size 10m autoextend on maxsize unlimited
sysaux datafile 'D:\oracle\product\10.1.0\oradata\ORACLE5\datafile\sysaux01.dbf' size 100m autoextend on maxsize unlimited
default temporary tablespace temp tempfile 'D:\oracle\product\10.1.0\oradata\ORACLE5 \datafile\temp01.dbf' size 10m autoextend on maxsize unlimited
character set US7ASCII
NATIONAL CHARACTER SET AL16UTF16
INIT.ORA FILE:-
# Copyright (c) 1991, 2001, 2002 by Oracle Corporation
# Archive
log_archive_format=ARC%S_%R.%T
# Cache and I/O
db_block_size=8192
db_cache_size=25165824
db_file_multiblock_read_count=16
# Cursors and Library Cache
open_cursors=300
# Database Identification
db_domain=""
db_name=oracle5
# Diagnostics and Statistics
background_dump_dest=D:\oracle\product\10.1.0\admin\oracle5\bdump
core_dump_dest=D:\oracle\product\10.1.0\admin\oracle5\cdump
user_dump_dest=D:\oracle\product\10.1.0\admin\oracle5\udump
# File Configuration
#db_create_file_dest=D:\oracle\product\10.1.0\oradata
db_recovery_file_dest=D:\oracle\product\10.1.0\flash_recovery_area
db_recovery_file_dest_size=2147483648
# Job Queues
job_queue_processes=10
# Miscellaneous
compatible=10.1.0.2.0
# Pools
java_pool_size=50331648
large_pool_size=8388608
shared_pool_size=83886080
# Processes and Sessions
processes=150
# Security and Auditing
remote_login_passwordfile=EXCLUSIVE
# Shared Server
dispatchers="(PROTOCOL=TCP) (SERVICE=oracle5XDB)"
# Sort, Hash Joins, Bitmap Indexes
pga_aggregate_target=25165824
sort_area_size=65536
# System Managed Undo and Rollback Segments
undo_management=AUTO
undo_tablespace=UNDOTBS1
control_files=("D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORACLE5\CONTROLFILE\control01.CTL", "D:\ORACLE\PRODUCT\10.1.0\FLASH_RECOVERY_AREA\ORACLE5\CONTROLFILE\control02.CTL")

Hi,
undo tablespace UNDOTBS1 datafile 'D:\oracle\product\10.1.0\oradata\ORACLE5 \datafile\undotbs1.dbf' size 10m autoextend on maxsize unlimited
After ORACLE5 there is a space character. Typo? Create proper directory or delete this character.
Bartek

ORA-01519: error while processing file '?/rdbms/admin/sql.bsq' near line 401

I receive the following error by creating dataabse as below sql:
ORA-01501: CREATE DATABASE failed
ORA-01519: error while processing file '?/rdbms/admin/sql.bsq' near line 401
ORA-00604: error occurred at recursive SQL level 1
ORA-01119: error in creating database file '/sisdev04/dbs/users01.dbf'
ORA-27039: create file failed, file size limit reached
Additional information: 1
CREATE DATABASE SISDEV04
     LOGFILE GROUP 1 ('/sisdev04/dbs/log01a.dbf','/sisdev04/dbs/log01b.dbf') SIZE 10M,
          GROUP 2 ('/sisdev04/dbs/log02a.dbf','/sisdev04/dbs/log02b.dbf') SIZE 10M
     NOARCHIVELOG
     CHARACTER SET ZHT16BIG5
     NATIONAL CHARACTER SET ZHT16BIG5
     DATAFILE '/sisdev04/dbs/system01.dbf'
          SIZE 210M AUTOEXTEND OFF
     , '/sisdev04/dbs/rbs01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/temp01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/users01.dbf'
SIZE 1050M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_index_ivr01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_ivr01.dbf'
SIZE 21M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_ems01.dbf'
SIZE 21M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_index_ems01.dbf'
SIZE 21M AUTOEXTEND OFF
I have issue df to check the directory sisuat04
/ >df
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 98304 45664 54% 2446 10% /
/dev/hd2 2097152 78992 97% 30280 12% /usr
/dev/hd9var 65536 20536 69% 308 4% /var
/dev/hd3 294912 262432 12% 147 1% /tmp
/dev/hd1 49152 5712 89% 1290 21% /home
/dev/mfcobol 81920 25520 69% 608 6% /mfcobol
/dev/magiclv 81920 22304 73% 346 4% /usr/magic
/dev/sispro02lv 6553600 6347792 4% 18 1% /sisdev02
/dev/sispro03lv 6553600 6347784 4% 19 1% /sisdev03
/dev/sisappllv 229376 173288 25% 1846 7% /sisappl
/dev/lvbaklog 98304 85552 13% 68 1% /app/bak_log
/dev/noteslv0 1048576 541352 49% 2195 2% /notes
/dev/noteslv1 4194304 1479304 65% 5269 2% /notesrv
/dev/noteslv2 2097152 898128 58% 51 1% /notesrv/sitesch
/dev/lvoracle 2097152 237240 89% 8280 4% /opt/oracle
/dev/u01lv 4194304 128720 97% 11697 3% /u01
/dev/sisdev01lv 4096000 3021552 27% 27 1% /sisdev04
Is is still a not enough space problem?

I receive the following error by creating dataabse as below sql:
ORA-01501: CREATE DATABASE failed
ORA-01519: error while processing file '?/rdbms/admin/sql.bsq' near line 401
ORA-00604: error occurred at recursive SQL level 1
ORA-01119: error in creating database file '/sisdev04/dbs/users01.dbf'
ORA-27039: create file failed, file size limit reached
Additional information: 1
CREATE DATABASE SISDEV04
     LOGFILE GROUP 1 ('/sisdev04/dbs/log01a.dbf','/sisdev04/dbs/log01b.dbf') SIZE 10M,
          GROUP 2 ('/sisdev04/dbs/log02a.dbf','/sisdev04/dbs/log02b.dbf') SIZE 10M
     NOARCHIVELOG
     CHARACTER SET ZHT16BIG5
     NATIONAL CHARACTER SET ZHT16BIG5
     DATAFILE '/sisdev04/dbs/system01.dbf'
          SIZE 210M AUTOEXTEND OFF
     , '/sisdev04/dbs/rbs01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/temp01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/users01.dbf'
SIZE 1050M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_index_ivr01.dbf'
SIZE 105M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_ivr01.dbf'
SIZE 21M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_ems01.dbf'
SIZE 21M AUTOEXTEND OFF
     , '/sisdev04/dbs/ts_index_ems01.dbf'
SIZE 21M AUTOEXTEND OFF
I have issue df to check the directory sisuat04
/ >df
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 98304 45664 54% 2446 10% /
/dev/hd2 2097152 78992 97% 30280 12% /usr
/dev/hd9var 65536 20536 69% 308 4% /var
/dev/hd3 294912 262432 12% 147 1% /tmp
/dev/hd1 49152 5712 89% 1290 21% /home
/dev/mfcobol 81920 25520 69% 608 6% /mfcobol
/dev/magiclv 81920 22304 73% 346 4% /usr/magic
/dev/sispro02lv 6553600 6347792 4% 18 1% /sisdev02
/dev/sispro03lv 6553600 6347784 4% 19 1% /sisdev03
/dev/sisappllv 229376 173288 25% 1846 7% /sisappl
/dev/lvbaklog 98304 85552 13% 68 1% /app/bak_log
/dev/noteslv0 1048576 541352 49% 2195 2% /notes
/dev/noteslv1 4194304 1479304 65% 5269 2% /notesrv
/dev/noteslv2 2097152 898128 58% 51 1% /notesrv/sitesch
/dev/lvoracle 2097152 237240 89% 8280 4% /opt/oracle
/dev/u01lv 4194304 128720 97% 11697 3% /u01
/dev/sisdev01lv 4096000 3021552 27% 27 1% /sisdev04
Is is still a not enough space problem? i think you have specified 1050m for users01.dbf file, your file system /sisdev04 has something like
1.5gb of free space. it failed because of lack of free space. may be you can reduce your datafile size or try to
create in another filesystem.

Issue while archiving the processed file in sender communication channel using SFTP adapter

Hi All,
In one of my scenario (File to IDOC), we are using SFTP sender communicationchannel.
we are facing an issue while archiving the processed file. Some times PI processed the file successfully but unable to archive it and in the next poll PI process & archives the same file successfully which will creates duplicate orders in ECC.
Please let us know how to resolve this issue.

Hi Anil,
Refer Archiving concepts in below links.
http://help.sap.com/saphelp_nw73/helpdata/en/44/682bcd7f2a6d12e10000000a1553f6/content.htm?frameset=/en/44/6830e67f2a6d12e10000000a1553f6/frameset.htm
http://scn.sap.com/docs/DOC-35572
Warm Regards,
DNK Siddhardha.

Post Processing file

I am reading file in labview which contains some comments separated by backslash but i dont want those commensts to be dispalyed after reading to be displayed in indicator. How to do this.
Below is attached vi and file
Solved!
Go to Solution.
Attachments:
post processing file.vi ‏8 KB
qwerty.txt ‏1 KB

I tried this match pattern also, with this i am facing two problems first, it is detecting only for one time whereas in my file i am having multiple comments, and it is detecting only the first comment it sees and second issue is that it is just separating the backslash from the string and i want to separate the whole string with backslash. May be i am not using it properly
below is the snippet of how i am doing.
Attachments:
read from file fp.png ‏219 KB
readfromfilebd.png ‏31 KB

My tv show and movies are getting stuck in the processing file stage

I purchased some TV show episodes and movies on itunes. They are stuck in the processing file stage, and wont let me pause them and restart. When I close itunes it wont open again. I have to restart my computer. Please help, they have been stuck for over a week now.

please contact me on [email protected]

TS3297 Hi I was down loading IOS 6 to my iPhone through I tunes got as far as processing file and the error message of error 5000 came up and said I would loose everything off my iPhone if I proceeded. I managed to down load for iPad fine. I have an iPhon

Just a beginner here first log in. Question is what do I do if I get an error message 5000 iF you progress you loose every thing - was trying to download IOS 6 onto my iPhone 4 and got to the very end when it ways processing file. Thank you

Errors 3000-3999 (3004, 3013, 3014, 3018, 3164, 3194, and so on): Error codes in the 3000 range generally mean that iTunes cannot contact the update server (gs.apple.com) on ports 80 or 443.
Update to the latest version of iTunes.
Verify the computer's date and time are accurate.
Check that your security or firewall software is not interfering with ports 80 or 443, or with the server gs.apple.com.
Follow Troubleshooting security software. Often, uninstalling third-party security software will resolve these errors.
An entry in your hosts file may be redirecting requests to gs.apple.com (see "Unable to contact the iOS software update server gs.apple.com" above).
Internet proxy settings can cause this issue. If you are using a proxy, try without using one.
Test restoring while connected to a known-good network.

Attempt to process file failed with Exception in XML Parser-format problem

Hi all,
Iam getting an unusual error in the J2EE stack in XI.
And the message is:
006-11-30 17:31:07 Error Attempt to process file failed with Exception in XML Parser (format problem?):'com.sap.engine.lib.xml.parser.NestedSAXParserException: Fatal Error: com.sap.engine.lib.xml.parser.ParserException: Invalid char #0xf(:main:, row:1, col:1044002)(:main:, row=1, col=1044002) -> com.sap.engine.lib.xml.parser.ParserException: Invalid char #0xf(:main:, row:1, col:1044002)' 2006-11-30 17:31:07 Error Exception caught by adapter framework: null 2006-11-30 17:31:07 Error Delivery of the message to the application using connection AFW failed, due to: RecoverableException.
My scenerio iam posting IDOC to a flat file with content conversion in the receiver side,the mapping got executed successfully and in the audit log i found that the error was after the 'Start converting XML document content to plain text'.
This means that error occured during content conversion of XML to the prescribed file format.
Can anyone suggest any better approach using which we may trace the junk data in IDoc. Manual adhoc approach could take time and is error prone.
Thanks in advance...
karun

Hi Bhavesh,
Thanks for the early reply. I checked the mapping and everything is fine and the output is also in valid XML format.
The audit log shows that the mapping got executed successfully and the error is after the step 'Start converting XML document content to plain text '. Is there any constraint in the file adapter regarding the message size for parsing.
2006-11-30 17:30:50 Success Transfer: "BIN" mode, size 2912595 bytes, character encoding -
2006-11-30 17:30:50 Success Start converting XML document content to plain text
2006-11-30 17:31:07 Error Attempt to process file failed with Exception in XML Parser (format problem?):'com.sap.engine.lib.xml.parser.NestedSAXParserException: Fatal Error: com.sap.engine.lib.xml.parser.ParserException: Invalid char #0xf(:main:, row:1, col:1044002)(:main:, row=1, col=1044002) -> com.sap.engine.lib.xml.parser.ParserException: Invalid char #0xf(:main:, row:1, col:1044002)'
2006-11-30 17:31:07 Error Exception caught by adapter framework: null
2006-11-30 17:31:07 Error Delivery of the message to the application using connection AFW failed, due to: RecoverableException.
2006-11-30 17:31:07 Success The asynchronous message was successfully scheduled to be delivered at Thu Nov 30 17:36:07 GMT 2006.
2006-11-30 17:31:07 Success The message status set to WAIT.
2006-11-30 17:31:08 Success Acknowledgement creation triggered for type: SystemErrorAck
2006-11-30 17:31:08 Success Acknowledgement sent successfully for type: SystemErrorAck
2006-11-30 17:36:08 Success Retrying to deliver message to the application. Retry: 1

How I can get rid of "Processing File" problem in downloading my apps?

hi, I have problems updating and sometimes downloading the apps in iTunse. when I want to update an app in iTunes( using windows 8) the app stuck says "processing file" and remain for hours and never finish again. what can I do for it. now that I live in Iran. and sometimes I have to use a vpn for downloading the files and when it downloaded a bit I disconnect the vpn and resume the download. please help me how to get rid of this annoying problem. I am using the latest iTunes. in fact in every iTunes update I was looking forward to seeing that this problem is solved but it wasn't. thanks in avanced and bye.

See Here for assistance...
iTunes Customer Service Contact
http://www.apple.com/support/itunes/contact.html

Can we process files with attachments using File Adapter

Hi All,
I want to process files which have attachments using File Adapter.
Is it possible to process such files?
May be using standard modules?
Regards,
CBKLP

Assuming you are talking about sender FILE channel....you can make use of PayloadSwapbean provided by SAP (standard module)...to read data from attachment.
Regards,
Abhishek.

How to process files one by one using sender file adapter

Hi,
I have to process file one by one using my file adapter (sender), because while doing the G/R materials get locked, if the same materal exist in other files.
I have maintained QOS as "Exactly Once in Order" & processing Sequence as "by date".
Processing sequence come in which tab? What will be the Queue name?
Thanks,
krishna

It will come under Processing tab
Here you need to select the first parameter QOS as EOIO
Then QUEUE name and processing sequence will be enabled.
NOTE: This will work only for NFS protocol not for FTP.

File adapter issue - files not processed (file name mask is *)

Hi guys!
I have a problem I never faced before. My file adapter doesn't process files in the source directory. When I put there exact file's name, it is processed. But when I use * , nothing happen. This is very strange. I was wondering, if problem can't be in that my PI box is UNIX and it's accessing Win filesystem.
Any ideas, what to try?
Thank you!
Olian

Yes, but I meant the one directory we're working in. We can write, delete there, so we should be able also use wildcards, don't we?
However, I tried it on a directory with 777 attributes. There is a file named clooo.ttt. When I run del .ttt I get Requested action not taken, file unavailable. The same for clooo. , clooo*,... But when I type del clooo.ttt, the file is deleted...
Any suggestions?
Olian

Could not process file 'Input 110110b.txt': com.sap.aii.adapter.file.

I have a FILE Adapter using FTP as a transport that is generating a very odd error.
The FTP sender channel is giving the following error is set up to retreive all files (*) from a specific directory. If you look at the error, it indicates an error processing the file 'Input 110110b.txt' as a file -- the only way this filename woudl be visible is if the adapater interrogated the directory, or the filename was hard-coded in the file mask. But the error indicates it can't change to the directory. The only way it would have come up with the file is if it had already CD'd t that directory.
1/7/11 8:12:01 AM   Retry interval started. Length: 60.0 seconds
1/7/11 8:12:01 AM   Could not process file 'Input 110110b.txt': com.sap.aii.adapter.file.ftp.FTPEx: 550 Failed to change directory.
1/7/11 8:12:00 AM   Processing started
My first thought was a simple permission issue, so I opened a connection to server with FileZilla (an FTP client) and was able to retrieve the file without a problem. I was also able to drop a file in the directory. There is no permissions problem.

Hi
Please check the below points before processing any file
a) check the File placing by a Local Plant FTP owner is using the same authentification credentials It means both PI and source team needs to use same FTP User ID and login credentials otherwise if suppose if source team is placing with one User and PI cannot able to pick that file as this User cannot have permissions in this case also u will get 550 error
b) I know what ur telling is u manualy palced one file and able retrive...but do one thing u can login with same PI FTP login server credentials and then place the file with different name also I suggest lets test the channel by simply keeping
. not simply * if you keep simply * then u cna use in receiver chanel while using dynamic connfiguration.
Else give *.txt and then activate the channel and also please check the CACHE as well
Regards
Amar Srinivas Eli

Processing file in chunks

Similar Messages

Maybe you are looking for