Ftp and File polling

Hi
Do anyone have sample code for file polling.
i am using linux environment.
If any client ftp a file i have to poll the file and parse the file csv , if file is parsed and good then send to oracle database and archive the file.
I have to do everything java
if one had done this kind of application, send me the sample code.
Thanks in advance
sami

Actually client using any ftp client programm and send or ftp a file on the my ftp server from there i have to check which file is new for every 1hour and i have to take that file, parse (csv parse) and insert the data to oracle database and archive the file, if file is good otherwise i have to send email to some dept saying file is bad.
could u have any idea or sample code.
Thanks
sami

Similar Messages

Using JCA FTP and File Adapters to transfer files as attachments

I am trying to develop a flow that picks a file up using the JCA FTP adapter and writes it using the JCA File Adapter. I can get this to work using opaqeElement but the issue I have is that the files that will be transferred are regularly very large > 1 GB and using the opaqueElement simply results in a OutOfMemoryException.
Reading the Adapter documentation it seems that Reading/Writing file as an attachment may be an answer. I therefore did the following.
. Created FTP Read and File Write adapters in JDeveloper
. Inmported this in to OEPE
. Created Proxy Service and Business Service and wired them together
The problem i am having is that even though there is no error and the file is being read on the FTP server, the resulting file only contains that attachment details, the file is not a copy of the file that was picked up from the ftp server.
I am assuming that this behaviour is not correct and that the file should be transferred in its entirity. Unfortunatly the documentation is not very detailed and I cannot find any examples on the internet.
Can anyone offer any advice or even better suggest where I can find an example.
Regards
My JCA adapters are as follows:
FileWriteAsAttachmentAdapter_file
<adapter-config name="FileWriteAsAttachmentAdapter" adapter="File Adapter"
     wsdlLocation="FileWriteAsAttachmentAdapter.wsdl"
     xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">
<connection-factory location="eis/HAFileAdapter"/>
<endpoint-interaction portType="Write_ptt" operation="Write">
<interaction-spec className="oracle.tip.adapter.file.outbound.FileInteractionSpec">
<property name="PhysicalDirectory" value="C:\fileserver\download\toprocess"/>
<property name="Append" value="false"/>
<property name="FileNamingConvention" value="attachment_%SEQ%.csv"/>
<property name="NumberMessages" value="1"/>
</interaction-spec>
</endpoint-interaction>
</adapter-config>
FileWriteAsAttachmentAdapter.wsdl
<wsdl:definitions
name="FileWriteAsAttachmentAdapter"
targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/file/AtatchmentApp/Project1/FileWriteAsAttachmentAdapter"
xmlns:jca="http://xmlns.oracle.com/pcbpel/wsdl/jca/"
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:tns="http://xmlns.oracle.com/pcbpel/adapter/file/AtatchmentApp/Project1/FileWriteAsAttachmentAdapter"
xmlns:imp1="http://xmlns.oracle.com/pcbpel/adapter/file/attachment/"
xmlns:plt="http://schemas.xmlsoap.org/ws/2003/05/partner-link/"
>
<plt:partnerLinkType name="Write_plt" >
<plt:role name="Write_role" >
<plt:portType name="tns:Write_ptt" />
</plt:role>
</plt:partnerLinkType>
<wsdl:types>
<schema targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/file/attachment/" xmlns="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified" >
<element name="attachmentElement" >
<complexType>
<attribute name="href" type="string" />
</complexType>
</element>
</schema>
</wsdl:types>
<wsdl:message name="Write_msg">
<wsdl:part name="body" element="imp1:attachmentElement"/>
</wsdl:message>
<wsdl:portType name="Write_ptt">
<wsdl:operation name="Write">
<wsdl:input message="tns:Write_msg"/>
</wsdl:operation>
</wsdl:portType>
</wsdl:definitions>
FtpGetAsAttachmentAdapter_ftp.jca
<adapter-config name="FtpGetAsAttachmentAdapter" adapter="FTP Adapter"
     wsdlLocation="FtpGetAsAttachmentAdapter.wsdl"
     xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">
<connection-factory location="eis/ftp/TlmsHAFtpAdapter"/>
<endpoint-activation portType="Get_ptt" operation="Get">
<activation-spec className="oracle.tip.adapter.ftp.inbound.FTPActivationSpec">
<property name="AsAttachment" value="true"/>
<property name="DeleteFile" value="true"/>
<property name="MinimumAge" value="300"/>
<property name="PhysicalDirectory" value="/ftp/tlms2atas"/>
<property name="Recursive" value="false"/>
<property name="FileModificationTime" value="FileSystem"/>
<property name="PollingFrequency" value="60"/>
<property name="FileType" value="ascii"/>
<property name="IncludeFiles" value=".*[_][0-9][0-9][-][0-9][0-9][-][0-9][0-9][_].*\.csv"/>
<property name="UseHeaders" value="false"/>
<property name="ModificationTimeFormat" value="dd/MM/yyyy HH:mm"/>
</activation-spec>
</endpoint-activation>
</adapter-config>
FtpGetAsAttachmentAdapter.wsdl
<wsdl:definitions
name="FtpGetAsAttachmentAdapter"
targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/ftp/AtatchmentApp/Project1/FtpGetAsAttachmentAdapter"
xmlns:jca="http://xmlns.oracle.com/pcbpel/wsdl/jca/"
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:tns="http://xmlns.oracle.com/pcbpel/adapter/ftp/AtatchmentApp/Project1/FtpGetAsAttachmentAdapter"
xmlns:attach="http://xmlns.oracle.com/pcbpel/adapter/file/attachment/"
xmlns:pc="http://xmlns.oracle.com/pcbpel/"
xmlns:plt="http://schemas.xmlsoap.org/ws/2003/05/partner-link/"
>
<plt:partnerLinkType name="Get_plt" >
<plt:role name="Get_role" >
<plt:portType name="tns:Get_ptt" />
</plt:role>
</plt:partnerLinkType>
<wsdl:types>
<schema targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/file/attachment/" xmlns="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified" >
<element name="attachmentElement" >
<complexType>
<attribute name="href" type="string" />
</complexType>
</element>
</schema>
</wsdl:types>
<wsdl:message name="Get_msg">
<wsdl:part name="attach" element="attach:attachmentElement"/>
</wsdl:message>
<wsdl:portType name="Get_ptt">
<wsdl:operation name="Get">
<wsdl:input message="tns:Get_msg"/>
</wsdl:operation>
</wsdl:portType>
</wsdl:definitions>

System resources are finite and have a threshold limit for processing. The Oracle SOA Suite, dependent on system resources, also has certain size limitations, largely due to the underlying resources beyond which the system cannot process incoming requests.
For example, Oracle JCA Adapters can process large payloads but the Oracle BPEL PM consumes huge memory when processing large payloads, which can cause OutOfMemory conditions and affect the whole system.
You can try File ChunkedRead option.This will read your large CSV file in chunks so chances of memory out will be less. You can define your own chunk size.
This is a feature of Oracle File and FTP Adapters that uses an invoke activity within a while loop to process the target file. This feature enables you to process arbitrarily
large files. See below link's section "4.5.5 Oracle File Adapter ChunkedRead"
http://www.orastudy.com/oradoc/selfstu/fusion/integration.1111/e10231/adptr_file.htm#BABDIABG
Thanks,
Ashu

FTP and File problems

I have Dreamweaver 8. Trying to manage a little league
website- not real computer savvy.
All of the files an folders in my panel group (right hand
column) are gone. Any idea on how I might get that back. I now have
to go up to 'file' drop down menu, choose 'open' and all folders
and file names are listed. Much easier when they were all on the
right column. Have no idea why they are gone.
I am also now getting a message everytime I try to update
something........ " FTP error occurred, cannot make connection to
host. Your login or password is incorrect". I am putting in the
correct password.
I recently updated to Windows Exp 7.0 and wonder if that had
anything to do with this program changing.
Thanks
Karen

> I recently updated to Windows Exp 7.0 and wonder if that
had anything to
> do
> with this program changing.
Yes - probably did. There are hundreds of posts here about
issues after
installing IE7. Have you updated to DW 8.0.2? Please do that
before doing
anything else.
Then, you need to create a new site definition for your site.
Clearly your
current definition has become hosed. That's a simple thing to
do.
Murray --- ICQ 71997575
Adobe Community Expert
(If you *MUST* email me, don't LAUGH when you do so!)
==================
http://www.dreamweavermx-templates.com
- Template Triage!
http://www.projectseven.com/go
- DW FAQs, Tutorials & Resources
http://www.dwfaq.com - DW FAQs,
Tutorials & Resources
http://www.macromedia.com/support/search/
- Macromedia (MM) Technotes
==================
"kwilson333" <[email protected]> wrote in
message
news:elhplt$2m0$[email protected]..
>
>
> I have Dreamweaver 8. Trying to manage a little league
website- not real
> computer savvy.
>
> All of the files an folders in my panel group (right
hand column) are
> gone.
> Any idea on how I might get that back. I now have to go
up to 'file' drop
> down
> menu, choose 'open' and all folders and file names are
listed. Much
> easier
> when they were all on the right column. Have no idea why
they are gone.
>
> I am also now getting a message everytime I try to
update
> something........ "
> FTP error occurred, cannot make connection to host. Your
login or
> password is
> incorrect". I am putting in the correct password.
>
> I recently updated to Windows Exp 7.0 and wonder if that
had anything to
> do
> with this program changing.
>
> Thanks
> Karen
>

FTP Adapter File Polling does not raise error when FTP server is down

Hi All,
Thanks in advance.
I am polling for new files in a FTP Server. However, if the server is down there is no way for me to catch the fault to notify someone. I am using fault-policies but that never gets triggered when polling an unavailable FTP Server.
One option is to have a completely separate scheduled process (SOA or not) that tries to connect to the servers but I am trying not to go that route just to keep things manageable and within the realm of the process.
I am sure that I am not the only one polling for new files in a FTP Server, so I thought to pick your brain to tackle this scenario.
Thanks for taking the time to read this.

I was contacted by our DBA team for an exception received while writing a file to a FTP location.
Then i looked at the exception message and asked them to check the whether they are able to contact the ftp server using command prompt then they said the user password got expired.
They reset the password and its working now.
Am i missing something here :(
Exception:
WSIF JCA Execute of operation 'Put' failed due to: Error sending file to FTP Server.
Unable to send file to server.
; nested exception is:
ORABPEL-11429
Error sending file to FTP Server.
Unable to send file to server.
Please ensure 1. Specified remote output Dir has write permission 2. Output filename has not exceeded the max chararters allowed by the OS and 3. Remote File System has enough space.
Thanks,
Vijay
Edited by: veejai24 on 25-Jul-2012 07:10

FTP and "File-in-use"

Dear all,
We have a scenario in which a file is written to the SAP application server which should be picked up by XI using the FTP adapter. The polling interval of the adapter is set to 60 seconds.
We running into a strange problem: it may occur that the program on the SAP application server is still busy writing the dataset to the server, however the file already exists (as soon as the OPEN DATASET statement is passed the file is created on the application server but with no content yet). If just at this time the interval of 60 seconds is passed the FTP-adapter picks up the empty file, does no processing because the file is empty, but still archives the file to an archive directory.
Obviously the OPEN DATASET does not put an exclusive lock to the file? Does anyone know a solution to this?
Kind regards,
John.

Hi,
use .tmp extension to the file untill it completes writing the data, then change it to original extension.
for eg: if it is a .txt file then use .tmp untill the file is written and then change it to .txt this is not a big logic at SAP, just renaming the file after completing the write.
Regards,
Karan.

Browsing is very slow on Mac Mini connected to 100mbit LAN. FTP and file download is very fast.

Hi all
I have a strange behaviour of my Mac MIni 2012. The mini is connected to the internet by LAN. When I download a file via http or ftp everything is very fast (arround 95mbit/s). But when I try to open a website in Safari, Chrome or Firefox i have to wait some seconds until the page starts rendering.after the page is rendered completely, everything works fast and smooth.
Do you have any ideas where the problem migth be? I have reinstalled Mavericks from zero yesterday. The problem resists. With my Macbook Air (conntected to the same router) and my Windows computer I dont have this problem.
THanks in advance for your help.
Greets
Marc

By any chance, do you run sophos? I had the same issue as you and fixed it on my mac mini and macbook pro by doing the following:
Sophos --> Open Preferences --> Web Protection --> General
You will need to enter your admin password after clicking the padlock in the bottom left to allow you to change settings.
From there, deactivate "Block access to malicious websites using realtime URL reputation checks". For good measure, I also deactivated "Block malicious downloads from websites".
Hope this helps!

Pure-ftp and file ownership

Hello,
I am running a oes2 server with pure-ftp. Everything is running fine with pure-ftp when login in with FileZilla or ftp command line, but I als have a ncp volume mapped to the ftp location on the server. When people in my organization put files in the ftp directories via the ncp connections ownership of the files will be of those who put them there and not the ownership of the ftp user which owns the ftp directory on the server.
Is it possible to configure the server/pure-ftp or ncp to change the ownership of the file from edit user to ftp user.
Regards

I will answer from the pure-ftpd perspective: There is not anything pure-ftpd can do to solve this issue.
You could develop a cron script that would frequently traverse these directories and chown the file according to your desires.
I don't have much experience on the NCP side, but if your real concern about ownership is that someone else needs permissions, then maybe setting a different umask in the NCP volume settings would grant more permissions to more people, and solve your issue that way.

Sql loader scheduling and file polling on Windows

Hi there,
I am looking for tool which can poll a particular folder on Windows box for files upload events and then kick of the sqlldr process by passing the file.
I don't want to write a custom utility but looking for something out of the box.
Let me know your thoughts.
Thanks,

oops I think i did not post correctly.
I have the sqlldr part all coded and working against the csv file and able to load into tables. this is fine.
Also In windows I can schedule a batch job and get this running. What I am looking for is a scheduler which also includes file listener utility. THis will run 24X7 on the windows box and as and when a file is uploaded will kickoff the sqlldr job by passing the appropriate file name to the sqlldr task.
I believe this has to be at the OS level if I am not mistaken. If there is way to code this in Oracle I am all for it. Let me know
Thanks In Advance.
Edited by: ssp on Feb 10, 2011 2:26 PM

File Polling is not working in FTP adapter

Experts,
Configured FTP adapter should poll the file from the FTP server which is in active mode .But this is not happening as expected. This feature was not available in Oracle soa 10g
http://download.oracle.com/technology/tech/soa/soa_best_practices_1013x_drop3.pdf
they say :
Does the FTP Adapter Support the ACTIVE Mode of Connections?
No, for security concerns the FTP adapter supports PASV (passive) only.
Do they have not change this feature in Oracle soa 11g.
Your thoughts will be very much appreciated.
Thanks

I doubt if the ftp path is not given correctly. Can you try this.
1. if you gave logical_file_path try giving the actual file path or vice versa and try.
2. having the highest level of logging enabled (development). You should see some error in the logs. Is this on linux or windows, if linux search is case sensitive
hence search for 'rror' without 'e' as you donot know the case it would be in.
3. last option try retiring the process and then turn it on. This works sometimes to pick the file

Pull compressed file (.gz format) via FTP and place on Application server

Hello!
Greetings.
We have a requirement where a compressed file in the format *.gz is to be pulled via FTP and saved to the application server after extracting.
I searched the forums and found options to pull text or XL files, but nothing about pulling a compressed file. I wished to know if there is any process for the same. After pulling the file, it is to be saved to Application server after extracting. My doubts are as below:
1. How to pull a *.gz file via FTP (Need batch processing)
2. Can I extract and rename th file before saving it to applciation server? Or I need a temporary location to place the file before extraction?
Any inputs are appreciated.
Thanks,
Shishir.

Hi Sandra.
Thanks for the confirmation.
There is a change is the requirement. We need to Poll the FTP server for the file for the duration of one week every month.
When the file is found, we are to take the *.gz file, extract and put on the application server.
My question is how do we poll the application FTP server? I searched the forums and found a few threads that say that an FTP adapter is to be setup for polling the FTP server and then we can schedule it using u201CAvailability Time Planningu201D.
/people/shabarish.vijayakumar/blog/2006/11/26/adapter-scheduling--hail-sp-19-
I wished to know if that is the only way to approach this requirement.
Any help is appreciated.
Thanks and Regards,
Shishir.
Edited by: Shishir Kinkar on Apr 26, 2011 11:07 AM

Issues with File Polling and File Copy in a Cluster

Hi,
I have a 2 node cluster of SOA 11.1.1.5. Running into an issue.
I have a File Poll Process which picks up new files and then copy them over to another directory [using COPY operation of file adapter]. After that using FTP Adapter to ftp the file [using MOVE operation of FTP]. Now if I place a file Poll is creating 2 instances. I have configured HAFile Adapter and HAFTPAdapter. Now if I remove the file COPY operation I see only one instance as soon as I put back file copy it creates multiple instances again. Any idea what might be wrong.
I am following the document for file copy and ftp move http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_file.htm#BABEBJAF section
4.5 Oracle File and FTP Adapters Use Cases
But in a cluster if I use file copy I see multiple instances of Polling instances.
Thanks

"puneet" <[email protected]> wrote in message
news:3fcf4fa4$[email protected]..
>
Hi I use WLI studio to add a user in a clustered environment (admin + 2managed
wli servers). I see the following message in the logs:
####<Dec 3, 2003 1:05:02 PM MST> <Warning> <Security> <suznd265><eflow_2323_wli
managed_2> <ExecuteThread: '9' for queue: 'default'> <kernel identity><14:5a2cc
dff10338da2> <090048> <Made an in-memory only change to the FileRealm.>
As a result, the user, that I added is not actually visible to the othermanaged
servers in the cluster. I have verified this by having only one managedserver
up when adding the user. Once I have added the user, I boot the othermanaged
server and bring the first managed server down. The user I have justcreated gets
lost.
This occurs if you make the change on the managed server. Can you make the
change on
the admin server? It should get pushed to the managed servers.

Sapevt and file adapter using ftp

Hi;
Using the file adapter, I ftp a file to our SAP R/3 system called T01 (=SID).
Target directory
//T01/010/OCS/TC/IN
File name scheme:
wka.dat
I specify in "Run Operating System Command After Message Processing"
sapevt WKA -t name=T01 nr=01
WKA is the event I defined (SM62) into our T01 R/3 system.
It seems not to work. Any help? Did I miss something?
Where do I need to search for logging (ABAP or J2EE stack)?
Wilbert

Hi Wilbert,
Please check these...
http://help.sap.com/saphelp_nw04/helpdata/en/bc/bb79d6061007419a081e58cbeaaf28/content.htm
FTP ReceiverChannel and OS commands help
Hope these help!
cheers,
Prashanth
P.S Please mark helpful answers

Need help for Scheduling a Spool file and FTP the file

I have one requirement like below...
1. Start Scheduling a job
2. Generate a Spool file (.csv file)
3. If Spool file generation is successful then start FTP the file
Else End job
4. After successful FTP process end the job.
We need to create a log file also for this job.
Can any body give some idea how will i proceed?
Thanks in advance.

Billy Verreynne wrote:
BluShadow wrote:
Chris' may be wrapped, but it includes the funky ability to query remote files directly in SQL due to it's use of pipelined functions. A feature I've also got in my own FTP package and very useful for monitoring logs on remote servers through our Apex applications. ;)Ditto - also rolled my own FTP package as doing wildcard file listing requires custom support depending on the type of FTP server you're dealing with. The text part of the responses differ (these are not RFC'ed) and makes parsing more complex than what it could have been.Yeah, the differing responses can be a pain, though fortunately they are far and few between. There was a bug in Chris' package in that it wasn't handling a two code response from a windows FTP server for one of the commands, but I emailed him and he fixed that. Not his fault, he didn't have a windows FTP server to test on at the time.
FTP is a pretty straight forward protocol and easy to wrap a PL/SQL package around (using <i>UTL_TCP</i>).Absolutely. and the RFC details almost everthing you need to handle. It was quite quick to knock up a package, similar to the code hoek has linked to.
I think there is a lot of common stuff that many of us do in this regard. Always wondered how well a proper GPL'ed open source project providing a PL/SQL development framework and libraries would do...I think there'd be a lot of arguments about what is the best way of doing things. :D

FTP a file to given mailbox loacted at ip-address and port number

I need technical suggestion on the following.
Actually I got struck up with a technical issue related to FTP a file to UNIX mailbox through ABAP.
We have tried with FM: SXPG_CALL_SYSTEM and FTP_CONNECT, which all works purely based on IP, USER ID and PWD.
But we have an extra PORT NUMBER, which any of this FM doesnu2019t take up.
Can you please suggest me any other FM that I can use to FTP a file to specified IP with Port Number along with User id and PWD.
Else are we supposed to go for a separate script for this purpose?
As i have no clue on UNIX, if i go for a script than what would be the reuirement for this script and where to place them and how to call them in to my ABAP?
Timely help would be appreciable.
Thanks in advance
Edited by: yamuna pillai on Nov 11, 2008 8:37 AM

Hi Yamuna,
Can you let us know hw u did this..because we are facing the same kind of issue.
Thanks
Gayathri

FTP Adapter stops polling on updating descriptors in BPEL Console

I have a bpel process activated by a file in a FTP dropzone. An FTP Adapter listens to this dropzone. I have set up a few preferences (Deployment descriptors) for this process. Once when I update any of these user defined preferences, ftp adapter stops polling the dropzone.
Any idea why this happens?
Workarounds?

Hi Shishir,
Bouncing the server working most of the times, which might not be possible all the time in case of production environments ;/
Do you find any errors on the logs when the cube engine and activation loggers are set to DEBUG ?
Could you please mail me your project to [email protected] ?
Regards
A

Ftp and File polling

Similar Messages

Maybe you are looking for