File/Hadoop CSV input : Discovery of file fails

Hello ,
I'm trying to make the project as described in -
"Hands-on Tutorial: Building a Simple SAP Sybase ESP Project" for SAP Sybase ESP version 5.1 SP04.
I create a project where I add an input adapter of type - File/Hadoop CSV input.
Then in edit properties I enter the directory and the file name . In advanced tab I also set Has Header = TRUE.
Then I click on Schema Discovery and this raises error - "Failed to parse discovery file : parse of file failed ".
I've imported the sample "towermon "project from SCN and this shows the same error !
I've done this before on SP02 and it used to work . Of course the File/Hadoop CSV input adapter did not exist in that release.
Can't quiet figure out what I'm doing wrong . Is this a bug in the software ?Any ideas how to fix this ?
Best regards,
Saty

Hello,
Two things to check:
1) I don't know which file you are using from the "towermon" project. One of the CSV files (input-streams.csv) prepends the stream name and opcode:
isSensorStream,i,2011-06-30T17:00:00.000,32.835,-114.188,Tower,Yuma T70,1,104,0
The other CSV files do not. So you will have to toggle the "csvExpectStreamNameOpcode" parameter to true/false (the default is true).
2) There is a bug in the adapter discovery mechanism where it will return the error Failed to parse discovery file : parse of file failed ":
749503 - Schema discovery does not work for toolkit adapters when unnecessary properties are listed on the adapter.
So for example, in text authoring mode, if you have this:
ATTACH INPUT ADAPTER File_Hadoop_CSV_Input1 TYPE toolkit_file_csv_input PROPERTIES
csvExpectStreamNameOpcode = TRUE ,
dir = 'c:/temp' ,
file = 'input-streams.csv' ,
dynamicMode = '' ,
removeAfterProcess = FALSE ,
csvDelimiter = '' ,
csvDateFormat = '' ,
csvTimestampFormat = '' ,
csvHasHeader = FALSE ,
pollingPeriod = 0 ,
scanDepth = 0 ;
You have to delete the unnecessary properties:
ATTACH INPUT ADAPTER File_Hadoop_CSV_Input1 TYPE toolkit_file_csv_input PROPERTIES
csvExpectStreamNameOpcode = TRUE ,
dir = 'c:/temp' ,
file = 'input-streams.csv' ;
Then switch back to visual authoring mode and try discovery again.
Hope this helps.
Thanks,
Neal

Similar Messages

Positional flat file schemas for input and output files to be generated with the required usecases

Hello all,
I need one help regarding the positional flat file schema which contains multiple headers, body and trailers.
I have attached the sample input file below. This is a batched input and we have to generate the output which I have given below:
We are unable to flat file schema which replicates the below input file. Please suggest better approach to achieve this.
Sample Input FIle:
010320140211ABC XYZ XYZ ABCD201402110 FSPAFSPA005560080441.02000006557.FSPA.IN AB-CD ABCD/AB-CD BETALKORT
1020140210AN194107123459.FSPA.IN
[email protected]
1020140210AN196202123453.FSPA.IN
[email protected]
1020140210AN198103123435.FSPA.IN
[email protected]
1020140210AN195907123496.FSPA.IN
[email protected]
1020140210AN195903123437.FSPA.IN
[email protected]
1020140210AN193909123434.FSPA.IN
[email protected]
1020140210AN195607123413.FSPA.IN
[email protected]
1020140210AN199205123408.FSPA.IN
[email protected]
1020140210AN196206123499.FSPA.IN
[email protected]
1020140210AN196709123410.FSPA.IN
[email protected]
1020140210AN194708123455.FSPA.IN
[email protected]
1020140210AN195710123443.FSPA.IN
[email protected]
1020140210AN198311123436.FSPA.IN
[email protected]
1020140210AN196712123471.FSPA.IN
[email protected]
1020140210AV197005123403.FSPA.IN
98000000000000014000000000000001000000000000015
010320140211ABC XYZ XYZ PEDB201402111 FSPAICA 005560080441.02000006557.FSPA.IN AB-CDABCD/ABCDBETALKORT
1020140210AN195111123491.ICA.IN
[email protected]
1020140210AV195309123434.ICA.IN
98000000000000001000000000000001000000000000002
Output FIle:
1020140210AN195607123413.FSPA.IN
[email protected]
1020140210AN199205123408.FSPA.IN
[email protected]
1020140210AN196206123499.FSPA.IN
[email protected]
1020140210AN196709123410.FSPA.IN
[email protected]
1020140210AN194708123455.FSPA.IN
[email protected]
1020140210AN195710123443.FSPA.IN
[email protected]
1020140210AN198311123436.FSPA.IN
[email protected]
1020140210AN196712123471.FSPA.IN
[email protected]
1020140210AN195111123491.ICA.IN
110019EPS [email protected]
1020140210AV197005123403.FSPA.IN
1020140210AV195309123434.ICA.IN
98000000000000001000000000000001000000000000002
99000000000000001
Note: Header is a single line till BETALKORT and also there is a space before email id. That is not getting pasted properly in the files.
Thanks and Regards,
Veena Handadi

Hi all,
Header is missed from the output file for the above post:
Please find the output file:
010320140211ABC XYZ XYZ ABCD201402110 FSPAFSPA005560080441.02000006557.FSPA.IN AB-CD ABCD/AB-CD BETALKORT
1020140210AN195607123413.FSPA.IN
[email protected]
1020140210AN199205123408.FSPA.IN
[email protected]
1020140210AN196206123499.FSPA.IN
[email protected]
1020140210AN196709123410.FSPA.IN
[email protected]
1020140210AN194708123455.FSPA.IN
[email protected]
1020140210AN195710123443.FSPA.IN
[email protected]
1020140210AN198311123436.FSPA.IN
[email protected]
1020140210AN196712123471.FSPA.IN
[email protected]
1020140210AN195111123491.ICA.IN
110019EPS [email protected]
1020140210AV197005123403.FSPA.IN
1020140210AV195309123434.ICA.IN
98000000000000001000000000000001000000000000002
99000000000000001

Xml file to csv.

Dear Friends,
Can anyone share blog to convert file from XML file to CSV.. which can handle multiple records.
my source structure is :
<ROOT>
     <HEADER>
           A1
           B1
      </HEADER>
      <ITEM>
           C1
           D1
       </ITEM>
</ROOT>
any inputs please ..
Thanks & Regards,
Suman

Hi,
What is the occurance of Header and Item based on that ur FCC will be decided.
refer the below link
Receiver FCC no need of Endseparator
File Adapter (Receiver) - Are we "really" sure about the concepts?
Receiver-
File Receiver with Content Conversion
chirag

When to export a numbers file to csv the minus signs don't work with R

Friends,
I have trouble getting the statistic program R to read my exported CSV-files from Numbers.
The minus sign doesn't work, R reads it as a factoric input rather than numeric.
Does anyone have a tip how to reformat this?
Jocke

Wayne, actually, I'm not sure but I think that factoric - or factorial is the denomination that R uses to say that somthing is rather a text than a numeric number - I'm pretty new to R.
But, anyway, we solved the problem but in a lillte bit of an awkward way.
The problem is that the minus sugns in numbers is percieved as dashes in R
(http://www.R-project.org, a free-ware statistical analysis program)
So where there is a numeric input with a minus sign R thinks it is a letter rather than a number.
We solved it by exporting the numbers file to CSV, opened it in text editor, copied the suspisios minus sign and pasted it into the search field to find all dashes/minus sign that didn't work and replaced them with a minus sign. After this operation, nothing different could be seen in the text but R saw the difference.
So this makes it difficult to explain by posting a screenshot - it looks the same but it isn't.
The funny thing is that the dashes that numbers use when to express a date are percieved as minus signs by R.
Can anyone understand what I tried to explain? And if so, is there a simpler way to do this? We have a lot of minus signs to export to R.....
best regard
Joachim

Need to create multiple files in CSV through UTL_File package

I have to extract data from table from csv format and maximum number should be 10 each file.how to do this through UTL_File utilities.
i am putting the code...any input would be apprecaited...
CREATE OR REPLACE PACKAGE BODY TEST
AS
PROCEDURE main_test(
errbuff OUT VARCHAR2,
retcode OUT VARCHAR2
IS
-- Variables
lv_file_handle UTL_FILE.FILE_TYPE;
lv_file_name_txt VARCHAR2(25);
lv_file_location_txt VARCHAR2(100) :='/dv1/gfp/ora01/dv1gfpcomn/temp';
v_error_code NUMBER;
v_error_text VARCHAR2(200);
lv_cursor_rowcnt NUMBER := 0;
lv_row_data VARCHAR2(32000);
lv_row_tran_data VARCHAR2(32000);
lv_transaction_number VARCHAR2(1000);
lv_row_head_data1 VARCHAR2(32000);
lv_row_trn_data1 VARCHAR2(32000);
lv_row_data1 VARCHAR2(32000);
lv_period VARCHAR2(100);
row_cnt number := 0;
c integer;
max_rows number;
-- Cursors
CURSOR cur_period IS
select
user_name name,
description description ,
email_address email,
start_date st_date,
end_date end_date
from fnd_user
WHERE ROWNUM <51
ORDER BY 1,4 desc;
BEGIN
lv_file_name_txt := 'test'||TO_CHAR (SYSDATE, 'DDMMYYYY')||'.txn';
lv_file_handle := UTL_FILE.FOPEN(lv_file_location_txt, lv_file_name_txt, 'w');
--code modification here
FOR cur_rec IN cur_period
LOOP
lv_row_data := NULL;
lv_cursor_rowcnt := lv_cursor_rowcnt + 1;
lv_row_data := rpad(nvl(substr(cur_rec.name,1,10),'') ,10,' ') || '+'
|| rpad(nvl(substr(cur_rec.description,1,10),'') ,10,' ') || '+'
|| rpad(nvl(substr(cur_rec.email,1,15),'') ,15,' ') || '+'
|| rpad(nvl(substr( cur_rec.st_date,1,10),'') ,10,' ') || '+'
|| rpad(nvl(substr( cur_rec.end_date ,1,10),'') ,10,' ') ;
END LOOP;
IF lv_cursor_rowcnt = 0 THEN
fnd_log.put_line ('No data found');
END IF;
UTL_FILE.FCLOSE(lv_file_handle);
EXCEPTION
WHEN OTHERS
THEN
v_error_code := SQLCODE;
v_error_text := SQLERRM;
retcode := 2;
fnd_log.put_line ('s-Error :' || v_error_code || '- ' || v_error_text );
fnd_log.put_line (SQLERRM);
UTL_FILE.FCLOSE_ALL();
END;
END TEST;
show err
---

Hi,
and maximum number should be 10 each file.utilities. I could not understand the above line. If you want to put 10 rows in one file , next 10 rows in another file and so on ......
There are two approaches.
1. Simple one is store all records in one file using UTL_FILE and then use unix
split utility to break 10 lines for each file.
2. Store all files name is an array ( VARRAY or Associative Arrays) .
(i) Fetch file name form Array
(ii) Open a File
(iii) Store 10 rows in File
(iv) Close file. Go to step 1.
Regards

2008R2 FOREACH csv file - move data to db - move file to folder weirdness

Greetings everyone,
I'm facing a weird problem. So I'm retrieving files from an ftp server and saving them in a local 'to process' folder. Then I use a foreach file iterator to process those files and save the data in the database through a data flow task. When this operation
finishes, I move the file in 2 possible folders: Processed or Error. This depend on the result of the dataflow task:
To ensure my loop continues if an error should occur i've placed and event handler on the OnError action of the DFT. There in the system variables I set the value for PROPAGATE to false. This way there isn't and error escalation that would fire the package
onerror event handler. Now all works well when all my files are correct regarding the content.
For testing purposes I''ve created a dummy data file that would fire a truncation error. When I run the package and the irerator arrives at the BAD dummy data file, the file gets placed in the error folder as expected and I get the correct error message
(truncation error). However when the next file, which is correct, processes in the DFT, it also produces an error:
Error: SSIS Error Code
DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "DS_STG" failed with error code 0xC0202009. There may be error messages posted before this with more information on why
the AcquireConnection method call failed.
I don't understand this. It seems like the previous error has a corrupting impact on the DFT. Is there a property that I have to set here? I've tried playing with the MaximumErrorCount and tried to run it with a configuration file to persist the connection
manager information, but the behaviour still persists.
So in summary: the iteration continues until the end, but after one bad file is being processed in the DFT, all the next good files get the connection manager error...
I can think of ways to workaround this issue, but I would think that this should work as it is, no? :)
Thanks for the answers in advance.

Hi Visakh,
I specify the folder which holds the csv files. Then I assign each file path to a variable
Then I use this variable (path) as connection string in the DFT flat file source:
I created a workaround which I don't like but it seems to do the job. I kept the original foreach file to determine which file is valid BUT I don't do the data insert in this DFT no more. I deleted the ole db destination so it generates an error only if
the data doesn't come through at the DFT.
Then I remove the bad files from the to process folder. I copy the foreach iterator with all the components, but now I add the ole db destination in the data flow task for data insert. But at this time there are only correct files in the 'to process' folder.
This works, but it isn't 'pretty' :)
Do you have an idea what could be wrong? It seems one bad file corrupts the destination connection.
When you say bad file is it metadata which is corrupted?
Please Mark This As Answer if it solved your issue
Please Mark This As Helpful if it helps to solve your issue
Visakh
My MSDN Page
My Personal Blog
My Facebook Page

Need help in writing data to a flat file in .csv format

Hi All,
could you please show with a sample example to write data in .csv format using UTL_file.
The result of the refcursor i want to write to a file in .csv format.
How can i achieve this.
Thx

Hi,
There are two ways to acheive this.
First and which I prefer is creating a SQL script which will generate csv file at the specified location in CSV format. And then this SQL script can be called from your cron job similar to how you call stored procedure. Following is pseudo code that can be used -
set linesize 500
set echo off
set feedback off
set prompt off
set pagesize 0
spool <name of the csv file>.csv
<Your select statement>;
spool offPut select statement delimiting columns you want to use with ','. E.g. following query should create a CSV file named as abc.csv with one record and 2 columns delimited by comma. First column is ENO and second column is ENAME.
set linesize 500
set echo off
set feedback off
set prompt off
set pagesize 0
spool abc.csv
select ENO || ',' || ENAME from employee;
spool offSecond, you can write a PLSQL procedure and then call this procedure from cron job. There is a generic procedure shared by BluShadow some time back which is a very good example for this. Please refer to following link for this solution -
REF Cursor creating CSV files
I will suggest to use first method above if possible since it will be faster and less complicated in my opinion. Second method is recommended for scenarios where select statement is created dynamically and cannot be written during development.
Hope this solves your purpose.
Cheers,
Anirudha

Issue while downloading file in .CSV format

Hi,
I need to download the file in .CSV format.
I hade used FM SAP_CONVERT_TO_CSV_FORMAT and then used GUI_Download.
Now when I am opening file which is downloaded, it gives all the data in a single column. If there are 5 fields in my table, the generated file gives the data of all the 5 fields in a single column.
Could you please help?

Hi try wi th the following code.
TYPE-POOLS : truxs.
DATA: t_file TYPE STANDARD TABLE OF type_file.
data:t_conv_data TYPE truxs_t_text_data.
CALL FUNCTION 'SAP_CONVERT_TO_TEX_FORMAT'
EXPORTING
    i_field_seperator          = ', '
TABLES
    i_tab_sap_data             = t_file
CHANGING
   i_tab_converted_data       = t_conv_data
EXCEPTIONS
   conversion_failed          = 1
   OTHERS                     = 2.
CALL FUNCTION 'GUI_DOWNLOAD'
EXPORTING
    filename                = 'C:\TESTCSV.CSV'
    filetype                = 'ASC'
    write_field_separator   = ' '
TABLES
    data_tab                = t_file
   fieldnames              = names
EXCEPTIONS
    file_write_error        = 1
    no_batch                = 2
    gui_refuse_filetransfer = 3
    invalid_type            = 4
    no_authority            = 5
    unknown_error           = 6
    header_not_allowed      = 7
    separator_not_allowed   = 8
    filesize_not_allowed    = 9
    header_too_long         = 10
    dp_error_create         = 11
    dp_error_send           = 12
    dp_error_write          = 13
    unknown_dp_error        = 14
    access_denied           = 15
    dp_out_of_memory        = 16
    disk_full               = 17
    dp_timeout              = 18
    file_not_found          = 19
    dataprovider_exception = 20
    control_flush_error     = 21
    OTHERS                  = 22.

An Error has occurred: File Repository server input is down

Hello Team,
I am getting the following error message "An Error has occurred: File Repository server input is down" when try to upload exel or any other document in repository using . NET Infoview. Even with the Admin account.
and
If i try the same steps from Java Infoview i am able to upload the excel.
I guess some issue with IIS .
Env Dtls:
BO XI 3.1 SP3
IIS 7
Windows server 2008.
Waiting for your valuable suggestions.
Regards,
Sonu Pandita

Hi Team,
I resolved this issue by giving one user a permission on temp folder.
Regards,
Sonu Pandita

Inputting an excel file to the java servlet program

Hi,
How can I input an excel file into the java servlet program? I am using POI HSSF programming to read the content of that excel file.I have created a form that will accept the path from the user and in the servlet program, I have called a post method to retrieve the path of the file. I am getting the path of the file as a string.Then I have to access the file to read the data. But it is not working. What should I do?.Please help me.
I have created an html file to create a form that will ask the path of the file to be loaded into the servlet program.Please give me a solution,thanks in advance. The servlet program that receives the file is shown below. It just only take the name of the file and prints if that is ok.
<html>
<body bgcolor="green">
<center>
<form method="post" action="http://localhost:8080/examples/servlet/Ron1">
<table>
<tr>
<td><B>ENTER THE PATH OF THE FILE</td>
<td><input type=file name="path"></td>
</tr>
</table>
<input type=submit value= "submit" >
</form>
</body>
</html>
import java.io.*;
import javax.servlet.*;
import java.lang.*;
import javax.servlet.http.*;
import java.sql.*;
import java.text.*;
import java.util.*;
import com.oreilly.servlet.MultipartRequest;
import org.apache.poi.hssf.usermodel.*;
public class Ron1 extends HttpServlet
   public void doPost(HttpServletRequest request, HttpServletResponse response)   throws IOException, ServletException
        response.setContentType("text/html");
        PrintWriter out = response.getWriter();
        String u = request.getParameter("FileToUpload");
        out.println("The Path is "+u);
        System.out.println("The path is "+u);
        MultipartRequest multi = new MultipartRequest(request,u,10 * 1024 * 1024);
        System.out.println("path is "+u);
        Enumeration files = multi.getFileNames();
        File fUploadedFile = null;
        String sFileName = "";
        out.println("path is "+u);
        while(files.hasMoreElements())
                        String name = (String)files.nextElement();
                        out.println("file is "+name);
                        sFileName = multi.getFilesystemName(name);
                        String type = multi.getContentType(name);
                        fUploadedFile = multi.getFile(name);
                FileInputStream fis= new FileInputStream(fUploadedFile);
               HSSFWorkbook wb=new HSSFWorkbook(fis);
                System.out.println("The path is "+u);
                out.println("File has identified successfully\n");
                out.close();
   }

Hi,
Can you try to use classes from java.net package (URL, URLConnection) to pass the contents of excel file to servlet?
HTH
Vishal

How to write to both .csv and binary format files

Hi, I am very new to labview. I have been given the programme to process measurement data and save the data into .csv format. But the file in .csv format is very large. So on top of the .csv, I would like to save the processed data into another binary file at the same time. I have attached the .vi to this post, I tried to add an write to binary function to the programme, but there are quite a few errors. Could someone give me a hand please? Thanks a lot.
Attachments:
RF Analyze_test.vi ‏40 KB

I agree that TDMS is worthwhile looking into.
As far as your issue: which errors are you referring to? Are you referring to the missing assignments to the tunnels? That's due to the fact that you did not wire a value for the other case. An output tunnel must generate a value for all cases. You can right-click on the output tunnel and select "Use default if unwired". In this case the value will be the default value for whatever datatype is being generated. If you are referring to other errors, please be specific.

Save as file format CSV in Excel

Does anyone know the correct syntax on this one?
I have an excel file which should be saved as CSV.
This is what I came up with, but it doesn't run.
tell application "Microsoft Excel"
activate worksheet "worksheet1"
set fileSaveName to get save as file format CSV file format filename
save as filename fileSaveName
if fileSaveName is not "" then
display dialog "Save as " & fileSaveName
end if
end tell

It's not surprising it doesn't run but that's not entirely your fault. Excel has pone of the funkiest dictionaries around - I think they forgot to read the AppleScript Implementor's guide and instead tried to bolt VB on instead.
Anyway, I digress...
I'm not sure what you expect this line to do:
set fileSaveName to get save as file format CSV file format filename
but it's completely invalid. For one, save as is a command verb, not a function, so you cannot get save as.
If you read the dictionary for Excel's save as command you'll find:
save as sheet ¬
filename unicode text ¬
file format as ¬
password unicode text ¬
write reservation password unicode text ¬
read only recommended boolean ¬
create backup boolean ¬
add to most recently used list boolean ¬
overwrite boolean ¬
save as local language boolean
so the first, required, parameter is the sheet to save followed by any number of the optional parameters. Given the above, I'd expect your script to look more like:
tell application "Microsoft Excel"
save as worksheet "worksheet1" file format CSV file format filename "whatever.csv"
end tell
which is completely bass-ackwards, if you ask me, but that's how it goes.

Set Adobe Acrobat XI Pro. When I want to save the file in WORD, EXCEL or esporta file into ... immediately throws an error "save as failed to process this document no file was created". What's the problem?

Set Adobe Acrobat XI Pro. When I want to save the file in WORD, EXCEL or esporta file into ... immediately throws an error "save as failed to process this document no file was created".
What's the problem?
Any help in finding a solution is greatly appreciated.
Thank you,

Installed AcrobatXI PRO 11.0.09 on seven computers and laptops. Two of them gives an error when you try to save a document in WORD, EXCEL, Power Point, or when exporting to... error: Save failed to process this document. No File was created.
But all good saves in the format of TXT and jpg.
I have uninstalled and restored and re-installed and updated and registry cleaned and removed using the special utility Cleaner Tool for Acrobat, but nothing helps.
On one notebook with Windows 8.1 and Microsoft office 2013, on another laptop with Windows 7 and Microsoft office 2010, the same problem, although there are computers with Windows 7 and Microsoft office 2010 and everything works.
Tell me where to find the problem and how to solve it.
Thank you.

UTL_FILE & User Input: Case Sensitive File Names

Hi.
I'm trying to reduce problems due to the whether the user uses upper or lower case or combo of them in entering file names.
Scenario is:
Existing file to be read: Orig_File.Txt (file name changes daily)
Output to be saved into file: RESULTS_FILE.TXT
In response to PL/SQL commands (please see below), the user enters:
ORIG_FILE.TXT and Results_FILE.txt
as the Input & Output file names. The problem is that the input file has been saved using Upper and Lower case and I get back a file not found error.
Is there a way to call a file with is a different case than the input?
Thanks.
Current commands are:
In_File          UTL__File.file_type;
Out_File          UTL__File.file_type;
Curr_Line     VarChar2(2000);
BEGIN
In_File := UTL__File.fopen('UTIL_FILE_DIR', '&input_file', 'R', 2048);
Out_File := UTL__File.fopen('UTIL_FILE_DIR', '&output_file', 'W', 2048);
LOOP
BEGIN
UTL__File.get_line(In_File, Curr_Line, 2048);
.......

A shell script would typically issue something along the lines of
newfile=$(ls -C1 | grep -i "\<${filename}\>")although of course it is possible for this to match multiple filenames, so it would have to check for that scenario and report it as an error. possibly you could call a shell script like this from PL/SQL via Java or DBMS_SCHEDULER. Good luck with that though.

I exported an archive of iCal from a computer running the last OS X. On a new computer running OS X (10.7.4), I tried to import the .icbu file and got a message "The calendar failed to restore". How do I fix this/get my calendar?

I exported an archive of iCal from a computer running the last OS X. On a new computer running OS X (10.7.4), I tried to import the .icbu file and got a message "The calendar failed to restore". How do I fix this/get my calendar?

I exported an archive of iCal from a computer running the last OS X. On a new computer running OS X (10.7.4), I tried to import the .icbu file and got a message "The calendar failed to restore". How do I fix this/get my calendar?

File/Hadoop CSV input : Discovery of file fails

Similar Messages

Maybe you are looking for