Problem in handling complex CSV file

Hi,
I am facing a strange problem in handling a complex CSV file.
The content of the file is as follows. It is getting executed through an XSD and the target is a table.
First time when the interface is executed ( for e.g session id - 19001) it is getting completed successfully.
Now when making changes in the CSV file suppose adding ( for e.g.-
DEPT,D05,Books,ABC Store
CUST,C01,Sayantan,[email protected]
CUST,C02,Shubham,[email protected])
or deleting ( for e.g.-
CUST,C02,Sarbajit,[email protected])
and running the interface I am not getting the added D05 in the target table nor my C02 data is getting removed i.e.- the updated data in CSV file is not getting fetched and I am getting the same records as i got when i ran interface of session id 19001.
I am not getting why it is happening??
The CSV file used in session id -19001 is:
DEPT,D01,Retail,World Mart
CUST,C01,Anindya,[email protected]
CUST,C02,Rashni,[email protected]
DEPT,D02,Food,Food Bazar
CUST,C01,Abhijit,[email protected]
CUST,C02,Anirban,[email protected]
CUST,C03,Sharmistha,[email protected]
DEPT,D03,Water,SW
CUST,C01,Nirmalya,[email protected]
DEPT,D04,Clothes,City style
CUST,C02,Sarbajit,[email protected]
CUST,C03,Abhishek,[email protected]

Here's what you can do to handle CSV files using HSQL.
Say the CSV file contains order data. Each order record contains data about a particular order like order number, product code, quantity, ordered by, date etc. Now lets take a hypothetical requirement which says, the user needs to know how many orders were ordered per product.
Option 1] At the basic level, read the order file, parse each line and write the logic to get count of orders per product.
Option 2] Load this CSV file into a database like MySQL and write database queries to get the orders per product.
So where does this leave us? Have we run out of options? I am sure we would have tried & used the above two options . But I wanted a different approach. Following questions were lingering in my mind:
1] Why can’t I write SQL queries against the CSV file itself. After all it’s like any RDBMS table.
2] Why should I load the CSV file into some database before I query it?
3] Why can’t I create a database table & attach the CSV file to it?
Continue Reading Here: Handling CSV Files

Similar Messages

Problem about Handling of Empty Files in File Adapter

Hello everyone,
NetWeaver 2004s --- XI
In Sender i have a File Adapter.
Now i meet a problem about Handling of Empty Files. When i send empty file, but don't cerate a leer message.
I have seen following text in help document. But in adapter configuration i can not find the correspond parameter.
can you give me some tips?
Thx in advance
best regards
Yaning
SAP Help Document über File Adapter
+Handling of Empty Files
Specify how empty files (length 0 bytes) are to be handled.
○       Do Not Create Message
No XI messages are created from empty files.
The files are processed according to the selected Processing Mode.
For example, if the processing mode is Delete, empty files are deleted in the source directory.
○       Process Empty Files
XI messages are created with an empty main payload.
The files are processed according to the selected Processing Mode.
○       Skip Empty Files
No XI messages are created from empty files.
Empty files are skipped and remain in the source directory.+
Help Docu

hi,
it's available since Sp19 for XI 3.0
and the corresponding SPS fpr XI 7.0
http://help.sap.com/saphelp_nw04/helpdata/en/44/f565854b7341e6e10000000a1553f6/frameset.htm
so probably you need to install the new SP
Regards,
michal
<a href="/people/michal.krawczyk2/blog/2005/06/28/xipi-faq-frequently-asked-questions"><b>XI / PI FAQ - Frequently Asked Questions</b></a>

Problem in handling multiple source of Complex CSV files in ODI

Hi,
I have a requirement where I have multiple CSV files.
In Details:
when I have one CSV file I use the following syntax in JDBC URL in creation of Data Server under Complex file of Topology navigator.
jdbc:snps:complexfile?f=C:\pqr\pqr.csv&d=C:\pqr\pqr.xsd&re=Root
When there are multiple CSV files what would be the syntax as? will it be like
jdbc:snps:complexfile?f=C:\pqr\*.csv&d=C:\pqr\pqr.xsd&re=Root (I am getting error as unexpected token * when I am using * and clicking on Test Connection Button).
Please help..
Thanks.

Hi,
first of all according to documentation use backslash
http://docs.oracle.com/cd/E21764_01/integrate.1111/e12644/appendix_complex_driver.htm#CCHDJGHF
in second place under my opinion you are not able to use mask to identify more file. A workaround is to use a variable to store the file name. BUT i've never used Complex Files, so i'm speculating about it.

Memory problem with loading a csv file and displaying 2 xy graphs

Hi there, i'm having some memory issues with this little program.
What i'm trying to do is reading a .csv file of 215 mb (6 million lines more or less), extracting the x-y values as 1d array and displaying them in 2 xy graphs (vi attacked).
I've noticed that this process eats from 1.6 to 2 gb of ram and the 2 x-y graphs, as soon as they are loaded (2 minutes more or less) are really realy slow to move with the scrollbar.
My question is: Is there a way for use less memory resources and make the graphs move smoother ?
Thanks in advance,
Ierman Gert
Attachments:
read from file test.vi ‏106 KB

Hi lerman,
how many datapoints do you need to handle? How many do you display on the graphs?
Some notes:
- Each graph has its own data buffer. So all data wired to the graph will be buffered again in memory. When wiring a (big) 1d array to the graph a copy will be made in memory. And you mentioned 2 graphs...
- load the array in parts: read a number of lines, parse them to arrays as before (maybe using "spreadsheet string to array"?), finally append the parts to build the big array (may lead to memory problems too).
- avoid datacopies when handling big arrays. You can show buffer creation using menu->tools->advanced->show buffer allocation
- use SGL instead of DBL when possible...
Message Edited by GerdW on 05-12-2009 10:02 PM
Best regards,
GerdW
CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
Kudos are welcome

Problem in writing to CSV file

Hi all,
i am retreiving a long String which is not in a proper format (ie) Lot of Spaces in between and my String contains comma(,) seperated and I am trying to put that string in to a CSV file. I am able to write the data in the CSV file but not in a proper format.i need to write the data in the below format.
For Ex:
Emp-No,Address,Name,Manager -------> this is Heading
1000,abcd,name,Man-name -----> After i have to put the data in next line
My String will be like this,
abcd , , , jsjsj, , ,saalkj,jfjfjfjf,mmm, nnn, , ,
I am able to get the data properly in the CSV format.
Please help me to solve this. Thanks in Advance!!!!

Are you using any reporting frameworks like Jasper
Reports or you using simple select queries in MySQL
to export the data to CSV format.Detail more clearly
to get assist from others.
Are you using Session beans or Entity beans on Weblogic?no no no, he can create the csv file, whether its in the correct format is the problem of the consumer of this csv file, isnt'it?
are you using a microsoft product?

Problem in uploading xls/csv file with Email address through oracle forms

Dear all,
I've created an interface to upload data from xls
and csv files to database tables.
I'm sucessed in this. Now i'm facing an issue.
If the xls/csv file having Email address,the
upload activity is not working..junk characters is
getting stored.
I came to know that, xls .csv file having email
address with hiperlink. so this may cause the
issue (just guessing)..
we can't restrict users to upload email without
hiperlink.. so what is the alternative to do
this..
Email address
[email protected]
[email protected]
[email protected]
etc...

Can you give some more information:
What versions of Forms, database, Java, browser are you using?
How are you uploading data? Please show us your code.
What do you mean by "email with a hyperlink"?

How To handle complex Flat File in BDC( V.V. Urgent)

Hi All,
Anybody help in these case.
I have to write the BDC for a complex File.
Here is the Scenario
I An recording for Tcode CN21.
1network -> many activities
1 Activity ->many components
Any body Help ifor this case,
Points will be awarded.
Regards,
Jayasimha

Hi Jayasimha
Can u explain what is the problem that u are facing? we will be able to help u if u let u know what is the problem.
as far as what u have mentioned, first do a recording in SHDB for the transaction CN21 and then in the program get all the data from the flat file into an internal table. then build a BDC.
Cheers

File Adapter Fault Handling for CSV files

I have been working on a requirement where the file adapter picks up the file and gives to the BPEL for processing the data.
In my composite, i have configured the file adapter as a service for reading files, and i defined the xsd using the native format builder....
<?xml version="1.0" encoding="UTF-8" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd"
xmlns:tns="http://TargetNamespace.com/ReadFile"
targetNamespace="http://TargetNamespace.com/ReadFile"
elementFormDefault="qualified"
attributeFormDefault="unqualified"
nxsd:version="NXSD"
nxsd:stream="chars"
nxsd:encoding="US-ASCII"
nxsd:hasHeader="true"
nxsd:headerLines="1"
nxsd:headerLinesTerminatedBy="${eol}"
>
<xsd:element name="Root-Element">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Student" minOccurs="1" maxOccurs="unbounded">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Name" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="Street1" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="Street2" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="City" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="State" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="Country" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="${eol}" nxsd:quotedBy=""" />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
If i put some test data like below , the .txt file is being picked up and the valid being passed to the BPEL and its not returning any error for the invalid records. In this case, just want to throw an error and exit the bpel.
Krish,abcd,abcd,chennai,tnin - comma is missing here
NP,abcd,abcd,chennai,tnin - - comma is missing here
Nutan,abcd,abcd,chennai,tn,in - this record being processed successfully.
Pari,abcd,abcd,chennai,tn,in - - this record being processed successfully.
Experts, any help is really appreciated..

Hi,
its better to opt for seeburger for EDI data processings..
http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/20ed5d5b-b188-2c10-76a2-997a616e48a9
http://www.cbs-consulting.com/EN/Services/ProcessesApplications/SAP__TechnologyConsulting/OpenPI/OpenPI__n,naviExpand=.html__nnn=true
Check the above links can be helpful..
HTH
Rajesh

Output CSV file problem

Hi Guru,
I encounted 1 problem where my output csv file content incorrect. Source is from SAP Proxy.
Example my csv file should have Material code, material description and date. However when i open the csv file, it is only have material code in one line and the rest of fields are not display in the csv file.
<?xml version="1.0" encoding="UTF-8"?>
<ns0:MT_PRD_MASTER_SND xmlns:ns0="http://com.starhub/sapprdcat">
   <RECORDSET>    1:1
      <RECORD>       1:unbounded
         <MATNR/>
         <MAKTX/>
         <DATE_ADD/>
      </RECORD>
   </RECORDSET>
</ns0:MT_PRD_MASTER_SND>
My source and target are same.
CC Receiver setting:
Transport Protocal : File System NFS
Message Protocal : File Content Conversion
Adapter Engine : Integration Server
Recordset Structure : RECORD
RECORD.fiedlNames : MATNR,MAKTX,DATE_ADD
RECORD.endSeparator : 'nl'
RECORD.fieldSeparator : ,
RECORD.processFieldNames: fromConfiguration
RECORD.addHeaderLine: 0
The INCORRECT result only display MATNR in one line only.
HNOK0000102     HNOK0000108     C032S31384A     C032S3XDATA     CFGSC300300     CRAWC300001
Expected Result should be :
HNOK0000102,Material test 1,01/01/2007
HNOK0000108,Material test 2,01/02/2007
C032S31384A,Material test 3,01/01/2007
C032S3XDATA,Material test 4,01/01/2007
CFGSC300300,Material test 5,01/01/2007
CRAWC300001,Material test 6,01/01/2007
Thank you very much for help.
Regards,

hi,
check this parameter:
Recordset Structure : RECORD [wrong]
Recordset Structure : RECORDSET,RECORD [right]
and the other parameters will be like this:
RECORD.fiedlNames : MATNR,MAKTX,DATE_ADD
RECORD.endSeparator : 'nl'
RECORD.fieldSeparator : ,
RECORD.processFieldNames: fromConfiguration
RECORD.addHeaderLine: 0
<b>RECORDSET.fieldSeparator: 'nl'</b>
Sachin
Message was edited by:
        Sachin Dhingra

Read a csv file in a PL/SQL routine with the path as a parameter

hello oracle community,
System Windows 2008, Oracle 11.1 Database
I have created a directory in the database and there is no problem to read a csv file from there. Since the files can be at different places, my java developer is asking me, if he can give me the path of the file as a parameter. So far I know I will need a directory to open the file with the UTL_FILE package, but I cannot create diretories for folders, that will be created in the future. Is there any way to solve that problem, so I can use the path as a parameter in my routine ?
Ikrischer

BluShadow wrote:
Ikrischer wrote:
as I said before, they are dynamic. As usal to most other companies, you get new partners and you will lose partners, so new folders will be added, old ones will be deleted. if you use different directories for different folders, you have more work to handle them.I wouldn't call that dynamic, I would call that business maintenance. Dynamic would mean that the same company is putting their files into different folders each time or on a regular basis. If it's one folder per company then that's not dynamic.the folder structure is changing, I call that dynamic, does not look static to me. but thats just a point of view, it is not important how we call it. lets just agree, the folder structure will change in the future, no matter how many folders one partner will have.
BluShadow wrote:
So you're receiving files and you don't even know the format of them? Some of them are not "correct"? Specify that the company must provide the files in the correct format or they will be rejected.
Computers work on known logic in most cases, not some random guessing game.we know the correct format, but we also know, that the partner wont always send the correct format, period. Cause of that we want to deliver an error log, record 11,15 and 20 is wrong cause of....that is a very usefull service.
BluShadow wrote:
The folders should be known.impossible, we do not know the future.
BluShadow wrote:
The file format should be known.see above, it has a fixed format, but thats not a gurantee you get the correct format. for me thats a normal situation, it happens every day, you have fixed rules and they will be broken.
BluShadow wrote:
The moment you allow external forces to start specifying things how they want it then you're shooting yourself in the foot. That's not good design and not good business practice.maybe we are talking about different things, we do not allow to load data into our system with a wrong format. but we do allow to to send us a wrong format, and even more, we do allow them to send wrong content, but still we dont load it into our system. format and content must be correct to be loaded in the real system, but both we are checking. and with external tables you cannot check a wrong format we a detailed error log.
Ikrischer

Tokenizing from .csv file

Hi everyone,
Here's the problem:
I have two .csv files. I need to replace the first, second and fifth field in the one file from the other. I would do it manually, but there's 2200 lines! Any help is greatly appreciated.
Thank you!

sabre150 wrote:
Which is why I suggested this. It takes care of all that for you.Yes, definitely. If it's anything more complex than what can be split using String.split, make use of that library.
@OP, you can choose to take this code and adjust it to your needs:
   public static void main(String[] args) throws Exception{
      BufferedReader f1 = null, f2 = null;
      PrintWriter f3 = null;
      String line1, line2;
      while ( (line1 = f1.readLine()) != null && (line2 = f2.readLine()) != null) {
         String[] fields1 = line1.split(",");
         String[] fields2 = line2.split(",");
         fields1[0] = fields2[0]; //etc
         StringBuilder buff = new StringBuilder();
         for ( int i = 0; i < fields1.length; i++ ) {
            buff.append(fields1);
if ( i != fields1.length-1) {
buff.append(",");

How to write the char value as is in the CSV file

Hi Everyone,
I am creating csv files which contains inventory details for all the products. I am able to create the csv file with utl file concepts. My problem starts after the csv file is being created.
some of the product numbers(Though I am saying Product number, it is varchar2 data type in the table) is like this 3E-12, 3E-54 and so on. I have totally 23 product numbers like this.
When the user opens the csv file it is changing to numbers like this 3.00E-12, 3.00E-54. I want to keep the product number as it is like char value. I tried many quoting and concat methods.
But none of them works for me.
I am using oracle 9i.
Oracle9i Enterprise Edition Release 9.2.0.7.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
Please help me to solve this problem.
Thanks in advance,
Vimal...

Hi Friends,
Thanks for the immediate response. Here is my code.
CREATE OR REPLACE PROCEDURE DATA_TO_CSV_FILE
IS
l_file_ptr UTL_FILE.file_type;
l_hdr_text VARCHAR2 (4000);
l_dynamic_sql_str VARCHAR2 (4000);
TYPE l_dynamic_sql IS REF CURSOR;
l_dynamic_cursor l_dynamic_sql;
l_org_id_select VARCHAR2 (200);
l_master_org_id NUMBER;
l_category VARCHAR2 (40);
l_itemno VARCHAR2 (40);
l_description VARCHAR2 (240);
l_brand VARCHAR2 (240);
l_organization_id NUMBER;
l_oh NUMBER;
l_status apps.mtl_system_items.inventory_item_status_code%TYPE;
BEGIN
l_file_ptr := UTL_FILE.fopen ('CSV_DIR', 'inventory.csv', 'w');
l_dynamic_sql_str :=
'SELECT mc.segment2 CATEGORY, itm.segment1 itemno '
|| 'itm.description description, LOG.brand brand,'
|| 'NVL (m.oh, 0) oh'
|| '(select msi.inventory_item_status_code from apps.mtl_system_items msi'
|| ' where msi.inventory_item_id = itm.inventory_item_id'
|| ' and msi.organization_id = '
|| l_master_org_id
|| ' ) status '
|| ' FROM xxx_custom_table m,'
|| ' apps.mtl_item_categories ic,'
|| ' apps.mtl_categories_b mc,'
|| ' apps.mtl_system_items itm,'
|| ' xxx_custom_table1 LOG,'
|| ' xxx_custom_table2 cap'
|| ' WHERE m.item_id(+) = itm.inventory_item_id'
|| ' AND m.org_id(+) = itm.organization_id'
|| ' AND itm.segment1 = cap.itemno(+)'
|| ' AND itm.inventory_item_id = ic.inventory_item_id'
|| ' AND itm.organization_id = ic.organization_id'
|| ' AND ic.category_id = mc.category_id'
|| ' AND mc.segment2 IS NOT NULL'
|| ' AND itm.inventory_item_id = LOG.inventory_item_id'
|| l_org_id_select
|| ' GROUP BY mc.segment2,'
|| ' itm.segment1,itm.inventory_item_id,itm.description,'
|| ' LOG.brand,LOG';
OPEN l_dynamic_cursor FOR l_dynamic_sql_str;
LOOP
FETCH l_dynamic_cursor
INTO
l_category, l_itemno, l_description, l_brand, l_status,l_oh;
EXIT WHEN l_dynamic_cursor%NOTFOUND;
UTL_FILE.put_line (
l_file_ptr,
l_CATEGORY
|| ','
|| l_itemno
|| ','
|| l_description
|| ','
|| l_status
|| ','
|| l_brand
|| ','
|| l_oh
END LOOP;
UTL_FILE.fclose (l_file_ptr);
END;
Hey damorgan,
Can you please give me little more detail about your workaround method. I think I did the ODBC Connection Once for MS ACCESS database. I
hope you are taking about the same method.
Thanks
Vimal....

Transfring from CSV file to LabVIEW 2011 and view it in XY Graph

Hi, I would like to seek for your assistance in my project.
I have an excel file which is in CSV format and intend to transfer it into LabVIEW 2011.
I want to create a storage to store all the excel file for viewing purpose and in the mean while I want to view it in graph form.
Hope to that you will be able to help me with this. Thank you.
Attachments:
History_power2013-10-27 12-53-32 ch1 tv ch 2 fan.csv ‏209 KB

Thank you for replying.
But I have problem displaying the whole CSV file.
However, I have a batch of CSV file to monitor, so is there a way to consolidate all the data in one page?
Attachments:
Open CSV file and history files.vi ‏33 KB

Importing a csv file into addressbook

So I did a search & found this question has come up quite often, but with the person having a slightly different problem, so here goes.
I am trying to import a csv file into address book, when I go import -> text tile it comes up like it should do and I pick the file, if I select ok at this point, it will quite happily import all of the data no trouble (but in the wrong categories). So I make the changes to what I want ie. Phone other --> Phone (work). Then when I click ok, the button goes from blue to white but nothing happens.
When I open the csv file in textedit I can see all the values separated by commas.
Any Ideas?

I too just had this problem. To clarify all fields work and you can map them. I found that the Outlook CSV file had two problems which were tripping up the AddressBook import function. Note these were both in street address fields so I suspect the poster who said that addresses don't work may have been having that problem.
The CSV load program does not work if you have commas "," in your data. In my case this was very common for some European addresses as well as some company names. And then the second problem was if there was a carriage return or new line character in a field. This later one was a problem with the address fields. I suspect an AddressBook export to CSV might have this same issue. Specifically, when there were multiple lines to the street address, they had been entered as multiple lines in street address one, as opposed to entering them in multiple fields such as "street 1", "street 2", etc.
To solve the problem I opened the CSV file using Numbers and did a global find and replace on: commas, substituting the string --- and on the newline character "¶" (ALT+Rreturn) with *.
Note to reduce some of the remapping I also changed all of the labels with "business" in the header row of the CSV file to "work". This made it a little easier for the AddressBook import function to map fields more correctly.
After these set of changes rather than the alternating blue then white with nothing behavior it cleanly loaded all of my names. I could then search for "---" and "*" in addressbook's search field to identify all addresses which I had to clean up.
I hope that is helpful to someone else. The import function really does work. However, it is not robust enough to deal with certain oddities in one's data.

Need to merge a csv file using external tables into a main table

Hi,
I have a csv file which contains the date(with time stamp), column1(number),column2(number), column3 (number). I am using external tables concept to load the data froom csv to this external table and then merging into the main table. Problem here is : the csv file is a system generated file and nothing can be edited under it. our aim is to automate this process of loading data from csv to the table. In this csv the date time stamp is not in the proper format.I mean the date is not visible and only minutes and seconds are visible.By changing the format in csv manually this can be overcome.but we donot need any manual intervention.
how can i overcome this problem ?? please help mee...
Excels data looks like:
(PDH-TSV 4.0) (India Standard Time)(-330)     \\DISAPPSER01\Processor(_Total)\% Privileged Time     \\DISAPPSER01\Processor(_Total)\% Processor Time     \\DISAPPSER01\Web Service(_Total)\Current Connections
56:59.0               47
57:09.0     0.72379582     4.204561281     46
57:19.0     0.916548537     4.006179927     44
57:29.0     0.663034771     3.674662541     43
57:39.0     0.750789844     4.093933999     42
57:49.0     0.721538487     2.650858026     40
57:59.0     0.594781604     3.333393703     40

please format your sample data giving header to the column so that we can make sense out of the values, also since the minutes and seconds are only given, what is the date to be considered for records to be moved to the master table, sysdate or will the date be passed as a parameter?

Problem in handling complex CSV file

Similar Messages

Maybe you are looking for