Reading csv files efficiently..

Hi..
I need to read,parse and insert required data from csv files of size 1 GB into database efficiently.
Which is the best approach.
Thanks,

I did this search:
http://www.google.com/search?q=site%3Ajakarta.apache.org+csv+parser
and found this:
http://jakarta.apache.org/commons/sandbox/csv/
It took only a few seconds.
Every java project should start with a google search against apache.org.

Similar Messages

Question about reading csv file into internal table

Some one (thanks those nice guys!) in this forum have suggested me to use FM KCD_CSV_FILE_TO_INTERN_CONVERT to read csv file into internal table. However, it can be used to read a local file only.
I would like to ask how can I read a CSV file into internal table from files in application server?
I can't simply use SPLIT as there may be comma in the content. e.g.
"abc","aaa,ab",10,"bbc"
My expected output:
abc
aaa,ab
10
bbb
Thanks again for your help.

Hi Gundam,
Try this code. I have made a custom parser to read the details in the record and split them accordingly. I have also tested them with your provided test cases and it work fine.
OPEN DATASET dsn FOR input IN TEXT MODE ENCODING DEFAULT.
DO.
READ DATASET dsn INTO record.
PERFORM parser USING record.
ENDDO.
*DATA str(32) VALUE '"abc",10,"aaa,ab","bbc"'.
*DATA str(32) VALUE '"abc","aaa,ab",10,"bbc"'.
*DATA str(32) VALUE '"a,bc","aaaab",10,"bbc"'.
*DATA str(32) VALUE '"abc","aaa,ab",10,"b,bc"'.
*DATA str(32) VALUE '"abc","aaaab",10,"bbc"'.
FORM parser USING str.
DATA field(12).
DATA field1(12).
DATA field2(12).
DATA field3(12).
DATA field4(12).
DATA cnt TYPE i.
DATA len TYPE i.
DATA temp TYPE i.
DATA start TYPE i.
DATA quote TYPE i.
DATA rec_cnt TYPE i.
len = strlen( str ).
cnt = 0.
temp = 0.
rec_cnt = 0.
DO.
* Start at the beginning
IF start EQ 0.
    "string just ENDED start new one.
    start = 1.
    quote = 0.
    CLEAR field.
ENDIF.
IF str+cnt(1) EQ '"'. "Check for qoutes
    "CHECK IF quotes is already set
    IF quote = 1.
      "Already quotes set
      "Start new field
      start = 0.
      quote = 0.
      CONCATENATE field '"' INTO field.
      IF field IS NOT INITIAL.
        rec_cnt = rec_cnt + 1.
        CONDENSE field.
        IF rec_cnt EQ 1.
          field1 = field.
        ELSEIF rec_cnt EQ 2.
          field2 = field.
        ELSEIF rec_cnt EQ 3.
          field3 = field.
        ELSEIF rec_cnt EQ 4.
          field4 = field.
        ENDIF.
      ENDIF.
*      WRITE field.
    ELSE.
      "This is the start of quotes
      quote = 1.
    ENDIF.
ENDIF.
IF str+cnt(1) EQ ','. "Check end of field
    IF quote EQ 0. "This is not inside quote end of field
      start = 0.
      quote = 0.
      CONDENSE field.
*      WRITE field.
      IF field IS NOT INITIAL.
        rec_cnt = rec_cnt + 1.
        IF rec_cnt EQ 1.
          field1 = field.
        ELSEIF rec_cnt EQ 2.
          field2 = field.
        ELSEIF rec_cnt EQ 3.
          field3 = field.
        ELSEIF rec_cnt EQ 4.
          field4 = field.
        ENDIF.
      ENDIF.
    ENDIF.
ENDIF.
CONCATENATE field str+cnt(1) INTO field.
cnt = cnt + 1.
IF cnt GE len.
    EXIT.
ENDIF.
ENDDO.
WRITE: field1, field2, field3, field4.
ENDFORM.
Regards,
Wenceslaus.

Use servlet read .csv file by ie, but why two more " ?

Hi , friends
following is the key method in my servlet used to read .csv file to ie , but , after I read file to client side , I found that in the both sides of every line " was added, so the csv file can have only one column (many columns was merged into one column), that was not what i want, who guys can find some idea? thanks for your enthusiasm!
void serveRemoteFile(String sFileName, String sContentType, HttpServletRequest req, HttpServletResponse res, StringBuffer sbLog, Runtime rt){
FileInputStream in = null;
ServletOutputStream out = null;
byte bBuf[] = null;
int nLen;
if (isBlank(sContentType) || isBlank(sFileName))
return;
res.setStatus(res.SC_OK);
res.setContentType(sContentType);
res.setHeader("Content-Disposition", "inline;filename= temp.csv");
try {
in = new FileInputStream(sFileName);
out = res.getOutputStream();
bBuf = new byte[1024];
while ( (nLen = in.read(bBuf,0,1024)) != -1) {
out.write(bBuf, 0, nLen);
out.flush();
out.close();
}catch (FileNotFoundException e){
errorPage(req,res,sbLog);
}catch (IOException e){
errorPage(req,res,sbLog);

Excel uses a weird CSV file format. You can find more information about it and a Java library to read data from it here:
http://ostermiller.org/utils/ExcelCSV.html

Drivers to read CSV file

Hi,
In one of my application, I need to read CSV file as a database and read the columns from the CSV file. Can I get any drivers which accepts CSV file as a datasource and I can query on that datasource?
I know this will work in Windows.. But I am working on UNIX and need drivers to be installed on Unix.
Does any one knows how to do it or if they can provide any links, That is of very helpful for me..
Please help me in resolving this problem..

Hi
You need to use a JDBC driver of Type 3 to do your task.
Swaraj

Read CSV file into a 1-D array

Hi
I would like to read a csv file into a cluster of 4 elements which would then be read into a 1-D array.
My cluster contains a typedef, a double, a boolean, and another typedef.
Basically it could be seen as:
Bob Runs, 4, T, Bob
Mary sits, 5, F, Mary
Bob Sits, 2, F, Bob
Mary Runs, 9, T, Mary
(keeps growing)
Are there any good examples for what I am trying to put together that I could leverage, or is it better to use a different input file than a csv. I am trying to make my program more flexable and easier to make adjustments even after the executable is created. My line items seem to be growing exponentially and is getting difficult to manage in the LV window.
Thanks
Solved!
Go to Solution.

Unless your CSV file is huge, I'd use "Read from Spreadsheet File" with the delimiter set as "," and the type as string. This will give you a 2D array of strings. You could then separate out each column of the array, convert to the appropriate data type, and use Index & Bundle Cluster Array to build your array of clusters. Something like this (except I'm using a string constant in place of reading from the file).

Read csv file and insert the data to a list in sharePoint

Hi everyone,
i wrote a code that reads from a csv file all his data but i need also to insert all the data to a new list in sharePoint.
how can i do this?
plus, i need to read from the csv file once a day in specific hour. how can i do this? thank you so much!!

Did you look at the example I posted above?
ClientContext in CSOM will allow you to get a handle on the SharePoint objects;
http://msdn.microsoft.com/en-us/library/office/ee539976(v=office.14).aspx
w: http://www.the-north.com/sharepoint | t: @JMcAllisterCH | YouTube: http://www.youtube.com/user/JamieMcAllisterMVP

SQL server 2014 and VS 2013 - Dataflow task, read CSV file and insert data to SQL table

Hello everyone,
I was assigned a work item wherein, I've a dataflow task on For Each Loop container at control flow of SSIS package. This For Each Loop container reads the CSV files from the specified location one by one, and populates a variable with current
file name. Note, the tables where I would like to push the data from each CSV file are also having the same names as CSV file names.
On the dataflow task, I've Flat File component as a source, this component uses the above variable to read the data of a particular file. Now, here my question comes, how can I move the data to destination, SQL table, using the same variable name?
I've tried to setup the OLE DB destination component dynamically but it executes well only for first time. It does not change the mappings as per the columns of the second CSV file. There're around 50 CSV files, each has different set off columns
in it. These files needs to be migrated to SQL tables using the optimum way.
Does anybody know which is the best way to setup the Dataflow task for this requirement?
Also, I cannot use Bulk insert task here as we would like to keep a log of corrupted rows.
Any help would be much appreciated. It's very urgent.
Thanks, <b>Ankit Shah</b> <hr> Inkey Solutions, India. <hr> Microsoft Certified Business Management Solutions Professionals <hr> http://ankit.inkeysolutions.com

The standard Data Flow Task supports only static metadata defined during design time. I would recommend you check the commercial COZYROC
Data Flow Task Plus. It is an extension of the standard Data Flow Task and it supports dynamic metadata at runtime. You can process all your input CSV files using a single Data Flow Task
Plus. No programming skills are required.
SSIS Tasks Components Scripts Services | http://www.cozyroc.com/

Problem in reading csv file in servlet

Hi everyone,
I m getting an ClassCastExeption while importing a .csv file to databse...
It works fine in case of .xls...
I m using jxl from Jakarta...and the code is as follows...
Thanks in advance..
FileItemFactory factory1 = new DiskFileItemFactory();
ServletFileUpload upload1 = new ServletFileUpload(factory1);
items1 = upload1.parseRequest(request);
Iterator iter = items1.iterator();
while (iter.hasNext()) {
FileItem item = (FileItem) iter.next();
InputStream uploadedStream = item.getInputStream();
importCSV(uploadedStream);
and in the import method.....
Workbook w;
w = Workbook.getWorkbook((FileInputStream)is);
In the above line i m getting ClassCastException...
Please help..

Thanks gimbal2 ..
w = Workbook.getWorkbook(is); throws the FileNotFoundException
I solved it by tokenizing the inputstream on ','(comma) and then getting the tokens for each column in the database....

Reading csv file how to get the Column name

Hi,
I am trying to read a csv file and then save the data to Oracle.
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
Connection c = DriverManager.getConnection("jdbc:odbc:;Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=.;Extensions=csv,txn");
Statement stmt = c.createStatement();
ResultSet rs = stmt.executeQuery("select * from filename.csv");
while(rs.next())
System.out.println(rs.getString("Num"));
My csv file looks like this:
"CHAM-23","COMPANY NAME","Test","12",20031213,15,16
Number,Environ,Envel,Date,Time
"1","2",3,"4",5
"6","7",8,"9",9
Now is there anyway using the above code I start processing the file from the second row that holds the names of the columns and skip the first row. And also can I get the name of the column using ResultSet something like:
if columnName.equals("Number")
Because I may have a csv file that could have more columns:
"CHAM-24","COMPANY NAME","Test","12",20031213,16,76
Number,Environ,Envel,Date,Time,Total,Count
"1","2","3","4","5",3,9
"6","7","8","9",9",,2
So I want to get the column name and then based on that column I do some other processing.
Once I read the value of each row I want to save the data to an Oracle Table. How do I connect to Oracle from my Application. As the database is on the server. Any help is really appreciated. Thanks

The only thing I could think of (and this is a cluj) would be to attempt to parse the first element of each row as a number. If it fails, you do not have a column name row. You are counting on the fact that you will never have a column name that is a number in the first position.
However, I agree that not always placing the headers in the same location is asking for trouble. If you have control over the file, format it how you want. If someone else has control over the file, find out why they are doing it that way. Maybe there is a "magic" number in the first row telling you where to jump.
Also, I would not use ODBC to simply parse a CSV file. If the file is formatted identically to Microsoft's format (headers in first row, all subsequent rows have same number of columns), then it's fine to take a shortcut and not write your own parser. But if the file is not adhering to that format, don't both using the M$ ODBC driver.
- Saish
"My karma ran over your dogma." - Anon

Reading CSV file, populating text fields, saving each as image to folder

I need to generate a bunch of images by reading a CSV file of records, populating various text fields over an image and then saving each iteration as a new image in a folder. Is Illustrator the best solution to this problem? And if so can someone help me get started?

In my opinion...
While Illustrator is an awesome tool its scripting DOM is a little limited, you may have better luck using InDesign as it has a much broader DOM.
If it is just simple text manipulations and you only require an image (not a document) Photoshop may work well for this as well.
Both the InDesign and Photoshop scripting forums seem to have much more activity as well.
For PhotoShop you can also use the awesome resources @: http://www.ps-scripts.com/bb/
Regards,
Herr Mueller

Reading csv file using file adapter

Hi,
I am working on SOA 11g. I am reading a csv file using a file adapter. Below are the file contents, and the xsd which gets generated by the Jdev.
.csv file:
empid,empname,empsal
100,Ram,20000
101,Shyam,25000
xsd generated by the Jdev:
<?xml version="1.0" encoding="UTF-8" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd" xmlns:tns="http://TargetNamespace.com/EmpRead" targetNamespace="http://TargetNamespace.com/EmpRead" elementFormDefault="qualified" attributeFormDefault="unqualified"
nxsd:version="NXSD"
nxsd:stream="chars"
nxsd:encoding="ASCII"
nxsd:hasHeader="true"
nxsd:headerLines="1"
nxsd:headerLinesTerminatedBy="${eol}">
<xsd:element name="Root-Element">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Child-Element" minOccurs="1" maxOccurs="unbounded">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="empid" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="empname" minOccurs="1" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="empsal" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="${eol}" nxsd:quotedBy=""" />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
For empname i have added minoccurs=1. Now when i remove the empname column, the csv file still gets read from the server, without giving any error.
Now, i created the following xml file, and read it through the file adapter:
<?xml version="1.0" encoding="UTF-8" ?>
<Root-Element xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://TargetNamespace.com/EmpRead xsd/EmpXML.xsd" xmlns="http://TargetNamespace.com/EmpRead">
<Child-Element>
<empid>100</empid>
<empname></empname>
<empsal>20000</empsal>
</Child-Element>
<Child-Element>
<empid>101</empid>
<empname>Shyam</empname>
<empsal>25000</empsal>
</Child-Element>
</Root-Element>
When i removed the value of empname, it throws the proper error for the above xml.
Please tell me why the behaviour of file adapter is different for the csv file and the xml file for the above case.
Thanks

Hi,
I am working on SOA 11g. I am reading a csv file using a file adapter. Below are the file contents, and the xsd which gets generated by the Jdev.
.csv file:
empid,empname,empsal
100,Ram,20000
101,Shyam,25000
xsd generated by the Jdev:
<?xml version="1.0" encoding="UTF-8" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd" xmlns:tns="http://TargetNamespace.com/EmpRead" targetNamespace="http://TargetNamespace.com/EmpRead" elementFormDefault="qualified" attributeFormDefault="unqualified"
nxsd:version="NXSD"
nxsd:stream="chars"
nxsd:encoding="ASCII"
nxsd:hasHeader="true"
nxsd:headerLines="1"
nxsd:headerLinesTerminatedBy="${eol}">
<xsd:element name="Root-Element">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Child-Element" minOccurs="1" maxOccurs="unbounded">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="empid" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="empname" minOccurs="1" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," nxsd:quotedBy=""" />
<xsd:element name="empsal" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="${eol}" nxsd:quotedBy=""" />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
For empname i have added minoccurs=1. Now when i remove the empname column, the csv file still gets read from the server, without giving any error.
Now, i created the following xml file, and read it through the file adapter:
<?xml version="1.0" encoding="UTF-8" ?>
<Root-Element xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://TargetNamespace.com/EmpRead xsd/EmpXML.xsd" xmlns="http://TargetNamespace.com/EmpRead">
<Child-Element>
<empid>100</empid>
<empname></empname>
<empsal>20000</empsal>
</Child-Element>
<Child-Element>
<empid>101</empid>
<empname>Shyam</empname>
<empsal>25000</empsal>
</Child-Element>
</Root-Element>
When i removed the value of empname, it throws the proper error for the above xml.
Please tell me why the behaviour of file adapter is different for the csv file and the xml file for the above case.
Thanks

Addressbook cant read .csv-files exported from Numbers (Solved)

Hello,
I can hardly believe this myself, but please try. Numbers separates fields by semicolons while exporting my address data to a .csv file, which seems completely feasible to me. When trying to import this into Addressbook, the applications moans that it cant import this file because it is not valid.
Do programmers from separate groups at Apple ever test whether their Apps work together? I just expect them to do so and I cant imagine a case more simple then that. This is really poor. The reason why I put this complaint in the Numbers discussion is, that there is no forum for Adressbook (whereas iCal got his own - why?).
And the reason why I pick up this topic again in a new thread is that the other thread I found related to transferring data from Numbers to Addressbook took a weird twist when they resorted to writing a complicated and elaborate Apple Script for this basic task. The Script provided there messed up with valuable Address data in the first version, so I am not going to try it here.
I just took the search and replace functionality of my text editor to solve it for me. But I still believe that this should work without headaches right out of the box. So Addressbook programmers please hurry up, go, fix it. It is just too distressing. Imagine, I did this in public with Windows guys around. I dont want to experience such again.
Yours, Christian Völker

Hello Yvan,
yes, I agree that Numbers is doing fine and by your explanation I even know why it does the way it does, but Addressbook remains broken.
Regarding your script I did not even bother to read it and I just dont trust it because I cant sue you in case something goes wrong. I prefer to do my own faults and in this case it was an even better and faster solution for me to use my text editor with search and replace.
To make the story complete, after the data was accepted, Addressbook wasnt able to recognize the field labeled "Email" as an Email Address and Street as a Street Name. After telling Addressbook about the meaning of the fields, it consistently crashed while importing. No, this is no good. A Script wont heal these flaws.

Labview not reading csv file properly.

I'm trying to use the read spreadsheet file vi to read the attached csv.
It does a few things incorrectly:
1) the first cell of the resulting array has ÿþ appended to the front of it, seems like labview is grabbing some special character it shouldn't? (Notepad, notepad++ and excel can all open the file without issue and none show ÿþ or anything like it).
2) It's mistreating the EOL characters as two line feeds, resulting in blank rows between each data row. (Looking at the csv in notepad++ confirms that the lines are ended with a CR and a LF (same as EOL).
3) I'm reading it in as a string, because one of the columns is non-numeric, but then I strip that column out and attempt to convert the remaining string array into a numeric array (this fails). It also fails if I pull out an individual element of the array and attempt to convert it to a number (in both cases using fract/exp string to number), below I have an example of trying to read the 2nd column of the 1st row (index: 0,1) which is formatted as a string: "1.37238168716431" and converting that into a number gives 0 as you can see in the snapshot of the front panel.
Solved!
Go to Solution.
Attachments:
REPORT02.CSV ‏4 KB

I guess the file is in unicode, that doesn't help me very much.
I don't use the "Read from Text File Function" - at least not directly, although it is called in Read from Spreadsheet File. Where is it called, however, right-click doesn't show an option for Convert EOL, instead it's a terminal and it's default is F and as the terminal is disconnected, I'm assuming it's not converting EOL. So it's already "unchecked" in a sense.
No idea how to convert the unicode string to ascii. (no functions show up when i search unicode or ascii in the manager). It displays it just fine in the text box, so I'm not sure why it's able to display it without issue but not read it as a number.
Using Labview 8.6, by the way.

Reading CSV file

Hello All,
As a requirement for employment retraining, I am trying to write a java program that reads a CSV and implements
three other methods: Even though, I have years of experience in non-objects oriented, language, I have a difficulty
with this IO stuff....as is the program doesn�t even compile.....help please ......
* This program expected to do 4 things:
* - Reads and prints the content of 'Book1.CSV' file.
* - It displays the total number columns present in the 'Book1.CSV' file.
* - It displays an alternate columns in the 'Book1.CSV' file, creats another method,
* in which you pass file, and it will display alternate columns of the file.
* - It takes an argument as number and a string as whole line from the file.
* First the argument number represents the column number in the string
* (second argument). This argument should return the value of column represented
* by first argument.
import java.io.*;
public class ReadFile {
     private static Object getColumnValue;
     private Object displayAlternateColumns;
     private Object getColumnCount;
     private String fileLine;
     private String targetString;
     private String searchString;
     private Object columnValue;
     private int columnNumber;
// It displays the total number columns present in the 'Book1.CSV' file.
     public int getColumnCount(String fileLine) {
          int columnValues = fileLine.spilt(",");
          return columnValues;
//Display an alternate columns in the 'Book1.CSV' file. Create another method,
//in which you pass file, and it will display alternate columns of the file.
public void displayAlternateColumns(targetString, String searchString,int
     columnNumber){
          String s = colptr(column, searchString);
          if (s == null)
               return -1;
          return s.compareTo(targetString)
//Create a method that takes an argument as number and a string as whole
//line from the file.First the argument number represents the column number in
//the string(second argument). This argument should return the value of
//column represented by first argument.
public String getColumnValue(int columnNumber, String fileLine) {
          String[] columnValues = fileLine.split(",");
          String value = columnValues[columnNumber];
          return value;
static final String NO_FILENAME_MSG = "No filename given!";
          public static void main(String[] args){
               String fileName = null;
               if(args.length > 0)
                    fileName = args[0];
               else
     System.out.println(NO_FILENAME_MSG);
               ReadFile RF = new ReadFile();
               RF.readFile(fileName);
          public void readFile(String fn){
               Object RF = null;
          File sourceFile = new File("c:\\Book1.csv");
          FileReader in = new FileReader(sourceFile);
          int character;
          while ((character = in.read()) != -1) {
          System.out.write(character);
          in.close();
//?????? I am npt sure what sort of logic needs to go here ?????
System.out.println(RF.getColumnCount(...???....));
System.out.println(RF.displayAlternateColumns(...????...));
System.out.println(RF.getColumnValue(...???..));
}

You failed to compile because there are many errors in your codes. Please go to the http://java.sun.com/j2se/1.5.0/docs/api/index.html to find each of the classes you used and their functions.
For reading CVS files, you want to read line by line and it looks like:
    try {
        FileReader     fr = new FileReader(your file name);
            BufferedReader br     = new BufferedReader(fr);
            String line = "";
                while ((line = br.readLine()) != null) {
                    // do something;
    } catch (IOException e) {
    } catch (FileNotFoundException e) {
    }Your first function should look like this
public int getColumnCount(String fileLine) {
    String[] array = fileLine.spilt(",");
    return array.length;
}...

Problem reading csv file with special character

Hai all,
i have the following problem reading a csv file.
442050-Operations Tilburg algemeen     Huis in t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
when reading this line with FM GUI_UPLOAD this line is split up in two lines in data_tab of the FM,
it is split up at this character
Line 1
442050-Operations Tilburg algemeen     Huis in
Line 2
t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
Anyone have a idea how to get this in one line in my interbal table??
Greetz Richard

Hi Greetz Richard
Problably character contains same binary code as line feed + carriage return.
You can use statement below as workaround.
OPEN DATASET file FOR INPUT IN TEXT MODE ENCODING UNICODE
In this case your system must support Unicode encoding
Kind regards

Reading csv files efficiently..

Similar Messages

Maybe you are looking for