Parsing BLOB (CSV file with special characters) into table
Hello everyone,
In my application, user uploads a CSV file (it is stored as BLOB), which is later read and parsed into table. The parsing engine is shown bellow...
The problem is, that it won't read national characters as Ö, Ü etc., they simply dissapear.
Is there any CSV parser that supports national characters? Or, said in other words - is it possible to read BLOB by characters (where characters can be Ö, Ü etc.)?
Regards,
Adam
|
| helper function for csv parsing
|
+-----------------------------------------------*/
FUNCTION hex_to_decimal(p_hex_str in varchar2) return number
--this function is based on one by Connor McDonald
--http://www.jlcomp.demon.co.uk/faq/base_convert.html
is
v_dec number;
v_hex varchar2(16) := '0123456789ABCDEF';
begin
v_dec := 0;
for indx in 1 .. length(p_hex_str) loop
v_dec := v_dec * 16 + instr(v_hex, upper(substr(p_hex_str, indx, 1))) - 1;
end loop;
return v_dec;
end hex_to_decimal;
|
| csv parsing
|
+-----------------------------------------------*/
FUNCTION parse_csv_to_imp_table(in_import_id in number) RETURN boolean IS
PRAGMA autonomous_transaction;
v_blob_data BLOB;
n_blob_len NUMBER;
v_entity_name VARCHAR2(100);
n_skip_rows INTEGER;
n_columns INTEGER;
n_col INTEGER := 0;
n_position NUMBER;
v_raw_chunk RAW(10000);
v_char CHAR(1);
c_chunk_len number := 1;
v_line VARCHAR2(32767) := NULL;
n_rows number := 0;
n_temp number;
BEGIN
-- shortened
n_blob_len := dbms_lob.getlength(v_blob_data);
n_position := 1;
-- Read and convert binary to char
WHILE (n_position <= n_blob_len) LOOP
v_raw_chunk := dbms_lob.substr(v_blob_data, c_chunk_len, n_position);
v_char := chr(hex_to_decimal(rawtohex(v_raw_chunk)));
n_temp := ascii(v_char);
n_position := n_position + c_chunk_len;
-- When a whole line is retrieved
IF v_char = CHR(10) THEN
n_rows := n_rows + 1;
if n_rows > n_skip_rows then
-- Shortened
-- Perform some action with the line (store into table etc.)
end if;
-- Clear out
v_line := NULL;
n_col := 0;
ELSIF v_char != chr(10) and v_char != chr(13) THEN
v_line := v_line || v_char;
if v_char = ';' then
n_col := n_col+1;
end if;
END IF;
END LOOP;
COMMIT;
return true;
EXCEPTION
-- some exception handling
END;
Uploading CSV files into LOB columns and then reading them in PL/SQL: [It’s|http://forums.oracle.com/forums/thread.jspa?messageID=3454184�] Re: Reading a Blob (CSV file) and displaying the contents Re: Associative Array and Blob Number of rows in a clob doncha know.
Anyway, it woudl help if you gave us some basic information: database version and NLS settings would seem particularly relevant here.
Cheers, APC
blog: http://radiofreetooting.blogspot.com
Similar Messages
-
Problem to get file with special characters in the message
Hi, I'm developing an application that read the email and save the attached file. However, some files have special characters in the name, like: Documento de EspecificaÇÂO.doc
I noticed the de name of the file in de message head is:
"=?iso-8859-1?Q?Documento_de_Especifica=E7=E3o.doc?="
I'm already using the JavaMil 1.4.5
Tha is my code:
package br.com.cesan.helpdesk;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
import javax.mail.Flags;
import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.Part;
import javax.mail.Session;
import javax.mail.Store;
import javax.mail.search.FlagTerm;
public class LerEmail2 {
* @param args
* @throws MessagingException
* @throws IOException
public static void main(String[] args) throws MessagingException, IOException {
// TODO Auto-generated method stub
// Get session
Session session = Session.getInstance(new Properties(), null);
// Get the store
Store store = session.getStore("pop3");
store.connect("pop.xxxxx.com.br", "user", "password");
Folder folder = store.getFolder("INBOX");
//folder.open(Folder.READ_ONLY);
folder.open(Folder.READ_WRITE);
// Show only unreaded Messages
FlagTerm ft = new FlagTerm(new Flags(Flags.Flag.SEEN), false);
// Get directory
Message message[] = folder.getMessages();
//Message messages[] = folder.search(ft);
for (int i=0, n=message.length; i<n; i++) {
System.out.println(i + " - "+ message.getSubject() +" - " + message[i].getSentDate() );
Object content = message[i].getContent();
if (content instanceof Multipart) {
handleMultipart((Multipart)content);
} else {
handlePart(message[i]);
// Close connection
folder.close(false);
store.close();
public static void handleMultipart(Multipart multipart) throws MessagingException, IOException {
for (int i=0, n=multipart.getCount(); i<n; i++) {
handlePart(multipart.getBodyPart(i));
public static void handlePart(Part part) throws MessagingException, IOException {
String disposition = part.getDisposition();
String contentType = part.getContentType();
if (disposition == null) { // When just body
System.out.println("Null: " + contentType);
// Check if plain
if ((contentType.length() >= 10) && (contentType.toLowerCase().substring(0, 10).equals("text/plain"))) {
part.writeTo(System.out);
} else { // Don't think this will happen
System.out.println("Other body: " + contentType);
part.writeTo(System.out);
} else if (disposition.equalsIgnoreCase(Part.ATTACHMENT)) {
System.out.println("Attachment: " + part.getFileName() + " : " + contentType);
saveFile(part.getFileName(), part.getInputStream());
} else if (disposition.equalsIgnoreCase(Part.INLINE)) {
System.out.println("Inline: " +
part.getFileName() +
" : " + contentType);
saveFile(part.getFileName(), part.getInputStream());
} else { // Should never happen
System.out.println("Other: " + disposition);
public static void saveFile(String filename, InputStream input) throws IOException {
if (filename == null) {
filename = File.createTempFile("xx", ".out").getName();
// Do no overwrite existing file
File file = new File(filename);
for (int i=0; file.exists(); i++) {
file = new File(filename+i);
System.setProperty("file.encoding", "iso-8859-1");
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
BufferedInputStream bis = new BufferedInputStream(input);
int aByte;
while ((aByte = bis.read()) != -1) {
bos.write(aByte);
bos.flush();
bos.close();
bis.close();
The problem occurs in:
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
BufferedInputStream bis = new BufferedInputStream(input);
int aByte;
while ((aByte = bis.read()) != -1) {
bos.write(aByte);
Thanks
Edited by: user10283976 on 30/03/2012 07:36
Edited by: user10283976 on 30/03/2012 07:37
Edited by: user10283976 on 30/03/2012 07:37
Edited by: user10283976 on 30/03/2012 07:38
Edited by: user10283976 on 30/03/2012 07:40
Edited by: user10283976 on 30/03/2012 07:41
Edited by: user10283976 on 30/03/2012 07:42http://www.oracle.com/technetwork/java/javamail/faq/index.html#encodefilename
-
Loading a CSV file with Umlaut characters (àáä)
Hai,
We are uploading a CSV file though a Custom JSP page built based on Oracle JTF framework.
The JSP page is loading the data into FND_LOBS table using JTF object, oracle.apps.jtf.amv.ServletUploader.
The CSV file in the FND_LOBS table stored properly with the umlaut characters.
Now the JSP page invokes a Java object to read and parse the data. We are selecting the data first into BLOB object and then using the Input Stream Reader to get the data.
Here is the sample code:
oraclepreparedstatement = (OraclePreparedStatement)oracleconnection.prepareStatement(" SELECT FILE_DATA FROM FND_LOBS WHERE FILE_ID = :1 ");
oraclepreparedstatement.defineColumnType(1, 2004);
oraclepreparedstatement.setLong(1, <file id>);
oracleresultset = (OracleResultSet)oraclepreparedstatement.executeQuery();
blob = (BLOB)oracleresultset.getObject(1);
InputStreamReader inputstreamreader = new InputStreamReader(blob.getBinaryStream());
lineReader = new LineNumberReader(inputstreamreader )
lCSVLine = lineReader.readLine();
I tried printing the character set used by the InputStreamReader and it returned as ASCII
Now I tried setting the different character sets to read Umlaut characters(german chars) but nothing has worked.
InputStreamReader inputstreamreader = new InputStreamReader(blob.getBinaryStream(),"UTF-8");
Can someone please let me know where and how to set the Character Set to accept the Umlaut characters like àáä?
Thanks,
AnjiThank you for the quick response.
Requirement:
I need to retrive the BLOB object with umlaut characters from database, parse the data with comma delimeter into strings and store in database.
I am viewing the umlaut data from the database table using TOAD utility tool.
I tried the same code example provided above but it is not working as expected. The umlaut characters are translated to 'ýýý.
CODE EXAMPLE:
Input:
test_umlaut (sno NUMBER, col1 VARCHAR2(100), col3 BLOB);
insert into test_umlaut(sno,col3) values(200, utl_raw.cast_to_raw('äöüÄÖÜ' ))
Note: Verified that the database is showing the umlaut characters on selecting the col3 and storing in a flat file
--- code
OraclePreparedStatement oraclepreparedstatement10 = null;
OracleResultSet rs = null;
oraclepreparedstatement10 = (OraclePreparedStatement)oracleconnection.prepareStatement(" SELECT col3 FROM test_umlaut WHERE sno = 200 ");
oraclepreparedstatement10.defineColumnType(1, 2004);
rs = (OracleResultSet)oraclepreparedstatement10.executeQuery();
while(rs.next()) {
BLOB b = (BLOB)rs.getObject(1);
InputStream is = b.getBinaryStream();
InputStreamReader r = new InputStreamReader(is,"utf8");
BufferedReader br = new BufferedReader(r);
String line;
while( (line = br.readLine()) != null) {
System.out.println(line);
OraclePreparedStatement oraclepreparedstatement12 = null;
OracleResultSet oracleresultset12 = null;
oraclepreparedstatement12 = (OraclePreparedStatement)oracleconnection.prepareStatement(" INSERT INTO test_umlaut(sno,col1) VALUES (300,?) ");
oraclepreparedstatement12.setString(1,line);
oraclepreparedstatement12.executeUpdate();
br.close();
r.close();
is.close();
Output: Verified the output from the database table which is inserted in the loop above.
select col1 from test_umlaut where sno=300
ýýý -
Parsing a csv file with carriage return replaced with #
Hi,
We have a weird problem. We are able to download a csv file using standard FM HTTP_GET. We want to parse the file and upload the data into our SAP CRM system. However, the file downloaded, has the carriage return replaced and the character # replaces it and everything seems like its one line.
I understand that the system replaces the Carriage return with the charater #. My question is, if I try to pass this file into my program to parse for the data, will there be any issues in the system recognizing that "#" that it is a carriage return and that the data in the file is not 1 record but multiple records?Hi
'#' is what you see in the SAP. But the actuall ascii associated will be of carraige return itself. So to identify if you have multiple records of not don't use hard coded '#' but instead use the constant CL_ABAP_CHAR_UTILITIES=>CR_LF.
Regards
Ranganath -
Error importing CSV files with 'hidden' characters using External Table
Hi Folks
Bit of a strange one here.
We're well used to using the External Table method of loading data from CSV files into the database but a recent event has presented us with a problem.
We have received some CSV files that 'look' like regular CSV files but Oracle will not load them.
When we examined the CSV file using VIM on a UNIX box we saw the following 'hidden' characters between every regular character in the file.
^@So a string that looks like this when opened in Excel/Wordpad etc
"TEST","TEXT"Looks like this when exmained with VIM
^@"^@T^@E^@S^@T^@"^@,^@"^@T^@E^@X^@T^@"Has anyone come across this before?
Many thanks
Simon Gadd
Oracle 11g 11.2.0.1.0Hi Simon,
^@ represents the NUL character (0x00).
So, most likely, you've got a Unicode-encoded file.
You'll have to specify the character set in the record specification (and if necessary the byte order mark), for instance :
CREATE TABLE ext_table
col1 VARCHAR2(10),
col2 VARCHAR2(10)
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY dump_dir
ACCESS PARAMETERS
RECORDS DELIMITED BY '
' CHARACTERSET 'UTF16'
FIELDS TERMINATED BY ','
LOCATION ('dump.csv')
REJECT LIMIT UNLIMITED;http://download.oracle.com/docs/cd/E11882_01/server.112/e16536/et_params.htm#i1009499 -
Creating Text files with special characters
I have a problem writing and then reading special characters in a text file. I am using FileWriter and FileReader and once I write (using FileWriter) in a file.txt many characters with ASCII code less than 31, I can not read them back using FileRead. I simply get a content which is different from the one originally written.
Does anyone have an idea about this problem ?The code is OK, problems may have occured during copy/paste. But in the original code it is written c.
Here is a copy of the output of those lines that i'm talking about:
Here is the ASCII Code >> 125
Here is the ASCII Code >> 126
Here is the ASCII Code >> 127
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 63
Here is the ASCII Code >> 160
Here is the ASCII Code >> 161
Here is the ASCII Code >> 162 -
File Adapter write XML file with special characters
Hi,
I have a BPEL processes which use DB adapter to retrieve record and output in a XML document using File Adapter.
In the table, some records contain these special characters - ">" , "<" , "." . I was expecting the file adapter will convert it to escape characters, but it didn't happen. it just put the same value back to the XML file, which cause other system to reject the file. Do anyone know how to resolve this ?
The version of SOA is 10.1.3.4
Thanks for the help.
Calvin
Edited by: user12018221 on May 25, 2011 1:48 PMone option is to specify validateXML on the partnerlink (that describes the file adapter endpoint) such as shown here
<partnerLinkBinding name="StarLoanService">
<property name="wsdlLocation"> http://<hostname>:9700/orabpel/default/StarLoan/StarLoan?wsdl</property>
<property name="validateXML">true</property>
</partnerLinkBinding>
hth clemens -
Problem reading csv file with special character
Hai all,
i have the following problem reading a csv file.
442050-Operations Tilburg algemeen Huis in t Veld, EAM (Lisette) Gebruikersaccount 461041 Peildatum: 4-5-2010 AA461041 1 85,92
when reading this line with FM GUI_UPLOAD this line is split up in two lines in data_tab of the FM,
it is split up at this character
Line 1
442050-Operations Tilburg algemeen Huis in
Line 2
t Veld, EAM (Lisette) Gebruikersaccount 461041 Peildatum: 4-5-2010 AA461041 1 85,92
Anyone have a idea how to get this in one line in my interbal table??
Greetz RichardHi Greetz Richard
Problably character contains same binary code as line feed + carriage return.
You can use statement below as workaround.
OPEN DATASET file FOR INPUT IN TEXT MODE ENCODING UNICODE
In this case your system must support Unicode encoding
Kind regards -
Splitting of a CSV File with Multiple Records into Multiple XML File
Dear All,
<b> I am doing a Scenario of CSV to XML Files. I am using BPM for the same. My incoming CSV File has got multiple records. I want to break this Multiple records into Multiple XML Files having one record each.</b>
Can someone suggest how can I break this rather Split this into Multiple XML Files.
Is Multimapping absoltely necesaary for this. Can't we do this without Multimapping. Can we have some workaround in the FCC parameters that we use in the Integration Directory.
Kindly reply ASAP. Thanks a lot to all in anticipation.
Pls Help.
Best Regards
Chakra and SomnathDear All,
I am trying to do the Multimapping, and have 0....unbounded also. Someways it is not working.
<b>
Smitha please tell me one thing...Assigning the Recordsets per Message to 1, does it mean that it will write multiple XML Files as I want.</b>
Also I am usinf Set to Read only. So once the File is read it becomes RA from A. Then will it write the other Records.
I have to use a BPM because there are certain dependencies that are there for the entire Process Flow. I cannot do without a BPM.
Awaiting a reply. Thanks a lot in anticipation.
Best Regards
Chakra and Somnath -
Hi friends, Problems with Special Characters in Table download to Excel.
Hi friends,
I am using Binary Cache method to download to excel. The problem is that there are certain fields in the back end R/3 that has special characters in it eg : & * £ # etc.
As a result, the download is not working for these rows. Please can any of you tell me how to get rid of this. As an example let's say I wan to replace the "&" with "and".
Please help.
Thank You.
Avikhi,
Hope you find tgis of some help !!
public void ReplaceMethod( )
//@@begin ReplaceMethod()
// get the string value of the cell (for you from the table cell, for eg, Pinki & Rakesh)
test = wdContext.currentTestStringConversionElement().getInput();
// calculate the total length of the String
n=test.length ();
if(n!=0)
for(i=1;i<=n;i++)
// catching each character from the String
c = test.charAt (i-1);
// checking if any character within the String (eg, Pinki & Rakesh) is &
if (c=='&')
wdContext.currentTestStringConversionElement().setOutput(Character.toString(c));
no=i;
break;
String val = Integer.toString(no);
//wdContext.currentTestStringConversionElement().setNo(val);
// breaking the String and adding and in place of &
sub1= test.substring(0,i-1);
sub2= " and ";
sub3= test.substring(i,n);
//Concatenating the String
test1 = sub1sub2sub3;
//Printing the String
wdContext.currentTestStringConversionElement().setOutput(test1); -
Replacing text in a text file with special characters
Hi all,
in my test text file i found an interesting issue, where i can't replace text that looks something like this "New Text Document - Copy (2).txt"
(gc $logeps)|foreach{$_ -replace "$($file.name)",""}|Out-File $logeps
the context of my $($file.name) is "New Text Document - Copy (2).txt"
how can i escape the parentheses characters inside of a variable that i am passing?
thanksThat should work if you're running PowerShell 3.0 or later, but in older versions, you'd run into errors if you try to run .Replace on an array of strings.
While I prefer the string .Replace method in this instance, for future reference, if you want to escape a literal string so it can be used in a regular expression, call the [regex]::Escape method:
$pattern = [regex]::Escape($file.Name)
(gc $logeps) -replace $pattern | Out-File $logeps -
Clementine can't play files with special characters
I saw Clementine can't play some mp3 files - it would just say "error loading file".
So I opened an mp3 from the terminal and found out that the problem was the characters in the file name.
For example, it wouldn't play "Schiller - Tired (mit Jaël).mp3" but works fine with "Schiller - Tired (mit Jael)"
Any ideas why?Well...
[bogdan@myhost ~]$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=
[bogdan@myhost ~]$ -
I also have a .csv file with the name of a jpeg in one column and a text description of each jpeg in a second column. Is there a way to automatically insert one jpeg (photo) and its corresponding text, each pair on one page, into a Indesign document?
I would also recommend to write the description into the meta data. This would allow to place a text frame above the image and it is possible to add meta information and file name automatically together with the image, when you place it or even in a prepared template.
Meta data information can be written easily in Bridge in the Meta File Workspace. -
Problem with special characteres
Hi. I have a problem with java. I�m making an application that works with special characters and displays correctly on my local machine, as I use escape characters such as ì for �. The problem is when I deploy the application on a unix machine. All of the special characters (�, �, �, etc) appear like a �?� (question mark). Any tips, please?
Thanks in advanceAndy2008 wrote:
Hi. I have a problem with java. I’m making an application that works with special characters and displays correctly on my local machine, as I use escape characters such as ì for �. The problem is when I deploy the application on a unix machine. All of the special characters (‘, �, �, etc) appear like a “?” (question mark). Any tips, please?First step: read [this excellent introduction into Unicode|http://www.joelonsoftware.com/articles/Unicode.html], it will give you the basics you need to solve this problem.
How do you input and output those characters? Simply print them to the console? Or write them to a HTML file (since you mention &<something>; notation)?
"?" is normally the result when you convert a String (which is always in UTF-16 encoding in Java) to an encoding that can't represent your characters (ASCII for example). -
Tag Query with Special Characters in Tag Name not working
Hello-
We are on SAP xMII 12.0 sp8.6, connecting to Honeywell's PHD. We are doing Tag Queries, and all is working well, except when there are special characters in the Tag Name (+, /, \) We have attempted to do the Encoding of the tag name, such as as is returned from the function xmlencodename and also tried URL Encoding.
Here is our tag which returns nothing.
82TK1INV.OCTANE_R+M/2
We have also tried these encoded versions, still to no avail:
82TK1INV.OCTANE_R%2BM%2F2
82TK1INV.OCTANER_M_2
Any help on how to return the values with special characters in the tag name would be very helpful!
thanks,
Paul MazeikaTo Chianti's question of why can't/shouldn't MII work with those bizarre tag names...
Actually, there are a number of reasons:
1) It's a poor design practice to use characters like that in tag names, regardless of the underlying system. That type of information is best assigned to the tag description, not the tag name. I've even seen idiots put leading spaces on tag names...figure that one out!
2) MII uses XML extensively, and XML's syntax rules dictate what is and is not acceptable for XML element and attribute names, and many of these special characters are not permitted.
3) Usage of characters in tags that also represent mathematical operators greatly complicate the parsing and processing of expressions that involve tag names
In general, it reminds of the story of the guy who walks into the doctor, bangs his fist against a specific spot on his head, and says "Doc, it hurts when I do this", to which the doctor responds, "so don't do that".
So, Don't Do That.
Maybe you are looking for
-
Mom does not see my iChat video, but I see her, and my wife can see mine
When I iChat with Mom, I can see her, but she cannot see my iChat video. All she sees is herself as if she is still in iChat preview mode. Audio works fine. It is like a 1-way video chat with a 2-way audio chat, only we did not request this mode. Thi
-
The recording device reported the media error: Unknown device error. (0x0C,
Hello I have a problem making a DUAL LAYER DVD, i have done anothers before. But now I can't burn it. It's a message error. (The recording device reported the media error: Unknown device error. (0x0C, 0x00.)) Next this my Mac configuration. Model Nam
-
How to implement a foreach cycle through Office 365 Calendar appointments?
Hi everyone, I am looking for a way to implement a foreach cycle that is better than what I could create already, so I would like to have your feedback. Basically, I have an Office365 Calendar and I want to count the number of appointments per day, s
-
HT1688 How come when I try to download an app, it doesn't ask for my password?
Hi. Usually when I download an app it always asks me for my password before it begins to download. But this time I can just download without them asking for my permission. How do I change that?
-
Ok, so one day, go to wake my iMac from sleep, it freezes, force a hard restart, it never boots again, get a white screen with nothing on it, no matter how long you leave it up. Took it to the geniuses, said it was the hard drive, replaced the hard d