Language Encoding

Hi all,
I have a question about detecting language encoding in a file. The w3c xml spec talks about identify the encoding and byte ordering via some lead bytes. Is there a java way to do this auto detection?

Sure, just open a FileInputStream on the file, read the first two bytes, and apply the W3C rules. This will probably involve closing that FileInputStream and opening a new FileReader.

Similar Messages

Problem in arabic language encoding

hi all
I have a problem in arabic language encoding when store data into database
i wrote my problem here in this page
http://www.010555555.com/Arabic/index.html
so please contact me if you know the solution on
[email protected]
thanks alot all

whos you sugest for me to solve my problem

Thai Language Encoding with TIS-620

Hi All,
I have a file which has few Thai language words.I need to decode with TIS-620 charset.
I have tried with a standalone program.When I run the program and try to print the local lang program..it simply prints ??? in Eclipse console......When I try to print the Byte Array with the charset Name=TIS-620, it gives me..[B@7d772e and when I try to print it after decode the array of bytes with given char set as ISO8859_1..it prints some garbage characters....��
I am attaching the java code I have been trying and also the file which I am trying to read.
{code}import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
public class te {
     private static final String ISO = "ISO8859_1";
     private static String charsetTobeChanged="TIS-620";
     public static void main(String args[]) throws Exception{
     BufferedReader brFile;
     String inFileName=null ;
     String inLine,inLine_BENE_NAME,decode_LL_BENE_NAME ;
     inLine_BENE_NAME="test";
     FileOutputStream fos2=new FileOutputStream("C:\\Documents and Settings\\Desktop\\asPerBaseFile.dat");
     DataOutputStream output = new DataOutputStream (fos2);
     final String NEXT_LINE = "\n";
     inFileName= "C:\\Documents and Settings\\D150911\\Desktop\\latestBaseFile.dat" ;
     String newStrLine,newDecodedBeneName;
     try{
          brFile = new BufferedReader(new InputStreamReader(new FileInputStream(inFileName), "UTF-8"));
          while ((inLine = brFile.readLine()) != null) {
               try{
                    inLine_BENE_NAME = getField(inLine, inLine.length(), 78, 128).trim();
                    if (inLine_BENE_NAME!=null && inLine_BENE_NAME.length()>0)
                                   decode_LL_BENE_NAME = decode(inLine_BENE_NAME.getBytes(charsetTobeChanged),ISO);
                                   newDecodedBeneName= fillSpace(decode_LL_BENE_NAME,50);
                                   newStrLine=inLine.substring(0, 78).concat(newDecodedBeneName).concat(inLine.substring(128));
                                   output.writeBytes(newStrLine);
                                   output.writeBytes(NEXT_LINE);
                    else
                         output.writeBytes(inLine);
               }//end try
               catch (Exception encodeE)
                              System.out.println(" exception in Encoding=="+encodeE);
          }//end while
     }//end of try
     catch(Exception e)
               System.out.println("File not found= or any other exception ==="+e);
     //getField
     public static String getField(String lineString, int lineLength, int startPos, int endPos) throws Exception {
          String outString;
          try {
               if (lineLength >= endPos) {
                    outString = lineString.substring(startPos, endPos);
               } else
                    if (lineLength >= startPos) {
                         outString = lineString.substring(startPos, lineLength);
                    } else {
                         outString = "";
               return(outString);
          catch (Exception ex) {
               throw ex;
     public static String decode(byte[] pvBytes, String pvTargetCodePage) throws Exception
     if(pvBytes == null)
     return "";
     return new String(pvBytes, pvTargetCodePage);
     public static String fillSpace(String field_value, int number_of_digit)
     String add_space = "";
     if (field_value.length() < number_of_digit) {
     for (int j=0; j<(number_of_digit - field_value.length()); j++) {
     add_space = add_space + " ";
     field_value = field_value + add_space;
     return field_value;
} // fillSpace
}//end class
And the file which I am trying to read is following..please copy and paste as a file...
000001 7441 7454797 2721001477 000000050030932 080429 0190014754              ผู้จัดการโรงเรียนมัธยมวิจัยระเบียบเกณฑ์ด้านดื้อดืฉ
000002 7441 7454797 1681016631 000000057153890 080429 0190014757              ศึกษาดื้อดื้อดื้อดื้อดื้อดื้อดื้อดื้อช่องวิทยาชนฉั
000003 7441 7454797 4162503389 000000090107942 080429 0190014762              ผู้จัดการโรงเรียนมัธยมวิจัยระเบียบเกณฑ์ด้านดื้อดืฉ
000004 9100 7454797 0000000000 000000197292764 000000
==============================================
The above line is just a seperator...It can be noticed in the given file above the 3rd last line after LL has a Thai word which needs to be encoded....
I mean ,I want to decode it with Char set TIS-620 so that when I open it in IE..Thai language can be viewed by Thai encoding.
Please help.
Edited by: InfoRequired on Jul 11, 2008 1:11 AM

Thank you for the help!.
True, I read the article you provided but it was a bit confusing if I read for the first time.
anyways, you summarised well.I tried with following,read the file,did string manipulation then simplay wrote to a file with require character encoding.
Preview:
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
public class forumHelp {
     private static String charsetTobeChanged="TIS-620";
     public static void main(String args[]) throws Exception{
     BufferedReader brFile;
     //OutputStreamWriter f1;
     String inFileName=null ;
     //String outFileName=null;
     String inLine,inLine_BENE_NAME;
     inLine_BENE_NAME="test";
     //ouput stream writer
     FileOutputStream fileOut = new FileOutputStream ("C:\\Documents and Settings\\Desktop\\forumHelp2.dat",true);
     OutputStreamWriter out1;
     out1 = new OutputStreamWriter (fileOut, charsetTobeChanged);
     out1.flush();
     final String NEXT_LINE = "\n";
     inFileName= "C:\\Documents and Settings\\Desktop\\latestBaseFile.dat" ;
     String newStrLine,newDecodedBeneName;
     try{
          brFile         = new BufferedReader(new InputStreamReader(new FileInputStream(inFileName), "UTF-8"));
          //f1         =new OutputStreamWriter(new FileOutputStream(outFileName), charsetTobeChanged);
          while ((inLine = brFile.readLine()) != null) {
               try{
                    inLine_BENE_NAME      = getField(inLine, inLine.length(), 78, 128).trim();
                    if (inLine_BENE_NAME!=null && inLine_BENE_NAME.length()>0)
                                   newDecodedBeneName= fillSpace(inLine_BENE_NAME,50);
                                   newStrLine=inLine.substring(0, 78).concat(newDecodedBeneName).concat(inLine.substring(128));
                                   out1.write(newStrLine);
                                   out1.write(NEXT_LINE);
                    else
                         out1.write(inLine);
               }//end try
               catch (Exception encodeE)
                              System.out.println(" exception in Encoding=="+encodeE);
          }//end while
     }//end of try
     catch(Exception e)
               System.out.println("File not found= or any other exception ==="+e);
     out1.close();
     //getField
     public static String getField(String lineString, int lineLength, int startPos, int endPos) throws Exception {
          String outString;
          try {
               if (lineLength >= endPos) {
                    outString = lineString.substring(startPos, endPos);
               } else
                    if (lineLength >= startPos) {
                         outString = lineString.substring(startPos, lineLength);
                    } else {
                         outString = "";
               return(outString);
          catch (Exception ex) {
               throw ex;
     public static String fillSpace(String field_value, int number_of_digit)
                    String add_space = "";
                  if (field_value.length() < number_of_digit) {
                          for (int j=0; j<(number_of_digit - field_value.length()); j++) {
                                  add_space = add_space + " ";
                  field_value = field_value + add_space;
                  return field_value;
     } // fillSpace
}//end classThank you all again!

Is my language encoded at all ??

Dear friends ,
How can one find out if a language is encoded in any coding system.My country language is Pashto. Java supports many encodings.But what if Pashto is not encoded and hence not supported by java ? I know many of the characters of my language are unicode encoded but unfortunately not all.If it is encoded in any encoding , how can I use it in java?
My second question is , how can if find out which encoding is used by a font ?
Please help me solve this issue.....and thanks in advance........

I deal mostly in applet and this is what I use:
setLocale(language,country,zone);
along with a default properties file that contains English text for various prompts that the user can localize with their own fonts. As far as find fonts, if you use Windows, you can get the necessary fonts by searching www.microsoft.com. I use Windows 98SE, Thai edition and I was able to search Thai web site for the fonts I need.
V.V.

My Firefox cannot display non-English characters, even though I have tried every language encoding I have!

I am a big fan of Japanese songs and websites, so I was very disappointed when I saw that Firefox could not handle any non-English characters. I have tried every encoding I can, but none work and I just see boxes with numbers and letters inside. I have only just got this older laptop for my birthday - my old laptop which ran Windows Vista and had Firefox 4 had no trouble at all. Please help me!

hello muoshui, please enter '''about:config''' into the firefox location bar (confirm the info message in case it shows up) & search for the preference named '''network.http.accept-encoding''' - right-click and reset that entry to the default value.
if this does not resolve the issue already, please also go through the steps offered at [[Websites look wrong or appear differently than they should]].

Need help with Locale or language encoding

Hi,
I'm not sure about the right solution for my problem, but here's the problem:
I'm from Israel, and I write some jdbc code using Oracle 10 server. (Oracle version isn't important, it used to happen also with Oracle 8).
The problem is, that when exceptions occur, Oracle will return a string like ????.???????? since it tries to convert it to my local settings, and not displaying the message in English.
I need to display sql exceptions in a dialog so the support team can deal with data problems.
However,
The dialog doesn't display the messages correctly (not with question marks, though, but still unreadable).
My solution for the time being is to copy the message to the clippboard and let my user search the web or a translation program he has for the correct encoding.
I tried both setting the Locale to US using JOptionPane.setDefaultLocale(Locale.US) and I also tried the hebrew local (maybe the exception content will not be translated), but ti didn't help, as Oracle allready translated the messages for me.
I guess it's the use of encoding schemes that will solve that problem, but I don't know how to do that in my application.
BTW,
I'm using Intellij and my console also displays unreadable data,
I didn't find how to change it in my IDE's settings.
How can I display the data correctly?
Thanks.

I can't change anything in Oracle.
However, the problem is solved.
We used to use the Oracle driver for Oracle 8 with all of our systems,
but now we upgraded it to the driver compatible with Oracle 10,
and it returns values correctly.
BTW, I don't see it as a good solution, since I didn't realy solve the problem, I only bypassed it.
When I'll encounter it again with a non Oracle produce that doesn't support hebrew, I'll get stuck again.
But for now, it'll do.
Thanks anyway.

Safari and Japanese websites / Language Encoding

Safari doesn't show correctly webpages with Japanese characters on them. The Japanese characters don't appear at all.
Also in the View menu, the Text Encoding option doesn't show any text... it is just a list of blank spaces...
Is there any solution available for this problem?
Thanks in advance.

Strange.
Yahoo.jp looks very different from my machine. The main problem for you seems to be that the hiragana get very wide. I do not know what could cause it. I guess that it is a font issue, and I guess it may be fixed in future versions, but it is good that you posted the pictures. Logging a bug report to Apple may also be good.
Chinese Yahoo is something I have experienced as well actually. Most sites with traditional Chinese display mostly fine, but the simplified characters tend to be substituted with a bad font. I thought that was because I was running a Japanese system, but there may be some other reason.
Do you have a local Simplified Chinese version of Windows?

Changing Language encoding

Hi all. I have a number of tracks that are encoded in MacRoman, and I need to convert the encoding to ISOLatinGreek.
Anyone know a way to do this?
Many thanks
David

There was no way around this other than to rewrite
the names in greek.
Sorry I did not see the earlier post. Normally I would suggest the apps mentioned in this page:
http://homepage.mac.com/thgewecke/mlingos9.html#itunes
OS X requires Unicode rather than ISO Greek however.

Palm Desktop language encoding

I have installed PiLoc in hebrew on my Palm T|X.
Problem is, Palm Desktop doesn't seem to support whatever encoding PiLoc uses for its hebrew. I assume it's either UTF8 or 8859-8.
Is there a way to set the encoding in Palm Desktop?
Post relates to: Palm TX

It has been a while but I haven't needed hebrew support until now again. I checked with piloc support and they tell me that there should be no problem as they use windows 1255 encoding. How can I change the default encoding for palm desktop?
Post relates to: Palm TX

How can I get Telugu typing keyboard and language encoding support ?

I could not able to use unicode or any other codepage based typing access like it's in XP for my iMac Intel 17" with OSX 10.4.6. I need to access them for my internet designs. Any help from Users or Apple?

I could not able to use unicode or any other codepage
based typing access like it's in XP for my iMac Intel
17" with OSX 10.4.6. I need to access them for my
internet designs. Any help from Users or Apple?
Not from Apple yet. By far the best thing would be for you to get this kit:
http://xenotypetech.com/osxTelugu.html
You can add a Telugu front from someplace like this
http://www.travelphrases.info/gallery/Fonts_Telugu.html
but they will most likely not render correctly and then you would still have to construct an input keyboard, which might not be easy.

IE 9 incorrectly encoding Unicode characters in URIs to ISO-8859-1 instead of UTF8

Lets take the example word
präsentation
In Firefox, if I specify that as a CGI parameter, on the receiving end, I recieve:
pr\\303\\244sentation
which decoded as UTF-8 gives me: pr{U+00E4}sentation or my submitted word präsentation.
What does IE give me, well let's see.
pr\\344sentation
which well, doesn't decode as UTF8 because 0o344 is 0xE4.
ä in Unicode is at the codeopint 0xE4. Which as we've seen above, encoded to UTF8 is
0xC3 0xA4
So question boils down to this.
Why does IE9 use ISO-8859-1 instead of UTF8 for non-ASCII characters in URIs?

Hi,
As my understanding, you could choose the encoding ways by yourself:
Change your Internet Explorer 9 language
encoding settings
Alex Zhao
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Arabic Language Problem

hi all
I have a problem in arabic language encoding when store data into database
i wrote my problem here in this page
http://www.010555555.com/Arabic/index.html
so please contact me if you know the solution on
[email protected]
thanks alot all

does snow leopard have any problem with Arabic language cause i noticed strange behavior with word the letters are not linked together every letter alone
Right, MS Word for OS X has never yet supported Arabic (or any Indic language either).
in pages its fine but the cursor don't move between letters and when i type it stays at beginning of the line although i can type why is that happening ?
Because, while Pages does support Arabic (and also Indic), it has various bugs which make it unsuitable for RTL scripts for most people (unless copy/paste from elsewhere is enough). Same for Keynote and Numbers and iWeb.
Use instead Mellel, TextEdit, Nisus Writer, OpenOffice, etc.
http://homepage.mac.com/thgewecke/TypingArabic.html

Why Outlook email messages show unreadable characters

When you receive a new email in Outlook, you may see unreadable characters in the email message body. This issue could be caused by one of the following reasons on Outlook client side:
The email you received is converted to plain text format or a different encoding by a virus scanner.
This always happens when you are using characters (Such as apostrophes, double quotes, dashes, and ellipsis) supported by the "Smart Quotes" feature. When the email you received is converted to plain text format or a different encoding by a virus
scanner, the character is converted from the ASCII character to a Unicode character, which is not supported on the recipient’s side.
To prevent this issue from happening, we can turn off the “Smart Quotes” Feature in Outlook:
1. Open Outlook 2013.
2. Click File > Options > Mail.
3. Click Spelling and AutoCorrect button.
4. In the Editor Options window, click AutoCorrect Options button to open the AutoCorrect dialog.
5. Click AutoFormat tab, and then clear the check box before “Straight quotes” with “smart quotes” under Replace heading.
6. Click OK.
The issue may be related to font substitution when a change is made to the Web Options in Microsoft Word.
If a change is made to the Web Options in Microsoft Word, registry data under the following key may be configured incorrectly:
HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Word\Internet\Fonts
This problem has been seen to occur when the Latin string value under the \Fonts key was empty (no value).
We can check the Web Options setting in Word via:
1. Start Word.
2. Go to File > Options > Advanced.
3. Scroll down to the General section, click Web Options button.
4. Switch to Font tab.
To resolve the problem, we need to delete the \Fonts registry key. To do this, please follow:
1. Exit Word.
2. Press Windows key + R to open the Run command, type regedit and press Enter.
3. Navigate to:
HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Word\Internet\Fonts
4. Right-click the Font key and select Delete.
5. Click Yes and exit Registry Editor.
The sender used an e-mail program that did not mark messages with a language encoding (character set), or marked messages with an incorrect encoding setting.
In this case, we will see unreadable characters when we open the received message in our Outlook. For example, European regional characters (like German) require UTF-8 encoding protocol. So if the encoding in your environment is set to ASCII, the regional
characters will not be available.
To resolve this issue, we need to change the encoding in Outlook to which will display your message correctly.
1. Double-click to open the message you received.
2. Click Actions button in the Move group under Message tab.
3. Click Other Actions > Encoding.
4. Click More and select the correct encoding setting for the email.
5. After you finish reading the message, close the message, and click Yes button to save changes to the message.
When we send an email with Outlook, we can set Outlook to automatically select encoding for outgoing email messages to increase the possibility that the recipient receiving the message will see all the characters rendered properly, even if they run older
e-mail programs. By default, Outlook 2013 will automatically select an optimal encoding for outgoing e-mail messages. We can change the settings here in Outlook 2013: File > Options > Advanced > International Options section.
Something you may need to know when you set the encoding settings:
* The Unicode (UTF-8) character enables the complete range of characters on all computers. It’s the official IANA code for the UTF-8 character encoding.
* Western European (ISO) and Western European (Windows) character set can be read by almost all computers.
* If you are using a language other than English, you must choose the correct character set for that language, or the characters you need will not be displayed correctly.
If you have any comments on this article, please feel free to post here.
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

The "problem" is not really a problem. I send and receive emails fine. But the in the panel between the Inbox (message queue) and the reading pane this is displayed:
From Mozilla Support Forum <[email protected]> (white star)
Subject Toad-Hall posted an answer to your question "Why do received email messages show they are addressed to an address other than my own?"
Reply to [email protected] (white star)
To [email protected] yellow star)
cfortso@gmail is the address of one of my contacts.
This glitch is not really bothersome, I just can't imagine why it happens.

Display Polish letters in a Java program

Hi,
I'm internationalizing a small app. I have properties file for Polish language encoded as UTF-8. When I run the program it displays some non-sense characters instead of the Polish letters. In what format do I have to encode the properties file? Or what do I have to do otherwise?
Thanks,
PR.

Aardenon wrote:
DrClap wrote:
If this "properties file" is a file which is read by a java.util.Properties object, and it's not in XML format, then the requirements are laid out pretty clearly in the API documentation for Properties.
Presumably your Properties object isn't using the load(Reader) object, or you are providing it with a Reader whose charset isn't UTF-8.Oh no, it's a ResourceBundle file. It's umm... MessagesBundle_pl_PL.properties
I load it through the ResourceBundle.getBundle() method. I haven't found anything about encoding in the Locale and ResourceBundle docs.Sorry... originally I hadn't seen your response, so my original post was based on your original question. But then I saw your response, so I edited my post. And "simultaneously" you were responding to the original version of my post.
Anyway, relativity aside, hopefully my edited post is slightly more useful than the original version.

JasperReports Integration PDF is created with ISO-8859-1 instead of UTF-8

Hi,
This is a strange problem i'm struggling with. Since about one month the PDF's are created with the wrong character set. Instead of UTF-8 (this is also given as a parameter calling the procedure as you see below) the PDF are always created with ISO-8859-1.
declare
l_jasper_report_url VARCHAR2(100) DEFAULT 'http://localhost:8180/JasperReportsIntegration/report';
l_rep_name VARCHAR2(80) DEFAULT'SLRReports/SLR_Statistic_Service_Level_Report_Monat';
l_rep_format VARCHAR2(10) DEFAULT 'pdf';
l_data_source VARCHAR2(20) DEFAULT 'default';
l_rep_locale VARCHAR2(10) DEFAULT 'de_CH';
l_rep_encoding VARCHAR2(10) DEFAULT 'UTF-8';
l_StartDate DATE;
l_EndDate DATE;
l_MinDowntime INTEGER DEFAULT 1;
l_ServiceHour CHAR DEFAULT 'Y';
l_name VARCHAR2(50);
l_typ VARCHAR2(20) DEFAULT 'monatsreport_ccps';
l_additional_params VARCHAR2(200);
l_mime_type VARCHAR2(30);
l_blob BLOB;
BEGIN
l_StartDate := to_date('01.'||to_char( add_months(SYSDATE,-12),'MM.YYYY'),'DD.MM.YYYY');
l_EndDate := to_date('30.'||to_char( add_months(SYSDATE,-1),'MM.YYYY'),'DD.MM.YYYY');
l_name := 'slr-statistik-'||to_char(l_StartDate,'YYYY-MM')||'.pdf';
l_additional_params := 'UserName=SYSTEM'||chr(38)||'String_StartDate='||to_char(l_StartDate,'dd.mm.yyyy')||chr(38)||'String_EndDate='||to_char(l_EndDate,'dd.mm.yyyy')||chr(38)||'SLR_MinDowntime='||l_MinDowntime||chr(38)||'SLR_ServiceHour='||chr(38)||l_ServiceHour;
xlog ('PRC_GET_REPORT_TUNNEL', 'url (orig):' || l_rep_name||':'||l_additional_params);
-- generate the report and return in BLOB
xlib_jasperreports.set_report_url (l_jasper_report_url);
xlib_jasperreports.get_report (
p_rep_name => l_rep_name,
p_rep_format => l_rep_format,
p_data_source => l_data_source,
p_rep_locale => l_rep_locale,
p_rep_encoding => l_rep_encoding,
p_additional_params => l_additional_params,
p_out_blob => l_blob,
p_out_mime_type => l_mime_type );
dbms_output.put_line('p_out_mime_type: '||l_mime_type);
-- insert into report (id,name,typ,mime_type,lob_text,datum) values (p_report_seq.nextval,l_name,l_typ,l_mime_type,l_blob,sysdate);
-- commit;
-- apex_application.g_unrecoverable_error := TRUE;
EXCEPTION
WHEN OTHERS THEN
xlog ('PUT_BLB', SQLERRM, 'ERROR');
RAISE;
END;
It looks like the procedure UTL_HTTP.get_header ignores the encoding parameter and overwrites it with ISO-8859-1. I also tried to set it with UTL_HTTP.set_header before calling UTL_HTTP.get_header buth that didn't help either.
Then I started the JVM is started with the UTF-8 option and that made no change.
There were no new installation on Oracle, tomcat or JasperReports side. So I'm wondering what has caused this string behaviour.
Does anyone had this problem before or has a solution for it?
I appreciate your help very much.
Thanks and regards,
Chris

Hi,
As my understanding, you could choose the encoding ways by yourself:
Change your Internet Explorer 9 language
encoding settings
Alex Zhao
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Language Encoding

Similar Messages

Maybe you are looking for