Native2ascii problem

Question

Hello Group,
I have some problem with native2ascii ant task for
non-english languages. When I run native2ascii task through ant task it
generates some junk characters for portuguese characters e.g.
Input char: Efectuar liga��o
Output : Efectuar liga\ufffd\ufffdo
But if I run antive2ascii in commandline without through ant task
then it generates correct characters e.g.
Input char: Efectuar liga��o
Output : Efectuar liga\u00e7\u00e3o
which is correct.
Could you please help me to find out what is the problem when it runs
through ant task?
- BR

Answer 1

Hold on - do you have only languages that can actually be represented in 8859-1? If you do, then you don't even need to run native2ascii, since Java Properties files are expected to be in 8859-1: http://java.sun.com/javase/6/docs/api/java/util/PropertyResourceBundle.html
But if you support 12 languages I would think it highly likely that you actually do have languages that cannot be represented in 8859-1, such as Japanese, Chinese, Russian, etc. In that case you either need to use a different legacy encoding for each distinct script (and therefore run native2ascii with a different encoding option for each group of scripts) or use UTF-8 for all (allowing you to run native2ascii once, on all files. It shouldn't be difficult to choose the best option from those.
With respect to ensuring that the properties files are in the correct encoding: simply saving a file with one of the options you see in Notepad does not convert the file to the encoding in question, it simply saves the bytes as they are - Notepad assumes that you know what kind of text you have in your file. In other words, if you have a file encoded in 8859-1, open it in Notepad and then save it as UTF-8, the bytes in the file are still encoded as 8859-1, and treating it as UTF-8 will just result in errors.
Of the Notepad options 'ANSI' would be the option you would choose for a file encoded in 8859-1 if you are on a system where the default system codepage is 1252 - if you are on a system where the default system codepage is e.g. Russian, however, ANSI would result in the file being saved in the legacy Russian Windows codepage, however. 'ANSI' is a Windows invention, and simply means whatever non-Unicode codepage the Windows box happens to be running in.
To actually convert the file from one encoding to another you need to use a conversion tool such as iconv. In order to use those tools correctly you will still need to know what codepage the file is in, however. So you really need to keep track of the encoding of the files you are dealing with. The easiest way to do that is to always use UTF-8, and these days that is fairly easy.
There are tools that can help you, if you have somehow lost track of the encoding, or the file comes to you from unknown sources, etc. Encoding detection is part of ICU, for instance, and a good Unicode editor like SC Unipad will at least tell you whether the file is encoded in Unicode or not. But none of those methods are work 100%, so again, you are back to keeping track of the encoding yourself if you want to be sure.