How to create an UTF-8 file.

Question

Hello,
I want to create a file with an UTF-8 encoding, but I am experiencing a lot of problems.
The problem is not in the encoding of the content but in the encoding of the file. What is happening is that if I open the file with an editor (ex. ULTRA-Edit), it doesn�t recognize the file as an UTF-8 file and some strange characters are displayed.
However, if I tell the editor that the file is a UTF-8 file the content is displayed as desired.
The content that I want put in the file is a XML, and I did another test. I open the file in a web browser, and it displays the characters in a correct way. Considering this, I can conclude that the content is well encoded, and the encoding of the file is not defined (maybe I have to put something in the header of the file that indicates the encoding of the file)
For me, it works fine in the web browser, because the encoding is defined in the preamble of the XML file, which indicates that the following content is encoded in UTF-8.
Does anybody know how to define the encoding in the file in order to say to the editor in which encoding the information is coded?
Regards,
Rui Torres

Answer 1

If you write a text file (i.e. not a XML file) using the UTF-8 encoding and the editor of your choice does not correctly recognize the encoding, then it is not to blame.
The problem is that there is no "magical marker" that tells the editor that the file is in UTF-8 encoding. For the editor it's just a stream of bytes and it tries to make as much sense of it, as possible.
And it can guess correctly or it can guess incorrectly, but since the "correct" encoding is never stored it can never be 100% sure that it got it right.
Read [this excellent introduction into Unicode|http://www.joelonsoftware.com/articles/Unicode.html] to learn more about the "plain text" myth.
With XML it's a bit different. The correct encoding is written in the header (*) and there are heuristics that guarantee that the header can be read correctly if its written in one of the common encodings. So if you use a good XML library to produce your XML file, then you can depend that any software that uses an equally good XML parser can correctly identify the encoding used for it.
The solution to this problem is simple: always make sure that the encoding of any text file is explicitly known. Either use a format that automatically guarantees that (such as XML), or standardize it for your application ("my application reads and writes *only* UTF-8" (+)) or store the information about each texts encoding somewhere together with the text.
(*) of course the information in the header could be wrong, but then someone went to the trouble of actually writing the wrong value there. That's a plain old software bug, then.
(+) this is only a half-way solution since your text files might be used with some other tool than your software and then the same problem would occur again.