Should I go utf8 all the way?

Question

I think I need some advise on the topic of encodings. My system right now is a mess: One old Windows partition (NTFS), one shared FAT32 partition, both with files from a long time ago in i don't know which encoding, filenames with and without non-ascii characters, etc.
Then there are the linux partitions with mostly new files created under my current locale, en_GB, although i tried out en_GB.utf8 for a while.
My files are partly English, partly Norwegian/Swedish, so I need to accomodate the special characters of those languages, but I have no need neither for Arabic nor Korean/Japanese etc.
When I would still like to switch to a fully unicoded system, it's partly to  avoid the mess, but mostly because some  of the programs I need and use on a daily basis, require unicode files (lilypond, etc.). I've had some programs complain about incorrect file name encoding (gwenview, I think, or perhaps gqview), and my file names with non-ascii characters show up in several different ways, and, it seems, differently in X-apps, in consoles, in terminals, in vim, etc.
So, there you have my situation. Does anyone have experience - good or bad - with this kind of situation? Ideally, I suppose, I'd like all my files to be properly unicoded, both regarding file names and contents. But I assume this involves a potentially "dangerous" situation with a batch job with iconv?
What are the pitfalls I need to keep in mind before I do this? Are there other solutions I might consider? What about apps which don't support unicode - would I be better off staying in latin1 - perhaps converting everything to that instead of unicode? As I said, in terms of characters, I really don't need more than what latin1 can offer, but some apps require unicode for the file encoding.
Any help, feedback, pointers to good sites, etc. are appreciated.

Answer 1

apps not supporting unicode tend to be rare these days, as most apps are built on qt, gtk, python, java, mono, readline, etc... and thus support utf8 OOTB.
IIRC (some part of) zsh does not support unicode.
you should also make clear distinction on multiple things:
1. display/input uft8
2. utf8 filenames in filesystem(s)
3. utf8 file contents
gentoo has good reference:
http://gentoo-wiki.com/HOWTO_Create_an_ … led_system
http://gentoo-wiki.com/HOWTO_Make_your_ … code/utf-8
as for my advice, I'd say go for utf8 all the way. when you're on the rails, it's just transparent, and if you happen to have some file/contents having some 'weird' character in it, you'll always be okay ('weird' being relative, you don't need to go as far as Arabic: that could simply be a simple accented character not present in your ISO-* set). why be restricted when you can just as well be open? you never know what might be your need tomorrow: e.g I'm french (accented chars), I got on to learn japanese (obvious chars), and I may be in relation occasionally with at least spanish (tilda chars), and german (umlauts and ss) someday in my work. I woudn't want to have to handle 5 charsets, or to convert to utf8 at that time, so I took things in hand right now, ready and set.
personnally the first thing I do when I install arch is locale-gen en_US.UTF-8 and set locale to it.
as for windows compatibility, NTFS filenames are stored as utf8 IIRC, and fat32 can have utf8 enabled, though IIRC it makes the fs case sensitive. regarding file contents, most not too old windows software usually handle utf8 correctly (at least never had a problem, when I still had windows). issues arise when files are stored in non-iso windows encoding (cp1252 and the like), which I was never able to convert successfully, yet it can just be my fault for not trying hard enough (because the few files I had being non-utf8 were not that important).
EDIT: some additions
Last edited by lloeki (2007-04-23 12:30:28)