Missing locales: UTF8 for X11

Question

Hello everyone
I have been trying to get Vietnamese input under X11 working for over two years now. My system keyboard layouts only partly work: I can use the normal keys, but the dead keys only create a space, not the added accent required. Since my language is largely composed of vowels with added accents, this disabled Vietnamese input completely.
I think one of my problems might be a lack of locale. /usr/X11R6/lib/X11/locale has two Vietnamese locales, neither of them UTF8. I've tried setting the legacy TCVN one in xterm_script and xterm_script.sh, but that doesn't help. It's a UTF8 locale I need, anyway. en_US.UTF-8 doesn't work for Vietnamese.
I don't know where to look for additional locales. Any advice you can offer would be very welcome. I've really been held back in testing our free-software translations, not to mention using the software itself, by this inability to input my language in X11.

Answer 1

Thankyou both for your replies. I'll answer them in order:
First I want to clarify the situation in native
MacOSX apps (e.g. Safari).
If I select US key layout from the Input Menu, then
option-` (and four more similar combinations)
act as dead-keys. In Safari (TextEdit, etc.), I get è
(e grave) by hitting option-` and then
e.
Yes. The same thing happens with the Australian English keyboard.
When I select Vietnamese key layout, then the
following twelve keys act as dead keys:
They're not dead keys. They simply input different characters from the English layout.
a e i o u y [ ] 1 2 3 4 (character printed
on the key-tops)
If I hit one of them, they produce the following
characters (Vietnamese vowels?)
a e i o u y ư ơ ă â ê ô (character they
actually produce)
Correct.
and await the next key. If I hit one of the
number keys 5-9 here, then I can add an accent to
those vowels. For example, in order to input è
(e grave), I must hit e and then 5, NOT
5 and then e. On your Mac, if you want
to input è into Safari, do you hit 5e or
e5?
I'm using a different layout. The default Vietnamese layout for OSX originally was a decomposed Unicode layout, which caused me a lot of problems. So I installed additional precomposed layouts. My current layout is TCVN: the only difference is that for that layout, you hit the dead key before the vowel, not after. However, I have also tried the default layout with X11, with the same results. I don't know if the default layout has been changed to precomposed Unicode yet, but I'm used to the TCVN one now, so I stick with that.
If you open Keyboard Viewer from the Input
Menu (if it is not in your Input Menu, then you can
add it via International System Preferences), the
dead keys will be indicated by orange color. It will
be interesting to keep Keyboard Viewer open when you
type.
Now let's see the situation in X11.
(I assume you have enabled "Use the system keyboard
layout" in the Preferences of X11)
Yes.
In my Mac, if US key layout is selected and the
locale is set to en_US.UTF-8, I can enter è
just as in Safari (by hitting option-` then
e), at least in xterm (It seems that luit must
be turned off by 'xterm +lc' (see xterm man page for
the option +lc)). xev tells that the keysym sent by
option-` is 'dead_grave'.
If Vietnamese layout is selected, then no dead keys
are available. This is not a surprise, because no
keySyms like “dead_e” are defined in X11 and so the
key e can't act as a dead key. If I hit
e then 5, I just get e` (e and a stand
alone `).
Ah. It's a lack of definition in X11. That's really useful to know.
In your post, the keysym sent by the number key 5 is
just a space:
state 0x0, keycode 31 (keysym 0x20, space),
same_screen YES,
In my mac, it is a simple grave:
state 0x0, keycode 31 (keysym 0x60, grave),
same_screen YES,
I don't know where this difference comes from. I'm
using a third party keyborad but it is recognized as
a US keyboard by the OS.
Yes, interesting. X11 may read the order of input slightly differently with different layouts.
Anyway, neither my Mac nor your Mac can't input è
into X11.
Yes.
I'm not sure whether there is a simple option (or
trick) which enables dead keys in X11 with Vietnamese
key layout. But as the last resort you can modify
keymap by yourself. For example, please create a file
'mykeymap' (or whatever name you like) and put the
following line to it
keycode 31 = dead_grave percent 5 percent
This will mean modifying it for all four dead keys. I will try that.
and run xmodmap as
$ xmodmap mykeymap
Then you can use the number key 5 as a dead key; you
may input è by hitting 5 and then e.
If you manually modify the keymap, then you should
disable "Use the system keyboard layout" in the
Preferences of X11; otherwise the keymap will be
overwritten by the system when you switch the key
layout.
OK.
I guess you can add four more lines to mykeymap so
that number keys 5-9 act as dead keys. You may find
the necessary keysym names in
/usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose. The
current keymap can be listed by
$ xmodmap -pke
Thankyou. I will try all that, then report back. Certainly it seems that the international features of X11 need work. Vietnamese is one of the languages/input methods which comes with OSX, so it's not exactly uncommon or esoteric to Apple. However, they don't enable it in Appleglot, either, and I've had an open bug on that for years now.
Jun then said:
It seems that luit must be turned off by 'xterm +lc'
Maybe I was wrong. You don't need +lc.
Tom added:
I notice that copy/paste from TextEdit works, so it's not a font problem. I >suppose that a custom keyboard that created all the required characters via >a single keypress would work ok, but that is not very practical, right?
No, since there are 72 vowels in total, and we use all the rest of the Roman alphabet except F, W and Z. It would have to be a very large keyboard! The dead-key system makes it all manageable on a standard keyboard.
Are you sure that your X11 has access to a font which contains all the >precomposed Vietnamese characters? I don't think you can rely on the OS X >fonts for this, but there may have to be a font in /usr/X11R6/lib/X11/fonts >or someplace similar.
After you first commented on this, I put all the Dejavu and URWVN fonts into that location. Dejavu is the best of the current free-software pan-Unicode fonts in displaying Vietnamese, due to keen attention by their developers to my feedback. URWVN is a package of GPL fonts designed for Vietnamese. After putting them in that location, I restarted X11, even restarted the Mac, but no change in my problem. I think the fonts are accessible, but X11 can't recognize the keypresses calling those characters.
You have said more than once (or the msg keeps being tacked on to the end of the thread):
Perhaps this fellow would know how to port this program to Darwin?
http://www.unikey.org/linux.php
As I said, I don't know anyone else in free-software who uses a Mac for Vietnamese. I'll keep an eye out, though.
Thankyou both for your help.
White MacBook 1.83 GHz, 2 GB RAM, 120 GB HDD   Mac OS X (10.4.8)