Yet another Base64 encoder / decoder

Question

Hi, I would like introduce a different Base64 encoding scheme which yields slightly more compact results when encoding international characters.
        This is an implementation of BASE64UTF9 encoder and decoder.
        Common practice converts Java strings to UTF8 before Base64 encoding.
        UTF8 is not so space efficient in representing non-ASCII characters.
        While UTF9 is impractical as raw data in most 8 bit machines, it fits
        well within Base64 streams since 3 symbols can represent 2 UTF9 nonets.
        In this implementation, a modified UTF9 (flipped bit 9) is chosen to
        ensure nonet '0x000' never appear but maybe be inserted if needed.
        This implementation is also coded in such a way that it is possible to
        use Base100 encoding. When run from command line, it will perform a
        test with random strings and display encoding compactness compared to
        to other existing schemes.
public class B64utf9 {
        private static final String look =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
//"\u0010\u0011\u0012\u0013\u0014!\"#$%&'()*+,-./ 0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~";
        private static final int BASE = look.length();
        private static final int SYMB = (int)java.lang.Math.pow(BASE, 1.5) / 2;
        public static String decode(String str) {
                long j = 1, k = 0, q = 0;
                StringBuffer buf = new StringBuffer();
                for(int p = 0; p < str.length(); p++) {
                        k += look.indexOf(str.charAt(p)) * j;
                        for(j *= BASE; j >= SYMB * 2; j /= SYMB * 2) {
                                q = q * SYMB + k % SYMB;
                                if(k % (SYMB * 2) >= SYMB) {
                                        buf.append((char)q);
                                        q = 0;
                                k /= SYMB * 2;
                return buf.toString();
        public static String encode(String str) {
                long j = 1, k = 0;
                StringBuffer buf = new StringBuffer();
                for(int p = 0; p < str.length(); p++) {
                        long r, q = str.charAt(p);
                        for(r = 1; r * SYMB <= q; r *= SYMB);
                        for(; r > 0; r /= SYMB) {
                                k += ((q / r) % SYMB + (r > 1 ? 0 : SYMB)) * j;
                                for(j *= SYMB * 2; j >= BASE; j /= BASE) {
                                        buf.append(look.charAt((int)k % BASE));
                                        k /= BASE;
                if(j > 1) buf.append(look.charAt((int)k));
                return buf.toString();
        public static void main(String arg[]) throws Exception {
                java.util.Random rnd = new java.util.Random();
                for(double q = 1; q < 9; q++) {
                        int x = 0, y = 0, z = 0;
                        int r = (int)java.lang.Math.pow(Character.MAX_VALUE, 1 / q);
                        for(int p = 0; p < 999; p++) {
                                StringBuffer buf = new StringBuffer();
                                while(rnd.nextInt(99) > 0) {
                                        char k = 1;
// varying ASCII density
                                        for(int j = 0; j < q; j++)
                                                k *= rnd.nextInt(r);
                                        buf.append(k);
                                String str = buf.toString();
// regression
                                if(!decode(encode(str)).equals(str))
                                        System.out.println(str);
// count encoded length
                                x += encode(str).length();
                                y += new sun.misc.BASE64Encoder().encode(str.getBytes("utf8")).length();
                                z += new sun.misc.BASE64Encoder().encode(str.getBytes("gb18030")).length();
                        System.out.println(x +","+ y +","+ z);
}{code}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

Answer 1

Good idea! Sometimes it depends on what you want to encode too, however.
I suppose there is a case where general purpose compression does not work very well - random generated data.
Here's how I tested it:
class Count {
        private static String enc(String src, String set) throws Exception {
                return new String(src.getBytes(set), "iso-8859-1");
        private static byte[] dec(byte[] src, String set) throws Exception {
                return new String(src, set).getBytes("iso-8859-1");
        public static void main(String[] arg) throws Exception {
                java.util.Random rnd = new java.util.Random();
                int x = 0, y = 0;
                for(int k = 0; k < 99; k++) {
                        byte[] raw = new byte[rnd.nextInt(99)];
                        for(int z = 0; z < raw.length; z++)
                                raw[z] = (byte)(rnd.nextInt(256));
                        String str = new String(raw, "utf16");
                        byte[] b6u = str.getBytes("x-base64utf9");
                        byte[] gz2 = enc(enc(str, "utf8"), "x-gzip").getBytes("x-base64");
                        x += b6u.length;
                        y += gz2.length;
                        if(!str.equals(new String(b6u, "x-base64utf9")))
                                System.out.println(str);
                        if(!str.equals(new String(dec(dec(gz2, "x-base64"), "x-gzip"), "utf8")))
                                System.out.println(str);
                System.out.println(x +","+ y);
}The above code does not include my my encoders, I put my encoders code as nio.charset package and can be downloaded from [SF.|http://sourceforge.net/project/showfiles.php?group_id=248737]
PS: sorry about the confusion, please allow me to clarify that the UTF9 encoder is not interchangeable with existing Base64 encoders. There is a compatible (hopefully) encoder in my package, however.