Issue with Japanese characters in files/filenames in terminal.

Question

I recently downloaded a zip file with Japanese characters in the archive and in the files within the archive. The name of the archive is "【批量下载】パノプティコン労働歌 第一等.zip"
The characters are properly displayed in firefox, chrome, and other applications, but in my terminal some of the characters appear corrupted. Screenshot: https://i.imgur.com/4R22m0D.png
Additionally, this leads to corruption of the files in the archive. When I try to extract the files, this is what happens:
% unzip 【批量下载】パノプティコン労働歌 第一等.zip
Archive: 【批量下载】パノプティコン労働歌 第一等.zip
extracting: +ii/flac/Let's -+-ʦ1,000,000-.flac bad CRC 5f603d51 (should be debde980)
extracting: +ii/flac/+ѦѾP++ -instrumental-.flac bad CRC 78b93a2d (should be 3501d555)
extracting: +ii/flac/----.flac bad CRC ddeb1d3e (should be c05ae84f)
extracting: +ii/flac/+ѦѾP++.flac bad CRC 0ccf2725 (should be be2b58f1)
extracting: +ii/flac/Let's -+-ʦ1,000,000--instrumental-.flac bad CRC 67a39f8e (should be ece37917)
extracting: +ii/flac/.flac bad CRC f90f3aa0 (should be 41756c2c)
extracting: +ii/flac/ -instrumental-.flac bad CRC 3be03344 (should be 0b7a9cea)
extracting: +ii/flac/---- -instrumental-.flac bad CRC 569b6194 (should be adb5d5fe)
I'm not sure what could be the cause of this. I'm using uxterm with terminus as my main font and IPA gothic (a Japanese font) as my secondary font. I have a Japanese locale set up and have tried setting LANG=ja_JP.utf8 before, but the results never change.
Also, this issue isn't just with this file. This happens with nearly all archives that have Japanese characters associated with it.
Has anyone encountered this issue before or knows what might be wrong?
Last edited by Sanbanyo (2015-05-21 03:12:56)

Answer 1

Maybe 7zip or another tool has workarounds for broken file names, you could try that.
Or you could try to go over the files in the zip archive one-by-one and write it to files out-1, out-2, ..., out-$n without concerning yourself with the file names. You could get file endings back via the mimetype.
This script might work:
#include <stdio.h>
#include <zip.h>
static const char *template = "./out-%04d.bin";
int main(int argc, char**argv)
int err = 0;
zip_t *arc = zip_open((const char*)argv[1], ZIP_RDONLY, &err);
if(arc == NULL)
printf("Failed to open ZIP, error %d\n", err);
return -1;
zip_int64_t n = zip_get_num_entries(arc, 0);
printf("%s: # of packed files: %d\n", argv[1], n);
for(int i = 0; i < n; i++)
zip_stat_t stat;
zip_stat_index(arc, i, ZIP_FL_UNCHANGED, &stat);
char buf[stat.size];
char oname[sizeof(template)];
zip_file_t *f = zip_fopen_index(arc, (zip_int64_t)i, ZIP_FL_UNCHANGED);
zip_fread(f, (void*)&buf[0], stat.size);
snprintf(&oname[0], sizeof(template), template, i);
FILE *of = fopen(oname, "wb");
fwrite(&buf[0], stat.size, 1, of);
printf("%s: %s => %lu bytes\n", argv[1], oname, stat.size);
zip_fclose(f);
fclose(of);
zip_close(arc);
return 0;
Compile with
gcc -std=gnu99 -O3 -o unzip unzip.c -lzip
and run as
./unzip $funnyzipfile
You should get template-named, numbered output files in the current directory.
Last edited by 2ion (2015-05-21 23:09:29)