Problem crawling filenames with national characters

Question

Hi
I have a big problem with filenames containing national (danish) characters.
The documents gets an entry in in wk$url but have error code 404 (Not found).
I'm running Oracle RDBMS 9.2.0.1 on Redhat Advanced Server 2.1. The
filesystem is mounted on the oracle server using NFS.
I configure the Ultrasearch to crawl the specific directory containing
several files, two of which contains national characters in their
filenames. (ls -l)
<..>
-rw-rw-r-- 1 user group 13 Oct 4 13:36 crawlertest_linux_2_fxeFXE.txt
-rw-rw-r-- 1 user group 19968 Oct 4 13:36 crawlertest_windows_fxeFXE.doc
<..>
(Since the preview function is not working in my Mozilla browser, I'm
unable to tell whether or not the national characters will display
properly in this post. But they represent lower and upper cases of the
three special danish characters.)
In the crawler log the following entries are added:
<..>
file://localhost/<DIR_PATH>/crawlertest_linux_2_B|C?C%C?C?.txt
file://localhost/<DIR_PATH>/crawlertest_linux_2_B|C?C%C?C?.txt
Processing file://localhost/<DIR_PATH>/crawlertest_linux_2_%e6%f8%e5%c6%d8%c5.txt
WKG-30008: file://localhost/<DIR_PATH>/crawlertest_linux_2_%e6%f8%e5%c6%d8%c5.txt: Not found
<..>
file://localhost/<DIR_PATH>/crawlertest_windows_B|C?C%C?C?.doc
file://localhost/<DIR_PATH>/crawlertest_windows_B|C?C%C?C?.doc
Processing file://localhost/<DIR_PATH>/crawlertest_windows_%e6%f8%e5%c6%d8%c5.doc
WKG-30008:
file://localhost/<DIR_PATH>/crawlertest_windows_%e6%f8%e5%c6%d8%c5.doc:
Not found
<..>
The 'file://' entries looks somewhat UTF encoded to me (some chars are
missing because they are not printable) and the others looks URL
encoded.
All other files in the directory seems to process just fine!.
In the wk$url table the following entries are added:
(select status url from wk$url where url like '%crawlertest%'; )
404 file://localhost/<DIR_PATH>/crawlertest_linux_2_%e6%f8%e5%c6%d8%c5.txt
404 file://localhost/<DIR_PATH>/crawlertest_windows_%e6%f8%e5%c6%d8%c5.doc
Just for testing purpose a
SELECT utl_url.unescape('%e6%f8%e5%c6%d8%c5') from dual;
Actually produce the expected resulat : fxeFXE
To me this indicates that the actual filesystem scanning part of the
crawler can sees the files, but the processing part of the crawler can
not open the file for reading and it therefor fails with error 404.
Since the crawler (to my knowledge is written in Java i did some
experiments, with the following Java program.
import java.io.*;
class filetest {
public static void main(String args[]) throws Exception {
try {
String dirname = "<DIR_PATH>";
File dir = new File(dirname);
File[] fs = dir.listFiles();
for(int idx = 0; idx < fs.length; idx++) {
if(fs[idx].canRead()) {
System.out.print("Can Read: ");
} else {
System.out.print("Can NOT Read: ");
System.out.println(fs[idx]);
} catch(Exception e) {
e.printStackTrace();
The performance of this program is very depending on the language
settings of the current shell (under Linux). If LC_ALL is set to "C"
(which is a common default) the program can only read files with
filenames NOT containing national characters (Just as the Ultrasearch
crawler). If LC_ALL is set to e.g. "en_US", then it is capable of
reading all the files.
I therefor tried to set the LC_ALL environment for the oracle user on
my oracle server (using locale_config, and .bash_profile) but that did
not seem to fix the problem at hand.
So (finally) my question is; is this a bug in the Ultrasearch crawler
or simply a mis configuration of my execution environment. If the
latter how do i configure my system correctly?
Yours sincerely
Martin Dahl Pedersen, Visanti ( mdp at visanti dot com )

Answer 1

I've posted my problems as a TAR on METALINK a little week ago.
And it turns out to be a new bug in UltraSearch.
It is now filed under BUG:2673282
-- mdp