Url char-encoding problem

Question

I am connecting to a web-server. The URL has Japanese characters embedded in it.
When run from NetBeans 6.8 everything works ok.
When run from the shell, the server appears to not understand the Japanese characters. For example, if the Japanese characters represent a user name, none of the names are ever understood.
Default charset when run from IDE is "UTF-8".
Default when run from shell: "Cp1252".
I don't want a systemic solution [for example a command-line switch, or config file setting]. I want to control every IO stream's encoding manually.
This is the test that comes closest to what I think should work:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
OutputStreamWriter baosStreamWrt = new OutputStreamWriter(baos, utf8.newEncoder());
BufferedWriter bufWrt = new BufferedWriter(baosStreamWrt);
bufWrt.write(url_with_jp_characters);
bufWrt.flush();
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
InputStreamReader inStreamRdr = new InputStreamReader(bais, utf8.newDecoder());
BufferedReader bufRdr = new BufferedReader(inStreamRdr);
String faulty_url = bufRdr.readLine();
URL webpage = new URL(faulty_url);
URLConnection urlConn = webpage.openConnection();
BufferedReader webIn = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), utf8.newDecoder()));
BufferedWriter response = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("/tmp/dump.txt"), utf8.newEncoder()));
response.("checking the the url: ")
response.write(faulty_url);
response.newLine();
while(true) {
  String webLine = webIn.readLine();
  if(webLine == null) { break; }
  response.write(webLine);
  response.newLine();
response.close();
webIn.close();
....Inspecting the response from the server in the file "/tmp/dump.txt":
(1) the file is formatted "UTF-8".
(2) the url, written in the first line of the file, is valid and a cut/paste into a browser works correctly.
(3) many correctly formed Japanese words are in the response from the server that is saying: "I have no idea how to understand/(decode?) the user name you sent me in the URL."
At this point I have a choice:
(1) I don't understand the source of the problem?
(2) I need to keep banging away at trying to get the url correctly encoded.
Finally, how can I debug this???
(1) It works in the IDE, so I don't have those debugging tools.
(2) My terminal cannot display asian characters.
(3) Writing output to files involves another encoder for the FileWriter which taints everything in the file.
(4) I don't have a webserver to act as a surrogate for the real one.
thanks.

Answer 1

Kayaman wrote:
rerf wrote:
And I don't know why NetBeans allowed me to get around not using this command.Because of the default charset of UTF-8 that's set in Netbeans.I see that, but something more subtle was happeneing I think. I still don't fully understand. Here is my best explanation. Exact code:
....for(File f : files) {
  BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(f), utf8.newDecoder()));
  String jpName = in.readLine().split(";")[0];
  String jpNameForUrlUsage = URLEncoder.encode(jpName, "UTF-8");
  String dumpFileName = (outputDir + f.getName().split(".txt")[0] + "-dump.txt");
  BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(dumpFileName), utf8.newEncoder()));
  URL url = new URL("http://www.abc.cp.jp?name=" + jpNameForUrlUsage);
  URLConnection urlConn = url.openConnection();
}....In my first post, I formed the complete url, then encoded it to utf-8 using ByteArray[Input / Output]Streams. But, my understanding is that you can't just encode Japanese characters to utf-8 and then append them to a url. A special method call is needed to transform the Japanese characters into something understandable by the URL (and its not as simple as just encoding the characters to utf-8). After posting, I actually made an effort to read all of javadoc for java.net.URL:
java.net.URL
The URL class does not itself encode or decode any URL components according to the escaping mechanism defined in RFC2396. It is the responsibility of the caller to encode any fields, which need to be escaped prior to calling URL, and also to decode any escaped fields, that are returned from URL. Furthermore, because URL has no knowledge of URL escaping, it does not recognise equivalence between the encoded or decoded form of the same URL. For example, the two URLs:
http://foo.com/hello world/ and http://foo.com/hello%20world
would be considered not equal to each other.
Note, the URI class does perform escaping of its component fields in certain circumstances. The recommended way to manage the encoding and decoding of URLs is to use URI, and to convert between these two classes using toURI() and URI.toURL().....
Then, it all made sense. I had already diagnosed that it was just the Japanese character part of the URL that caused the failure. And I remember reading that the Chinese were upset that they could not use asian characters in URLs. So, even though in my browser it looks like a Japanese character is in it, its really not. Chrome browser is doing some transform from what I see in my browser URL and what the actual URL is. That is why cutting/pasting the url I was forming in Java into the browser is not a good representation of what is going on. Chrome, behind the scenes does what the URLEncoder class does.
Yet, I could very easily be wrong:
On the one hand NetBeans worked, and the shell did not. The relevant difference being default charset.
But on the other hand, in my initial post, I completely encoded that url into utf-8 using ByteArray[Input / Output]Streams. I am no expert on character encodings, but if the url I created in the initial posting is not utf-8 encoded then I need a few pointers on how to character encode.
final thought:
I can't just encode a Japanese character to UTF-8, append it to a URL, and expect it to work. A browser can make it look that way so its deceiving. That is why there is the class URLEncoder. I don't understand why NetBeans appears to invoke it for me without my knowledge. And maybe its not. Having the default charset as utf-8 might obviate the need for a URLEncoder() call (but I don't see why it should).