Send Search request to SearchEngine

Question

Hi,
I want to write an application in Java, which sends a search request to e.g. Yahoo and evaulates the result.
How can I send this request?
One solution:
URL url = new URL("http://de.search.yahoo.com/search?p=wunsch&fr=yfp-t-501&ei=UTF-8&meta=vl%3D");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream(), "UTF-8"));With these 2 lines I get the an result, but the html code is encoded in ISO-8859-1 and it contains special signs like the EURO sign. (�)
It is not possible to work with this string, because any XML or HTTP parser throws this Exception an stopps working:
An invalid XML character (Unicode: 0x0) was found in the element content of the document.
Changing the Encoding by using the URLEncoder / URLDecoder classes does not work.
How can I remove the special signs, or how can I change the signs coding to valid UTF-8?
Another solution for the search request could be:
URL url = new URL("http://de.search.yahoo.com:80/");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream(), "UTF-8");
wr.write("GET /search?p=wunsch&ei=ISO-8859-1&fr=yfp-t-501&x=wrt&meta=vl%3D HTTP/1.0\r\n");
wr.write("Host: de.search.yahoo.com\r\n");
wr.write("Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01\r\n");
wr.write("Accept-Encoding: gzip\r\n");
wr.write("Accept-Language: en\r\n");
wr.write("User-Agent: Lynx/2.8.6rel.4 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/1.6.2\r\n");
wr.write("\r\n");
wr.flush();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));But with this HTTP Request I only get the startpage of Yahoo and not the resultpage.
Please can someony help me... I don't know, how to fix the problem...
Thanks.

Answer 1

Actually, the page you reach is encoded in UTF-8, not ISO-8859-1.
Your problem is different: you can not parse HTML with an XML parser. Use an HTML parser for this.