Decoding HTML Characters

Question

A while ago I wrote in java an application which would sit on my college server and every 15mins parse an rss feed and create a duplicate of that feed but with full body content .The reason I have done this is so that I can view the feed offline.
All works well except I have one issue and Im sure it lies in decoding the stream, for the most part all the characters come out fine but for certain special characters they come out in junk
So again, I'll
listen to some Utada Hikaru to begin with and instantly I know something'sGets decoded to
So again, Iâ??ll listen to some Utada Hikaru to begin with and instantly I know somethingâ??sSo obviously the character ' is throwing out the decoding (actually as I see it the character is not ' as above but a curly version of it, this format appers to be cleaning it up).
So I would like to know how to decode the stream correctly I tried using InputStreamReader and setting the CharsetDecoder but to no avail though of course I may not have done this correctly.
Thanks Ger.
Edited by: Ger@newToProgramming on Nov 3, 2009 4:04 AM

Answer 1

I tired again to set the CharSet and no luck.
To explain whats is happening to see if the problem lies anywhere else.
A screen scrape is preformed the scrapped page is run through JTidy the main article located and storged in an sql database using Apache it is then later retrieved and written to a rss feed enclosed in CData blocks.
[Example Output|http://www.redbrick.dcu.ie/~gleesog4/rss2/feeds/Breaking/Bloomberg/index.rss]
Code for reading the page:
     public static String GetPage(String url) throws Exception
          BufferedReader buffer = new BufferedReader( new InputStreamReader(GetURLStream(url),"UTF-8") );
          StringBuilder builder = new StringBuilder();
          int byteRead;
          while ((byteRead = buffer.read()) != -1)
                        builder.append((char) byteRead);
          buffer.close();
          return builder.toString();
    public static InputStream GetURLStream(String url) throws Exception
         InputStream strIn = null;
         boolean isFile = false;
         boolean isURL = false;
          if(!isURL)
                   try
                        strIn = new FileInputStream(url);
                        isFile = true;
                   }catch(Exception e){}
          if(!isFile)
               try
                    strIn = new URL(url).openStream();
                    isURL = true;
               }catch(Exception e)     {}
          if(!isFile && !isURL)
               throw new FileNotFoundException("Can not locate as either file or url\n" + url);
          return strIn;
    }