Problems reading unicode characters with BufferedReader or PrintWriter

Question

I am creating a pretty standard chat server and client with the exception that I want to be able to send special math symbols back and forth to aid in helping people with math homework. The unicode characters will appear on the buttons, in the textField (where users type in their messages), in the messageArea (where users see all of the messages) correctly. However, as soon as the server reads it, it is converted to a ? character by either the BufferedReader or the PrintWriter, but I am fairly certain it is the BufferedReader. All of the Sun examples I have read recommend doing exactly what I have done to be able to support unicode characters so I am somewhat at a loss. Any help would be appreciated. I will include all of the source code (which is rather large) but the main problem is the with the BufferedReader or PrintWriter when the servers reads the string in and then broadcasts it to all the clients (at least I'm pretty sure it is).
// ChatServer.java
import java.net.Socket;
import java.net.ServerSocket;
import java.io.InputStreamReader;
import java.io.BufferedReader;
import java.io.PrintWriter;
import java.io.IOException;
import java.util.Set;
import java.util.HashSet;
import java.util.HashMap;
import java.util.Iterator;
* A multithreaded chat room server. When a client connects the
* server requests a screen name by sending the client the
* text "SUBMITNAME", and keeps requesting a name until
* a unique one is received. After a client submits a unique
* name, the server acknowledges with "NAMEACCEPTED". Then
* all messages from that client will be broadcast to all other
* clients that have submitted a unique screen name. The
* broadcast messages are prefixed with "MESSAGE ".
* Because this is just a teaching example to illustrate a simple
* chat server, there are a few features that have been left out.
* Two are very useful and belong in production code:
* 1. The protocol should be enhanced so that the client can
* send clean disconnect messages to the server.
* 2. The server should do some logging.
public class ChatServer {
* The port that the server listens on.
private static final int PORT = 9001;
* The set of all names of clients in the chat room. Maintained
* so that we can check that new clients are not registering name
* already in use.
private static HashSet names = new HashSet();
* The set of all the print writers for all the clients. This
* set is kept so we can easily broadcast messages.
private static HashMap writers = new HashMap();
private static HashMap userSockets = new HashMap();
private static Set keys;
* The appplication main method, which just listens on a port and
* spawns handler threads.
public static void main(String[] args) throws Exception {
System.out.println("The chat server is running.");
ServerSocket listener = new ServerSocket(PORT);
try {
while (true) {
new Handler(listener.accept()).start();
     } catch (Exception e) {
} finally {
listener.close();
* A handler thread class. Handlers are spawned from the listening
* loop and are responsible for a dealing with a single client
* and broadcasting its messages.
private static class Handler extends Thread {
private String name;
private Socket socket;
private BufferedReader in;
private PrintWriter out;
* Constructs a handler thread, squirreling away the socket.
* All the interesting work is done in the run method.
public Handler(Socket socket) {
this.socket = socket;
public void disconnect(String userName) {
if (userName != null) {
names.remove(userName);
if (writers.get(userName) != null) {
writers.remove(userName);
try {
((Socket)userSockets.get(userName)).close();
} catch (IOException e) {
e.printStackTrace();
System.out.println(userName +" has disconnected from the Chat Server.");
keys = writers.keySet();
for (Iterator i = keys.iterator(); i.hasNext(); ) {
((PrintWriter)writers.get(i.next())).println(
"USERQUIT" +name);
public void disconnectAll() {
for (Iterator i = names.iterator(); i.hasNext();) {
String userName = i.next().toString();
if (!userName.equals("administrator")) {
names.remove(userName);
keys = writers.keySet();
for (Iterator it = keys.iterator(); it.hasNext(); ) {
((PrintWriter)writers.get(it.next())).println(
"USERQUIT" +name);
writers.remove(userName);
userSockets.remove(userName);
* Services this thread's client by repeatedly requesting a
* screen name until a unique one has been submitted, then
* acknowledges the name and registers the output stream for
* the client in a global set, then repeatedly gets inputs and
* broadcasts them.
public void run() {
try {
// Create character streams for the socket.
in = new BufferedReader(new InputStreamReader(
socket.getInputStream()));
out = new PrintWriter(socket.getOutputStream(), true);
// Request a name from this client. Keep requesting until
// a name is submitted that is not already used. Note that
// checking for the existence of a name and adding the name
// must be done while locking the set of names.
while (true) {
out.println("SUBMITNAME");
name = in.readLine();
if (name == null) {
return;
synchronized (names) {
if (!names.contains(name) && name.equalsIgnoreCase("administrator")) {
out.println("SUBMITPASSWORD");
String password = in.readLine();
if (password.equals("s3cure")) {
out.println("GRANTEDADMIN");
if (!names.contains(name)) {
names.add(name);
break;
// Now that a successful name has been chosen, add the
// socket's print writer to the set of all writers so
// this client can receive broadcast messages.
out.println("NAMEACCEPTED");
keys = writers.keySet();
for (Iterator i = keys.iterator(); i.hasNext(); ) {
((PrintWriter)writers.get(i.next())).println(
"USERJOIN" +name);
writers.put(name, out);
userSockets.put(name, socket);
System.out.println(name +" has successfully connected to the Chat Server.");
// Accept messages from this client and broadcast them.
// Ignore other clients that cannot be broadcasted to.
while (true) {
String input = in.readLine();
if (input == null) {
return;
          else if (input.startsWith("LISTUSERS")) {
               Iterator i = names.iterator();
               String namesString = new String();
               while (i.hasNext()) {
               namesString += " " +i.next();
               out.println("USERLIST " +namesString);
     catch (Exception e) {
     finally {
          disconnect(name);
// ChatClient.java
import java.util.HashSet;
import java.util.StringTokenizer;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.ListIterator;
import java.awt.event.WindowEvent;
import java.awt.event.WindowAdapter;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.MouseListener;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import java.awt.event.KeyListener;
import java.awt.event.KeyEvent;
import java.awt.Point;
import java.awt.Button;
import java.awt.Font;
import java.awt.BorderLayout;
import java.awt.FlowLayout;
import java.awt.Dimension;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JLabel;
import javax.swing.JTextField;
import javax.swing.JTextArea;
//import javax.swing.JTextPane;
import javax.swing.JScrollPane;
import javax.swing.JOptionPane;
import javax.swing.JList;
import javax.swing.DefaultListModel;
import javax.swing.JMenuBar;
import javax.swing.JMenu;
import javax.swing.JMenuItem;
import javax.swing.JPopupMenu;
import java.net.Socket;
import java.io.InputStreamReader;
import java.io.BufferedReader;
import java.io.PrintWriter;
import java.io.IOException;
* A simple Swing-based client for the chat server. Graphically
* it is a frame with a text field for entering messages and a
* textarea to see the whole dialog.
* The client follows the Chat Protocol which is as follows.
* When the server sends "SUBMITNAME" the client replies with the
* desired screen name. The server will keep sending "SUBMITNAME"
* requests as long as the client submits screen names that are
* already in use. When the server sends a line beginning
* with "NAMEACCEPTED" the client is now allowed to start
* sending the server arbitrary strings to be broadcast to all
* chatters connected to the server. When the server sends a
* line beginning with "MESSAGE " then all characters following
* this string should be displayed in its message area.
public class ChatClient {
BufferedReader in;
PrintWriter out;
JFrame frame = new JFrame("Math Chatter version 0.3");
JTextField textField = new JTextField(40);
JTextArea messageArea = new JTextArea(25, 40);
//JTextPane messageArea = new JTextPane();
// Menu
JMenuBar menuBar = new JMenuBar();
// File Menu and MenuItems
JMenu fileMenu = new JMenu();
JMenuItem connectMenuItem = new JMenuItem();
JMenuItem disconnectMenuItem = new JMenuItem();
JMenuItem exitMenuItem = new JMenuItem();
// Help Menu and MenuItems
JMenu helpMenu = new JMenu();
JMenuItem aboutMenuItem = new JMenuItem();
JOptionPane aboutDialog = new JOptionPane();
// Message Area Popup Menu and MenuItems
JPopupMenu messageAreaPopupMenu = new JPopupMenu();
JMenuItem connectPopupMenuItem = new JMenuItem();
JMenuItem disconnectPopupMenuItem = new JMenuItem();
JMenuItem clearTextMenuItem = new JMenuItem();
JMenuItem exitPopupMenuItem = new JMenuItem();
MouseListener messageAreaPopupListener = new messageAreaPopupListener();
// User Area Popup Menu and MenuItems
JPopupMenu userAreaPopupMenu = new JPopupMenu();
JMenuItem pingPopupMenuItem = new JMenuItem();
JMenuItem disconnectUserPopupMenuItem = new JMenuItem();
JMenuItem disconnectAllUsersPopupMenuItem = new JMenuItem();
MouseListener userAreaPopupListener = new userAreaPopupListener();
// Panel with textfield, send, and clear buttons
JPanel sendTextPanel = new JPanel();
// Panel that has all the math buttons
JPanel buttonPanel = new JPanel();
// Panel that has messageArea and users panell
JPanel displayPanel = new JPanel();
DefaultListModel listModel = new DefaultListModel();
JList userNamesList = new JList(listModel);
//JTextArea userNamesTextArea = new JTextArea(25, 10);
// buttonPanel stuff
Button sendButton = new Button();
Button clearButton = new Button();
Button infinityButton = new Button();
Button thetaButton = new Button();
Button limitButton = new Button();
Button thereforeButton = new Button();
JLabel buttonLabel = new JLabel("Useful math stuff: ");
HashSet usersHashSet = new HashSet();
LinkedList sentMessageList = new LinkedList();
ListIterator sentMessageIterator;
boolean isAdmin = false;
* Constructs the client by laying out the GUI and registering a
* listener with the textfield so that pressing Return in the
* listener sends the textfield contents to the server. Note
* however that the textfield is initially NOT editable, and
* only becomes editable AFTER the client receives the NAMEACCEPTED
* message from the server.
public ChatClient() {
// -----------------> Layout MenuBar
frame.setJMenuBar(menuBar);
fileMenu.setText("File");
connectMenuItem.setText("Connect");
connectMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
showWhoopsDialog();
/*out.println("CONNECT");
try {
run();
catch (Exception ex) {
ex.printStackTrace();
fileMenu.add(connectMenuItem);
disconnectMenuItem.setText("Disconnect");
disconnectMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
chatDisconnect();
fileMenu.add(disconnectMenuItem);
exitMenuItem.setText("Exit");
exitMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
frame.dispose();
System.exit(0);
return;
fileMenu.add(exitMenuItem);
menuBar.add(fileMenu);
helpMenu.setText("Help");
aboutMenuItem.setText("About");
aboutMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
aboutDialog.setDoubleBuffered(true);
aboutDialog.showMessageDialog(frame,
"Chat Server and Client were originally written by Ray Toal and were found at \n http://www.technocage.com/~ray/viewsource.jsp?java/networking/ChatServer.java and \n http://www.technocage.com/~ray/viewsource.jsp?java/networking/ChatClient.java. \n\n Math Chatter and all revisions have been done by William Ready.",
"About Math Chatter Server and Client version 0.3",
JOptionPane.INFORMATION_MESSAGE);
helpMenu.add(aboutMenuItem);
menuBar.add(helpMenu);
// -----------> End Menu
// -----------> Popup Menus
// -----------> Message Area Popup Menu
connectPopupMenuItem.setText("Connect");
connectPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
showWhoopsDialog();
disconnectPopupMenuItem.setText("Disconnect");
disconnectPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
chatDisconnect();
clearTextMenuItem.setText("Clear text");
clearTextMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
messageArea.setText("");
exitPopupMenuItem.setText("Exit");
exitPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
frame.dispose();
System.exit(0);
messageAreaPopupMenu.add(connectPopupMenuItem);
messageAreaPopupMenu.add(disconnectPopupMenuItem);
messageAreaPopupMenu.add(clearTextMenuItem);
messageAreaPopupMenu.add(exitPopupMenuItem);
messageArea.addMouseListener(messageAreaPopupListener);
// -----------> End Message Area Popup Menu
// -----------> User Area Popup Menu
pingPopupMenuItem.setText("Ping User");
pingPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
//System.out.println("Yo yo yo");
disconnectUserPopupMenuItem.setText("Disconnect User");
disconnectUserPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
if (userNamesList.getSelectedValue() != null) {
//System.out.println("Yo yo yo");
out.println("DISCONNECTUSER " +userNamesList.getSelectedValue().toString());
//disconnectUserPopupMenuItem.setEnabled(false);
disconnectUserPopupMenuItem.setVisible(false);
disconnectAllUsersPopupMenuItem.setText("Disconnect All Users");
disconnectUserPopupMenuItem.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
if (userNamesList.getSelectedValue() != null) {
//System.out.println("Yo yo yo");
out.println("DISCONNECTALLUSERS");
//disconnectUserPopupMenuItem.setEnabled(false);
disconnectAllUsersPopupMenuItem.setVisible(false);
userAreaPopupMenu.add(pingPopupMenuItem);
userAreaPopupMenu.add(disconnectUserPopupMenuItem);
userAreaPopupMenu.add(disconnectAllUsersPopupMenuItem);
userNamesList.addMouseListener(userAreaPopupListener);
// -----------> End User Area Popup Menu
// -----------> End Popup Menus
// -----------> Layout GUI
textField.setEditable(false);
textField.requestDefaultFocus();
messageArea.setEditable(false);
sendTextPanel.setLayout(new FlowLayout());
sendTextPanel.add(textField);
sendButton.setLabel("Send");
sendButton.setEnabled(false);
sendTextPanel.add(sendButton);
clearButton.setLabel("Clear");
sendTextPanel.add(clearButton);
buttonPanel.setLayout(new FlowLayout());
buttonPanel.add(buttonLabel);
infinityButton.setLabel("\u221e");
buttonPanel.add(infinityButton);
     thetaButton.setLabel("\u03b8");
buttonPanel.add(thetaButton);
limitButton.setLabel("lim");
buttonPanel.add(limitButton);
thereforeButton.setLabel("\u2234");
buttonPanel.add(thereforeButton);
userNamesList.setVisibleRowCount(26);
userNamesList.setFixedCellWidth(110);
JScrollPane nameTextScrollPane = new JScrollPane(userNamesList);
JScrollPane messageAreaScrollPane = new JScrollPane(messageArea);
displayPanel.setLayout(new FlowLayout());
displayPanel.add(messageAreaScrollPane);
displayPanel.add(nameTextScrollPane);
frame.getContentPane().add(sendTextPanel, "North");
frame.getContentPane().add(displayPanel, "Center");
frame.getContentPane().add(buttonPanel, "South");
frame.pack();
// ---------------> Add Listeners
textField.addKeyListener(new KeyListener() {
public void keyPressed(KeyEvent e){ }
public void keyTyped(KeyEvent e){
if (e.getKeyChar() == KeyEvent.VK_UP) {
if (sentMessageIterator.hasNext()) {
textField.setText((String)sentMessageIterator.next());
if (e.getKeyChar() == KeyEvent.VK_DOWN) {
if (sentMessageIterator.hasPrevious()) {
textField.setText((String)sentMessageIterator.previous());
if (e.getKeyChar() == KeyEvent.VK_ENTER) {
String text = textField.getText();
if (text != null && !(text.equals(""))) {
sentMessageList.addFirst(textField.getText());
sentMessageIterator = sentMessageList.listIterator();
out.println(textField.getText());
textField.setText("");
public void keyReleased(KeyEvent e){}
sendButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
out.println(textField.getText());
textField.setText("");
textField.requestFocus();
clearButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
textField.setText("");
textField.requestFocus();
infinityButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
textField.setText(textField.getText() +"\u221e");
textField.requestFocus();
thetaButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
textField.setText(textField.getText() +"\u03b8");
textField.requestFocus();
limitButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
String limit = JOptionPane.showInputDialog(frame,
"Enter the value that the variable approaches. \n If the value is infinity, type in inf. \n (Note: the variable name is irrelevant)",
"Value that variable approaches",
JOptionPane.QUESTION_MESSAGE);
if (limit != null && limit.equalsIgnoreCase("inf"))
textField.setText(textField.getText() +"lim [x->\u221e] ");
else if (limit != null)
textField.setText(textField.getText() +"lim [x->" +limit +"] ");
textField.requestFocus();
thereforeButton.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
textField.setText(textField.getText() +"\u2234");
textField.requestFocus();
// -------------> End Listeners
// -------------> End GUI
* Prompt for and return the address of the server.
private String getServerAddress() {
return JOptionPane.showInputDialog(
frame,
"Enter IP Address of the Server:",
"Welcome to the Chatter",
JOptionPane.QUESTION_MESSAGE);
private void showWhoopsDialog() {
JOptionPane.showMessageDialog(frame,
"Unfortunately this feature has not been implemented yet. \n The only way to connect currently is to exit the program \n and start it again",
"Whoops!",
JOptionPane.ERROR_MESSAGE);
private void chatDisconnect() {
messageArea.append("You have disconnected from the Chat Server\n");
textField.setEditable(false);
messageArea.setEnabled(false);
userNamesList.setEnabled(false);
fileMenu.requestFocus();
out.println("DISCONNECT");
* Prompt for and return the desired screen name.
private String getName() {
return JOptionPane.showInputDialog(
frame,
"Choose a screen name:",
"Screen name selection",
JOptionPane.PLAIN_MESSAGE);
private String getPassword() {
return JOptionPane.showInputDialog(
frame,
"Type in your password:",
"Password Input",
JOptionPane.PLAIN_MESSAGE);
* Connects to the server then enters the processing loop.
private void run() throws IOException {
// Make connection and initialize streams
String serverAddress = getServerAddress();
if (serverAddress != null && !(serverAddress.equals(""))) {
Socket socket = new Socket(serverAddress, 9001);
in = new BufferedReader(new InputStreamReader(
socket.getInputStream()));
out = new PrintWriter(socket.getOutputStream(), true);
// Process all messages from server, according to the protocol.
while (true) {
try {
String line = in.readLine();
if (line.startsWith("SUBMITNAME")) {
out.println(getName());
          } else if (line.startsWith("NAMEACCEPTED")) {
textField.setEditable(true);
          out.println("LISTUSERS");
sendButton.setEnabled(true);
//messageArea.setEnabled(true);
} else if (line.startsWith("SUBMITPASSWORD")) {
out.println(getPassword());
} else if (line.startsWith("GRANTEDADMIN")) {
isAdmin = true;
//disconnectUserPopupMenuItem.setEnabled(true);
disconnectUserPopupMenuItem.setVisible(true);
disconnectAllUsersPopupMenuItem.setVisible(true);
} else if (line.startsWith("DISCONNECT")) {
sendButton.setEnabled(false);
return;
} else if (line.startsWith("USERLIST")) {
          StringTokenizer st = new StringTokenizer(line.substring(8), " ");
          while (st.hasMoreTokens()) {
          usersHashSet.add(st.nextToken());
listModel.removeAllElements();
for (Iterator i = usersHashSet.iterator(); i.hasNext(); ) {
          listModel.addElement(i.next());
} else if (line.startsWith("USERJOIN")) {
          String userJoinName = line.substring(8).trim();
          usersHashSet.add(userJoinName);
          listModel.addElement(userJoinName);
} else if (line.startsWith("USERQUIT")) {
          String userQuitName = line.substring(8).trim();
          usersHashSet.remove(userQuitName);
listModel.removeElement(userQuitName);
} else if (line.startsWith("MESSAGE")) {
messageArea.setFont(new Font("Lucida Sans", Font.PLAIN, 12));
messageArea.append(line.substring(8) + "\n");
catch (Exception e) {
//e.printStackTrace();
return;
* Runs the client as an application with a closeable frame.
public static void main(String[] args) throws Exception {
ChatClient client = new ChatClient();
client.frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
client.frame.setLocation(200, 100);
client.frame.show();
client.run();
class messageAreaPopupListener extends MouseAdapter {
public void mousePressed(MouseEvent e) {
maybeShowPopup(e);
public void mouseReleased(MouseEvent e) {
maybeShowPopup(e);
private void maybeShowPopup(MouseEvent e) {
if (e.isPopupTrigger()) {
messageAreaPopupMenu.show(e.getComponent(),
e.getX(), e.getY());
class userAreaPopupListener extends MouseAdapter {
public void mousePressed(MouseEvent e) {
maybeShowPopup(e);
public void mouseReleased(MouseEvent e) {
maybeShowPopup(e);
private void maybeShowPopup(MouseEvent e) {
if (e.isPopupTrigger()) {
userAreaPopupMenu.show(e.getComponent(),
e.getX(), e.getY());

Answer 1

Yes, the code is large, isn't it. It is a good idea, when you have a problem you can't solve, to remove unnecessary code. When the problem disappears, then you know the code you just removed was probably the source of the problem.
But in your case, I think that none of the 1000 lines of code you posted are the problem. You appear to be using Lucida Sans, which is a positive sign; it can render many of the characters you have hard-coded in your program. But apparently the problem is with characters that are coming from somewhere else? And you are using a PrintWriter to send them there and a BufferedReader to get them back? These convert your data from characters to bytes and back again using the default encoding for your system. Chances are that this encoding is some "extended ASCII" encoding that doesn't know how to map anything above \u00ff. You need to get an InputStreamReader and an OutputStreamWriter into your transmission. (Wrap your socket.getInputStream() in an InputStreamReader and wrap that in a BufferedReader, etc.) These objects can take an encoding name as their second parameter; try "UTF-8" or "UTF8" for the encoding. Their API documentation has a link to a page about encodings; follow that if you need more info.