Multi-threaded performance server using 2 ports to two-way communication

Question

Hi, I'm new in the forums so welcome everyone. I'm developing now an online computer game based on orginal Polish board-game, but it doesn't mean. Important is that I need to develope a high-performance multi-threaded server, which can serve... let's say few hundres of games, and a thousend of players simulateously. My server works on two ports/sockets.
First one is represented by "ServerSocketChannel clientConSsc" in my code. The main thread in my server class which invokes the method "run()" (which You can see below), registers clientConSsc with Selector and then repeatingly accepts incoming connections from clients on this socket. When connection(channel) is estabilished and clients wants to write some request to this channel, the main thread on server creates a new instance of class "RequestHandler" which extends Thread and handles this request (reads the request from channel, makes some changes on server, spreads some new messages to other clients in the same game and so on... everything on the same socket/port/channel). Then the channel/connection is closed by the server - I call such connections: "a short connections". In the same time the main thread continues the loop and is still able to accept connections from another players. That's the idea of multi-threaded server, right? :) And to this point, everything works fine, but... what if the server wants to trigger some new command, write new information to client?
Server could have tried to open a SocketChannel to client's IP address, but then the client programme would have to create and ServerSocketChannel object, bind it to some InetAddress and then... and then client becomes a server! - that breaks the idea of client-server cooperation and demands on players of my game to have routed some port on their machines. To resolve this problem I made some sort of "system" which, when some player/client is logging into my server, accepts second (this time constant, not "short") connection on the second port I mentoined on the beginning. This connection is held in variable "SocketChannel serverCon" of class "Player" - which reflects a player logged onto server. Server maintains every such connection till the client logs off. After the connection is accepted, the player whom connection it was is added to collection called "playersToRegisterToWrite". After the Selector.selectNow() is invoked (as You can see in the code below), each and every player's connection is registered in selector (with selection key - OP_WRITE and attachment pointing on owning player). In the same time client's code is trying to read some bytes from this connection and blocks on it until some bytes are written to this connection. When server wants to "tell" something to client, deliver some message/command, it creates a new instance of class which extends an abstract class called "ServerAction" (another new thread) and adds it to collection "actionsToDo". In ServerAction's method "run()" there are whole code to interact with client (e.g. send an update of players' list, an update of games' list, a chat message). Finnaly when Selector informs the server that some connection is writable (some client is waiting for commands), then server checks if there's any "actionToDo" involving this client/player. If there is, it unregisters the connection from Selector (so no other ServerAction (thread) can write on this connection) and starts the action. At the end of every "run()" method of subclass of ServerAction there's a code, which again adds the player to collection "playersToRegisterToWrite", so the player's connection can again be registered in Selector and the player can again receive informations from server - after it consumed THIS "ServerAction".
It looks to me like it should be working fine, but it's not. When I test my server and clients on the same machine (no ping/latency) everything seems to be working fine. But when it comes to play my game with somebody even on LAN or through the Internet there comes the problems. My first socket I describe (and first type of connections - short connections) works perfectly fine, so the requests triggered by client are delivered and handled properly. But the second socket is cirppling. For example: the server triggers a command refering to an update of clients logged to server' list. It is triggered by Selector so the client should be ready to read data from server, so the server sends data! And the client never receives it... Why?
It experimented a whole lot of time on SocketChannel's method "configureBlocking(boolean)", but it never helps. I think the client should always block it's listening thread on connection, contratry to server which is ment to be fast, so it should send the data through non-blocking channels. Am I right? Are there any rules refering blocking configuration of SocketChannels?
I will be grateful for every answer. To help You out I attach also drafts from run()'s methods of my classes.
Server's main method - main thread :
    public void run() {
       try {
            selector = Selector.open();
            clientConSsc.configureBlocking(false);
            clientConSsc.register(selector , SelectionKey.OP_ACCEPT);
            while(serverRunning) {
                try {
                    selector.selectNow();
                    Iterator it;
                    synchronized(playersToRegisterToWrite) {
                        it = playersToRegisterToWrite.iterator();
                        while(it.hasNext()) {
                            Player player = (Player)it.next();
                            it.remove();
                            player.serverCon.configureBlocking(false);
                            player.serverCon.register(selector , SelectionKey.OP_WRITE , player);
                    Set keys = selector.selectedKeys() {
                    it = keys.iterator();
                    while(it.hasNext()) {
                        SelectionKey key = (SelectionKey)it.next();
                        if(key.isAcceptable()) {
                            it.remove();
                            clientCon = clientConSsc.accept();
                            clientCon.configureBlocking(false);
                            clientCon.register(selector , SelectionKey.OP_READ);
                            continue;
                        if(key.isReadable()) {
                            it.remove();
                            key.cancel();
                            new RequestHandler(this , (SocketChannel)key.channel()).start();
                            continue;
                        if(key.isWritable()) {
                            if(key.attachment() != null ) {
                                ServerAction action = null;
                                synchronized(actionsToDo) {
                                    Iterator aIt = actionsToDo.iterator();
                                    while(aIt.hasNext()) {
                                        ServerAction temp = (ServerAction)aIt.next();
                                        if(temp.getPlayer().equals((Player)key.attachment())) {
                                            action = temp;
                                            aIt.remove();
                                            break;
                                if(action != null) {
                                    key.channel().configureBlocking(false);
                                    it.remove();
                                    key.cancel();
                                    action.start();
                            continue;
                } catch(Exception e) {
                    statusArea.append(e.toString() + "\n");
                    e.printStackTrace();
                    continue;
        } catch(ClosedChannelException e) {
            statusArea.append(e.toString() + "\n");
        } catch(IOException e) {
            statusArea.append(e.toString() + "\n");
            A part of server's RequestHandler's class' method run() - the method which handles requests triggered by client on first port:
public void run() {
        boolean served = false;
            try {
                channel.configureBlocking(true);
                retryCount++;
                retry = false;
                String command = ns.readText(channel).trim();
                ns.statusArea.append("ktos przesyla komende: " + command + "\n");
                if(command != null && !command.equals("ERROR")) {
                    String[] cmd = command.split("�");
                    ns.statusArea.append("komenda: " + cmd[0] + "\n");
                    if(cmd[0].equals("CHAT")) {
                        if(cmd.length != 5) {
                            retry();
                        } else {
                            if(cmd[2].equals("NOGAME")) {      // jezeli nie ma okreslonej gry czyli rozsylamy do wszystkich
                                for(int i = 0 ; i < ns.loggedPlayersListItems.size() ; i++) {
                                    Player current = (Player)ns.loggedPlayersListItems.get(i);
                                    if(current.state == Player.CHOOSING && !current.equals(new Player(Integer.parseInt(cmd[1]))))   // jezeli gracz jest w okienku wybierania gry to wysylamy mu broadcast
                                        ns.actionsToDo.add(new SendMsg(ns , current , "CHAT�" + cmd[1] + "�" + cmd[3] + "�" + cmd[4]));
                            } else {
                                Game game = (Game)ns.gamesListItems.get(ns.gamesListItems.indexOf(new Game(Integer.parseInt(cmd[2]))));
                                for(int i = 0 ; i < game.players.size() ; i++) {
                                    Player current = (Player)game.players.get(i);
                                    if(!current.equals(new Player(Integer.parseInt(cmd[1]))))
                                        ns.actionsToDo.add(new SendMsg(ns , current , "CHAT�" + cmd[1] + "�" + cmd[3] + "�" + cmd[4]));
                            served = true;
                    } else if(cmd[0].equals("GETGAMEINFO")) {
                        if(cmd.length != 3)
                            retry();
                        else {
                            int gameID = Integer.parseInt(cmd[2]);
                            ns.statusArea.append("poproszono o informacje o grze nr: " + gameID + "\n");
                            Game checkedGame = new Game(gameID);
                            if(ns.gamesListItems.contains(checkedGame)) {
                                Game game = (Game)ns.gamesListItems.get(ns.gamesListItems.indexOf(checkedGame));
                                channel.configureBlocking(true);
                                ObjectOutputStream oos = new ObjectOutputStream(channel.socket().getOutputStream());
                                oos.writeObject(game);
                                oos.writeObject(game.players);
                                oos.flush();
                                ns.statusArea.append("wyslano informacje o grze nr: " + gameID + "\n");
                            } else {
                                ns.statusArea.append("brak gry o nr: " + gameID + "\n");
                            served = true;
            } catch(IOException e) {
                e.printStackTrace();
        if(!served) {
            ns.findAndDisconnectPlayer(channel);
        try {
            channel.close();
        } catch(IOException e) {
            e.printStackTrace();
    }An example of ServerAction's subclass - the class which triggers server commands to client on second port:
class UpdateLoggedPlayersList extends ServerAction {
    public UpdateLoggedPlayersList(NeuroServer ns , Player player) {
        super(ns , player);
    public void run() {
        try {
            serverCon.configureBlocking(true);
            ns.sendText("UPDATELOGGEDPLAYERSLIST" , serverCon);
            if(ns.readText(serverCon).equals("OK")) {
                ObjectOutputStream oos = new ObjectOutputStream(serverCon.socket().getOutputStream());
                oos.writeObject(ns.loggedPlayersListItems);
                oos.flush();
            synchronized(ns.playersToRegisterToWrite) {
                 ns.playersToRegisterToWrite.add(player);
        } catch(IOException e) {
            e.printStackTrace();
}A part of client's CmdHandler's class' method run() - the method which handles commands triggered by server:
public void run() {
    try {
        while(works) {
            String command = nc.readText(nc.serverCon);
            System.out.println("Server przesyla komende: " + command + "\n");
            String[] cmd = command.split("�");
            if(cmd[0] != null && !cmd[0].equals("ERROR")) {
                if(cmd[0].equals("CHAT")) {
                    if(nc.chooseGameDialog != null && nc.chooseGameDialog.isVisible()) {     // jezeli na wybieraniu gry
                        newChatEntry(cmd[2] , null , cmd[3] , nc.chooseGameDialog.chatPane);
                    } else if(nc.readyDialog != null && nc.readyDialog.isVisible()) {  // jesli na przygotowywaniu
                        Player sender = (Player)nc.actGame.players.get(nc.actGame.players.indexOf(new Player(Integer.parseInt(cmd[1]))));
                        newChatEntry(cmd[2] , sender.fraction , cmd[3] , nc.readyDialog.chatPane);
                    } else if(nc.ng != null) {                   // jesli w grze
                        Player sender = (Player)nc.actGame.players.get(nc.actGame.players.indexOf(new Player(Integer.parseInt(cmd[1]))));
                        newChatEntry(cmd[2] , sender.fraction , cmd[3] , nc.ng.inGameChatPane);
                } else if(cmd[0].equals("UPDATELOGGEDPLAYERSLIST")) {
                    nc.sendText("OK" , nc.serverCon , false);
                    nc.serverCon.configureBlocking(true);
                    ObjectInputStream ois = new ObjectInputStream(nc.serverCon.socket().getInputStream());
                    DefaultListModel players = (DefaultListModel)ois.readObject();
                    nc.chooseGameDialog.updateLoggedPlayersList(players);
    } catch(IndexOutOfBoundsException e) {
        System.out.println(e);
    } catch(IOException e) {
        System.out.println(e);
    } catch(ClassNotFoundException e) {
        System.out.println(e);
}And two methods I used in codes above: sendText(String text , SocketChannel sc) and readText(SocketChannel sc) - they are my "utility" methods, which I use to send and receive Strings through specified SocketChannels.
boolean sendText(String text , SocketChannel sc) {
    ByteBuffer bbuf = ByteBuffer.allocate(BSIZE);
    boolean sendRetry;
    boolean sent = false;
    do {
        sendRetry = false;
        try {
            StringBuffer cmd = new StringBuffer();
            cmd.setLength(0);
            if(text.length()+4 < 100)
                cmd.append("0");
            if(text.length()+4 < 10)
                cmd.append("0");
            cmd.append(Integer.toString(text.length()+4) + "�");
            cmd.append(text);
            cmd.append("\n");
            bbuf = charset.encode(CharBuffer.wrap(cmd));
            sc.write(bbuf);
            bbuf.clear();
            int n = sc.read(bbuf);
            if(n == 1) {
                bbuf.flip();
                Byte re = bbuf.get();
                if(re == 1) {
                    sendRetry = true;
                } else {
                    sent = true;
        } catch(Exception e) {
            findAndDisconnectPlayer(sc);
            try {
                sc.close();
            } catch(IOException f) {
                f.printStackTrace();
            return false;
    } while(!sent && sendRetry);
    return true;
String readText(SocketChannel sc) {
    ByteBuffer bbuf = ByteBuffer.allocate(BSIZE);
    int readRetryCount = -1;
    boolean readRetry;
    do {
        readRetry = false;
        readRetryCount++;
        StringBuffer cmd = new StringBuffer();
        cmd.setLength(0);
        bbuf.clear();
        try {
            readLoop:
            while(true) {
                int n = sc.read(bbuf);
                if(n > 0) {
                    bbuf.flip();
                    CharBuffer cbuf = charset.decode(bbuf);
                    while(cbuf.hasRemaining()) {
                        char c = cbuf.get();
                        if(c == '\r' || c == '\n') break readLoop;
                        cmd.append(c);
                } else break;
            statusArea.append(Thread.currentThread().getId() + " readText() odczytuje: " + cmd.toString() + "\n");
            if(cmd.length() < 3 || Integer.parseInt(cmd.substring(0 , 3)) != cmd.length()) {
                sc.write(ByteBuffer.wrap(new byte[] {1}));
                readRetry = true;
            } else {
                sc.write(ByteBuffer.wrap(new byte[] {0}));    // length OK
                return cmd.toString().substring(4 , cmd.toString().length());
        } catch(Exception e) {
            findAndDisconnectPlayer(sc);
            try {
                sc.close();
            } catch(IOException f) {
                f.printStackTrace();
            return "ERROR";
    } while(readRetry && readRetryCount < 3);
    findAndDisconnectPlayer(sc);
    try {
        sc.close();
    } catch(IOException e) {
        e.printStackTrace();
    return "ERROR";
}Edited by: Kakalec on Jul 23, 2008 11:04 AM

Answer 1

You seem to be using a horrendous mixture of PrintWriters, BufferedReaders, ObjectOutputStreams, and no doubt ObjectInputStreams. You can't do that.
Some rules about this:
(a) Whenever you use a stream or reader or writer on a socket, you must allocate it once for the life of the socket, not every time you want to do an I/O.
There are many reasons for this including losing buffered data in the old stream and auto-closure of the socket on garbage-collection of the old stream.
(b) You can't mix stream types on the same socket. (i) If you're writing objects, use an ObjectOutputStream, you must read with an ObjectInputStream, and you can't read or write anything that isn't supported by the API of ObjectOutputStream. (ii) If you're writing lines, use a BufferedWriter and a BufferedReader at the other end. If you're writing characters, use a Writer and a Reader. (iii) If you're writing primitive data or UTF strings, use a DataOutputStream and a DataInputStream. (iv) If you're writing bytes, use DataOutputStream and a DataInputStream or an OutputStream and an InputStream. You can't mix (i), (ii), (iii), and (iv).
(c) Don't use PrintWriter or PrintStream over the network. They swallow exceptions that you need to know about. They are for writing to the console, or log files, where you don't really care about exceptions.
(d) Call flush() before reading.
(e) Always close the 'outermost' output stream or writer of a socket, never the input stream or the socket itself. Closing any of these closes the socket and therefore makes the other stream unusable, but only closing the output stream/writer will flush it.