Server Process taking all CPU time intermittently on Solaris

Question

Hi,
What does my program do :-
We have been facing this problem from many days, We have a server
process which is continuously listening at a port on Solaris 5.8.
Whenever a request is sent to this server a new thread & a socket is created for that client.
This process is running as a proxy server, which connects to some other machine on client request.
Problem:
The process start taking whole CPU after few days, happens in a weeks
period. The process remain in this state for few hours & after that it
stops responding to any requests.
& the only thing which can be done, to make it work is to restart the
process.
If someone has faced a similar problem then please let me know if you
have found a solution for that.
I will give 10 duke dollars to you if you can provide a solution for this problem.
Regards,
Sachin

Answer 1

Here is another class which is used in my code:
package com.ge.med.service.olea.telnetapplet.proxy;
import java.io.*;
import java.net.*;
import java.util.*;
* This class is a generic framework for a flexible, multi-threaded server.
* It listens on any number of specified ports, and, when it receives a
* connection on a port, passes input and output streams to a specified Service
* object which provides the actual service. It can limit the number of
* concurrent connections, and logs activity to a specified stream.
public class Server {
// This is the state for the server
ConnectionManager connectionManager; // The ConnectionManager object
Hashtable services; // The current services and their ports
ThreadGroup threadGroup; // The threadgroup for all our threads
PrintWriter logStream; // Where we send our logging output to
* This is the Server() constructor. It must be passed a stream
* to send log output to (may be null), and the limit on the number of
* concurrent connections. It creates and starts a ConnectionManager
* thread which enforces this limit on connections.
public Server(OutputStream logStream, int maxConnections) {
setLogStream(logStream);
log("Starting server");
threadGroup = new ThreadGroup("Server");
connectionManager = new ConnectionManager(threadGroup, maxConnections);
connectionManager.start();
services = new Hashtable();
* A public method to set the current logging stream. Pass null
* to turn logging off
public void setLogStream(OutputStream out) {
if (out != null) logStream = new PrintWriter(new OutputStreamWriter(out));
else logStream = null;
/** Write the specified string to the log */
protected synchronized void log(String s) {
if (logStream != null) {
logStream.println("[" + new Date() + "] " + s);
logStream.flush();
/** Write the specified object to the log */
protected void log(Object o) { log(o.toString()); }
* This method makes the server start providing a new service.
* It runs the specified Service object on the specified port.
public void addService(Service service, int port) throws IOException {
Integer key = new Integer(port); // the hashtable key
// Check whether a service is already on that port
if (services.get(key) != null)
throw new IllegalArgumentException("Port " + port + " already in use.");
// Create a Listener object to listen for connections on the port
Listener listener = new Listener(threadGroup, port, service);
// Store it in the hashtable
services.put(key, listener);
// Log it
log("Starting service " + service.getClass().getName() +
" on port " + port);
// Start the listener running.
listener.start();
* This method makes the server stop providing a service on a port.
* It does not terminate any pending connections to that service, merely
* causes the server to stop accepting new connections
public void removeService(int port) {
Integer key = new Integer(port); // hashtable key
// Look up the Listener object for the port in the hashtable of services
final Listener listener = (Listener) services.get(key);
if (listener == null) return;
// Ask the listener to stop
listener.pleaseStop();
// Remove it from the hashtable
services.remove(key);
// And log it.
log("Stopping service " + listener.service.getClass().getName() +
" on port " + port);
* This nested class manages client connections for the server.
* It maintains a list of current connections and enforces the
* maximum connection limit. It creates a separate thread (one per
* server) that sits around and wait()s to be notify()'d that a connection
* has terminated. When this happens, it updates the list of connections.
public class ConnectionManager extends Thread {
int maxConnections; // The maximum number of allowed connections
Vector connections; // The current list of connections
* Create a ConnectionManager in the specified thread group to enforce
* the specified maximum connection limit. Make it a daemon thread so
* the interpreter won't wait around for it to exit.
public ConnectionManager(ThreadGroup group, int maxConnections) {
super(group, "ConnectionManager");
this.setDaemon(true);
this.maxConnections = maxConnections;
connections = new Vector(maxConnections);
log("Starting connection manager. Max connections: " + maxConnections);
* This is the method that Listener objects call when they accept a
* connection from a client. It either creates a Connection object
* for the connection and adds it to the list of current connections,
* or, if the limit on connections has been reached, it closes the
* connection.
synchronized void addConnection(Socket s, Service service) {
// If the connection limit has been reached
if (connections.size() >= maxConnections) {
try {
PrintWriter out = new PrintWriter(s.getOutputStream());
// Then tell the client it is being rejected.
out.println("Connection refused; " +
"server has reached maximum number of clients.");
out.flush();
// And close the connection to the rejected client.
s.close();
// And log it, of course
log("Connection refused to " + s.getInetAddress().getHostAddress() +
":" + s.getPort() + ": max connections reached.");
} catch (IOException e) {log(e);}
else {  // Otherwise, if the limit has not been reached
// Create a Connection thread to handle this connection
Connection c = new Connection(s, service);
// Add it to the list of current connections
connections.addElement(c);
// Log this new connection
log("Connected to " + s.getInetAddress().getHostAddress() +
":" + s.getPort() + " on port " + s.getLocalPort() +
" for service " + service.getClass().getName());
// And start the Connection thread running to provide the service
c.start();
* A Connection object calls this method just before it exits.
* This method uses notify() to tell the ConnectionManager thread
* to wake up and delete the thread that has exited.
public synchronized void endConnection() { this.notify(); }
/** Change the current connection limit */
public synchronized void setMaxConnections(int max) { maxConnections=max; }
* Output the current list of connections to the specified stream.
* This method is used by the Control service defined below.
public synchronized void printConnections(PrintWriter out) {
for(int i = 0; i < connections.size(); i++) {
Connection c = (Connection)connections.elementAt(i);
out.println("CONNECTED TO " +
c.client.getInetAddress().getHostAddress() + ":" +
c.client.getPort() + " ON PORT " + c.client.getLocalPort()+
" FOR SERVICE " + c.service.getClass().getName());
* The ConnectionManager is a thread, and this is the body of that
* thread. While the ConnectionManager methods above are called by other
* threads, this method is run in its own thread. The job of this thread
* is to keep the list of connections up to date by removing connections
* that are no longer alive. It uses wait() to block until notify()'d by
* the endConnection() method.
public void run() {
while(true) {  // infinite loop
// Check through the list of connections, removing dead ones
for(int i = 0; i < connections.size(); i++) {
Connection c = (Connection)connections.elementAt(i);
if (c != null && !c.isAlive()) {
connections.removeElementAt(i);
               try      {
                    //close server socket. SPR SVCge16539. Sachin Joshi.
                    Socket server = ProxyServer.Proxy.getServerSocket(c.client);                    
                    if (server != null)
                         server.close();
                    //close client socket.
                    if (c.client != null)
                         c.client.close();                    
               catch (java.io.IOException e)     {
                    System.err.println("Error closing connection " + e);
               catch (Exception e)     {
                    System.err.println("Cannot Establish Connection !! " + e);
log("Connection to " + c.client.getInetAddress().getHostAddress() +
":" + c.client.getPort() + " closed.");
          System.out.println("Total Connections now = " + connections.size());
// Now wait to be notify()'d that a connection has exited
// When we wake up we'll check the list of connections again.
try { synchronized(this) { this.wait(); } }
catch(InterruptedException e) {}
* This class is a subclass of Thread that handles an individual connection
* between a client and a Service provided by this server. Because each
* such connection has a thread of its own, each Service can have multiple
* connections pending at once. Despite all the other threads in use, this
* is the key feature that makes this a multi-threaded server implementation.
public class Connection extends Thread {
Socket client; // The socket to talk to the client through
Service service; // The service being provided to that client
* This constructor just saves some state and calls the superclass
* constructor to create a thread to handle the connection. Connection
* objects are created by Listener threads. These threads are part of
* the server's ThreadGroup, so all Connection threads are part of that
* group, too.
public Connection(Socket client, Service service) {
super("Server.Connection:" + client.getInetAddress().getHostAddress() +
":" + client.getPort());
this.client = client;
this.service = service;
* This is the body of each and every Connection thread.
* All it does is pass the client input and output streams to the
* serve() method of the specified Service object. That method
* is responsible for reading from and writing to those streams to
* provide the actual service. Recall that the Service object has been
* passed from the Server.addService() method to a Listener object
* to the ConnectionManager.addConnection() to this Connection object,
* and is now finally getting used to provide the service.
* Note that just before this thread exits it calls the
* ConnectionManager.endConnection() method to wake up the
* ConnectionManager thread so that it can remove this Connection
* from its list of active connections.
public void run() {
try {
InputStream in = client.getInputStream();
OutputStream out = client.getOutputStream();
// service.serve(in, out);
((ProxyServer.Proxy)service).serve(in, out,client);
catch (IOException e) {log(e);}
finally { connectionManager.endConnection(); }
* Here is the Service interface that we have seen so much of.
* It defines only a single method which is invoked to provide the service.
* serve() will be passed an input stream and an output stream to the client.
* It should do whatever it wants with them, and should close them before
* returning.
* All connections through the same port to this service share a single
* Service object. Thus, any state local to an individual connection must
* be stored in local variables within the serve() method. State that should
* be global to all connections on the same port should be stored in
* instance variables of the Service class. If the same Service is running
* on more than one port, there will typically be different Service instances
* for each port. Data that should be global to all connections on any port
* should be stored in static variables.
* Note that implementations of this interface must have a no-argument
* constructor if they are to be dynamically instantiated by the main()
* method of the Server class.
public interface Service {
public void serve(InputStream in, OutputStream out) throws IOException;     
     }