Solaris Kernel and TCP/IP Tuning Parameters (Continued)

Question

This page describes some configuration optimizations for Solaris hosts running ATG Page Serving instances (application servers) that will increase server efficiency.
Note that these changes are specific to Solaris systems running ATG application servers (+page serving+ instances). Do not use these on a web server or database server. Those systems require entirely different settings.
h3. Solaris 10 Kernel
Adjust /etc/system (parameters below) and reboot the system.
set rlim_fd_cur=4096
set rlim_fd_max=4096
set tcp:tcp_conn_hash_size=32768
set shmsys:shminfo_shmmax=4294967295
set autoup=900
set tune_t_fsflushr=1h4. Set limits on file descriptors
{color:blue}set rlim_fd_max = 4096{color}
{color:blue}set rlim_fd_cur = 4096{color}
Raise the file-descriptor limits to a maximum of 4096. Note that this tuning option was not mentioned in the "Sun Performance And Tuning" book.
[http://download.oracle.com/docs/cd/E19082-01/819-2724/chapter2-32/index.html]
h4. Increase the connection hash table size
{color:blue}set tcp:tcp_conn_hash_size=8192{color}
Increase the connection hash table size to make look-up's more efficient. The connection hash table size can be set only once, at boot time.
[http://download.oracle.com/docs/cd/E19455-01/816-0607/chapter4-63/index.html]
h4. Increase maximum shared memory segment size
{color:blue}set shmsys:shminfo_shmmax=4294967295{color}
Increase the maximum size of a system V shared memory segment that can be created from roughly 8MB to 4GB.
This provides an adequate ceiling; it does not imply that shared memory segments of this size will be created.
[http://download.oracle.com/docs/cd/E19683-01/816-7137/chapter2-74/index.html]
h4. Increase memory allocated for dirty pages
{color:blue}set autoup=900{color}
Increase the amount of memory examined for dirty pages in each invocation and frequency of file system synchronizing operations.
The value of autoup is also used to control whether a buffer is written out from the free list. Buffers marked with the B_DELWRI flag (which identifies file content pages that have changed) are written out whenever the buffer has been on the list for longer than autoup seconds. Increasing the value of autoup keeps the buffers in memory for a longer time.
[http://download.oracle.com/docs/cd/E19082-01/819-2724/chapter2-16/index.html]
h4. Specify the time between fsflush invocations
Specifies the number of seconds between fsflush invocations.
{color:blue}set tune_t_fsflushr=1{color}
[http://download.oracle.com/docs/cd/E19082-01/819-2724/chapter2-105/index.html]
Again, note that after adjusting any of the preceding kernel parameters you will need to reboot the Solaris server.
h3. TCP
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_conn_req_max_q 16384
ndd -set /dev/tcp tcp_conn_req_max_q0 16384
ndd -set /dev/tcp tcp_ip_abort_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 7200000
ndd -set /dev/tcp tcp_rexmit_interval_initial 4000
ndd -set /dev/tcp tcp_rexmit_interval_max 10000
ndd -set /dev/tcp tcp_rexmit_interval_min 3000
ndd -set /dev/tcp tcp_smallest_anon_port 32768
ndd -set /dev/tcp tcp_xmit_hiwat 131072
ndd -set /dev/tcp tcp_recv_hiwat 131072
ndd -set /dev/tcp tcp_naglim_def 1h4. Tuning the Time Wait Interval and TCP Connection Hash Table Size
{color:blue}/usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 60000{color}
The tcp_time_wait_interval is how long a connection stays in the TIME_WAIT state after it has been closed (default value 240000 ms or 4 minutes). With the default setting, this socket will remain for 4 minutes after you have closed the FTP connection. This is normal operating behavior. It is done to ensure that any slow packets on the network will arrive before the socket is completely shutdown. As a result, a future program that uses the same socket number won't get confused upon receipt of packets that were intended for the previous program.
On a busy Web server a large backlog of connections waiting to close could build up and the kernel can become inefficient in locating an available TCP data structure. Therefore it is recommended to change this value to 60000 ms or 1 minute.
h4. Tuning the maximum number of requests per IP address per port
{color:blue}ndd -set /dev/tcp tcp_conn_req_max_q 16384{color}
{color:blue}ndd -set /dev/tcp tcp_conn_req_max_q0 16384{color}
The {color:blue}tcp_conn_req_max_q{color} and {color:blue}tcp_conn_req_max_q0{color} parameters are associated with the maximum number of requests that can be accepted per IP address per port. tcp_conn_req_max_q is the maximum number of incoming connections that can be accepted on a port. tcp_conn_req_max_q0 is the maximum number of “half-open” TCP connections that can exist for a port. The parameters are separated in order to allow the administrator to have a mechanism to block SYN segment denial of service attacks on Solaris.
The default values are be too low for a non-trivial web server, messaging server or directory server installation or any server that expects more than 128 concurrent accepts or 4096 concurrent half-opens. Since the ATG application servers are behind a DMZ firewall, we needn't starve these values to ensure against DOS attack.
h4. Tuning the total retransmission timeout value
{color:blue}ndd -set /dev/tcp tcp_ip_abort_interval 60000{color}
{color:blue}tcp_ip_abort_interval{color} specifies the default total retransmission timeout value for a TCP connection. For a given TCP connection, if TCP has been retransmitting for tcp_ip_abort_interval period of time and it has not received any acknowledgment from the other endpoint during this period, TCP closes this connection.
h4. Tuning the Keep Alive interval value
{color:blue}ndd -set /dev/tcp tcp_keepalive_interval 7200000{color}
{color:blue}tcp_keepalive_interval{color} sets a probe interval that is first sent out after a TCP connection is idle on a system-wide basis.
If SO_KEEPALIVE is enabled for a socket, the first keep-alive probe is sent out after a TCP connection is idle for two hours, the default value of the {color:blue}tcp_keepalive_interval{color} parameter. If the peer does not respond to the probe after eight minutes, the TCP connection is aborted.
The {color:blue}tcp_rexmit_interval_*{color} values set the initial, minimum, and maximum retransmission timeout (RTO) values for a TCP connections, in milliseconds.
h4. Tuning the TCP Window Size
{color:blue}/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 65535{color}
{color:blue}/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 65535{color}
Setting these two parameters controls the transmit buffer and receive window. We are tuning the kernel to set each window to 65535 bytes. If you set it to 65536 bytes (64K bytes) or more with Solaris 2.6, you trigger the TCP window scale option (RFC1323).
h4. Tuning TCP Slow Start
{color:blue}/usr/sinb/ndd -set /dev/tcp tcp_slow_start_initial 4{color}
tcp_slow_start_initial is the number of packets initially sent until acknowledgment, the congestion window limit.
h4. Tuning the default bytes to buffer
{color:blue}ndd -set /dev/tcp tcp_naglim_def 1{color}
{color:blue}tcp_naglim_def{color} is the default number of bytes to buffer. Each connection has its own copy of this value, which is set to the minimum of the MSS for the connection and the default value. When the application sets the TCP_NODELAY socket option, it changes the connection's copy of this value to 1. The idea behind this algorithm is to reduce the number of small packets transmitted across the wire by introducing a short (100ms) delay for packets smaller than some minimum.
Changing the value of tcp_naglim_def to 1 will have the same effect (on connections established after the change) as if each application set the TCP_NODELAY option.
{note}
The current value of any of the TCP parameters can be displayed with the command ndd get. So to retrieve the current setting of the {color:blue}tcp_naglim_def parameter{color}, simply execute the command:\\
{color:blue}ndd -get /dev/tcp tcp_naglim_def{color}
{note}
h3. References
Solaris Tunable Parameters Reference Manual
[http://download.oracle.com/docs/cd/E19455-01/816-0607/index.html]
WebLogic Server Performance and Tuning
[http://download.oracle.com/docs/cd/E11035_01/wls100/perform/OSTuning.html]

Answer 1

For example,
Socket.setSoTimeout() sets SO_TIMEOUT option and I
want to what TCP parameter this option corresponds in
the underlying TCP connection.This doesn't correspond to anything in the connection, it is an attribute of the API.
The same questions
arises fro other options from SocketOptions class.setTcpNoDelay() controls the Nagle algorithm. set{Send,Receive}BufferSize() controls the local socket buffers.
Most of this is quite adequately described in the javadoc actually.