Use of CountOfRefreshFailuresTillDisable

Question

Hi,
I have recently run into a problem that requires me to use CountOfRefreshFailuresTillDisable...we've been running into network or DB availability problems that are causing our getConnection call on the DataSource to take up to 8 minutes to fail (socket level failure). The failure occurs when we try to reserve/create a connection.
Now the documentation is somewhat vague on some of the specifics of CountOfRefreshFailuresTillDisable and I'm not sure I have it configured properly as I see no indication of WebLogic trying to reable the pool. The docs say "WebLogic Server periodically runs the refresh process" to reenable the pool.
I am already setting TestConnectionsOnReserve="true" with an appropriate test table
My questions are:
(1) Is all I have to do is set CountOfRefreshFailuresTillDisable="2"?
(2) Do I have to set TestFrequencySeconds? Hopefully not as I don't want to periodically test the pool (I'm already doing it on reserve).
(3) Is there any way to control the period of the refresh process tied to the CountOfRefreshFailuresTillDisable functionality i.e. how often it tests in order to reenable the pool? What is it by default?
(4) Should I see logs in the WLS logs indicating retries in order to reenable the pool? I'd feel more confortable if I had a visual indication of it.
Thanks in advance for any responses and hope to hear back soon.
I'm running WLS 8.1 SP4.
Cheers,
Mich

Answer 1

Michel Crichton wrote:
Hi,
I have recently run into a problem that requires me to use CountOfRefreshFailuresTillDisable...
we've been running into network or DB availability problems that are causing our getConnection
call on the DataSource to take up to 8 minutes to fail (socket level failure). The failure
occurs when we try to reserve/create a connection.
Now the documentation is somewhat vague on some of the specifics of CountOfRefreshFailuresTillDisable
and I'm not sure I have it configured properly as I see no indication of WebLogic trying to reable
the pool. The docs say "WebLogic Server periodically runs the refresh process" to reenable the pool.
I am already setting TestConnectionsOnReserve="true" with an appropriate test table
My questions are:
(1) Is all I have to do is set CountOfRefreshFailuresTillDisable="2"?
(2) Do I have to set TestFrequencySeconds? Hopefully not as I don't want to periodically test
the pool (I'm already doing it on reserve).
(3) Is there any way to control the period of the refresh process tied to the
CountOfRefreshFailuresTillDisable functionality i.e. how often it tests in order to reenable
the pool? What is it by default?
(4) Should I see logs in the WLS logs indicating retries in order to reenable the pool? I'd
feel more confortable if I had a visual indication of it.
Thanks in advance for any responses and hope to hear back soon.
I'm running WLS 8.1 SP4.
Cheers,
MichHi Mitch. I'm well aware of the problem. The TCP timeout is a OS level
variable, typically minutes long, that governs how long a JVM/driver
will have to wait on a JDBC call before being told that the socket is
dead.
If you have test-on-reserve, you should set two properties:
CountOfTestFailuresToFlush, and CountOfRefreshFailuresToDisable.
The first one will cause to pool empty the whole pool if that many
consecutive test failures occur. This is good because it prevents
the pool from having to test every connection before replacing it.
This setting is good for quick, transient network glitches that
kill all the connections, but don't prevent a reconnect.
The second setting is when the pool is unable to replace dead
connections, during protracted network or DBMS outages. Whether
you have asynchronous pool testing turned on or not, if a pool
(with this setting) fails to replace that many consecutive dead
connections (I'd set it to 1), the pool will self-disable, and
will asyncronously try to reconnect. Once it can reconnect, it
will self-re-enable. Applications requesting connections while
the pool is disabled will get a Resource exception, and the instant
the pool can reconnect and re-enables, the next request will
work. I believe that in 81sp5 we also alter the console display
to show a pool is back online when it re-enables itself.
All that being said, there is still a big multi-minute
hole where all the server threads might fall, each testing
a dead connection, because the pool won't be reset/disabled until
the first test finally returns. We are working on some sort
of internal solution to this case. In the mean time, if it were
me, I might write a small standalone program that tries to
make or use a direct simple JDBC connection to your DBMS, and
if it fails or even delays too long, I would have the program
(via another thread if necessary) log into WebLogic and
manually disable the relevant pool. I would have the program
continue to try to reconnect, and when it can, re-enable the
WebLogic pool(s).
Joe