Sockets remaining open after connection failure

Question

Hi - I have a multithreaded application that runs as a daemon (always up) and uses OCI (10.2.0, 64-bit client). If a database connection or query fails for any reason, it is coded to keep on retrying on a set schedule until it succeeds. The application runs on RHEL 4 (64-bit) and is compiled with g++ 3.4.6. (It's nominally C++, but is really what I call "C with objects" as it doesn't use a lot of the standard C++-isms, though there are objects -- the OCI code in question is in a C++ object wrapper that I wrote.)
Anyway, about 5 days ago, the database where most of the queries are run went down (it's a 10g server; I don't remember the exact version, but since it's down, I'm not sure it matters), and it has not come back up, although the hosts it runs on are up. Meanwhile, my application kept failing to connect and retrying, and the number of queries that this was happening to kept on growing as new ones are added every day. Unknown to me, the failed connection attempts were leaving open TCP sockets to the database that was down, until eventually the maximum open files on the system was exceeded and I was forced to reboot. Twice. (It happened again the next day.)
From what I can tell, I am properly dropping the handles and such when a connection fails, so why are the sockets staying open? Is there anything that can be done to force the socket to close after a failed connect? This morning I changed all the queries to use a different database (that's actually up), but according to lsof, there are still 56 open sockets to the old (non-working) database, even though none of my program threads are actively trying to connect there anymore.
The sockets do close, by the way, if the application is stopped and restarted. But they remain open as long as the instance that created them is still running. (I added a periodic restart to clean things up, but I'd really like to fix the problem instead of using a stopgap like this!)
Sample code is below. This first bit is what is used to connect and log in. (Note that I have this mutex-locked -- I know OCI is supposed to be thread-safe if you use the OCI_THREADED attribute, but I've been having so many issues that I thought it was safer.)
if (check_err(OCIEnvCreate(&env, (OCI_THREADED), (void *) 0, 0, 0, 0, (size_t) 0, (void **) 0)) != 0)
fprintf(stderr, "Error allocating OCI environment handle\n");
return;
// Allocate error handle.
if (check_err(OCIHandleAlloc((void *) env, (void **) &err, OCI_HTYPE_ERROR, 0, (void **) 0)) != 0)
fprintf(stderr, "Error creating OCI error handle.\n");
return;
// Allocate server handle.
if (check_err(OCIHandleAlloc((void *) env, (void **) &server, OCI_HTYPE_SERVER, 0, (void **) 0)) != 0)
fprintf(stderr, "Error allocating OCI server handle.\n");
return;
// Allocate service handle.
if (check_err(OCIHandleAlloc((void *) env, (void **) &svc, OCI_HTYPE_SVCCTX, 0, (void **) 0)) != 0)
fprintf(stderr, "Error allocating OCI service handle.\n");
return;
[Note: these are snippets from two different functions; this is a C++ wrapper that uses OCI, so the above code is in the database object constructor, and below is a separate login function.]
// Attach to server.
retcode = OCIServerAttach(server, err, (text *) curDS, strlen(curDS), OCI_DEFAULT);
if (check_err(retcode))
fprintf(stderr, "Error attaching to Oracle server.\n");
return(retcode);
// Set server attribute in service handle.
retcode = OCIAttrSet((void *) svc, OCI_HTYPE_SVCCTX, (void *) server, 0, OCI_ATTR_SERVER, err);
if (check_err(retcode))
fprintf(stderr, "Error setting OCI server attribute.\n");
return(retcode);
// Allocate session handle.
retcode = OCIHandleAlloc((void *) env, (void **) &sess, OCI_HTYPE_SESSION, 0, (void **) 0);
if (check_err(retcode))
fprintf(stderr, "Error allocating OCI session handle.\n");
return(retcode);
// Set username attribute in session handle.
retcode = OCIAttrSet((void *) sess, OCI_HTYPE_SESSION, (void *) curUser, strlen(curUser), OCI_ATTR_USERNAME, err);
if (check_err(retcode))
fprintf(stderr, "Error setting OCI username.\n");
return(retcode);
// Set password attribute in session handle.
retcode = OCIAttrSet((void *) sess, OCI_HTYPE_SESSION, (void *) curPass, strlen(curPass), OCI_ATTR_PASSWORD, err);
if (check_err(retcode))
fprintf(stderr, "Error setting OCI password.\n");
return(retcode);
// Start session.
retcode = OCISessionBegin(svc, err, sess, OCI_CRED_RDBMS, OCI_DEFAULT);
if (check_err(retcode) != 0)
return(retcode);
// Set session attribute in service handle
retcode = OCIAttrSet((void *) svc, OCI_HTYPE_SVCCTX, (void *) sess, 0, OCI_ATTR_SESSION, err);
if (check_err(retcode))
fprintf(stderr, "Error setting OCI session attribute.\n");
return(retcode);
If any of the above calls returns an error, the next call is to the database object destructor (also mutexed), which looks like this:
if (connected && svc && err && sess && check_err(OCISessionEnd(svc, err, sess, OCI_DEFAULT)))
fprintf(stderr, "Oracle - Error ending session\n");
if (connected && server && err && check_err(OCIServerDetach(server, err, OCI_DEFAULT)))
fprintf(stderr, "Oracle - Error detaching from server\n");
if (connected && server && check_err(OCIHandleFree((void *) server, OCI_HTYPE_SERVER)))
fprintf(stderr, "Oracle - Error freeing server handle\n");
if (connected && svc && check_err(OCIHandleFree((void *) svc, OCI_HTYPE_SVCCTX)))
fprintf(stderr, "Oracle - Error freeing service handle\n");
if (connected && err && check_err(OCIHandleFree((void *) err, OCI_HTYPE_ERROR)))
fprintf(stderr, "Oracle - Error freeing error handle\n");
if (connected && env && check_err(OCIHandleFree((void *) env, OCI_HTYPE_ENV)))
fprintf(stderr, "Oracle - Error freeing environment handle\n");
Anyone else had this happen? Any insights/help would be greatly appreciated!
Oh, one more note: though the application is multithreaded, none of the database objects or connections is shared -- each thread makes its own connections and maintains database objects separately. I added the mutex locks because Helgrind was reporting race conditions on handle allocations and deallocations.
Thanks,
--Tina                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Answer 1

Hmm, I believe I did have an OCIHandleFree on the session handle in an earlier version of the code. I can't remember why I took it out, but I do remember that there was a reason -- it was causing a problem. (Yeah, I know, vague enough for ya?) Anyway, the OCI documentation said that when you call OCIHandleFree on the environment handle, any child handles would be implicitly freed, so I assumed it wouldn't be a problem. Still, I'll try putting it back and see what happens. Thanks.
Oh, and to answer your question (about netstat), it said the processes were ESTABLISHED. But the Oracle server was down (although the hosts it resided on were up). The connection attempt failed with the error message:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Message was edited by:
user613364