Euthanasia by strace

Question

Thought it might be worthwhile posting a technique I discovered by accident. I occasionally need to kill runaway sessions, and sometimes those sessions don't want to die ( "session is marked for kill" ). Sometimes they hang around for days, or at least they used to until I discovered this little trick that works on Suse Linux with 10.2.0.4 ( and maybe a few other 10.2s although I haven't bothered to take notes). Log in to the database server, strace -p <the spid for the session you tried to kill>. Often - not always, but quite often, what will happen when you strace the undead process is that it seems to wake up from whatever it was doing, realizes it's supposed to be killed, and exits politely all in very short order.
Hope that helps..
Paul Kelley

Answer 1

It was an impulsive post... Coincidence crossed my mind, and to be honest it still might be coincidence. However I've done it many times, it's become a standard technique for the oddly stuck sessions that don't want to die. I can't prove they wouldn't have died at exactly the same time they did when I was stracing them, but it seems strange that they usually die very quickly after I strace them - within seconds. Might be coincidence. I should have set up some sort of test plan before posting - documenting the different kinds of sessions and whether or not they were rolling back. A long-term project, might take a year to collect all the data. Now that I posted I'll really have to do the tests.. I sort of expect there will be some class of stuck process states that get nicely cleaned up by strace..
I think kill -9 on a process that is rolling back will cause smon to take over the rollbacks, which in my experience in early 10.2s was a real disaster. Much slower than just letting the process roll back, at least with our configurations. Then again I often had processes with a lot of rolling back to do...
we had a magic application where a user could use his/her thumbs to fire off a select ... for update nowait for every row in a 25 million row table - that's select .. for update without a where clause. Developers couldn't find/fix the bug. Users had to use the problem program, thumbs and all. The older rows in the table were all locked first - 20 million or so, then eventually it got to the point where it locked newer rows that other sessions needed and then it would iitself be blocked and would have to roll back.. the whole thing could take 90 minutes, and if you kill -9 the shadow process it could take 2-3 hours. I got used to watching used_ublks in v$transaction, got to where I could estimate when the problems would clear up by watching the rate of rollback. It will be ready in 45 minutes, .. 25 minutes...
The eventual ugly bug fix was to lock the first row in the table and keep it locked during working hours. The problem went away, as did 75% of help desk calls from this database, and redo bursts of 4-5gb/hour.
By the way, selecting 20 million rows for update will generate a lot of redo in a hurry, which didn't help our somewhat undersized servers at all.
Also, there seemed to be a pattern that developed - if I used kill -9 on one of these problem processes then server-side load-balancing would get hosed and we would have to do lots of tweaking to undo the unbalanced workloads that would result. When I stopped using kill-9 on the runaways, I stopped having to re-load-balance multiple RAC instances - we never had a single load serious balancing problem since I stopped using kill-9 that way. It was like flipping a switch... This was 10.1, maybe early 10.2.
Truth is stranger than fiction.