Help in profiling a complex application needed

Question

Hi,
I have a rather complex, general, problem here I would like to solve using
dtrace so any hint, advise, similar experience etc would be more than
welcome.
Basically, my company has created a complex database system for acquisition
and management of reference stock market data and the process has several
phases (this is my understanding of it): acquire the data from
e.g. Bloomberg in their native format, transforms it in some universal
format and then use that format to load it into a database where it can be
analysed, processed etc using more or less (in house developed) standard
SQL with calendar extensions (e.g every Friday, last Monday in month etc
etc). There are several processes (servers) involved: data server,
scheduler, database server etc etc.
Now, when we have installed the product at one of our customers (running
E6500 with 24CPUs and 20GB of RAM, all that on Solaris 8 --I think), our
product was given a two hours window every day to do a certain amount of
work and to process several GBs of data (don't have the figures) and
everything was fine, since we were able to finish that in 90 minutes. But
now, as the amount of data within database grows (although still within
preallocated 80 or 100GB size -- split into 2GB files to work around (ex)
large file problem), for the same amount of work the product in question
needs something like 180 minutes -- which is 1 hour too much.
My task is to try to find out where the bottleneck is and to give some
suggestions for speeding things up. The customer claims that the
application is I/O bound and prstat -m more or less confirms that (almost
no CPU latency, large amount of voluntary context switches etc etc).
How do I confirm this using dtrace? I can't run dtrace on their production
system since it's Solaris 8 but I have set up a Solaris 10 (s10_60)
machine that has all the components installed and will try to run similar
kind of work there.
Where do I start? What shall I look for? The problem is also that there is
a lot of programs involved, a lot of IPC (shared memory, semaphores, pipes,
sockets) and in general when the system is busy, it is quite busy.
I guess in ideal world, we would call Sun and ask them to figure the
problem themselves, but it is not an ideal world and the golden age of .com
is gone :-(
Thanks and bye, Dragan

Answer 1

Where do I start? What shall I look for? The problem
is also that there is
a lot of programs involved, a lot of IPC (shared
memory, semaphores, pipes,
sockets) and in general when the system is busy, it is
quite busy.Well, this is definitely the hard part. DTrace makes it much easier to ask questions of the system, but you're asking "what questions do I ask?" There are lots of valid ways to go about this, but I often start by by looking at how the physical resources on the system are being used: CPU, I/O, networking and so on. For starters, squeeze all of the idle out of the system: as long as there is idle in the system, you're obviously not going as fast as you could. You want to start asking questions like "why am I doing I/O?" or "why am I coming off CPU?" If you could post some mpstat/vmstat/iostat data, perhaps we could provide some more concrete directions...
- Bryan
Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc