How to remove wrlock calls ?

Question

We have analysed the performance of our applications using Quantify and find that a lot of the time is spent in calls to llrw_wrlock/unlock etc.
Our application is written in C++ built using SunWS4.2 on Solaris 2.6.
The application is single-threaded but sits on top of a thread-safe 'architecture' designed in-house.
We make use of the RogueWave libaries Tools.h/Money.h, and of Oracle 8.1.6.
I know from dbx that our application consists of 3 threads. 1 of which is the 'application' thread (where the application functionality is executed), 1 thread sits on a signotify wait, another sits on a condtimedwait.
I have looked on google to try to identify what the llrw_wrlock is (I had already guessed it was mutex lock related) - and came to the conclusion that it may be related to thread context switching. (We are using unbound LWP threads). I also determined that we were using a SCHED_OTHER thread scheduling policy. So I concluded that if I could keep the 'application' thread on CPU (or more to the point - on LWP) then I might improve performance.
I experimented with changing thread scheduling policy (using sched_setscheduler) but eventually found that only SCHED_OTHER is supported on Solaris when using native Solaris threads. (Note to Sun: the man pages are very poor in this area - man sched_setscheduler gives the impression that the other policies are supported - and it is not until one does 'man thr_setprio' that one realises that only SCHED_OTHER is supported - or if one is lucky enough to have the Sun Education "Multi-threaded Application Programming" manual to hand ;-[ ).
I also experimented with changing thread priority for the 'application' thread (trying priorities of 120 through to 127, and also a priority of 10 - in case I had misunderstood the man pages).
All my experiments have been to no avail - there was no performance change between my modified applications and the unmodified.
I also created a stock C++ application
int k=0;
for (int i = 0; i< 1000;++i)
for (int j=0;j < 1000000; ++j)
k++;
I compiled this and compared it against the same algorithm executed from within one of our 'architecture' wrappers - again no difference in performance, which gave me the impression that there was no negligible performance overhead caused by the"multi-threaded" nature of our architecture.
This has led me to believe that the Quantify figures produced are a red-herring ?
Can anyone else confirm my findings or has had similar performance issues that they have now overcome ? (If so - could you please indicate what the solution is ?)
Many thanks
Paul McGrath

Answer 1

We have analysed the performance of our applications
using QuantifyYou may be interested to know that the latest version of Workshop,
called Sun ONE Studio 7 Enterprise Edition, comes with a tool
called Performance Analyzer that will give you a lot more information.
In particular, it will tell you not only where you are spending your
time but how you got there (call stack) and which of multiple paths
through the call stack is taking the most time. It will also analyze
time spent at various types of locks.
and find that a lot of the time is
spent in calls to llrw_wrlock/unlock etc.As you probably know, rw_wrlock is an interface to something
called a readers-writer lock. It is a form of mutual exclusion lock
that allows many readers into a protected section but only one
writer. llrw_wrlock is new to me, but it's virtually certain to be
related.
So I
concluded that if I could keep the 'application'
thread on CPU (or more to the point - on LWP) then I
might improve performance.
I experimented with changing thread scheduling policy
(using sched_setscheduler) but eventually found that
only SCHED_OTHER is supported on Solaris when using
native Solaris threads.Based on your comment that you want to keep the app on
one LWP, I think you are more interested in the scope of
contention than in priority or policy. Scope of contention
is the scope within which the LWP competes for resources.
Threads are not visible to Solaris or the Solaris 2.6 scheduler.
Only LWPs are visible. By using unbound threads, you are
potentially using fewer kernel-scheduled LWPs than user-
created threads. Setting the scope of contention of your app
thread to systemwide rather than its present setting will
allow the thread to contend for resources (CPU time, for
example) directly at a system level. It will make it a bound
thread, which means that your thread (a user-level concept)
will be bound one-to-one with an LWP (an OS-level concept).
Binding the thread to an LWP allows the thread to compete,
through its LWP, for resources.
However, that is not your problem because...
I know from dbx that our application consists of 3
threads. 1 of which is the 'application' thread (where
the application functionality is executed), 1 thread
sits on a signotify wait, another sits on a
condtimedwait....and that means that two of your three threads are just
sitting there and not taking or wasting any resources.
They will just sit quietly until they are allowed to proceed
by getting notified in the case of the first one or by either
having the condition on which it is waiting met or having
the timeout expire in the case of the second one. And
if those two threads aren't taking any time then...
I also experimented with changing thread priority for
the 'application' thread (trying priorities of 120
through to 127, and also a priority of 10 - in case I
had misunderstood the man pages).... adjusting the priority is not going to help because
a higher priority only lets you go ahead of another LWP
with a lower priority that is taking resources that you
want. However, the other LWPs are sitting quietly and
not taking any resources. You can have any priority you
want, high or low, and your results will not change
because priority determines your place in line and you
are the only one in line at the moment.
This has led me to believe that the Quantify figures
produced are a red-herring ?Sounds good to me. Although I have never used Quantify,
the story that Quantify is telling (and that I am able to
judge only second-hand from your retelling thereof) is not
correct, or is at least incomplete.
Can anyone else confirm my findings or has had similar
performance issues that they have now overcome ? (If
so - could you please indicate what the solution is
?)The solution is to get Sun ONE Studio 7 Enterprise Edition
and use Performance Analyzer. Performance Analyzer
will tell you where your time is going in terms of inclusive
or exclusive elapsed wall clock time, CPU time, system
time or user time. On an UltraSPARC-III, it will use the
hardware counters to show you where you are spending
time on cache misses, pipeline stalls, mispredicted
branches, and a dozen other factors. It will attribute the
time to specific methods and even a specific line of
source. It will analyze your code built with compilers from
Sun, gnu, PGI, Fujitsu, and any other. If you have no money
then get the Try-and-Buy and at least take it for a spin on
this problem.