Weird exec failure

Question

Hello,
This is not a DTrace question, rather a question about suggestions on how I could find more information (possibly with DTrace's help) related to the problem that I am going to describe:
One user has a simple Fortran program - one single source file. The user has about 15 copies of the source file, the difference between them being the value of a parameter. After he starts 12 such processes, when he tries to start another one the answer at the shell prompt is simply 'Killed'.
I assume nothing about what the program tries to do although I can get the source for it. What I tried to find out was why this process was killed.
The system on which the user tried this has 24 processors and runs Solaris 9.
First I thought it could be some resource limit, but I tried as root and the same thing happened ('Killed' when running more than 12 processes at the same time).
The BSM record for a failed exec looks like this:
============================================
<record version="2" event="execve(2)" modifier="fe" time="Sun Sep 12 01:38:19 EEST 2004" msec="914">
<path>/export/home/keyusers/vgrama/rez18.1.5</path>
<attribute mode="100750" uid="vgrama" gid="other" fsid="155" nodeid="4748945" device="0"/>
<exec_args><arg>./rez18.1.5
</arg></exec_args>
<subject audit-uid="vgrama" uid="vgrama" gid="keyusers" ruid="vgrama" rgid="keyusers" pid="23211" sid="20107" tid="0 52862 e420"/>
<return errval="failure: Resource temporarily unavailable" retval="-1"/>
</record>
=============================================
The 24 processor system on which I ran this has just been upgraded. Since I already have Sun ONE Studio 9 installed on other systems, I tried compiling and running the program on a 2-processor server, one running Solaris 9 as well. On this one, after running 3 processes concurrently I got the same message.
For the record, when running only one process, I can do ldd and the output is
==========
libf77compat.so.1 => /opt/SUNWspro/lib/libf77compat.so.1
libfui.so.1 => /opt/SUNWspro/lib/libfui.so.1
libfai.so.2 => /opt/SUNWspro/lib/v8plus/libfai.so.2
libfai2.so.2 => /opt/SUNWspro/lib/v8plus/libfai2.so.2
libfsumai.so.1 => /opt/SUNWspro/lib/v8plus/libfsumai.so.1
libfprodai.so.1 => /opt/SUNWspro/lib/v8plus/libfprodai.so.1
libfminlai.so.1 => /opt/SUNWspro/lib/v8plus/libfminlai.so.1
libfmaxlai.so.1 => /opt/SUNWspro/lib/v8plus/libfmaxlai.so.1
libfminvai.so.1 => /opt/SUNWspro/lib/v8plus/libfminvai.so.1
libfmaxvai.so.1 => /opt/SUNWspro/lib/v8plus/libfmaxvai.so.1
libfsu.so.1 => /opt/SUNWspro/lib/v8plus/libfsu.so.1
libsunmath.so.1 => /opt/SUNWspro/lib/libsunmath.so.1
libm.so.1 => /lib/libm.so.1
libc.so.1 => /lib/libc.so.1
libdl.so.1 => /lib/libdl.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfai_isa.so.2
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfai2_isa.so.2
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfsumai_isa.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfprodai_isa.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfminlai_isa.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfmaxlai_isa.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfminvai_isa.so.1
/opt/SUNWspro/prod/lib/cpu/sparcv8plus+vis/libfmaxvai_isa.so.1
/opt/SUNWspro/prod/lib/v8plus/../cpu/sparcv8plus+vis/libfsu_isa.so.1
/usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1
==========
What surprises me is that even 'ldd <program>' doesn't work (after having enough instances of the process running)
=============
# ldd ./rez_1.5
ldd: ./rez_1.5: execution failed due to signal 9
=============
On a workstation (1 processor) with Solaris Express b58 I ran the command while doing a
dtrace -n '::*kill*:{printf ("Entered %s\n", probefunc);}'
in another shell. Nothing appeared, although ldd reported "execution failed due to signal 9" as well.
No matter what that program may be trying to do, this all seems weird to me. Am I wrong ?
Just getting a SIGKILL so abruptly ?
Any ideas ?
What other probes could be helpful for seeing signals while they are being sent ?
Thank you,
Vlad.

Answer 1

Once exec(2) has passed the point where the old
address space has been destroyed, the only way it can
fail is by killing the process itself. My bet is that
the program has a rather large BSS segment, and there
isn't enough swap to reserve the space. (ldd(1) works
by setting up some environment variables, then running
the program itself, which is why it would fail)
To verify this, you can run /usr/ccs/bin/size
/path/to/program The third field is the BSS
segment.
You are right:
$ size ./rez15
28468 + 940 + 1200000976 = 1200030384which is roughly 1GB.
You could also look at the proc:::exec-failure probe,
which will probably give the same answer BSM does --
EAGAIN.
What other probes could be helpful for seeingsignals
while they are being sent ? You should look at the proc provider -- in particular,
proc:::signal-send.Thank you very much for the answer.
I first thought memory could be the limiting resource. I dismissed the idea because vmstat showed a lot of free space. So did '::memstat' in mdb -k - reported 87% free.
Now that you told me what the problem is I see the only command that shows me I don't have enough memory is swap -s :
$ swap -s
total: 129136k bytes allocated + 11656752k reserved = 11785888k used, 3454144k availableWhich shows a lot of memory reserved.
And running ps -o vsz I can see 1 GB for each process.
In general I think the performance parameters from the output of commands like vmstat,mpstat,iostat,swap could be documented in more detail.
At least the basics.
For instance, could I find documentation on reserved memory ?
Thanks again for your reply,
Vlad