Tracing what is happening in the connect() system call

Question

I've been playing around with some of the recipies in dtrace today trying to find out why it takes so long to connect from a linux machine to an ultra 5 running sendmail but I'm not having a lot of luck.
The one thing I can find is that the amount of time for the connect system call to return takes a long time. Here's the dtrace script I wrote (borrowing from Bryan Cantrill's blog):
bash-2.05b# cat functiontime.d
#!/usr/sbin/dtrace -s
#pragma D option flowindent
syscall::connect:entry
/execname == "sendmail"/
self->start = vtimestamp;
syscall:::entry,
syscall:::return
/self->start/
trace(vtimestamp - self->start);
syscall::connect:return
/execname == "sendmail"/
self->start = 0;
exit(0);
This is the output:
bash-2.05b# ./functiontime.d
dtrace: script './functiontime.d' matched 454 probes
CPU FUNCTION
0 => connect 0
0 | connect:return 303079
0 <= connect
I know there are other syscalls going on from the time connect enters to the time it returns but I have no idea how to get them to print.
Any ideas how I can make this script more useful to see exactly where sendmail is getting hung up?

Answer 1

You might also try something like
#!/usr/sbin/dtrace -s
#pragma D option flowindent
syscall::connect:entry
/execname == "sendmail"/
        self->depth = 1;
        self->syscallstart = timestamp
fbt:::entry
/self->depth/
        self->start[self->depth] = timestamp;
        trace(self->start[self->depth++] - self->syscallstart);
fbt:::return
/self->depth/
        this->delta = timestamp - self->start[--self->depth];
        @func[probefunc] = quantize(this->delta);
        printf("delta is %d\n", this->delta);
        self->start[self->depth] = 0;
syscall::connect:return
/self->depth/
        exit(0);
}This will give you the function call flow inside the kernel along with the following:
- on function entry, a time delta since connect was called
- on function exit, how long we spent in the function
- on return from the connect system call, a list of histograms showing the call times for all functions called
You are probably going to want to redirect the output as it gets big.
Also note, I've written this to give the information only for the first call to connect.
Alan.
Alan Hargreaves - http://blogs.sun.com/tpenta
Senior Technical Support Specialist/VOSJEC Engineer
Product Technical Support (APAC)
Sun Microsystems