What is DTrace? in a nutshell?

Question

G'Day Folks,
It's great to see Sun making this open source, however I imagine many new people will wander to this forum to figure out what DTrace is. So lets discuss it.
I'd describe DTrace as a mix of,
.... 1. truss, struss, apptrace
.... 2. mdb -p, mbd -k
.... 3. C, awk
.... 4. ???
1. truss, struss, apptrace: These tools can probe system calls, library functions, and user level functions. They have been crucial for Unix troubleshooting, but have had their quirks. eg, how do we truss a syscall if we don't know the program or PID? Or, if we want to get serious about processing the output, we'd better know awk or Perl. Ok, so DTrace does all that.
2. mdb -p, mdb -k: mdb was the super adb enhancement; these tools can navigate user space or kernel space, walking down structures and translating symbols. DTrace does that stuff too.
3. C, awk: The language is somewhat event based like awk (but probes not lines), with the syntax of C. comfy stuff.
4. ???: Dynamically probe whatever we feel like with minimal impact on the system, including user level functions down to nanosecond timestamps, stack backtraces and user variables; and even user level functions down to nanosecond timestamps, stack backtraces and user variables (I thought I'd better say that twice).
Ok - if anyone else can come up with a nice summary for what DTrace is, lets hear it - post to the forum!
I think it's quite hard to describe DTrace to someone who hasen't heard of it before - so it would be interesting if someone can come up with a neat summary in a sentence or two.
... or maybe we should just stick to the intro from the answerbook,
http://docs.sun.com/app/docs/doc/817-6223/6mlkidlf1?a=view
Brendan Gregg
[Sydney, Australia]

Answer 1

I think it's quite hard to describe DTrace to someone
who hasen't heard of it before - so it would be
interesting if someone can come up with a neat
summary in a sentence or two.
... or maybe we should just stick to the intro from
the answerbook,
http://docs.sun.com/app/docs/doc/817-6223/6mlkidlf1?a=
view
Brendan Gregg
[Sydney, Australia]Well, I'll try to give my summary, although I think the the documentation is self-explanatory
DTrace is tightly coupled with the Solaris 10 kernel; together, this system allows for an event-driven model. The kernel has a lot of points to which you can dinamically attach - you ask to be notified when that event happens (in DTrace these points are called probes and the event handler that you associate with a certain event is called "probe clause" ); when that point in kernel is reached (for example: a system call is invoked) your event handler is "called". The nice thing is that you don't have to periodically poll for events - you are notified just as events happen and you receive these notifications "straight from the source" - from inside the kernel.
You then need to use your imagination (and Solaris knowledge) to relate different events in order to obtain the aggregate data you're interested in; for this you mainly "tag" the process (actlually a thread within a process) that was on the CPU at the time the probe fired (see thread-local variables in the documentation)
That's how, when an event handler (DTrace probe clause) is invoked you can identify if the thread has previously reached another probe of interest for your DTrace script. You can also print stack traces to see what the thread "was doing" at the time the event happened (in DTrace terminologi - "the probe fired" )
You can watch processes, function entry and return points and even ask to be notified when a certain assembler instruction (part of a function in a process) is reached.
That's a very simplified view of DTrace; I've used an event-driven analogy because for me it was helpful to think that way.
Go check the http://docs.sun.com/db/doc/817-6223 manual and start playing with the many examples there; this should give you a taste of what you can do.
Go see the the nice examples at http://users.tpg.com.au/adsln4yb/dtrace.html
Vlad.