Approachability of DTrace

I was lucky enough to be the first user of DTrace outside of Sun, and started cutting my teeth on real-world applications in the 2003 time frame. During these past 5+ years, I have had the opportunity to look at hundreds of applications and systems. In addition, I have done training, seminars, presentations, and boot camps. This has given me opportunities not only to interact with systems, but also with those that maintain and develop them.

One common comment that has been heard since the beginning of time (for me, that is when DTrace shipped) is that DTrace is only for [developers | kernel engineers | people who read Solaris Internals cover to cover]. Many believe that without a full understanding of the Solaris kernel, you can never be a true practitioner of DTrace. In fact, the final straw that prompted me to write this blog entry can be found in the comments of one of Bryan's recent (and as always insightful) blog entries. Look specifically at "UX-adm,in  on July 02, 2008 at 01:35 AM PDT" and "Brendan Gregg  on July 02, 2008 at 03:01 PM PDT" (ignore all the licensing issue nonsense. If you are an attorney and reading this blog, you need to be disbarred). Well, I am here to tell you that this is completely false. Let me explain.

When I do any form of presentation (i.e. not looking at real customer applications), I start with a discussion of DTrace's origins, syntax, and then lastly examples. Now, by the time I get to examples, we are way past slideware and on to the command line. I can't tell you how many times I have had people in the audience catch syntax errors I have made before I finish the DTrace invocation - in an hour presentation. Therefore, I believe that the syntax is no barrier to entry.

Secondly, as I said in my first blog entry at Forsythe, about 90% of the time I have only looked at the application. I didn't even use the syscall or fbt providers - pure userland. So clearly, DTrace is for more folks than just kernel engineers. Ahh, but that means you have to be a developer, right? Nope. In this case, you just have to know the developer. Be it internal to your company, Sun, Oracle, or another vendor; you just need to know who to send your findings. I will go into more details about how to do this in future entries (by tomorrow I hope), but rest assured, it is quite simple to get information that will help developers understand where to look at their code.

Now for the pure SA play. The system is yours and you should know what is happening on it. Granted, you may not know the implementation details, but certainly you know what your system should be doing. If all else fails, fall back on one of the most powerful (and simple) DTrace one-liners imaginable - get a kernel profile:

# dtrace -n profile-303'/arg0/{@[stack()] = count()}END{trunc(@, 30)}'

Let's say, for example, that you see that your top stack includes page_get_contig_pages(). Well, just go to bugs.opensolaris.org and enter that as a search term. You might find that you are hitting CR 5005024. No rocket science here and no kernel knowledge required. Even if you get no hits searching the open bug list, providing this information to Sun could result in a significantly shorter root cause time. The point here is that you don't have to know the universe of possible issues, simply how to identify possibilities using the most powerful software observability tool on Earth. I, for one, have no idea what the heck all of those obfuscated Oracle function names mean, but I have been able to file Oracle TAR's with the DTrace data I have collected that have led to patches being created.

Now, to get back to Brendan Gregg's point about DTrace training in the comments on Bryan's blog entry - I feel that the best way to learn DTrace is in using DTrace. Sun has a great course on DTrace that covers all of the basic concepts, but the approach and methodology behind the appropriate application of DTrace is the real enabler. At Forsythe, we have a training regimen that addresses this specifically. We can come into your environment and train your staff using your own applications. This accomplishes two goals: 1) You get DTrace training and 2) You are left with a repository of DTrace scripts that are specific to your environment. Pretty nifty. I don't want to sound like a commercial, but I think that this is one of the core strengths that we have at Forsythe (that can't be found anywhere else) and would benefit many of those that feel overwhelmed by the power of DTrace. Contact us if you want more information.

I am not saying that being successful with DTrace is easy. What I am saying is that you don't need to be a subject matter expert on all things Solaris and application development (although it doesn't hurt). DTrace is just the simplest and most powerful way to shorten the diagnosis loop to allow customers to solve problems quickly in production environments. Be afraid of Bungee jumping, not DTrace.