Eric Schrock's Blog

Solaris Paleontology

August 14, 2004

In the footnote a few days ago, I commented on the fact that the history of Solaris debugging could rougly be divded into three ‘eras’. As someone interested in UNIX history, I decided to dig through the Solaris archives and put together a chronology of Solaris debuggability and observability tools. For fun, I divided it into eras to parallel Earth’s history. And I swear I’m not out to make anyone feel like a dinosaur (or a prokaryote, for that matter).

I’ve only been around for one of these “dawn of a new era” arrivals, DTrace1. When one of these revolutionary tools arrive, it’s amazing to see how quickly engineers avoid their own past. Try asking Bryan to debug a performance problem on Solaris 9, and you’ll probably get some choice phrases politely explaining that while he appreciates the importance of your problem, he would rather throw himself down a slide of broken glass and into a vat of rubbing alcohol. Being the neophyte that I am, I’ve only ventured into the ‘Paleozoic era’ on one occasion. After an MDB session on a Solaris 8 crashdump (paraphrased slightly):

$ mdb 0
> ::print
mdb: invalid command '::print': unknown dcmd name
> ::help print
mdb: unknown command: print
> ::please print
mdb: invalid command'::please': unknown dcmd name
> ::ihateyou

I quickly ran away screaming, never to return. I think I ended up hiding in a corner of my office for two hours, cradling my DTrace answerbook and whispering “there’s no place like home” over and over. I’m still a spoiled brat, but at least I have respect and admiration for those Solaris veterans who crawled through debugging hell so that I could live a comfortable life2. It’s also made me feel sorry for the Linux (and Windows) developers out there. Not in the Nelson Muntz “Ha ha! You don’t have DTrace!” sense. More like “Poor little guy. It’s not his fault his species never evolved opposable thumbs.” There are a lot of brilliant Linux developers out there, stuck in a movement that doesn’t embrace debugging or observability as fundamental goals. But this post is supposed to be about history, not Linux. So without further ado, my brief history of Solaris (soon to be available in refrigerator magnet form):

<1989 HADEAN SunOS 4.X
adb, ptrace, crash
1990 ARCHAEAN SVr4 merge
1991 vtrace
1992 SOLARIS 2.0
1993 mpstat
1994 Kernel slab allocator
basic ptools
1996 SOLARIS 2.5.1
PROTEROZOIC Next generation /proc
Userland watchpoints
1997 lockstat(1M)
pkill and pgrep
1998 savecore on by default
1999 libproc for corefiles
lockstat kernel profiling
2000 SOLARIS 8
EOL of crash(1M)
2001 live process control for MDB
EOL of adb(1)
pargs and preap
MESOZOIC kernel CTF data
2002 SOLARIS 9
libumem(3LIB) and umem_debug(3MALLOC)
::typegraph for mdb(1)
2003 Userland CTF
coreadm(1M) content control
2004 DTrace pid provider for x86
pfiles with pathnames
DTrace sched, proc providers
CTF for core libraries
DTrace I/O provider
DTrace MIB, fpuinfo providers
Per-thread ptools

These are my choices based on SCCS histories and putback logs. Obviously, I’ve failed to include some things. Leave a comment or email if you think something’s not getting the recognition it deserves (keeping in mind this is a blog post, not a book).

1 I actually started exactly one day before DTrace integrated. But I had some experience (albeit limited) as an intern the previous year.

2 In all seriousness, it’s not that I don’t have to ever debug anything, or that the problems we have today are somehow orders of magnitude simpler than those in the past. What these tools provide is a dramatic reduction in time to root-cause. You still need the same inquisitive and logical mind to debug hard problems, its just that good tools let you form questions and get answers faster than you could before. Really good tools (like DTrace) let you ask the previously unanswerable questions. You may have been able to debug the problem before, but you would have ended up running around in circles trying to get data that’s now immediately available thanks to DTrace.

5 Responses

  1. The lack of those commands in earlier Solarises (Solarii?) is one of the reasons that we find Solaris CAT so useful in addition to the other tools available. I too love the debugging features available in Solaris 10, however I don’t have the luxury of not looking at problems in earlier systems (given that it’s what they pay me for). Solaris CAT probably belongs in that timeline somewhere. I know that we were using it internally long before 4.0 was released.
    That being said, it’s a rare day when you don’t hear me say something like, “if we could dtrace this, we’d have the answer by now”.

  2. I’m just relieved to see that crash(1M) (deceased) didn’t make the list. Or perhaps Mike ripping out crash(1M) in June 2000 should be marked as progress? Regarding CAT: it’s fair to keep it (and its ilk) off the list; Eric’s history is only of tools in the OS, not of tools layered on top of it.

    One milestone that probably should be added, however, is Bonwick’s kmem debugging facilities circa 1994. (And likewise, umem_debug(3MALLOC) in 2002.) And were I not so modest and self-effacing, I would probably also argue for ::findleaks (1998, shipped 2000) and ::typegraph (2003). 😉

  3. The EOL of crash(1M) was in my notes but never made it to the final list; I’ve added it along with the EOL of adb for good measure. I totally forgot about libumem (and the original kmem allocator). I also threw in ::typegraph since it’s saved me a few times.
    As Bryan points out, there are many useful tools like CAT, dbx, and lsof which won’t make this list simply because they’re not part of the OS.

  4. I’ll lose the modesty: you’ve got to add ::findleaks. ::findleaks has found more bugs in the system than any single one of those technologies — over 250 bugs to date! On the one hand, it’s not nearly as radical as something like DTrace — but it has made a substantial, quantifiable difference in the quality of software that we deliver. And thanks to libumem (and to the work of people like Alan to document its capabilities), ::findleaks is now finding lots of bugs outside Solaris as well. It may be hard for us to remember, but before ::findleaks, memory leaks were actually difficult to track down.

    Okay, now back to my regular modest, self-effacing self… 😉

Recent Posts

April 21, 2013
February 28, 2013
August 14, 2012
July 28, 2012