sample is a program to take periodic stack traces of all processes
running on the system.  It then presents the stack history and
combined count.  (This is modeled on the Mac OS X programs sample
and spindump.)

There are two parts to the program:  the device driver, and the
user-mode program.  In retrospect, the name "sample" may be a poor
choice here, as seeing "sample_driver.ko" in the list of loaded kernel
modules does not really convey information about the driver.

The device driver creates a node in /dev, called "sample"; it uses an
ioctl to start the sampling process.  The command specifies number of
samples to take, and the duration between them.

Internally, the driver then schedules a number of callout(9) tasks --
one for each CPU, and then an additional one.  The CPU-specific ones
examine the thread that is running on that CPU at that time; the
remaining one examines the stacks for all sleeping threads on the
system.

When finished, the user-mode program, sample, then collects the data
and presents it.  It also includes a list of executable files mapped
into each process' space (this includes the main executable, as well
as shared libraries).  With the -S option, it will also attempt to
symbolicate.

To build the user-mode program, you need to have libbfd installed;
this also then requires libiberty.  It builds the program staticly
linked, so they are not required to run.  (libbfd could be avoided
with a compile-time flag to disable symbolication, however.)
Otherwise, to build, just

	make

To use it, you must first load the device driver.  E.g.:

	sudo kldload ./driver/sample_driver.ko

After that, you can then run the sample program (which must be run
as root):

	sudo ./sample/sample -n 100 -S > /tmp/sample.txt

which will take 100 samples, at 10ms, and attempt to symbolicate
them.

WARNING

This consists of a work-in-progress KERNEL MODULE.  As such, the
system may panic, or deadlock.  Use at your own risk.

The symbolication does not do the right thing at all for kernel
modules.  And since the system is installed stripped, there are no
symbols for most programs.

Currently only amd64 support is provided.

HOW TO READ THE OUTPUT

First up is the process, which includes the process ID, name, and the
path.  After that, it says how many samples were taken.  N.B.:  this
is the _total_ number of samples taken for the process, so it is the
sum of all the samples for all of the threds.  (In the case, below,
sample only has one thread.)

After that is a list of each thread, and then the samples.  Each
sample line output has the format:

	<number> <addresss> (<file> + <offset>) [(<file>:<symbol> + <offset>)]

The last set is optional, and only if -S is given.

The line is indented to indicate stack depth.

	Process 31049 (sample, pathname /home/sef/src/sample/sample/sample):
	101 samples

	Thread ID 100913
	   100 0x40030f (/home/sef/src/sample/sample/sample + 783) (/home/sef/src/sample/sample/sample:_start + 367)
	    99 0x72709a (/home/sef/src/sample/sample/sample + 3305626) (/home/sef/src/sample/sample/sample:_read + 10)
	     99 0xffffffff80d18efb (/boot/kernel/kernel + 11636475) (/boot/kernel/kernel:Xfast_syscall + 251)
	      99 0xffffffff80d33e3c (/boot/kernel/kernel + 11746876) (/boot/kernel/kernel:amd64_syscall + 844)
	       99 0xffffffff80993aa3 (/boot/kernel/kernel + 7944867) (/boot/kernel/kernel:sys_read + 99)
		99 0xffffffff80993b18 (/boot/kernel/kernel + 7944984) (/boot/kernel/kernel:kern_readv + 104)
		 99 0xffffffff80993df5 (/boot/kernel/kernel + 7945717) (/boot/kernel/kernel:dofileread + 149)
		  99 0xffffffff8082d7d8 (/boot/kernel/kernel + 6477784) (/boot/kernel/kernel:devfs_read_f + 216)
		   99 0xffffffff81e11289 (./driver/sample_driver.ko + 649)
		    99 0xffffffff80949d8d (/boot/kernel/kernel + 7642509) (/boot/kernel/kernel:_sleep + 637)
		     99 0xffffffff80988cbf (/boot/kernel/kernel + 7900351) (/boot/kernel/kernel:sleepq_wait_sig + 15)
		      99 0xffffffff80988e0b (/boot/kernel/kernel + 7900683) (/boot/kernel/kernel:sleepq_catch_signals + 171)
		       99 0xffffffff8094a371 (/boot/kernel/kernel + 7644017) (/boot/kernel/kernel:mi_switch + 225)
	    1 0x7050ea (/home/sef/src/sample/sample/sample + 3166442) (/home/sef/src/sample/sample/sample:_nanosleep + 10)
	     1 0xffffffff80d18efb (/boot/kernel/kernel + 11636475) (/boot/kernel/kernel:Xfast_syscall + 251)
	      1 0xffffffff80d33e3c (/boot/kernel/kernel + 11746876) (/boot/kernel/kernel:amd64_syscall + 844)
	       1 0xffffffff80953c61 (/boot/kernel/kernel + 7683169) (/boot/kernel/kernel:sys_nanosleep + 81)
		1 0xffffffff80953b7e (/boot/kernel/kernel + 7682942) (/boot/kernel/kernel:kern_nanosleep + 270)
		 1 0xffffffff80949d48 (/boot/kernel/kernel + 7642440) (/boot/kernel/kernel:_sleep + 568)
		  1 0xffffffff80989340 (/boot/kernel/kernel + 7902016) (/boot/kernel/kernel:sleepq_timedwait_sig + 16)
		   1 0xffffffff80988e0b (/boot/kernel/kernel + 7900683) (/boot/kernel/kernel:sleepq_catch_signals + 171)
		    1 0xffffffff8094a371 (/boot/kernel/kernel + 7644017) (/boot/kernel/kernel:mi_switch + 225)
	  1 0x72709a (/home/sef/src/sample/sample/sample + 3305626) (/home/sef/src/sample/sample/sample:_read + 10)
	   1 0xffffffff80d18efb (/boot/kernel/kernel + 11636475) (/boot/kernel/kernel:Xfast_syscall + 251)
	    1 0xffffffff80d33e3c (/boot/kernel/kernel + 11746876) (/boot/kernel/kernel:amd64_syscall + 844)
	     1 0xffffffff80993aa3 (/boot/kernel/kernel + 7944867) (/boot/kernel/kernel:sys_read + 99)
	      1 0xffffffff80993b18 (/boot/kernel/kernel + 7944984) (/boot/kernel/kernel:kern_readv + 104)
	       1 0xffffffff80993df5 (/boot/kernel/kernel + 7945717) (/boot/kernel/kernel:dofileread + 149)
		1 0xffffffff8082d7d8 (/boot/kernel/kernel + 6477784) (/boot/kernel/kernel:devfs_read_f + 216)

	Mapped Files:
		Start	End	File
		0x400000	0x8a2000	/home/sef/src/sample/sample/sample
		0xaa1000	0xac2000	/home/sef/src/sample/sample/sample