The Sleeper Agent is a Python library that, when loaded, allows inspecting live state with gdb.
It supports CPython version 2.6 and 2.7.
The library is developed at https://github.com/mpasternacki/sleeper_agent
Its unit tests run automatically at http://travis-ci.org/mpasternacki/sleeper_agent
When initializing your Python code, just import the module:
import sleeper_agent
There is nothing more to do. The sleeper agent is loaded, and it does nothing.
When you want to peek into what your code is doing, run
sleeper_agent_activate PID to get backtrace of all its
threads. There are some options available; run
sleeper_agent_activate --help to see the docs.
Below is the example of activating the agent on an ipython
session. Note that one of the threads has the function
sleeper_agent._get_state_info in its stack trace - this is just
appended to this thread's stack, and the information is
harmless. There is also some diagnostic info from gdb that I wasn't
able to suppress - it is also harmless.
japhy@portinari:sleeper_agent 20% sleeper_agent_activate 18055
Attaching to process 18055.
Reading symbols for shared libraries . done
Reading symbols for shared libraries ........................................................................... done
0x00007fff97396df2 in select$DARWIN_EXTSN ()
### Thread 4326428672:
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 525, in __bootstrap
self.__bootstrap_inner()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "<string>", line 2, in run
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/history.py", line 61, in needs_sqlite
return f(*a,**kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/history.py", line 647, in run
self.history_manager.save_flag.wait()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 404, in wait
self.__cond.wait(timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 244, in wait
waiter.acquire()
### Thread 140735285004640:
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/ipython", line 8, in <module>
load_entry_point('ipython==0.12', 'console_scripts', 'ipython')()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 403, in launch_new_instance
app.start()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 377, in start
self.shell.mainloop()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 290, in mainloop
self.interact(display_banner=display_banner)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 368, in interact
line = self.raw_input(prompt)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 436, in raw_input
line = py3compat.str_to_unicode(self.raw_input_original(prompt))
File "sleeper_agent.py", line 15, in _get_state_info
for thread_id, frame in sys._current_frames().items() )
The library is composed of two pieces: a Python introspection module, and some C glue.
Python module, sleeper_agent.py, defines one function named
_get_state_info. This function takes no arguments, and returns
program state as string. Currently, the state is list of threads and
their respective backtraces.
Python module also imports the C module _sleeper_agent_activation to
load the glue.
The core of the C module is the char * sleeper_agent_state(void)
function. It acquires the GIL, calls out to
sleeper_agent._get_state_info Python function, and returns its
result as a C string, which can be then printed from gdb.
The C module also exports the sleeper_agent_state_pyobject function
to Python as sleeper_agent_state. This function calls the C
sleeper_agent_state function, and returns its result as a Python
string, thus completing the cycle.
So, a smoke check for particular pieces without actually resorting to use gdb would look like this:
>>> import sleeper_agent
>>> sleeper_agent._get_state_info()
'### Thread 140735285004640:\n File "<stdin>", line 1, in <module>\n File "sleeper_agent.py", line 15, in _get_state_info\n for thread_id, frame in sys._current_frames().items() )\n'
>>> sleeper_agent._sleeper_agent_activation.sleeper_agent_state()
'### Thread 140735285004640:\n File "<stdin>", line 1, in <module>\n File "sleeper_agent.py", line 15, in _get_state_info\n for thread_id, frame in sys._current_frames().items() )\n'
>>> sleeper_agent._sleeper_agent_activation.sleeper_agent_state() == sleeper_agent._get_state_info()
True
>>>
You can activate the agent manually by attaching gdb to the process
and calling out to the sleeper_agent_state() C function:
$ gdb -p PID
(gdb) printf "%s", sleeper_agent_state()
### Thread 4326428672:
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 525, in __bootstrap
self.__bootstrap_inner()
[...]
Some of the code and ideas have been inspired by Leonardo Rochael
Almeida's talk at Europython 2012,
sys._current_frames(): Take real-time X-rays of your software for fun and performance.
The returned info can be extended to include locals and globals on each call stack level.
The call info can also be used to perform statistical profiling, by semi-regularly dumping the stack somewhere to be analyzed later on.
Code can be expanded to interactively talk with Python and explore the stack levels. This would need storing the stack trace objects, and probably also some scripting on the gdb side to have a usable interface. Maybe some pieces of pdb code could be reused for that.