A real-world experience with sysdig

After a config change, I needed to debug these messages which began appearing in bursts in the Apache error logs:
Error opening file for reading: Permission denied
with no other information like filename or even timestamp. These occurred no more frequently than hourly. Using strace ran out of gas quickly because -etrace=file still had too much output to reasonably filter. Instead I tried out sysdig!

I began with a few of the examples from the sysdig site, and pretty quickly pinned it down to this command I left running until the next time the problem occurred:

sysdig -p "%12user.name %6proc.pid %8evt.res %20proc.name %3fd.num %fd.typechar %fd.name"
evt.res=EACCES or evt.res=EPERM and proc.name'!=irrelevantprocess' |
egrep --line-buffered -v '/foo/a-flag-file'

It specifically looks for no-permission events, and has filters to remove some noperm errors which turn out to be quite common: one from an irrelevant process, and a flagfile access by apache (which I couldn’t exclude by process name, since apache is what I was trying to debug!). The next time a burst of Permission denied errors showed up in the error log, sysdig showed these:
root 32060 EACCES /usr/sbin/apach -1 f /proc/self/auxv
with the various specific PIDs. And indeed:
# ls -l /proc/32060/auxv
-r-------- 1 root root 0 Dec 15 16:55 /proc/32060/auxv

After that, a quick web search revealed that apparently in 12.04, Ubuntu edited an imaging library so that it checks /proc/self/auxv for CPU capabilities, I presume so it can use certain CPU instructions if they are available.  Apache starts as root, forks, and much later the children are non-root by the time our code resizes an image or some such and libjpeg tries to look up the capabilities.

This is the sort of thing a sysadmin has to debug all the time, and certainly there are other ways to approach it, but sysdig turned out to be rapid, not frustrating to learn or use, and successful in debugging the issue. Recommended!

See also:
http://www.sysdig.org/
http://www.sysdig.org/wiki/sysdig-user-guide/
Installation instructions at http://www.sysdig.org/install/, (installation was easy and did not require a reboot)
and Gareth Rushgrove’s article from this year’s SysAdvent:
http://sysadvent.blogspot.com/2014/12/day-3-so-server-tell-me-about-yourself.html