Friday, January 3, 2014

Searching for the right Upstart signal or job

If you want to use Upstart to start/stop a job on any of the not-obvious triggers (like "startup"), then you need to do some digging to find the right trigger.


initctl show-config


Be careful, there are TWO sets of Upstart jobs: System jobs and user jobs. Use sudo to distinguish between them.

$ sudo initctl show-config dbus   # Use sudo for system-level jobs
dbus
  start on local-filesystems
  stop on deconfiguring-networking

$ initctl show-config dbus        # Omit sudo for user-level jobs
dbus
  start on starting xsession-init 


Searching for a job or a signal using grep


The initctl show-config command without any job name prints all the jobs. That means you can use grep on the full list. Here is an example of using grep to look for all root jobs that care about the system "startup" signal:

$ sudo initctl show-config | grep -B8 startup
  start on (starting mountall or (runlevel [016] and ((desktop-shutdown or stopped xdm) or stopped uxlaunch)))
resolvconf
  start on mounted MOUNTPOINT=/run
  stop on runlevel [06]
ssh
  start on runlevel [2345]
  stop on runlevel [!2345]
udev-fallback-graphics
  start on (startup and (((graphics-device-added PRIMARY_DEVICE_FOR_DISPLAY=1 or drm-device-added PRIMARY_DEVICE_FOR_DISPLAY=1) or stopped udevtrigger) or container))
--
mountall
  emits virtual-filesystems
  emits local-filesystems
  emits remote-filesystems
  emits all-swaps
  emits filesystem
  emits mounting
  emits mounted
  start on startup
--
acpid
  start on runlevel [2345]
  stop on runlevel [!2345]
checkfs.sh
  start on mounted MOUNTPOINT=/
checkroot-bootclean.sh
  start on mounted MOUNTPOINT=/
kmod
  start on (startup and started udev)
--
  start on runlevel S
  stop on runlevel [!S]
wait-for-state
  stop on (started $WAIT_FOR or stopped $WAIT_FOR)
flush-early-job-log
  start on filesystem
friendly-recovery
  emits recovery
  emits startup
--
  start on runlevel [2345]
  stop on runlevel [!2345]
socket-test
  start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1
tty2
  start on (runlevel [23] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!23]
udevtrigger
  start on ((startup and started udev) and not-container)
--
  emits not-container
  start on mounted MOUNTPOINT=/run
mounted-dev
  start on mounted MOUNTPOINT=/dev
tty3
  start on (runlevel [23] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!23]
udev-finish
  start on ((((startup and filesystem) and started udev) and stopped udevtrigger) and stopped udevmonitor)
alsa-state
  start on runlevel [2345]
cryptdisks-udev
  start on block-device-added ID_FS_USAGE=crypto
hostname
  start on startup
--
network-interface
  emits net-device-up
  emits net-device-down
  emits static-network-up
  start on net-device-added
  stop on net-device-removed INTERFACE=$INTERFACE
plymouth-ready
  emits plymouth-ready
  start on (startup or started plymouth-splash)
--
  start on (started plymouth and ((graphics-device-added PRIMARY_DEVICE_FOR_DISPLAY=1 or drm-device-added PRIMARY_DEVICE_FOR_DISPLAY=1) or stopped udev-fallback-graphics))
plymouth-upstart-bridge
  start on (started dbus or runlevel [06])
  stop on stopping plymouth
tty1
  start on (stopped rc RUNLEVEL=[2345] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!2345]
udevmonitor
  start on (startup and starting udevtrigger)

We found one job that emits startup (friendly-recovery).
We found seven jobs that listen for it: udev-fallback-graphics, mountall, kmod, udevtrigger, hostname, plymouth-ready, and udevmonitor


Searching for a signal using upstart-monitor


The upstart-monitor application is a handy GUI and command-line tool to listen to all the signal chatter in Upstart. The application is provided by the upstart-monitor package in the Ubuntu repositories. A bug in 13.10 prevents it from running on a non-GUI system like Ubuntu Server, but it's also easy to fix the bug yourself...

Here are the signals emitted by Upstart when I switch over to a TTY, login, wait ten seconds, and then logout. This isn't an example of monitoring logins (do that using consolekit or logind) - this is an example of monitoring the Upstart signals emitted by a change in tty2.

$ upstart-monitor --no-gui --destination=system-bus
# Upstart Event Monitor (console mode)
#
# Connected to D-Bus system bus
#
# Columns: time, event and environment

2014-01-03 23:23:43.013436 stopping JOB='tty2' INSTANCE='' RESULT='ok'
2014-01-03 23:23:43.020309 starting JOB='tty2' INSTANCE=''
2014-01-03 23:23:43.031193 starting JOB='startpar-bridge' INSTANCE='tty2--started'
2014-01-03 23:23:43.033055 started JOB='startpar-bridge' INSTANCE='tty2--started'
2014-01-03 23:23:43.040671 stopping JOB='startpar-bridge' INSTANCE='tty2--started' RESULT='ok'
2014-01-03 23:23:43.042496 stopped JOB='startpar-bridge' INSTANCE='tty2--started' RESULT='ok'
2014-01-03 23:23:43.044271 started JOB='tty2' INSTANCE=''
^C

You can see the progression of signals: starting, started, stopping, stopped.
You can also see the nesting of jobs. startpar-bridge starts on starting tty2, and runs it's entire task of starting-started-stopping-stopped for tty2 to transition from starting to started.

If you want to trigger a job when tty2 is starting or started, you now know the signals that get emitted. Your job can listen for those signals.


Drawing out relationships using dotfiles


Dot diagram of Upstart user jobs
The initctl2dot application creates dotfile graphics of xdot application. initctl2dot is included with the upstart package, part of all Ubuntu installations (even ubuntu-minimal). xdot is a separate package available in the Ubuntu repositories (Software Center).


As the name implies, initctl2dot's input is initctl's output.You can manually trim an initctl show-show-config output, and input that to initctl2dot if you really want a specific diagram.

You can easily diagram and display the entire system job tree...though it's perhaps less useful than you may expect:

$ initctl2dot --system --outfile /tmp/upstart_root_tree.dot
$ xdot /tmp/upstart_root_tree.dot


You can also diagram the user job tree:

$ initctl2dot --user --outfile /tmp/upstart_user_tree.dot
$ xdot /tmp/upstart_user_tree.dot

Limiting the dotfile size


The initctl2dot manpage includes options for showing/hiding various relationship types (emit, start on, stop on, etc) for clarity.

Another handy option is the --restrict-to-jobs flag, to draw much smaller charts.

For example, let's diagram the system "startup" signal relationships we already discovered using grep:

$ initctl2dot --system --outfile /tmp/upstart_startup_tree.dot \
              --restrict-to-jobs=friendly-recovery,udev-fallback-graphics,\
                                 mountall,kmod,udevtrigger,hostname,\
                                 plymouth-ready,udevmonitor 
$ xdot /tmp/upstart_startup_tree.dot 


And there you have it. How to search system jobs and user jobs for useful signals, and how to easily diagram out the relationships among signals and jobs.

No comments: