Buildbot and EOFError

More SEO-bait, after tracking down an poorly documented problem:

# buildbot start master
Following twistd.log until startup finished..
2016-01-17 04:35:49+0000 [-] Log opened.
2016-01-17 04:35:49+0000 [-] twistd 14.0.2 (/usr/bin/python 2.7.6) starting up.
2016-01-17 04:35:49+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-01-17 04:35:49+0000 [-] Starting BuildMaster -- buildbot.version: 0.8.12
2016-01-17 04:35:49+0000 [-] Loading configuration from '/home/user/buildbot/master/master.cfg'
2016-01-17 04:35:53+0000 [-] error while parsing config file:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/", line 1155, in gotResult
        _inlineCallbacks(r, g, deferred)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/", line 1099, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/lib/python2.7/dist-packages/buildbot/", line 189, in startService
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/buildbot/", line 156, in loadConfig
        exec f in localDict
      File "/home/user/buildbot/master/master.cfg", line 415, in <module>
        extra_post_params={'secret': HOMU_BUILDBOT_SECRET},
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/", line 404, in __init__
        secondaryQueue=DiskQueue(path, maxItems=maxDiskItems))
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/", line 286, in __init__
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/", line 208, in popChunk

2016-01-17 04:35:53+0000 [-] Configuration Errors:
2016-01-17 04:35:53+0000 [-]   error while parsing config file:  (traceback in logfile)
2016-01-17 04:35:53+0000 [-] Halting master.
2016-01-17 04:35:53+0000 [-] Main loop terminated.
2016-01-17 04:35:53+0000 [-] Server Shut Down.

This happened after the buildmaster’s disk filled up and a bunch of stuff was manually deleted. There were no changes to master.cfg since it worked perfectly.

The fix was to examine master.cfg to see where the HttpStatusPush was created, of the form:

    extra_post_params={'secret': HOMU_BUILDBOT_SECRET},

Digging in the Buildbot source reveals that wants to unpickle a cache file from / if there was nothing in / To fix this the right way, create that file and make sure Buildbot has +rwx on it.

Alternately, you can give up on writing your status push cache to disk entirely by adding the line maxDiskItems=0 to the creation of the HttpStatusPush, giving you:

   extra_post_params={'secret': HOMU_BUILDBOT_SECRET},

The real moral of the story is “remember to use logrotate.