Mitigating a thundering herd: Run Puppet out of Cron 📍

I’ve determined that I have a thundering herd. I'd like to prevent future thundering herds using Cron.

Version and installation information

Puppet version: 5.0, 6.0
Installation type: All

Solution

Prevent thundering herds by disabling the puppet service and using a Cron job to run puppet agent -t on each agent at a time randomly chosen by the fqdn_rand() function. This solution keeps agent check-ins evenly distributed, even after the puppet agent service is restarted by rebooting agents or running mco puppet runonce.

Add the following Puppet code to a profile class to:

• Stop and disable the puppet service
• Install a Cron job that runs puppet agent -t using a randomized runinterval.

...
    $runinterval = 30 #minutes

    $first_run = fqdn_rand($runinterval)
    $second_run = $first_run + $runinterval

    cron { 'cron.puppet':
      command => "/opt/puppetlabs/bin/puppet agent -t > /dev/null",
      user => "root",
      minute => [ $first_run, $second_run ],
    }

    # make sure we haven't started the puppet daemon ever
    # this may cause the report on an agent that triggers this via a daemonized run to not be submitted

    service { 'puppet':
      enable => false,
      ensure => stopped,
      require => Cron['cron.puppet'],
    }
...

You can also have the reidmv/puppet_run_scheduler module do this for you. Follow the instructions in the module's README to configure it.