Iโve determined that I have a thundering herd. I'd like to prevent future thundering herds using Cron.
Version and installation information
Puppet version: 5.0, 6.0
Installation type: All
Solution
Prevent thundering herds by disabling the puppet
service and using a Cron job to run puppet agent -t
on each agent at a time randomly chosen by the fqdn_rand()
function. This solution keeps agent check-ins evenly distributed, even after the puppet
agent service is restarted by rebooting agents or running mco puppet runonce
.
Add the following Puppet code to a profile class to:
- Stop and disable the
puppet
service - Install a Cron job that runs
puppet agent -t
using a randomized runinterval.
...
$runinterval = 30 #minutes
$first_run = fqdn_rand($runinterval)
$second_run = $first_run + $runinterval
cron { 'cron.puppet':
command => "/opt/puppetlabs/bin/puppet agent -t > /dev/null",
user => "root",
minute => [ $first_run, $second_run ],
}
# make sure we haven't started the puppet daemon ever
# this may cause the report on an agent that triggers this via a daemonized run to not be submitted
service { 'puppet':
enable => false,
ensure => stopped,
require => Cron['cron.puppet'],
}
...
You can also have the reidmv/puppet_run_scheduler
module do this for you. Follow the instructions in the module's README to configure it.
Comments
0 comments
Please sign in to leave a comment.