Use tasks to stop a thundering herd in Puppet📍

If Puppet is experiencing performance issues, it could be that too many nodes are checking in simultaneously. You can use tasks to stop a thundering herd on Unix (including MacOS) and Windows nodes. On targeted agent nodes, these tasks stop puppet agent service and restart it, delaying agent check-in for each node from anywhere between 1 second up to the configured runinterval.

Note: Each task might up to the time of the runinterval to complete. Each node will be offline for 10 or 15 seconds while the task is running.

Additional information

Determine a thundering herd condition in Puppet

Version and installation information

Puppet version: 6.0 and later

Solution

Before you begin:

• Install the puppetlabs-support_tasks module.
• To use the tasks in the puppetlabs-support_tasks module with Puppet, install Bolt on a workstation.

Learn more about:

• Installing modules using a Puppetfile
• Running tasks with Bolt
• The support_tasks module

1. Run the task st0346_herd_resolver on the target nodes. For example:

   bolt task run support_tasks::st0346_herd_resolver --targets opensource-centos.puppetlabs.net,opensource-ubuntu.puppetlabs.net,opensource-win.puppetlabs.net

To prevent future thundering herds:

Note: Choose only one of these solutions.

• If you’re using Puppet 5.3 or later, use the solution in Prevent a thundering herd: Use max-queued-requests.
• If you’re able to use Cron, you can spread out agent catalog requests by running Puppet out of Cron.