If Puppet is experiencing performance issues, it could be that too many nodes are checking in simultaneously. You can use tasks to stop a thundering herd on Unix (including MacOS) and Windows nodes. On targeted agent nodes, these tasks stop puppet agent service and restart it, delaying agent check-in for each node from anywhere between 1 second up to the configured runinterval.
Note: Each task might up to the time of the runinterval to complete. Each node will be offline for 10 or 15 seconds while the task is running.
Additional information
Determine a thundering herd condition in Puppet
Version and installation information
Puppet version: 6.0 and later
Solution
Before you begin:
- Install the
puppetlabs-support_tasks
module. - To use the tasks in the
puppetlabs-support_tasks
module with Puppet, install Bolt on a workstation.
Learn more about:
-
Run the task
st0346_herd_resolver
on the target nodes. For example:bolt task run support_tasks::st0346_herd_resolver --targets opensource-centos.puppetlabs.net,opensource-ubuntu.puppetlabs.net,opensource-win.puppetlabs.net
To prevent future thundering herds:
Note: Choose only one of these solutions.
- If you’re using Puppet 5.3 or later, use the solution in Prevent a thundering herd: Use
max-queued-requests
. - If you’re able to use Cron, you can spread out agent catalog requests by running Puppet out of Cron.
Comments
0 comments
Please sign in to leave a comment.