puppetlabs-puppet_metrics_collector module to diagnose common performance issues that can result in symptoms such as failed catalog compilation and/or a growing command queue in PuppetDB.
Version and installation information
Puppet version: 5.0, 6.0
Installation type: All
Note: Links to our documentation are for the latest release of Puppet. Use the version selector on our docs site to make sure you've got the right version of our docs for your deployment.
The first step to resolving a performance issue is to gather data on the performance of services. You can easily gather metrics from Puppet Server and PuppetDB by installing and using the
Note: If you already have the module installed, please update it to the latest version.
In the following sections, we'll take a look at key metrics for Puppet Server and PuppetDB and use them to optimize your deployment.
Learn more about configuring Grafana, Telegraf, and InfluxDB to use metrics from Puppet's services using the puppetlabs-puppet_metrics_dashboard module.
Puppet Server metrics
One of the most important metrics for Puppet Server is
average-free-jrubies. See that metric over time by running:
grep average-free-jrubies puppetserver/127.0.0.1/*.json
cd /opt/puppetlabs/puppet-metrics-collector grep average-free-jrubies puppetserver/127.0.0.1/*.json puppetserver/127.0.0.1/20170404T170501Z.json: "average-free-jrubies": 0.9950009285369501 puppetserver/127.0.0.1/20170404T171001Z.json: "average-free-jrubies": 0.9999444653324225 puppetserver/127.0.0.1/20170404T171502Z.json: "average-free-jrubies": 0.9999993830655706
average-free-jrubies is consistently below one, improve performance by increasing the
max-active-instances setting to increase the number of JRubies available to Puppet Server.
The default recommendation for this setting, CPUs-1, assumes that your master node is dedicated to running only Puppet Server. If Puppet Server and PuppetDB are on the same node, be careful not to set the number of JRubies so high that Puppet Server is competing with PuppetDB for resources. We recommend increasing
max-active-instances by one and watching CPU utilization in conjunction with Puppet Server metrics. Depending on how many CPUs are allocated to your master, you might need to increase the number of available CPUs before you can further increase JRubies.
When you increase the number of JRubies, increase the heap size of Puppet Server linearly using the instructions in our documentation.
|Number of JRubies||Java heap||Number of JRubies||Java heap|
|3||2GB||4||2.66GB = 2GB + (2GB/3 JRubies)|
|6||5GB||7||5.83GB = 5GB + (5GB/6 JRubies)|
average-free-jrubies is occasionally (but not consistently) dropping to zero, you probably have a thundering herd. Read more about:
- Detecting a thundering herd
- Preventing thundering herd completely by running puppet out of Cron
- Prevent a thundering herd: Use
One of the most important metrics for PuppetDB is
queue_depth, also referred to as the Command Queue in the PDB dashboard. See that metric over time by running:
grep queue_depth puppetdb/127.0.0.1/*.json
cd /opt/puppetlabs/puppet-metrics-collector grep queue_depth puppetdb/127.0.0.1/*.json puppetdb/127.0.0.1/20170404T170501Z.json: "queue_depth": 0 puppetdb/127.0.0.1/20170404T171001Z.json: "queue_depth": 0 puppetdb/127.0.0.1/20170404T171502Z.json: "queue_depth": 0
queue_depth metric should stay around zero, or slightly above zero in busy systems. The
queue_depth can ebb and flow, but it should not show a pattern of consistently increasing. If the
queue_depth is persistently larger than 100 and/or is continually growing, then increase your command processing threads.
Command processing threads defaults to CPUs/2. If PuppetDB is on its own node, increase the number of command processing threads. If PuppetDB is on the same node as the master, increase the CPUs available to the master to provide enough resources to increase the number of command processing threads.