Resolve Puppet run failures with "(SystemStackError) stack level too deep" errors for Puppet 4.10 and later 📍

When I run puppet agent -t, I get a (SystemStackError) stack level too deep error.

Error messages and logs

Error message:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep

Logs:

In the Puppet Server log: /var/log/puppetlabs/puppetserver/puppetserver.log

2015-11-02 11:17:10,496 ERROR [p.p.ringutils] Exception while handling HTTP request org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613) ~[puppet-server-release.jar:na] at org.jruby.RubyEnumerable.inject(org/jruby/RubyEnumerable.java:866)

You received this error because the puppetserver process exceeded the JVM stack size.

Version and installation information

Puppet version: 4.10 to latest

Solution

To resolve the error, increase the JVM stack size from 1MB (default) to 2MB by completing the following steps or by installing a task and using a module.

To increase the JVM stack manually:

1. Open /etc/sysconfig/puppetserver (on EL-based systems) or /etc/default/puppetserver (on Ubuntu). Add an -Xss2m option to the existing JAVA_ARGS, for example:

   JAVA_ARGS="-Xms<XMS SIZE> -Xmx<XMX SIZE> -Xss2m"

   Note: Add the -Xss option if it doesn’t exist, edit it if it already exists. No other changes to JAVA_ARGS are necessary.

2. Restart puppetserver:

   puppet resource service puppetserver ensure=stopped ; puppet resource service puppetserver ensure=running

3. Verify that the change resolved the error by running puppet agent -t.

To increase the JVM stack with a task:

Before you begin: Install the support_tasks module and Bolt on a workstation to run the tasks.

If you’re using a version of the puppetlabs-support_tasks module older than 1.1.1 (17 January 2020), please update the module to use these steps. We renumbered all the tasks in the module at that time.

1. Execute the task via Bolt, targeting your primary server: bolt task run support_tasks::st0149_resolve_stack_level_too_deep --targets neptune --modulepath ~/modules.

   The task will exit with one of the following:

   Message	Result
   Puppetserver node detected - followed by -ST#0149 Task ended	The task increased the Puppet Server’s JVM stack size to 2MB.
   Argument Already Present	A value is already configured for the JVM stack. Either this task has already been executed or this task will not resolve your issue.
   Not a Puppet primary server node exiting	The task was executed on a node that is not a Puppet Server, no change was made.

   For example, when the task changes the Puppet Server’s JVM stack size, the output is:

   -Puppetserver node detected - 
   EL Notice: /Service[puppetserver]/ensure: ensure changed ‘running’ to ‘stopped’ 
   service { ‘puppetserver’: 
     ensure => ‘stopped’, 
   } 
   Notice: /Service[puppetserver]/ensure: ensure changed ‘stopped’ to ‘running’ 
   service { ‘puppetserver’: 
     ensure => ‘running’, 
   } 
   -ST#0149 Task ended 1579883459 – 
   { 
   }

Learn more about:

• Installing modules using a Puppetfile
• Running tasks with Bolt
• The support_tasks module