Advanced troubleshooting: dig deeper with pry

Sometimes, reading the debug logs doesn’t give you enough information about what’s going on. You may need to pry into the parts of Puppet that happen after the execution starts, but before the PowerShell code is executed.

Pry is a powerful tool for interactively debugging ruby. Using it with Puppet on Windows is a bit involved but it can be an extremely useful tool in your kit.

Set up Ruby and Puppet for debugging

First, you’ll want to get a functional ruby environment installed separate from the PDK.

Right now, the PDK does not support prying into a Puppet run due to limitations around native gems. If you don’t know what this means right now, don’t worry too much. As you get more used to how Ruby and Puppet works it will make more sense.

You can accomplish this however you want, but make sure there is an installation of both Ruby and the associated Devkit installed when you are done. We suggest you do this via Glenn Sarti’s RubyInstaller PowerShell module:

Install-Module RubyInstaller -Scope CurrentUser
# This will kick off an interactive installation of Ruby, including the necessary dependencies
# It relies on Chocolatey (and will install it if needed)
# Strongly suggest installation option G ('2.5.1-2 (x64)') at the time of this writing.
Install-Ruby

The next step is to find the module files on disk. They’re probably in a location like this one:

C:/ProgramData/PuppetLabs/code/environments/<your environment name>/modules/

The actual location will depend on how you configured your Puppet Environments.

Next, you’re going to Set-Location to the module you want to debug, like so:

Set-Location 'C:/ProgramData/PuppetLabs/code/environments/production/modules/powershellget'

We strongly suggest opening this location in VSCode to make editing a bit easier on yourself.

In any case, you will want to delete the file Gemfile.lock in this directory and create a new file, Gemfile.local, with the following content:

gem 'fuubar'
gem 'pry-byebug'
gem 'pry-stack_explorer'

Next, you will run the following commands:

• uru 2.5 - this loads ruby 2.5 for use. If you don't do this, you won't be able to run other commands.
• gem install bundler - this is only necessary after you first install a ruby version; it's a tool to manage ruby dependencies
• bundle config set path '.bundle/gems' - this will cause bundler install all of the ruby dependencies into the folder you're currently in.
• bundle install - this will install all of your ruby dependencies.
• bundle exec rake spec_prep - this will pull down a copy of the `pwslib` module for use and editing in the rest of your debugging.

At this point, you have a functioning Ruby installation and a Puppet Module setup and ready to be debugged.

Add your test manifest code

The next step is to create a new file in the module folder at examples/test.pp and put your test manifest code in there.

Now, you can run your test manifest with the following command:

bundle exec puppet apply ./examples/test.pp --modulepath ./spec/fixtures/modules

If you want to run in debug mode, append --debug to the end of that command. The output you saw during a normal Puppet run will show, and the command runs until it stops at the error you are currently investigating.

Now, you’re ready to start adding pry statements to the base provider and digging around.

Pry into the base provider

For the rest of this section we’ll be looking at the dsc_base_provider.rb file in the puppetlabs-pwshlib module, which you can find at ./spec/fixtures/modules/pwshlib/puppet/provider/dsc_base_provider in the module folder. That file is the shared provider that all Puppetized DSC Resources rely on.

For this brief tutorial, we’re going to pry into invoke_get_method, but these techniques apply to all of the included methods you may want to debug.

First, find the method in the provider file (at the time of this writing, that’s around line 231):

def invoke_get_method(context, name_hash)
  ...
end

We’re going to add a pry binding to this method. For the purposes of this tutorial, we’re going to add it just before the DSC invocation (around line 241 at the time of this writing):

context.debug("Script:\n #{redact_secrets(script_content)}")
# This is the new code we're adding, which will tell ruby to enter a pry debugging session:
require 'pry' ; binding.pry
output = ps_manager.execute(script_content)[:stdout]

This will get you an interactive prompt inside the Puppet run:

You can call ls in that context to see what methods and variables are available:

You can type the name of one of the variables, such as name_hash and hit enter to see what it is set to:

You can use the whereami command to show where you are in the code. If you pass an integer to this command, it will show you that many lines around your current line.

Next, we’ll execute the invocation script and assign it to our own variable (this will take a bit as it’s spinning up PowerShell for the first time and running Invoke-DscResource):

In my case, it looks like we got some data for stdout, an exit code of zero, and no errors.

If we call next, pry will execute the next line of code.

Note that the marker for our current line has updated to 243. We can keep using next to walk through the code execution, investigating any variables or running any methods for ourselves to see. I’ll keep doing that til we hit line 267, where the code manipulates the output data a little bit:

Before we run these lines, lets check on some state:

Okay, so that shows us the current state of the data, the list of valid attributes, and the parameters respectively. We can then run next and check the value of data again to see what the data.select! line did:

It looks like the list of key-value pairs has been trimmed down - we do this so only the values that Puppet cares about will be evaluated in the rest of the Puppet run.

If we use next again and look at data a final time:

We see that the key for PSDscRunAsCredential has been removed, which happens because it was the only key which was in the list of parameters.

This brings us to a loop in the code. If we want to see what happens inside the loop, we need to use the step command. This command also lets you shift contexts inside of any method being called.

Note that we’re now inside the loop - if we had called next, it would’ve evaluated the whole loop and gone to the next line after it.

If we want to let Puppet run its course, we can use the continue command - this will keep us in the pry session, looking for the next pry.binding in our code. Right now we only have the one, but you can have any number of them.

If instead we want to exit the puppet run entirely, we can use the exit! command, which will bring you back to your PowerShell prompt.

And that’s it for basic pry debugging! A few commands and tools to help you look around and try stuff out to investigate where you think something has gone wrong.

Additional commands to use with pry

Here are a few methods you may want to investigate with a pry binding and reasons for that:

• canonicalize - this method retrieves the state of the DSC Resource on the node, if it exists, and loads it into a cache for comparing later. If something goes wrong here, chances are you’ll get idempotency issues.
• invoke_get_method - this is responsible not just for querying the state of the DSC Resource but also for puppetizing the returned data
• ps_script_content - this method builds the PowerShell script that gets invoked to query or set the state of a DSC Resource. The methods included in it, particularly prepare_credentials, prepare_cim_instances, and invoke_params are where things are most likely to go wrong when turning Puppet code into PowerShell code.