Extend the expiry date of your CA certificate in Puppet 5.0 and later📍

When the expiry date for Puppet Server’s certificate authority (CA) certificate has passed, your agents won’t be able to check in. You can extend the expiration date of the certificate (cert) in-place using Bolt plans and tasks in the puppetlabs-ca_extend module. If you use the CA cert for other services, such as SSL encrypted PuppetDB traffic or other integrations, you must also update the copy of the certificate used by those services.

Error messages

During an agent run, if the CA certificate is expired, you get an error similar to the following:

Info: Not using expired certificate for ca from cache; expired at <DATE> Error: Could not run: stack level too deep

Version and installation information

Puppet version: 5.0 and later
OS: Any
Installation type: Any
Bolt version: 1.21.0 and later
Bolt OS: A *nix OS (to run Bolt plans)
Bolt installation: On a client machine or on the primary server

Solution

Update the CA certificate, and then the copies used by SSL encrypted PuppetDB traffic and other integrations. This article uses an update for SSL-enabled PuppetDB traffic as an example integration.

1. Follow the instructions included with the ca_extend module module to install the module and its dependencies using Bolt. Use the Bolt plans and tasks to:

   A. Generate a CA certificate with a new expiry date.

   B. Distribute the CA cert to your agents.

   C. Check the expiry date of the CA cert and agent certificates.

2. Copy the new ca.pem to the locations used by PuppetDB and PostgreSQL:

   • If set, the location of the CA for PostgreSQL, set in postgresql.conf in ssl_ca_file

   • The ssldir specific to PuppetDB, by default: /etc/puppetlabs/puppetdb/ssl

3. You can embed the certificate information in the JDBC connection using LibPQFactory or by using a keystore such as the keytool utility. Using the JDBC connection is preferred and takes fewer steps.

   A. If you are embedding the certificate information in the JDBC connection using LibPQFactory:

   1. Restart the PostgreSQL and PuppetDB services.

      systemctl restart postgresql-9.6 puppetdb

   2. Confirm that services are working by reviewing the postgresql-* and puppetdb.log to make sure that new entries are being added.

   B. If you are using a keystore such as the keytool utility, consider using the JDBC as in this example in Enable SSL encrypted communication between PuppetDB and PostgreSQL. This uses LibPQFactory and does not require the creation of a keystore. If you wish to continue using a keystore, complete the following steps.

   1. First, identify the keystore entry containing the CA cert, for example,

       $ keytool -list -v -keystore /etc/puppetlabs/puppetdb/ssl/truststore.jks
      
       Alias name: puppet ca
       Creation date: Dec 18, 2019
       Entry type: trustedCertEntry
       Owner: CN=Puppet CA: foss-server.puppetdebug.vlan
      
       Issuer: CN=Puppet CA: foss-server.puppetdebug.vlan
        Serial number: 1
       Valid from: Mon Dec 16 23:33:23 UTC 2019 until: Sun Dec 17 23:33:23 UTC 2034
       Certificate fingerprints:
           MD5:  AC:F1:D4:56:D5:4F:25:FE:53:43:9E:B8:13:70:05:2E
       SHA1: D3:5A:7C:8D:BB:8E:25:A0:B6:3D:0D:66:87:BF:A9:1A:1C:1A:C8:2B
       SHA256:21:CB:6C:2C:12:33:AF:57:CD:8D:23:53:5F:82:C5:01:9D:D4:D6:F5:6F:6F:C0:7F:67:4E:59:3B:0B:79:7F:E1
       Signature algorithm name: SHA256withRSA
       Subject Public Key Algorithm: 4096-bit RSA key
       Version: 3
       ...
       SubjectKeyIdentifier [
       KeyIdentifier [
       0000: AA E3 59 59 C7 98 B1 04   2C 34 C9 62 4A B8 35 28  ..YY....,4.bJ.5(
       0010: F3 63 CB 64                                        .c.d
       ]
       ]

   2. Ensure this is the same certificate issued by the CA by comparing the identifying information to the certificate on the primary server:

       $ openssl x509 -text -inform pem -in /etc/puppetlabs/puppet/ssl/certs/ca.pem
      
       Certificate:
         Data:
           Version: 3 (0x2)
           Serial Number: 1 (0x1)
         Signature Algorithm: sha256WithRSAEncryption
           Issuer: CN=Puppet CA: foss-server.puppetdebug.vlan
           Validity
             Not Before: Dec 17 17:13:29 2019 GMT
             Not After : Dec 18 17:13:29 2034 GMT
           Subject: CN=Puppet CA: foss-server.puppetdebug.vlan
       ...
      
           X509v3 extensions:
             X509v3 Basic Constraints: critical
               CA:TRUE
             X509v3 Key Usage:
               Certificate Sign, CRL Sign
             X509v3 Subject Key Identifier:
               AA:E3:59:59:C7:98:B1:04:2C:34:C9:62:4A:B8:35:28:F3:63:CB:64
             X509v3 Authority Key Identifier:
               DirName:/CN=Puppet CA: foss-server.puppetdebug.vlan
               serial:01

   3. In the keystore, delete the old CA certificate and import the new CA certificate. For example:

      $ keytool -delete -alias "Puppet CA" -keystore /etc/puppetlabs/puppetdb/ssl/truststore.jks

      $ keytool -import -alias "Puppet CA" -file /etc/puppetlabs/puppet/ssl/certs/ca.pem -keystore /etc/puppetlabs/puppetdb/ssl/truststore.jks

   4. Restart the PuppetDB and PostgreSQL services.

       $ systemctl restart postgresql-9.6 puppetdb

   5. Confirm that services are working by reviewing the postgresql-* and puppetdb.log to make sure new entries are being added.