Getting to Know the Puppet Development Kit (PDK) Part 2

Back to Listing

Hanover, MD, 14 December 2018


This post is the second post in a three-part series on Puppet module development using the Puppet Development Kit (PDK), adapted from a presentation for the St. Louis Puppet Users Group.

To recap from part 1, PDK will let you create a new Puppet module:

pdk new module author-modulename

You can then create a new class, defined type, or task:

pdk new class modulename
pdk new defined_type modulename::typename
pdk new task modulename::taskname

If you update PDK, you can update your existing PDK-format module from the templates:

pdk update

You can also convert an existing module into PDK format:

pdk convert

PDK Templates

Most of the magic of PDK is driven by the templates that PDK uses. The default PDK templates are bundled with the PDK package in a bare Git repository. On non-Windows systems, they’re located in /opt/puppetlabs/pdk/share/cache/pdk-templates.git. The upstream source can be found here: https://github.com/puppetlabs/pdk-templates

If you look through the templates, the behavior of PDK is relatively simple to understand.

  • Files in the moduleroot_init directory are created only if they don’t exist when you run pdk update or pdk convert (and are always created with pdk new module).
  • Files in the moduleroot directory will replace any existing file when you run pdk update or pdk convert (and are also created with pdk new module).
  • Files in the object_templates directory are used by the various pdk new commands.

The templates are all ERB (embedded Ruby). The default settings for the template output can be found in the file config_defaults.yml.

config_defaults.yml contains a hash. The keys of the hash are the names of the generated files (which are the filenames of the templates with the .erb extension removed). The filename-specific parts are passed as the Ruby variable @configs to the corresponding template.

As an example, let’s look at moduleroot/.gitattributes.erb:

<%
  excludelist = @configs['exclude'] || []
  includelist = @configs['include'] || {}
-%>
<% (includelist.keys - excludelist).each do |key| -%>
<%= key -%> <%= includelist[key] %>
<% end -%>

And here’s the corresponding portion of config_defaults.yml:

.gitattributes:
  include:
    '*.rb': 'eol=lf'
    '*.erb': 'eol=lf'
    '*.pp': 'eol=lf'
    '*.sh': 'eol=lf'
    '*.epp': 'eol=lf'

Those two snippets get put together by pdk new module to generate the following .gitattributes:

*.rb eol=lf
*.erb eol=lf
*.pp eol=lf
*.sh eol=lf
*.epp eol=lf

Customizing Template Output

Data in the file .sync.yml (in the root of your module) can be used to customize the output of the templates. The contents of .sync.yml are merged with config_defaults.yml in the template. Many of the valid values that you can use in .sync.yml can be found by looking at the file README.md in the PDK templates. You can find additional values by examining config_defaults.yml. Ultimately though, if you really want to know how to customize what PDK will generate, read through the code of the templates.

Example .sync.yml

Gemfile:
  optional:
    ':acceptance':
      - gem: beaker
      - gem: beaker-rspec
      - gem: beaker-puppet_install_helper
      - gem: beaker-module_install_helper
      - gem: vagrant-wrapper
spec/spec_helper.rb:
  spec_overrides: |-
    # Add coverage report.
    RSpec.configure do |c|
      c.after(:suite) do
        RSpec::Puppet::Coverage.report!
      end
    end

There are two additional keys that you can use in .sync.yml: unmanaged and delete. If you set unmanaged: true for a file, PDK will ignore the file and stop managing its contents. If you set delete: true for a file, PDK will delete the file and won’t try to manage its contents.

For example, if you wanted to delete appveyor.yml but you wanted a hand-customized .gitlab-ci.yml, you could put the following in .sync.yml:

appveyor.yml:
  delete: true
.gitlab-ci.yml:
  unmanaged: true

After you make changes to .sync.yml, run pdk update to regenerate files from the templates.

Using Non-Default Templates

If the templates included with PDK don’t match your expectations for whatever reason, you can use a custom template URL when creating a new module with the --template-url option. Let’s say, for example, that you want to track upstream pdk-templates development:

$ pdk new module mynewmodule --skip-interview --template-url=https://github.com/puppetlabs/pdk-templates.git
pdk (INFO): Creating new module: mynewmodule
pdk (INFO): Module 'mynewmodule' generated at path '/home/user/mynewmodule', from template 'https://github.com/puppetlabs/pdk-templates.git'.
pdk (INFO): In your module directory, add classes with the 'pdk new class' command.

(Note that you may need to remove template-url from ~/.pdk/cache/answers.json afterwards to revert to the previous behavior of using the default packaged templates for other new modules.)

You can also switch to other templates in an existing PDK module by using pdk convert and the --template-url option:

$ pdk convert --template-url=https://github.com/puppetlabs/pdk-templates.git

----------Files to be modified----------
metadata.json
spec/spec_helper.rb
appveyor.yml

----------------------------------------

You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

3 files modified.

The template URL is stored in the module’s metadata.json, so you only need to set it once per module.

Converting Existing Modules, Revisited

Speaking of converting modules, let’s look at the process (which we covered briefly in Part 1) again…

$ pdk convert

------------Files to be added-----------
spec/default_facts.yml
appveyor.yml
.yardopts
.rubocop.yml
.rspec
.pdkignore
.gitlab-ci.yml

----------Files to be modified----------
metadata.json
spec/spec_helper.rb
Rakefile
Gemfile
.travis.yml
.gitignore

----------------------------------------

You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

7 files added, 6 files modified.

After converting a module, you are going to want to examine convert_report.txt thoroughly to validate the changes and make sure none of them will cause undesirable effects. By default, the converted module is listed (in metadata.json) as being compatible with the following operating systems:

  • CentOS, Oracle Linux, Red Hat, and Scientific Linux version 7
  • Debian 8
  • Ubuntu 16.04
  • Microsoft Windows 2008R2, 2012R2, and 10

There is no way (currently) to modify this list with .sync.yml or any pdk commands. Unfortunately, you’ll need to edit metadata.json if it isn’t correct.

After converting a module, run pdk validate to see what needs to be fixed up in your code to make the linters (syntax and style checkers) happy.

$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✖] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✖] Checking Ruby code style (**/**.rb).
warning: metadata-json-lint: metadata.json: Dependency puppetlabs-stdlib has an open ended dependency version requirement >= 1.0.0
warning: metadata-json-lint: metadata.json: License identifier Apache 2.0 is not in the SPDX list: http://spdx.org/licenses/
warning: puppet-lint: manifests/init.pp:13:5: arrow should be on the right operand's line
convention: rubocop: lib/facter/customfact.rb:4:12: Layout/SpaceAroundOperators: Surrounding space missing for operator `=`.
convention: rubocop: lib/facter/customfact.rb:5:10: Style/SpecialGlobalVars: Prefer `$CHILD_STATUS` from the stdlib 'English' module (don't forget to require it) over `$?`.
info: task-name: ./: Target does not contain any files to validate (tasks/**/*).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).

In this example, we hit issues with metadata-json-lint, puppet-lint, and rubocop. The metadata-json-lint warnings are easy to fix, but they require editing metadata.json directly. In this case, simply changing

 "license": "Apache 2.0",

to

  "license": "Apache-2.0",

and

  "dependencies": [
    {
      "name": "puppetlabs-stdlib",
      "version_requirement": ">= 1.0.0"
    }
  ],

to something like

"dependencies": [
    {
      "name": "puppetlabs-stdlib",
      "version_requirement": ">= 4.19.0 < 6.0.0"
    }
  ],

will eliminate the warnings.

The warnings from puppet-lint and rubocop are even easier to fix. puppet-lint’s -f option and rubocop’s -a option automatically fix issues. You can run both commands via pdk bundle exec:

$ pdk bundle exec puppet-lint -f manifests/init.pp
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
FIXED: arrow should be on the right operand's line on line 13
$ pdk bundle exec rubocop -a lib/facter/customfact.rb
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
Inspecting 1 file
C

Offenses:

lib/facter/customfact.rb:4:12: C: [Corrected] Layout/SpaceAroundOperators: Surrounding space missing for operator =.
      value=`command`
           ^
lib/facter/customfact.rb:5:10: C: [Corrected] Style/SpecialGlobalVars: Prefer $CHILD_STATUS from the stdlib 'English' module (don't forget to require it) over $?.
      if $?.exitstatus != 0
         ^^

1 file inspected, 2 offenses detected, 2 offenses corrected

With those changes, pdk validate now runs cleanly:

$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✔] Checking Ruby code style (**/**.rb).
info: task-name: ./: Target does not contain any files to validate (tasks/**/*).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).

(Note that the last two lines are not errors. They just tell you that this module has no tasks.)

In part 3, we’ll explore adding additional test coverage to your converted module.

Click here to learn more about how Onyx Point, Inc's professional services and development teams can help you.

Steven is a consultant and trainer for Onyx Point, focusing on Puppet, compliance automation, and all things DevOps.

At Onyx Point, our engineers focus on Security, System Administration, Automation, Dataflow, and DevOps consulting for government and commercial clients. We offer professional services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other solutions in place that keep your systems running securely and efficiently. We offer Open Source Software support and Engineering and Consulting services through GSA IT Schedule 70. As Open Source contributors and advocates, we encourage the use of FOSS products in Government as part of an overarching IT Efficiencies plan to reduce ongoing IT expenditures attributed to software licensing. Our support and contributions to Open Source, are just one of our many guiding principles

  • Customer First.
  • Security in All We Do.
  • Pursue Innovation with Integrity.
  • Communicate Openly and Respectfully.
  • Offer Your Talents, and Appreciate the Talents of Others

puppet, pdk, rspec, testing, programming, technical

Share this story

We work with these Technologies + Partners

puppet
gitlab
simp
beaker
redhat
`
AFCEA
GitHub
FOSSFeb