Hiera v5: Part 1: Globbing

Back to Listing

Dylan Cochran

Hanover, MD, 11 July 2017

(This is part 1 of a multi part series of posts about cool things you can do with Hiera v5 on modern Puppet versions)


In this blog post, I’m going to talk about a new feature added in Hiera v5, which is yaml file globbing.

Hiera is the data access layer for Puppet, and it is organized into hierarchies. When Puppet calls lookup(), it looks through each layer in the hierarchy, in order, and attempts to retrieve the value for a key from the layer. When it can’t find a key in the current layer, it continues on to the next one down, one at a time, until it finds a match.

Explicit Files

Here is an example of a common hierarchy setup. We have a layer for per-node data, we have a layer for hostgroups, a layer for osfamily (eg ‘Windows’ vs ‘RedHat’), and a common layer.

  - name: "Per-node data (yaml version)"
    path: "nodes/%{trusted.certname}.yaml"
  - name: "Other YAML hierarchy levels"
      - "hostgroup/%{::hostgroup}.yaml"
      - "os/%{facts.os.family}.yaml"
      - "common.yaml"

Each file is explicitly specified in the hierarchy. But this leads to a problem with organization. What if we want to keep all of the GitLab settings in common.yaml together? or the Nginx settings? We could separate them in the yaml files with headers and whitespace. We could also try to use separate files, but because each file in each layer is specified in the file, we have to explicitly add a new layer for each file.

A Better Way

This is where globbing comes in.

  - name: "Per-node data (yaml version)"
    path: "nodes/%{trusted.certname}.yaml"
  - name: "Other YAML hierarchy levels"
      - "hostgroup/%{::hostgroup}.yaml"
      - "os/%{facts.os.family}.yaml"
  - name: "Common Settings"
      - "common/*.yaml"

If we replace paths with globs, we can use * and ? as wildcards to the filename. Puppet will automatically expand that into a real file when lookup runs. Now we can have a set of Hiera data files like this:

helio@helio-laptop scratch1> tree data
├── common
│   ├── gitlab.yaml
│   ├── nginx.yaml
│   └── ntp.yaml
├── hostgroup
│   └── masters.yaml
└── nodes
    └── mom.test.yaml

3 directories, 5 files

And all of our GitLab settings, Nginx settings, and NTP settings are neatly organized into their own files. We can do this on any hierarchy layer we want, so we can even change the per-node data to match this scheme. In this way data can be stored in nodes/mom.test/gitlab.yaml or nodes/mom.test/nginx.yaml instead of all sharing nodes/mom.test.yaml

At Onyx Point, our engineers focus on Security, System Administration, Automation, Dataflow, and DevOps consulting for government and commercial clients. We offer professional services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other solutions in place that keep your systems running securely and efficiently. We offer Open Source Software support and Engineering and Consulting services through GSA IT Schedule 70. As Open Source contributors and advocates, we encourage the use of FOSS products in Government as part of an overarching IT Efficiencies plan to reduce ongoing IT expenditures attributed to software licensing. Our support and contributions to Open Source, are just one of our many guiding principles

  • Customer First.
  • Security in All We Do.
  • Pursue Innovation with Integrity.
  • Communicate Openly and Respectfully.
  • Offer Your Talents, and Appreciate the Talents of Others

hiera, puppet

Share this story

We work with these Technologies + Partners