Storing Puppet Provider Metadata for Single Instance Application

Back to Listing

Hanover, MD, 20 July 2013


Background

When initially working on the Concat Module, Morgan and I were quite frustrated by the lack of ability to wait until the last resource of a given type had run and then apply all changes all at once.

As I’ve delved more into the workings of Puppet providers, I’ve discovered that there is a way to do this with some creative use of the way providers are built.

One of the main uses for this technique is to, as the Concat Module does, perform more creative manipulations of target files on the system than is able to be done with either templates or Augeas. For instance, to perform logical file ordering based on custom rules across multiple declarations of a type from disparate modules.

Example of a Multi-Part Type

demo { 'section one':
  order   => '5',
  content => 'This is section one'
}
 
demo { 'section two':
  order   => '1',
  content => 'This is section two but will be before section one'
}

Goal

This post will walk you through a custom provider called demo. At the end of this post you should be able to use this technique in your own providers if the need arises. It is expected that you understand the basics of custom Provider development. I highly recommend the Puppet Types and Providers book by Dan Bode and Nan Liu.

Preparing the Type

The first step in this process is to prepare our demo type for use.

Prepare the workspace

mkdir -p /etc/puppet/modules/demo/lib/puppet/type
mkdir -p /etc/puppet/modules/demo/lib/puppet/provider/demo

Now, create your type in the file /etc/puppet/modules/demo/lib/puppet/type/demo.rb.

Demo type

# 'demo' Puppet Type
#
# Copyright 2013 Onyx Point, Inc.
# 
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
#   http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
module Puppet
  newtype(:demo) do
    @doc = <<-EOM
      A demo type for showing how to use final provider execution.
 
      Puts together a file at /tmp/demo_provider.txt with ordered
      lines based on the combination of the 'order' and 'name'
      parameters.
    EOM
 
    def initialize(args)
      super(args)
    end
 
    newparam(:name) do
      desc "An arbitrary, but unique, name for the resource."
 
      isnamevar
    end
 
    newparam(:order) do
      desc <<-EOM
        The numeric order in which items should be arranged.
        In the case of a tie, resources are ordered by :name.
 
        Default: 100
      EOM
 
      newvalues(/^(\d+)$/)
 
      defaultto('100')
    end
 
    newproperty(:content) do
      desc <<-EOM
        The actual content of the line in the file.
      EOM
      
      isrequired
 
      def change_to_s(current_value,new_value)
        return "Changing\n'#{current_value[:orig]}'\nTo\n'#{current_value[:new]}'"
      end
    end
 
    validate do
      if not self[:content] then
        raise(ArgumentError,"Hey, where's the content?")
      end
    end
  end
end

This is a very simple type just for illustrating the functionality of this technique. You would obviously adjust the content based on your needs.

Preparing the Provider

We are now ready to being preparing the provider. This section will step you through each section of the provider, culminating with the entire file.

Open the file /etc/puppet/modules/demo/lib/puppet/provider/demo/demo.rb with your favorite text editor and let us begin.

Start your file the usual way:

Provider start

Puppet::Type.type(:demo).provide(:demo) do

The Special Sauce

The key to making this all work is to use a class variable that can hold your state throughout the Puppet run. The most important thing to remember here is that class variables are global to ALL providers. This means that you can not re-use the name of a class variable across providers or you will have unexpected results.

The best way to handle this is to name your class variables something like @@_classvars since provider names are guaranteed to be unique within the entire compiled space.

Add, the demo classvars to the provider.

Demo Classvars

@@demo_classvars = {
  # This provider is set up to only target *one* file. To target
  # multiple files, you'll need to do some creative adjustment to
  # how your provider works.
  :target_file => "/tmp/demo_provider.txt",
 
  # The original content of the target file (if any).
  :old_content => "",
 
  # The new content of the target file.
  # Since we're ordered, we'll hash off of the order and title.
  :new_content => {},
 
  # Don't read the file every time.
  # This is essentially a prefetch without the magic.
  :initialized => false,
 
  # How many resources do we have of the demo type?
  :num_demo_resources => 0,
 
  # How many times have we hit the demo provider?
  :num_runs => 0
}

Now, add your initialize define.

initialize

def initialize(*args)
  super(*args)
 
  # Check to see if we've already initialized.
  if not @@demo_classvars[:initialized] then
 
    # Load the old file content (if any).
    if File.file?(@@demo_classvars[:target_file]) then
      Puppet::Util::FileLocking.readlock(@@demo_classvars[:target_file]) { |fh|
        # In this case, we're ignoring trailing spaces.
        @@demo_classvars[:old_content] = fh.read.strip
      }
    end
 
    @@demo_classvars[:initialized] = true
  end
end

You’ll notice that we’ve very carefully ensured that the file loading portion of the initialize define only runs once. There’s really no reason to read the file multiple times and waste all that glorious I/O.

Now, we’re ready to start processing the meat of the provider and will add the content comparator.

Comparator

def content
  @@demo_classvars[:num_runs] += 1
 
  # Only run through the catalog once.
  if @@demo_classvars[:num_demo_resources] == 0 then
    # How many resources (lines) are we managing?
    @@demo_classvars[:num_demo_resources] =
      # If you had other requirements for matching, then you could
      # process on other type attributes in the catalog.
      resource.catalog.resources.find_all{ |x|
        x.is_a?(Puppet::Type.type(:demo))
      }.count
  end
 
  # Normally, you would do any target manipulation in the
  # prop=(should) # method.
  #
  # However, in this case, this resource may not actually be
  # handling the manipulation of the target file so you have to do
  # the in-memory manipulation in the comparator.
    @@demo_classvars[:new_content]["#{resource[:order]}_#{resource[:name]}"] =
      resource[:content]
 
  # You do not want each resource to drop a log into the log file,
  # only the last one. So we only declare a change on the last
  # entry.
  if @@demo_classvars[:num_runs] == @@demo_classvars[:num_demo_resources] then
    if collate_output != @@demo_classvars[:old_content] then
      return {
        :orig => @@demo_classvars[:old_content],
        :new  => collate_output
      }
    end
  end
 
  # We are not ready to do anything yet, just return yourself so that
  # the resource doesn't trigger.
  return resource[:content]
end

If you were creating a more complex provider, you could do any level of processing here. One very important thing to remember is that properties are executed in the order that they are declared in the type. This means that, should you want to roll all of your logic into one property, you should do it in the last property declared.

Now, we need to add the content setter since Puppet expects it to be declared. However, we aren’t actually going to do any work here since we’re going to wait for flush do to the real work in the provider. This is so that you can be sure that all your properties have run before actually applying your actions to the target file.

Content Setter

def content=(should)
    # Don't do anything here, just wait for the flush.
    # We have to wait for all other potential properties to be handled
    # prior to the final dump.
end

We are now finished with the meat of the provider and simply have to flush the data to disk. To do this, we will use the aptly named flush method of the Puppet provider.

Remember, you won’t actually get here unless there is something to do! Be careful with this though. Any of your properties can trigger a flush so you have to be very careful to wrap them all in such a way that they don’t trigger every time. Alternately, you can flush your target every time and build a file piece by piece. The disadvantage to this being that you’re going to write the file each time a provider runs. Pick what works for you for each given situation.

Flush

def flush
  # If we've gotten here, then there was something to do. However,
  # we should only get here if all resources have run since nothing
  # else will actually trigger the event.
  Puppet::Util::FileLocking.writelock(@@demo_classvars[:target_file],0644) { |fh|
    fh.rewind
    fh.puts(collate_output)
  }
end

Now, we are not quite done. I’ve used a couple of helper methods in this code called collate_output and human_sort respectively and we need to add them as private methods.

Add the final end tag to your file and you are ready to roll! You can now add the following code to your node declaration to have your custom content written to /tmp/demo_provider.txt.

Demo Manifest

demo { 'name one':
  order   => '5',
  content => 'This is some line'
}
 
demo { 'name two':
  content => 'Just some random stuff'
}
 
demo { 'name three':
  order   => '2',
  content => 'W00t!'
}
 
demo { 'name four':
  content => 'Some more random stuff'
}

Having done this, the resulting file will contain:

W00t!
This is some line
Some more random stuff
Just some random stuff

Wrap Up

I hope that this has been a clear example of how to maintain state across providers on a client. Please feel free to post any questions that you may have about this.

The two links below will provide you with the complete type and provider.

Alternatively, you can check out a copy of the code at http://git.io/SrfZ0g

Trevor has worked in a variety of IT fields over the last decade, including systems engineering, operating system automation, security engineering, and various levels of development.

At OP his responsibilities include maintaining overall technical oversight for Onyx Point solutions, providing technical leadership and mentorship to the DevOps teams. He is also responsible for leading OP’s solutions and intellectual property development efforts, setting the technical focus of the company, and managing OP products and related services. In this regard, he oversees product development and delivery as well as developing the strategic roadmap for OP’s product line.

At Onyx Point, our engineers focus on Security, System Administration, Automation, Dataflow, and DevOps consulting for government and commercial clients. We offer professional services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other solutions in place that keep your systems running securely and efficiently. We offer Open Source Software support and Engineering and Consulting services through GSA IT Schedule 70. As Open Source contributors and advocates, we encourage the use of FOSS products in Government as part of an overarching IT Efficiencies plan to reduce ongoing IT expenditures attributed to software licensing. Our support and contributions to Open Source, are just one of our many guiding principles

  • Customer First.
  • Security in All We Do.
  • Pursue Innovation with Integrity.
  • Communicate Openly and Respectfully.
  • Offer Your Talents, and Appreciate the Talents of Others

puppet, ruby, technical

Share this story

We work with these Technologies + Partners

puppet
gitlab
simp
beaker
redhat
AFCEA
GitHub
FOSSFeb