Puppet Notes and How To

(special thanks to Dean Wilson the puppetcookbook guy)

What is Puppet:

Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to orchestration and reporting. Using Puppet, you can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage change, with high scalability.

How Puppet Works

Puppet uses a declarative, model-based approach to IT automation.

Define the desired state of the infrastructure’s configuration using Puppet’s declarative configuration language.
Simulate configuration changes before enforcing them.
Enforce the deployed desired state automatically, correcting any configuration drift.
Report on the differences between actual and desired states and any changes made enforcing the desired state.
Define reusable configuration modules.

Enforce Desired State

After you deploy your configuration modules, the Puppet Agent on each node communicates regularly with the Puppet Master server to automatically enforce the desired states of the nodes.

The Puppet Agent on the node sends Facts, or data about its state, to the Puppet Master server.
Using the Facts, the Puppet Master server compiles a Catalog, or detailed data about how the node should be configured, and sends this back to the Puppet Agent.
After making any changes to return to the desired state (or, in “no-op mode,” simply simulating these changes), the Puppet Agent sends a complete Report back to the Puppet Master.
The Reports are fully accessible via open APIs for integration with other IT systems.

Puppet Quick Overview:

Puppet is a configuration management tool

Written by Luke Kanies
Supported by Puppet Labs (formerly Reductive Labs)
Client/Server model
Written in Ruby

Puppet Language

Puppet decouples the syntax of the configuration management tool from the syntax of the underlying OS and applications

This allows you to define a high level idea like user, application, or service and puppet will translate that in to the commands required by the client OS

Configuration information is specified in “recipes”

Recipe files end with .pp

The recipes define what the system should look like (be configured as)

Puppet then issues OS/applications specific commands to change the configuration to match the desired result

Recipes are written in ruby

How Puppet Works

Puppet has a passive server called the “puppet master”:

Runs the RPC-XML/HTTPS server listening on port 8140

Acts a the certificate authority for the puppet clients

Has recipes the clients can download

Has repositories of files the clients can download

Puppet clients pull configuration information from the puppet master

Client first collects local host configuration information using factor

Client then requests a master recipe to configure the client

This master recipe then pulls in additional recipes based on the client’s configuration

The puppet client then translates this information into host specific commands to run

Puppet is a Declarative Language

Configuration components are organized into resources

Resources are grouped into collections

Resources are made up of a type, title and a series of attributes:

file { “/etc/hosts”:

owner => “root”;

group => “root:

}

Type is file

Title is /etc/hosts

Attributes define the owner and group is root

Installing Puppet

I use RHEL/CentOS so I use the Fedora Project EPEL repository

rpm -ivh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-3.noarch.rpm

I then use yum on the puppet master:

yum install puppet puppet-server

And on each client I install just the client:

yum install puppet

Make sure the puppet user exists:

id puppet

Creating a Simple Puppet Configuration

The puppet master is configured in two places:

/etc/puppet/puppet.conf

/etc/puppet/manifests/site.pp

You do not typically need to change the puppet.conf file

The site.pp pulls in all of the puppet recipes

The simplest site.pp file is:

file { “/etc/hosts”:

owner => “root”,

group => “root”,

mode => “644”,

}

Running the Puppet Master

The puppetmasterd runs only on the puppetmaster:

chkconfig puppetmasterd on

service puppetmasterd start

Look for error in:

/var/log/puppet/masterhttp.log

Running the Puppet client

Normally the puppet client runs as a daemon

You can run it manually

puppetd -v –test

If we break the hosts file:

chmod 664 /etc/hosts

And run puppet in verbose mode:

puppetd -v –no-daemonize

We should see:

notice: Starting Puppet client version 0.25.5

info: Caching catalog for puppet.kvm.harker.com

info: Applying configuration version ‘1279238728’

notice: //sysfiles/File[/etc/hosts]/mode: mode changed ‘664’ to ‘644’

notice: Finished catalog run in 0.02 seconds

Puppet Defaults In The site.pp File

import “classes”

import “modules”

import “nodes”

# The filebucket option allows for file backups to the server

filebucket { main: server => ‘puppet.kvm.harker.com’ }

# Set global defaults – including backing up all files to the main

# filebucket and adds a global path

File { backup => main }

Exec { path => “/usr/bin:/usr/sbin/:/bin:/sbin” }

Puppet Nodes and Classes

Puppet has three types of recipe files:

Nodes: host or node control (configuration) files
Classes: action files that define what to do
Modules: Reusable classes

Puppet Classes

Puppet classes define how to install and configure files, applications, services, etc…

A class is defined with:

class Title {

}

Resources are then added to the class

A class can have multiple resources:

#/etc/puppet/manifests/classes/sysfiles.pp

class sysfiles {

file { “/etc/hosts”:

owner => “root”,

group => “root”,

mode => “644”,

}

file { “/etc/passwd”:

owner => “root”,

group => “root”,

mode => “644”,

}

This class is then included in a node definition with:

include sysfiles

The Nodes File

The nodes defines what classes and modules are applied to which hosts

Typically the node definitions are put in a node.pp file:

/etc/puppet/nodes.pp

The default node is used if there is no match for a specific host

A simple site.pp file that includes the sysfiles class:

# /etc/puppet/manifests/nodes.pp

node default {

include sysfiles

}

Nodes have inheritance

A complex node can be configured by inheriting a simpler node

I start with a basenode that all hosts inherit

This includes things I want done on all nodes:

Applications installed or removed
Services enabled or disabled
Site wide configuration files

You can then make a more complex node based on this inheritance

Webserver = basenode + apache
MySQLserver = basenode + MySQL

You can then make a specific node or host:

fooMysql = MySQLserver + foo specific additions
barMysql = MySQLserver + bar specific additions

Two cautions:

You cannot redefine resources
Once a resource is added it is very difficult to remove it

A Class to Install an Application/Service

# /etc/puppet/manifests/classes/ntpd.pp

class ntp {

package { ntp: ensure => present }

file { “/etc/ntp.conf”:

owner => root,

group => root,

mode => 444,

backup => false,

source => “puppet:///files/etc/ntp.conf”,

require => Package[“ntp”],

}

service { “ntpd”:

enable => true ,

ensure => running,

subscribe => [Package[ntp], File[“/etc/ntp.conf”],],

}

Puppet Modules

A puppet module is a portable collection of classes, configuration resources, templates and files that configures a particular application or function

You typically make a modules sub-directory:

mkdir -p /etc/puppet/modules

Then a sub-directory for each module

mkdir -p /etc/puppet/modules/sudo

In a manifest sub-directory puppet related files get added:

mkdir -p /etc/puppet/modules/ntpd/manifests

You would then add the ntpd.pp file:

mv /etc/puppet/manifests/classes/ntpd.pp /etc/puppet/modules/ntpd/manifests/ntpd.pp

You can also have a module install files:

/etc/puppet/modules/ntpd/files

Puppet Templates

Modules can also edit files on the fly:

/etc/puppet/modules/ntpd/templates

In /etc/puppet/modules/ntpd/templates/ntp.conf:

# /etc/ntp.conf, configuration for ntpd

. . .

fudge 127.127.1.0 stratum <%= local_stratum %>

. . .

includefile /etc/ntp.server.conf

includefile /etc/ntp.client.conf

The ntp.pp recipe file:

. . .

$local_stratum = $ntp_local_stratum ? {

” => 13,

default => $ntp_local_stratum,

}

config_file { “/etc/ntp.conf”:

content => template(“ntp/ntp.conf”),

require => Package[$ntp_package];

}

Puppet Packages:

Install a package

Challenge

You want to install a package

Solution

package { “screen”:

ensure => “installed”

}

Explanation

The simplest use of the package type only requires you to specify the package name and the desired status of the package. The ‘ensure’ attribute will accept either ‘present’ or ‘installed’.

This will use whichever installation provider Puppet considers to be the default for this platform and install the latest version of the package that it finds in the hosts configured software repos when the manifest first runs. Once the package is installed Puppet will not track new versions of the package.

Install and track the newest version of a package

Challenge

You want to install a package if it’s not present and track and upgrade the version if it is.

Solution

package { “screen”:

ensure => “latest”

}

Explanation

Sometimes just having a package present isn’t enough, you also want to ensure it stays up to date. By specifying ‘latest’ Puppet will install a package if absent and upgrade the package to newer versions when they become available. This last part is where latest differs from ‘installed’.

Install multiple packages

Challenge

You want to install multiple packages

Solution

# the first, most obvious solution is

package { “screen”: ensure => “installed” }

package { “strace”: ensure => “installed” }

package { “sudo”: ensure => “installed” }

# you can use a global package parameter

Package { ensure => “installed” }

package { “screen”: }

package { “strace”: }

package { “sudo”: }

# you can specify the packages in an array …

$enhancers = [ “screen”, “strace”, “sudo” ]

package { $enhancers: ensure => “installed” }

# … and even combine it a global package parameter

Package { ensure => “installed” }

$enhancers = [ “screen”, “strace”, “sudo” ]

package { $enhancers: }

Explanation

There are often times when you’ll need to install more than a single package in one of your modules or manifests. While you can install multiple packages with multiple package resources, one per package you want to install, there are a couple of extra formatting options to reduce duplication slightly.

As a reminder the package type only requires you to specify the package name and the desired status of the package. The ‘ensure’ attribute will accept either ‘present’ or ‘installed’.

It’s worth mentioning that no matter which of the above formats you specify the packages will still be installed one by one. None of those snippets will install multiple packages simultaneously in a way that will satisfy a circular dependency. So if A requires B and B requires A there is no way in pure puppet to install them. And you should probably rethink your packages.

This will use whichever installation provider Puppet considers to be the default for this platform and install the latest version of the packages that it finds in the hosts configured software repos when the manifest first runs. Once the packages are installed Puppet will not track new versions of the packages.

Remove a package

Challenge

You want to remove a package

Solution

# remove a package but leave its config files alone

package { “screen”:

ensure => “absent”

}

# remove a package and purge its config files

package { “screen”:

ensure => “purged”

}

Explanation

Removing a package via Puppet is as simple as installing one. Simply setting “ensure” to “absent” will remove the package. If your platforms package provider supports purging packages (you can check in the table under packages then you can specify “Purged” as a value and it should remove the config files along with the package.

Installing packages with names that vary per distro

Challenge

You manage multiple operating systems or Linux distributions and sometimes they disagree on what a package should be named.

Solution

class apache-server {

# determine the apache-server package based on the operatingsystem fact

$apache_server = $operatingsystem ? {

Fedora => “httpd”,

default => “apache2”,

}

package { “$apache_server”:

ensure => “present”,

alias => “apache-server”,

}

file { “/var/www/html/index.html”:

source => “puppet:///index.html”,

require => Package[“apache-server”],

}

Explanation

With the handy combination of a selector and a fact we can tailor the package name to suit the clients operating system. This construct takes the value of the ‘$operatingsystem’ fact and compares it to the left hand value on each line of the body. If it matches one then the variable ($apache_server in this case) is set to the right hand side value of that line. If none match then the variable is set to the default value (apache2 in this case). You can also, using leaning toothpick style, use regular expressions (/Fedora|CentOS/) as the left-hand side.

If you need to subdivide versions of an operating system you can do so with facts like $lsbdistcodename (which on Debian for example will return etch, lenny or squeeze)

You’ll also notice we have an alias line in the package resource. By explicitly setting an alias to this resource we can refer to it later, such as in the files require without needing to constantly use the variable. Is this a huge win? Probably not, but I like to reserve variables for places where things actually change a lot.

Install a Ruby Gem

Challenge

You want to install a Ruby Gem

Solution

package { ‘sinatra’:

ensure => ‘installed’,

provider => ‘gem’,

}

# notice: /Stage[main]//Package[sinatra]/ensure: created

Explanation

Installing a Ruby Gem is very similar to how you install a normal package using your systems package manager. However in this case you need to explicitly state that the gem provider will handle the install.

Much like a package that is installed via the default system provider Puppet will not track new versions of the package unless “ensure => ‘latest’” is set.

Change the Package provider

Challenge

You want to change the default package provider for your systems.

Solution

# in your site.pp

Package { provider => “aptrpm” }

Explanation

Not all package providers are created equal. Some of them, as shown in the table under packages, have additional functionality that you may want. Changing the provider is simple, including a line like the above somewhere high up in your configuration file chain (such as site.pp) will ensure all the declared package types use the newly chosen provider.

Puppet Files:

You want to create a directory

Challenge

You want to create a new directory

Solution

# create a directory

file { “/etc/site-conf”:

ensure => “directory”,

}

# a fuller example, including permissions and ownership

file { “/var/log/admin-app-log”:

ensure => “directory”,

owner => “root”,

group => “wheel”,

mode => 750,

}

# this example is incorrect and creates a file

file { “/etc/site-conf/”:

ensure => “present”,

}

Explanation

Creating a directory is another task that is handled by the powerful and flexible file type. In this case specifying a directory is done by setting ensure to (unsurprisingly) “directory”. The second example is a (semi-)common mistake made by people coming from certain other tools. Adding a ’/’ to the end of a filename isn’t enough to tell puppet to make a directory and so it creates a file instead, and silently drops the ’/’ off the end of the name.

You can use a number of the parameters from the file type to control the directories properties, such as the owner, group and permissions.

It’s also worth noting that puppet will not create missing parent directories. There is a long standing bug regarding this feature request “Directory creation fails if parent directory does not exist” and while there are a couple of workarounds there’s no official fix.

You want to create a directory tree

Challenge

You want to create a new directory tree

Solution

# create a directory tree, list the directories in order

# and puppet will “do the right thing”.

file { [ “/usr/local/whisper/”, “/usr/local/whisper/2.0”,

“/usr/local/whisper/2.0/bin”, “/usr/local/whisper/2.0/log” ]:

ensure => “directory”,

}

# or you can assign them to a variable and use them in the resource

$whisper_dirs = [ “/usr/local/whisper/”, “/usr/local/whisper/2.0”,

“/usr/local/whisper/2.0/bin”, “/usr/local/whisper/2.0/log”,

]

file { $whisper_dirs:

ensure => “directory”,

owner => “root”,

group => “wheel”,

mode => 750,

}

# doesn’t work – will fail unless whisper and 2.0 already exist

file { “/usr/local/whisper/2.0/bin”:

ensure => “directory”,

}

Explanation

Creating a directory tree in puppet is slightly harder than you’d first expect as puppet lacks a native mkdir -p equivalent. Instead you can use an array of directories, each one progressing slightly further down the tree and puppet will create them in turn. Starting with puppet 2.7 (File type should auto-require all parents) each directory will auto-require its nearest ancestor directory, which will add another layer of guaranty to this method of creating a directory tree.

You can use a number of the parameters from the file type to control the directories properties, such as the owner, group and permissions. It’s also worth reiterating that puppet will not create missing parent directories as this is an often raised “bug”.

Managing Symlinks

Challenge

You want to create a symlink using puppet.

Solution

# preferred symlink syntax

file { ‘/tmp/link-to-motd’:

ensure => ‘link’,

target => ‘/etc/motd’,

}

# older, less clear syntax

file { ‘/tmp/link-to-motd’:

ensure => ‘/etc/motd’,

}

Explanation

Although it may seem slightly counter intuitive at first you create and manage symlinks through the file type. It’s worth mentioning that the title (in this example /tmp/link-to-motd) is the name of the link to create and the filename given in ensure is the file to link to. If you get these two options the wrong way around then you’ll probably need to start reading the filebucket section.

Select a file based on a fact

Challenge

You want to deploy a specific file based on the value of a fact.

Solution

file { “/etc/mysql/conf.d/tuning.cnf”:

ensure => “present”,

source => [

“puppet:///modules/mysql-server/tuning.$hostname.cnf”,

“puppet:///modules/mysql-server/tuning.$domain.cnf”,

“puppet:///modules/mysql-server/tuning.cnf”

}

Explanation

Sometimes you want to have specific versions of a file for a given host, location or operating system while having a default fallback for the standard cases. By using an array of sources in a file resource, with the value of a fact as part of the on-disk file name, you can easily make your configs more granular where needed while keeping sensible defaults.

In the mysql-server module you would have file names like these:

files/

tuning.host1.cnf

tuning.host2.cnf

tuning.example.org.cnf

tuning.cnf

When using multiple sources in this way puppet will use the first file that matches a source line, and cause an error if none match, so you should nearly always have a default file.

Only add a file if it’s absent.

Challenge

You only want puppet to add a file if the file isn’t already present. Once it’s added, or if it’s already there, you want to leave it alone.

Solution

file { “/tmp/hello-file”:

replace => “no”, # this is the important property

ensure => “present”,

content => “From Puppet\n”,

mode => 644,

}

Explanation

Adding the “replace” property with the value of ‘no’ (or the uglier ‘false’) to a file will stop puppet managing its content if the file already exists. If it’s not already present then the file will be created as per the file types parameters and then left alone – even if its content changes on either the puppet master or client side.

In general this isn’t the best approach as all configuration should be under centralised management. Having local changes that can drift over time will cause pain in the long run and make your systems baseline configuration harder to predict.

It’s worth noting that the file type will still maintain permissions and other such file properties if they change on the client. Even with replace set to ‘no’.

Restart a service when a file changes

Challenge

You want to restart a service when its config file changes

Solution

# define the service to restart

service { “sshd”:

ensure => “running”,

enable => “true”,

require => Package[“openssh-server”],

}

# add a notify to the file resource

file { “/etc/ssh/sshd_config”:

notify => Service[“sshd”], # this sets up the relationship

mode => 600,

owner => “root”,

group => “root”,

require => Package[“openssh-server”],

content => template(“ssh/sshd_config.erb”),

}

Explanation

It would be very annoying if puppet allowed you to deploy a new config file without providing a way to restart a service to take advantage of the change. Using the “notify” metaparameter we can tell a resource to signal another resource, often a file notifying a service, and cause it to refresh, which in the case of a service causes a restart.

Puppet will accumulate multiple notifies over a single run and only refresh the service once. This is useful for services like Nagios where a large number of config files can change for one amendment and nothing would be gained from multiple restarts.

It’s worth noting that you probably don’t want to use this with larger, more important, services like mysql where a small white space change in a config file could cause a restart of your mysql database and begin a domino effect.

Reduce Duplicated File Attributes

Challenge

You’ve had enough of specifying owner, mode, group and other settings on every file resource and you want a solution with less duplication

Solution

File {

ensure => “present”,

owner => “root”,

group => “root”,

mode => 644,

}

file { “/etc/cobbler/modules.conf”:

content => template(“cobbler/modules.conf”),

}

file { “/etc/cobbler/dhcp.template”:

content => template(“cobbler/dhcp.template”),

}

Explanation

When you start using file resources in puppet you’ll probably write each and every one of them out in the full long hand style shown below:

file { “/etc/cobbler/modules.conf”:

ensure => “present”,

content => template(“cobbler/modules.conf”),

owner => “root”,

group => “root”,

mode => 644,

}

file { “/etc/cobbler/dhcp.template”:

ensure => “present”,

content => template(“cobbler/dhcp.template”),

owner => “root”,

group => “root”,

mode => 644,

}

While there is nothing incorrect with this approach it can lead to ballooning manifests, a large amount of duplication between resources and easy to miss one line differences. Did both of those resources have the same permissions?

As you become more comfortable with Puppet you can start to make more use of intermediate features such as resource defaults in order to clarify and declutter your manifests. By grouping the common attributes together we can convert the previous example to this cleaner version:

File {

ensure => “present”,

owner => “root”,

group => “root”,

mode => 644,

}

file { “/etc/cobbler/modules.conf”:

content => template(“cobbler/modules.conf”),

}

file { “/etc/cobbler/dhcp.template”:

content => template(“cobbler/dhcp.template”),

}

# override the permissions for this one file

file {“/etc/cobbler/users.digest”:

source => “puppet:///modules/cobbler/users.digest.live”,

mode => 660,

}

While this isn’t a huge saving in raw characters typed (although in longer manifests they do start to mount up) it moves all the common settings in to a single location (keeping us clear of DRY violations) and leaves only the differences between file type definitions. There is one caveat though, any declared resource of this type that doesn’t override a given setting, from this point and onwards in the include chain, get the value of the default setting, which can lead to odd action from afar head scratching.

Depending on how you like to structure your resources you can condense them even further and nest all the declarations inside one resource statement:

File {

ensure => “present”,

owner => “root”,

group => “root”,

mode => 644,

}

file {

“/etc/cobbler/modules.conf”:

content => template(“cobbler/modules.conf”);

“/etc/cobbler/dhcp.template”:

content => template(“cobbler/dhcp.template”);

# override the permissions for this one file

“/etc/cobbler/users.digest”:

source => “puppet:///modules/cobbler/users.digest.live”,

mode => 660;

}

It’s worth pointing out that when using this form each resource is separated with a semicolon (;), not a comma (,).

Hostfiles

Add a host entry

Challenge

You want to add an entry to your local systems equivalent to /etc/hosts

Solution

# create a simple hostname and ip host entry

host { ‘syslog’:

ip => ‘10.10.10.10’,

}

# create a fully qualified full host entry with an alias

host { ‘ntpserver.example.com’:

ip => ‘10.100.10.50’,

host_aliases => ‘timeserver’,

}

# host entry with multiple aliases

host { ‘dashboard’:

ip => ‘10.120.100.111’,

host_aliases => [ ‘nagios’, ‘munin’ ],

}

Explanation

The simplest use of the host type requires just a name (fully qualified or relative as needed) and the ip address it should resolve to. On the next puppet run in addition to your new host entry you’ll also see the following Puppet warning banner, indicating that parts of this file are now under puppet management.

# HEADER: This file was autogenerated at Sun Jan 02 17:46:16 +0000 2011

# HEADER: by puppet. While it can still be managed manually, it

# HEADER: is definitely not recommended.

The next two examples follow the same basic pattern, requiring that a host and ip address be specified, and write the same records, just with additional host aliases. In the third example you can see that in order to pass more than a single host_alias to a host entry you’ll need to supply them as an array.

You want some resource examples

Challenge

You’d like to see some example resources in the puppet configuration format

Solution

$ puppet resource package puppet # puppet 2.6 or greater

$ ralsh package puppet # puppet 2.5 or less

package { ‘puppet’:

ensure => ‘2.6.4-0.5.fc14’

}

# an example host file entry resource

$ puppet resource host localhost.localdomain # puppet 2.6 or greater

$ ralsh host localhost.localdomain # puppet 2.5 or less

host { ‘localhost.localdomain’:

ensure => ‘present’,

target => ‘/etc/hosts’,

ip => ‘127.0.0.1’,

host_aliases => [‘localhost’,’puppet’]

}

# an example user resource

$ puppet resource user daemon # puppet 2.6 or greater

$ ralsh user daemon # puppet 2.5 or less

user { ‘daemon’:

home => ‘/sbin’,

password_min_age => ‘0’,

ensure => ‘present’,

uid => ‘2’,

shell => ‘/sbin/nologin’,

password_max_age => ‘99999’,

password => ‘*’,

gid => ‘2’,

groups => [‘bin’,’daemon’,’adm’,’lp’],

comment => ‘daemon’

}

Explanation

When you are starting out in puppet the first few resources of each type you use can be awkward to write. There are never enough examples and sometimes you just want to see what something looks like on your local system through the eyes of puppet. Thanks to puppet resource (or ralsh if you’re on an older version) you can do just that.

When you run with a provider and a resource on your local machine puppet resource will try to load them in the way it does during a normal puppet run. If this succeeds it will present the resource in the same format as you’d use in your manifest. While some examples, such as the user above, provide more options than you’d normally specify having the ability to introspect and crib from your existing setup can help the learning process and speed up those first few manifests.

Remove a host entry

Challenge

You’ve found an old host entry that you no longer needed. Whether the old record is now in DNS or should no longer exist puppet makes removing it simple.

Solution

host { ‘syslog’:

ensure => “absent”,

}

host { ‘ntpserver.example.com’:

ensure => “absent”,

}

Explanation

Removing puppet managed host entries is as easy as the code snippet illustrates. A simple “ensure => ‘absent’” is all you need.

Remove all unmanaged host entries

Challenge

Sometimes you need to rule with an iron fist and remove ANY host entries that are not managed by puppet.

Solution

# always test with noop first!

resources { ‘host’:

purge => true,

noop => true,

}

# remove ALL unmanaged host resources

resources { ‘host’: purge => true }

Explanation

Once you’re feeling more confident with puppet you may want to start removing hand added configs from your systems. While puppet makes this amazingly easy (maybe too easy…) it also provides some useful logging capabilities to show you what would have changed.

Using the ‘resources’ metatype (a metatype is used to manage other types) the first example above will ‘pretend’ to remove all host file entries that are not puppet managed, including the entry for localhost (which you’ll need to puppet manage if you try resource purging). It will log all the resources that will be purged, when you remove the ‘noop => true’, in a format like this:

notice: /Host[localhost.localdomain]/ensure: is present, should be absent (noop)

When you remove the ‘noop => true’ puppet will remove any unmanaged host entries while leaving all the ones it controls in place. This is one of the places where centralised configuration management gives you more than enough rope to hang yourself so always run in noop mode first, double check the edge cases like the localhost entry and use the power wisely.

Remove a host entry

Challenge

You’ve found an old host entry that you no longer needed. Whether the old record is now in DNS or should no longer exist puppet makes removing it simple.

Solution

host { ‘syslog’:

ensure => “absent”,

}

host { ‘ntpserver.example.com’:

ensure => “absent”,

}

Explanation

Removing puppet managed host entries is as easy as the code snippet illustrates. A simple “ensure => ‘absent’” is all you need.

Adding a per-domain hostname

Challenge

In certain network architectures you have roles defined that are implemented on a number of machines in different domains. A local package mirror, a syslog forwarding host and a local network time servers are all examples of this. You need a host entry that points to a different ip address depending on the location of the client.

Solution

host { “syslog”:

ip => $domain ? {

/production/ => “10.10.10.10”,

/staging/ => “192.168.23.10”,

default => “10.100.100.100”,

}

Explanation

Using a combination of whichever fact best describes your clients location (we’re using the domain part of the hostname in the example above) and a selector you can choose the correct IP address for your local version of which ever machine you’re dealing with.

While most lookups of this type should live in DNS sometimes there are a handful of services that are important enough to have their host/ip mappings configured on the servers themselves. By using this approach you can localise where possible while always having the ‘default’ entries IP address as the common fall back.

EXEC

Run a command with Puppet

Challenge

Despite all the types and modelling available sometimes you just want puppet to run a command for you.

Solution

exec { “refresh_cache”:

command => “refresh_cache 8600”,

path => “/usr/local/bin/:/bin/”,

# path => [ “/usr/local/bin/”, “/bin/” ], # alternative syntax

}

Explanation

No matter how far Nix tools evolve sometimes you just need the ease or power of an existing command or script. The exec type provides a simple way to run those commands via puppet (on the puppet client, not the master) and harness them in your modelling, whether as a dependency of another resource, an easy way to accomplish something puppet doesn’t yet provide or as part of a gradual migration.

In the, very minimal, example above we first give the command a nice, human readable name to ease future manifest reading. We then specify the command itself (and any arguments it should receive) using “command”. The last piece of this simple example is to specify the $PATH that this command is located in. It’s worth mentioning that any commands you run via an exec should be idempotent as they may be invoked multiple times and they must have an exit code of ‘0’ for puppet to consider them successful.

Most uses of exec (at least in my manifests) have another option or two specified to restrict when the exec should run. For details on these have a look at the other posts in the exec section.

In general you should carefully consider any execs you add, while they are often the quickest, most familiar solution to a problem they are frequently a sign that your management is a little too coarse grained.

You want to set a default $PATH for all execs

Challenge

You’ve had enough of specifying a complete ‘path =>’ on each exec and you want a more global solution

Solution

# put this somewhere global, like site.pp

Exec { path => [ “/bin/”, “/sbin/” , “/usr/bin/”, “/usr/sbin/” ] }

# uses the globally specified path

exec { “make_foo_state”:

command => “mkdir -p /tmp/foo/state”,

}

# overrides the global path for this command

exec { “my_local_command”:

command => “my_special_local_command”,

path => “/usr/local/sbin/”,

}

Explanation

Adding a path to every exec is annoying, easy to forget and leads to unneeded duplication. By placing the Exec shown above in a global location (like your site.pp file) all execs will use that path by default, while still allowing you to add extra directories to the path where needed.

You’d like nicer names for your exec resources

Challenge

You’re using the command in the exec resource name and it’s ugly, unwieldy and awkward to use from other resources.

Solution

# exec with the command as the name

exec { “/bin/mkdir -p /tmp/needed/directory”:

}

package { “needed”:

ensure => “installed”,

# awkward, long require

require => Exec[“/bin/mkdir -p /tmp/needed/directory”],

}

# nicer, human friendly naming

exec { “create_needed_directory”:

command => “/bin/mkdir -p /tmp/needed/directory”,

}

package { “needed”:

ensure => “installed”,

# easier to read require

require => Exec[“create_needed_directory”],

}

Explanation

Older manifests, as well as newer ones written by less experienced puppet users, often use the actual command as the name of the exec. While this works, and can even be used in other parameters such as require, it is a little unwieldy and leads to a certain unneeded leaking of information to other resources.

It’s much nicer to use a human readable explanation of what the command is for as the execs name and then put the actual command in the “command” property.

Don’t run exec if $file exists

Challenge

You have an exec that creates a file or directory that should only run if it doesn’t already exist.

Solution

exec { “create_needed_directory”:

command => “/bin/mkdir -p /tmp/needed/directory”,

creates => “/tmp/needed/directory”

}

Explanation

This is such a common requirement that puppet has a single property that “does the right thing”. As long as the file / directory specified in “creates” exists puppet won’t run the exec. It doesn’t even cause clutter in the logs!

Selective exec running

Challenge

Sometimes you have a more complicated prerequisite that must be satisfied before an exec should run.

Solution

# run exec only if command in onlyif returns 0.

exec { “run_account_purger”:

command => “/usr/local/sbin/account_purger”,

onlyif => “grep -c old_account /etc/passwd”,

}

# or run multiple commands – all must succeed for exec to run

exec { “run_account_purger”:

command => “/usr/local/sbin/account_purger”,

onlyif => [

“grep -c old_account /etc/passwd”,

“test -d /home/old_account/”

]

}

Explanation

Sometimes a simple ‘creates’ isn’t enough to determine if an exec should run or not. Using “onlyif” allows you to use the return value of one or more commands as your execution criteria; run the exec on a return of 0. The test commands will use the same $PATH as the exec itself and can be simple or as complicated as required, or as extreme as your patience for escaping string quoting allows!

If your monitoring system of choice has checks that you can manually run on the command line, and nagios is a great example of this, you can often get quite complicated and comprehensive checks with no extra coding required.

Don’t run exec if $file exists

Challenge

You have an exec that creates a file or directory that should only run if it doesn’t already exist.

Solution

exec { “create_needed_directory”:

command => “/bin/mkdir -p /tmp/needed/directory”,

creates => “/tmp/needed/directory”

}

Explanation

Selective exec running

Challenge

Sometimes you have a more complicated prerequisite that must be satisfied before an exec should run.

Solution

# run exec only if command in onlyif returns 0.

exec { “run_account_purger”:

command => “/usr/local/sbin/account_purger”,

onlyif => “grep -c old_account /etc/passwd”,

}

# or run multiple commands – all must succeed for exec to run

exec { “run_account_purger”:

command => “/usr/local/sbin/account_purger”,

onlyif => [

“grep -c old_account /etc/passwd”,

“test -d /home/old_account/”

]

}

Explanation

Logging an Execs output

Challenge

Your exec has output you want puppet to log or to silently ignore.

Solution

# on_failure logs the commands output on failure

exec { “delete_str_tmp”:

path => “/usr/local/bin/:/bin:/usr/sbin”,

command => ‘find /tmp/ -name “*.str” -type f | xargs -n 1 rm’,

logoutput => “on_failure”,

}

Explanation

When an exec runs a command that has output it can be handled by puppet in a number of different ways. If you want the output to be logged (and then passed back in reports or to Dashboard) you can specify “logoutput => true,” (note – the true isn’t quoted), if you want the command to be silent even on failure (puppet will still show an err if the exec fails) you can specify “logoutput => false”. Otherwise if you only care about the output when something breaks (as we often do) you can pass the string “on_failure” to logoutput.

Services

Ensure service is running

Challenge

You want to ensure a service is running and have puppet restart it when needed.

Solution

service { “cron”:

ensure => “running”,

}

Explanation

There are services that should always be running and if they are ever stopped should be restarted. By setting the ensure service property to “running” (or true) puppet will check for the presence of the service on each run and restart it when it’s absent.

Ensure service starts on boot

Challenge

You want to ensure a service starts on boot.

Solution

service { “puppet”:

enable => true,

}

# let’s show this snippet in action

# puppet won’t start on boot

$ chkconfig –list puppet

puppet 0:off 1:off 2:off 3:off 4:off 5:off 6:off

# “enable” the service

$ sudo puppet apply -e ‘service { “puppet”: enable => true, } ‘

notice: /Stage[main]//Service[puppet]/enable: enable changed ‘false’ to ‘true’

notice: Finished catalog run in 0.25 seconds

# double check it (on Redhat derived systems)

$ chkconfig –list puppet

puppet 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Explanation

Sometimes you want to ensure that a service starts when the host boots. In puppet you accomplish this by setting the enable property on a service resource.

Ensure service is stopped

Challenge

You want to ensure a service is down and have puppet stop it whenever it discovers it running.

Solution

service { “cron”:

ensure => “stopped”,

}

Explanation

There are services that should never be running on your hosts and if they are discovered should be stopped. By setting the ensure service property to “stopped” (or false) puppet will check for the presence of the service on each run and stop it when ever it encounters it running.

Ensure service is stopped on boot

Challenge

You want to ensure a service is stopped on boot.

Solution

service { “puppet”:

enable => false,

}

# let’s show this snippet in action

$ sudo puppet apply -e ‘service { “puppet”: enable => false, }’

notice: /Stage[main]//Service[puppet]/enable: enable changed ‘true’ to ‘false’

notice: Finished catalog run in 0.13 seconds

$ chkconfig –list puppet

puppet 0:off 1:off 2:off 3:off 4:off 5:off 6:off

Explanation

Sometimes you want to ensure that a service does not automatically start when the host boots. In puppet you accomplish this by setting the enable property on a service resource.

Restart a service when a file changes

Challenge

You want to restart a service when its config file changes

Solution

# define the service to restart

service { “sshd”:

ensure => “running”,

enable => “true”,

require => Package[“openssh-server”],

}

# add a notify to the file resource

file { “/etc/ssh/sshd_config”:

notify => Service[“sshd”], # this sets up the relationship

mode => 600,

owner => “root”,

group => “root”,

require => Package[“openssh-server”],

content => template(“ssh/sshd_config.erb”),

}

Explanation

FACTS

Show all facter facts

Challenge

You want to see all the facts available to you.

Solution

# show all facter facts

facter

# show all facter facts, and those defined for use in puppet (run as root)

facter -p

Explanation

While facter is heavily used by puppet the facter tool itself can be run as a standalone program or included as a library in your own ruby code.

Running facter with no arguments will list all the facts and their values that facter knows about. When invoked with the “-p” option facter will load the puppet libraries and also show the puppet specific facts, including any custom ones you have deployed via puppet.

View all the puppet variables on this client

Challenge

You want to see all the variables that are set for this client when puppet runs

Solution

file { “/tmp/facts.yaml”:

content => inline_template(“<%= scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }.to_yaml %>”),

}

Explanation

Sometimes you want to see all the variables available to a puppet client when it runs. Using an inline template the above snippet dumps all the variables that look like a string (and so helpfully filters some internal puppet settings) to a file. The output’s both human readable, which can help with debugging and easily machine par-sable. This ensures an up to date copy of the information is available for later processing on the client.

http://www.semicomplete.com/about/

Enable LSB facts

Challenge

You want to make the LSB facts available to facter and puppet

Solution

# on Debian

package { “lsb-release”: ensure => “installed” }

# on Redhat / Fedora

package { “redhat-lsb”: ensure => “installed” }

Explanation

Before the LSB facts can be used the package that provides the commands they wrap must be installed. While this is simple to do it’s not part of the base install of many Linux distributions and so should be done via Puppet.

Override a Facter fact

Challenge

You want to change the value of a fact retrieved from facter.

Solution

# show the operating system we are running (retrieved from facter)

$ puppet -e ‘notify { “We are running on $operatingsystem”: }’

notice: We are running on Fedora

# override $operatingsystem for testing purposes

$ FACTER_operatingsystem=Debian puppet -e ‘notify { “We are running on $operatingsystem”: }’

notice: We are running on Debian

Explanation

There will often be times, when writing manifests or facts, that you will want to test slightly different conditions than you are currently running under. By specifying the “FACTER_factname” environmental variable you can tweak the value you care about while leaving everything else in the run untouched.

Although the examples given in “Solution” are simplistic this technique can be very helpful when testing branching code, like the below, within your manifests:

$apache_server = $operatingsystem ? {

Fedora => “httpd”,

Debian => “apache2”,

}

Display fact resolution timings

Challenge

Facter takes a long time to run and you want to know which fact is the culprit.

Solution

# present in facter 1.5.9+

$ facter -t

…

facterversion: 0.04ms

arp: 25.87ms

puppetversion: 401.19ms

sshrsakey: 0.17ms

netmask_lo: 7.05ms

…

Explanation

There are times, mostly when you are writing your own facts, that something causes facter or puppet to take a lot longer than you would normally expect. Running facter with the “-t” option will trace each facts resolution and print how long it takes. This will often be enough to pinpoint the misbehaving fact and for the real work to begin.

USERS

Create a home directory for managed users

Challenge

You’ve added a user through puppet but its home directory was not created

Solution

user { “mcfakey”:

ensure => “present”,

managehome => true,

}

Explanation

When you create a user puppet does not automatically create its home directory because not all the providers support this functionality. If your systems provider can handle home directories (which you can check in the manages_homedir column of the user features table, and most Linux systems can, you can tell puppet to create the users home directory by specifying ‘managehome => true’.

User Password Management Fails in Puppet

Challenge

You want to control a users password through puppet but receive a warning.

Solution

user { ‘mcfakey’:

ensure => ‘present’,

password => ‘$1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv.’

}

# info: /User[mcfakey]: Provider useradd does not support features manages_passwords; not managing attribute password

Explanation

You are trying to manage a users password through puppet but you are receiving a message like this:

info: /User[mcfakey]: Provider useradd does not support features manages_passwords; not managing attribute password

This means that the ruby language support that puppet needs in order to control a users password is absent. If you’re running on CentOS / Fedora this can be fixed by installing the ‘ruby-shadow’ package and on Debian / Ubuntu ‘libshadow-ruby’ will fulfil the requirement. This is most often seen on gem or source based installs as your package manager should handle this dependency when installing puppet via a package.

It’s worth noting that the user resource will still be added but the password will not be set to the value you chose.

GROUPS

Create a group

Challenge

You want to create a group.

Solution

# minimum required.

group { “logusers”:

ensure => “present”,

}

# create a group with a specific GID.

group { “stats”:

gid => 2001,

}

Explanation

Creating a new group via puppet only requires the resource to be named and that ‘ensure => “present”’ be specified. If you specify a GID then you can even skip the ensure, otherwise you must provide it; and the new groups GID will be chosen by your operating system.

If you specify a different GID on a group that already exists then puppet will change the GID in /etc/group but it will not change the ownership of any existing files and directories. By default most Linux distributions will use the ‘groupadd’ provider, which doesn’t allow you to manage group members, so you’ll have to do it on the user resources instead.

Remove a group

Challenge

You want to remove an existing group.

Solution

group { “logusers”:

ensure => absent,

}

Explanation

Removing an existing group is simple in puppet, you only need to specify the group name and that it should be absent.

Change a Unix groups group ID

Challenge

You want to change a Unix groups group ID

Solution

# create a group with a specific GID.

group { “apacheadmins”:

gid => 2002,

}

# find files owned by group for permission cleanup

$ find -group apacheadmins

Explanation

Changing the group id (GID) of an existing group is very easy to do, but it’s often only part of the actual goal. While Puppet makes the change it doesn’t alter the GID of any of the groups existing files. You’ll need to do those separately. Either in puppet or with a handful of shell commands.

DEBUGGING

Adding debug messages to your manifests

Challenge

You want to add some debug messages to puppet so you can see what’s happening in your manifests.

Solution

# for debug output on the puppet master

notice(“Running with \$mysql_server_id ${mysql_server_id} ID defined”)

# for debug output on the puppet client

notify {“Running with \$mysql_server_id ${mysql_server_id} ID defined”:}

# for debug output on the puppet client – with full source information

notify {“Running with \$mysql_server_id ${mysql_server_id} ID defined”:

withpath => true,

}

Explanation

Sometimes you need to double check the value of a fact or variable at a given point or even just output a short message to show what puppet is doing. Puppet gives you a number of ways to do this and the simplest is by using the notice function if you want the message to appear on the puppet master or the notify type if you want the message displayed on the client. These messages are written to the log when puppet runs and can also be seen in the output of a debug run.

The addition of the ‘withpath’ attribute to a ‘notify’ resource causes it to print the full resource path with the message.

notify { ‘Hello World’: }

# notice: Hello World

notify { ‘Hello World’: withpath => true }

# notice: /Stage[main]/My::Test/Notify[Hello World]/message: Hello World

The last common variant is using a separate title and message:

notify { ‘FqdnTest’:

withpath => true,

name => “my fqdn is $fqdn”,

}

# notice: /Stage[main]/Not::Base/Notify[FqdnTest]/message: my fqdn is pcbtest.udlabs.private

View all the puppet variables on this client

Challenge

You want to see all the variables that are set for this client when puppet runs

Solution

file { “/tmp/facts.yaml”:

content => inline_template(“<%= scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }.to_yaml %>”),

}

Explanation

Sometimes you want to see all the variables available to a puppet client when it runs. Using an inline template the above snippet dumps all the variables that look like a string (and so helpfully filters some internal puppet settings) to a file. The output’s both human readable, which can help with debugging and easily machine parsable. This ensures an up to date copy of the information is available for later processing on the client.

Show Puppet Config Settings

Challenge

How do I find the value of a config setting in puppet?

Solution

# show the value of a single setting

$ puppet –configprint modulepath

/home/dean/.puppet/modules:/usr/share/puppet/modules

# show all config settings

$ puppet –configprint all

Explanation

Sometimes you just need to know what puppet “thinks” the value of a setting actually is. The easiest way to find this is to ask puppet directly by running “puppet –configprint modulepath” with either the setting you’re interested in or the argument “all” to all the current settings.

You should run the –configprint commands as the user that puppet runs as to avoid receiving information based on your user account:

# as me

$ puppet –configprint modulepath

/home/dwilson/.puppet/modules:/usr/share/puppet/modules

# as the root user – which puppet runs as on this system

$ sudo puppet –configprint modulepath

/etc/puppet/modules:/usr/share/puppet/modules

Display fact resolution timings

Challenge

Facter takes a long time to run and you want to know which fact is the culprit.

Solution

# present in facter 1.5.9+

$ facter -t

…

facterversion: 0.04ms

arp: 25.87ms

puppetversion: 401.19ms

sshrsakey: 0.17ms

netmask_lo: 7.05ms

…

Explanation

MISC

Why have trailing commas in resources?

Challenge

You’ve noticed how messy I am, leaving trailing commas throughout my examples and you’d like to know how I can be so inattentive. Well, there are a couple of reasons for leaving those stray little commas…

Solution

file { “/path/to/filename”:

owner => “root”,

group => “wheel”, # <– Unneeded comma?

}

Explanation

While those commas are not actually needed for the manifest to work I like to leave them in for two related reasons. The first is that they reduce the chance of a simple syntax error when you later go back and add some more settings to a resource. By always leaving a trailing comma you avoid all those niggling little errors.

The second reason is that it makes running diff over your version controlled manifests easier – and you are keeping them under version control aren’t you?

We have a very simple file resource that doesn’t have a trailing comma:

file { “/path/to/filename”:

owner => “root”,

group => “wheel”

}

We then update this file resource to include a mode – so it looks like this:

file { “/path/to/filename”:

owner => “root”,

group => “wheel”,

mode => 755

}

Now when we diff the changes, as we always do before committing them, we see that apparently two lines have changed. The line we made a real difference to and the line where the only amendment is the addition of a comma.

$ diff -u no-trailing updated

— no-trailing 2010-12-29 16:36:32.000000000 +0000

+++ updated 2010-12-29 16:38:24.000000000 +0000

@@ -1,4 +1,5 @@

file { “/path/to/filename”:

owner => “root”,

– group => “wheel”

+ group => “wheel”,

+ mode => 755

}

On the other hand, if we leave the trailing comma at the end of the line the diff becomes smaller and the actual change becomes more obvious

file { “/path/to/filename”:

owner => “root”,

group => “wheel”,

}

$ diff -u trailing updated

— trailing 2010-12-29 16:39:16.000000000 +0000

+++ updated 2010-12-29 16:39:22.000000000 +0000

@@ -1,4 +1,5 @@

file { “/path/to/filename”:

owner => “root”,

group => “wheel”,

+ mode => 755,

}

While this may not seem like a huge difference (pun intended) when you consider that you, and possibly others, will be re-reading these logs in the future, probably when somethings gone wrong, keeping changes and the resulting diffs as simple as possible is worth the little bit of additional effort

Autosigning Client Certificates

Challenge

You want to autosign any new client certificates that are sent to the puppet master. Be sure to understand the lack of security this presents.

Solution

$ cat /etc/puppet/autosign.conf

Explanation

By adding a single * to the autosign.conf file you tell the puppet master to accept the first certificate it sees for each client host. This allows machines to come up on build and immediately connect to puppet and begin their configuration. If you rebuild a machine, or do anything that triggers a change in the clients certificate, the puppet master will not allow the new certificate to connect until the old one has been removed.

To reiterate – having this option enabled may seem like a time saver but the risk is that any machine can connect without authorisation and request your manifests, which may contain privileged information such as passwords, certificates, shared keys etc.

You want some resource examples

Challenge

You’d like to see some example resources in the puppet configuration format

Solution

$ puppet resource package puppet # puppet 2.6 or greater

$ ralsh package puppet # puppet 2.5 or less

package { ‘puppet’:

ensure => ‘2.6.4-0.5.fc14’

}

# an example host file entry resource

$ puppet resource host localhost.localdomain # puppet 2.6 or greater

$ ralsh host localhost.localdomain # puppet 2.5 or less

host { ‘localhost.localdomain’:

ensure => ‘present’,

target => ‘/etc/hosts’,

ip => ‘127.0.0.1’,

host_aliases => [‘localhost’,’puppet’]

}

# an example user resource

$ puppet resource user daemon # puppet 2.6 or greater

$ ralsh user daemon # puppet 2.5 or less

user { ‘daemon’:

home => ‘/sbin’,

password_min_age => ‘0’,

ensure => ‘present’,

uid => ‘2’,

shell => ‘/sbin/nologin’,

password_max_age => ‘99999’,

password => ‘*’,

gid => ‘2’,

groups => [‘bin’,’daemon’,’adm’,’lp’],

comment => ‘daemon’

}

Explanation

COMMANDLINE

Adhoc Puppetry with puppet apply execute

Challenge

Sometimes you want to try out a tiny snippet of puppet code, maybe to see what a property does to a file or check if the file type takes owner or user as a parameter. The puppet apply command, with the ‘-e’ option allows you to run these kind of small snippets, often without requiring root.

Solution

$ puppet apply -e ‘file { “/tmp/adhoc”: content => “Written from puppet on $hostname\n” }’

notice: /Stage[main]//File[/tmp/adhoc]/ensure: defined content as ‘{md5}c1047ebc91c191f0ef6ad5fedcc5a0df’

$ cat /tmp/adhoc

Written from puppet

$ ls -al /tmp/adhoc

-rw-r–r–. 1 deanw deanw 20 Dec 30 13:59 /tmp/adhoc

$ puppet apply -e ‘file { “/tmp/adhoc”: content => “Written from puppet\n”, mode => 640 }’

notice: /Stage[main]//File[/tmp/adhoc]/mode: mode changed ‘644’ to ‘640’

$ ls -al /tmp/adhoc

-rw-r—–. 1 deanw deanw 20 Dec 30 13:59 /tmp/adhoc

Explanation

This is a simple way to allow easy testing of small puppet snippets while you become more familiar with the configuration language. As this is a full instance of puppet you can use facts in your snippets but when you find yourself going over a line or two it may well be time to investigate stand alone module testing.

Syntax check your manifests

Challenge

We all make mistakes, the trick is to quickly find and fix them.

Solution

# if we have this (note the missing comma after mode) in our manifest

file { “/tmp/broken_resource”:

mode => 644

owner => “root”,

}

# to syntax check it we run

$ puppet –parseonly unixdaemon/manifests/init.pp

# and we get –

err: Could not parse for environment production: Syntax error at ‘owner’;

expected ‘}’ at /etc/puppet/modules/unixdaemon/manifests/init.pp:5

# check all .pp files

$ find -name ‘*.pp’ | xargs -n 1 -t puppet –parseonly

Explanation

Finding a typo in your manifest when you’re writing it is much better than discovering it in the puppet master logs. By syntax checking your manifests as you’re working, and always before a commit, you can catch some of the more obvious errors before they escape on to the network.

This is only a basic syntax check, it won’t find misspelled properties or even incorrect resource names.

Which files are puppet managed?

Challenge

How do I tell if filename is under puppet control?

Solution

# show all the puppet managed files in a directory

$ puppet-ls /etc/mcollective

/etc/mcollective/facts.yaml

/etc/mcollective/server.cfg

# show all the puppet managed files in /etc and any of its subdirectories

$ puppet-ls -r /etc/

# show all unmanaged files in /etc/nagios and any subdirectories

$ puppet-ls -r -i /etc/nagios/

Explanation

While it’s possible to search through the clients catalog and state.yaml files for each file you want to check the simplest way is to download and run puppet-ls from github.

The puppet-ls script will show all the puppet managed files in the given directory (or the current directory if called without arguments), can be made to check recursively (with -r) or can reverse its behavior (when passed -i for invert) to show all files that are not yet puppet managed.

TESTING

Syntax check your manifests

Challenge

We all make mistakes, the trick is to quickly find and fix them.

Solution

# if we have this (note the missing comma after mode) in our manifest

file { “/tmp/broken_resource”:

mode => 644

owner => “root”,

}

# to syntax check it we run

$ puppet –parseonly unixdaemon/manifests/init.pp

# and we get –

err: Could not parse for environment production: Syntax error at ‘owner’;

expected ‘}’ at /etc/puppet/modules/unixdaemon/manifests/init.pp:5

# check all .pp files

$ find -name ‘*.pp’ | xargs -n 1 -t puppet –parseonly

Explanation

This is only a basic syntax check, it won’t find misspelled properties or even incorrect resource names.

Override a Facter fact

Challenge

You want to change the value of a fact retrieved from facter.

Solution

# show the operating system we are running (retrieved from facter)

$ puppet -e ‘notify { “We are running on $operatingsystem”: }’

notice: We are running on Fedora

# override $operatingsystem for testing purposes

$ FACTER_operatingsystem=Debian puppet -e ‘notify { “We are running on $operatingsystem”: }’

notice: We are running on Debian

Explanation

Although the examples given in “Solution” are simplistic this technique can be very helpful when testing branching code, like the below, within your manifests:

$apache_server = $operatingsystem ? {

Fedora => “httpd”,

Debian => “apache2”,

CRON

Managing entries under /etc/cron.d

Challenge

You want to manage one or more entries in a file located under “/etc/cron.*”

Solution

Manage the entire crontab file. As a single file, a concat, a define or via Augeas.

Explanation

The built in Puppet type only manages per user crontab files (the kind that exist under /var/spool/cron on Linux). There is currently no built in way to explicitly manage cron file entries under /etc, although you can use many of the other types as a work around.

ERRORS

User Password Management Fails in Puppet

Challenge

You want to control a users password through puppet but receive a warning.

Solution

user { ‘mcfakey’:

ensure => ‘present’,

password => ‘$1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv.’

}

# info: /User[mcfakey]: Provider useradd does not support features manages_passwords; not managing attribute password

Explanation

You are trying to manage a users password through puppet but you are receiving a message like this:

info: /User[mcfakey]: Provider useradd does not support features manages_passwords; not managing attribute password

It’s worth noting that the user resource will still be added but the password will not be set to the value you chose.

Fileserver In Template call

Challenge

Puppet reports “module names must be alphanumeric (plus ‘-‘), not ‘puppet:”

Solution

# this is invalid

file { ‘/tmp/filesources’:

content => template(‘puppet:///modules/mymodule/mymodule.conf.erb’),

}

# and will cause an error like this to appear

# Invalid module name; module names must be alphanumeric (plus ‘-‘), not ‘puppet:’

# at /my/path/init.pp:4 on node pcb.unixdaemon.private

# instead it should look like this

file { ‘/tmp/filesources’:

content => template(‘mymodule/mymodule.conf.erb’),

}

Explanation

When running puppet you see lines that look like this on your screen or in the logs:

Invalid module name; module names must be alphanumeric (plus ‘-‘), not ‘puppet:’

at /my/path/init.pp:4 on node pcb.unixdaemon.private

This is often caused when you confuse the ‘content’ and ‘source’ attributes and mostly when you are refactoring a resource from a static file to one that uses a template for dynamic content. This is an annoying mistake as you can easily miss it due to the syntax being correct – just not in this context.

File source without specifying ‘modules’

Challenge

You want to clear “DEPRECATION NOTICE: Files found in modules without specifying ‘modules’” messages

Solution

# this will cause the deprecation notice

file { ‘/etc/mymodule.conf’:

source => ‘puppet:///mymodule/mymodule.conf’,

}

# this won’t cause the deprecation notice

file { ‘/etc/mymodule.conf’:

source => ‘puppet:///modules/mymodule/mymodule.conf’,

}

Explanation

When running puppet you see lines that look like this on your screen or in the logs:

DEPRECATION NOTICE: Files found in modules without specifying ‘modules’

in file path will be deprecated in the next major release. Please fix

module ‘mymodule’ when no 0.24.x clients are present

This is an ever decreasingly seen deprecation notice that was introduced, and was most often seen, when upgrading from the 0.24 to .25 series of puppet releases. The fix is simple, adding the literal ‘modules’ to the path in your source lines after specifying the server is enough to fix the complaint, and could often be automatically corrected with a little scripting or sed.

It is now most often seen when copying example code from older blog posts or sample modules.

Puppet SSL additional information

Purpose of Puppet SSL PKI

The current puppet security layer has 3 aims:

authenticate any node to the master (so that no rogue node can get a catalog from your master)
authenticate the master on any node (so that your nodes are not tricked into getting a catalog from a rogue master).
prevent communication eavesdropping between master and nodes (so that no rogue users can grab configuration secrets by listening to your traffic, which is useful in the cloud)

A public key cryptographic system works like this:

every components of the system has a secret key (known as the private key) and a public key (this one can be shared with other participant of the system). The public and private keys are usually bound by a cryptographic algorithm.
authentication of any component is done with a simple process: a component signs a message with its own private key. The receiver can authenticate the message (ie know the message comes from the original component) by validating the signature. To do this, only the public key is needed.

There are different public/private key pair cryptosystem, the most known ones are RSA, DSA or those based on Elliptic Curve cryptography.

Usually it is not good that all participants of the system must know each other to communicate. So most of the current PKI system use a hierarchical validation system, where all the participant in the system must only know one of the parent in the hierarchy to be able to validate each others.

X509 PKI

X509 is an ITU-T standard of a PKI. It is the base of the SSL protocol authentication that puppet use. This standard specifies certificates, certificate revocation list, authority and so on…

A given X509 certificate contains several information like those:

Serial number (which is unique for a given CA)
Issuer (who created this certificate, in puppet this is the CA)
Subject (who this certificate represents, in puppet this is the node certname or fqdn)
Validity (valid from, expiration date)
Public key (and what kind of public key algorithm has been used)
Various extensions (usually what this certificate can be used for,…)

You can check RFC1422 for more details.

The certificate is usually the DER encoding of the ASN.1 representation of those informations, and is usually stored as PEM for consumption.

A given X509 certificate is signed by what we call a Certificate Authority (CA for short). A CA is an infrastructure that can sign new certificates. Anyone sharing the public key of the CA can validate that a given certificate has been validated by the CA.

Usually X509 certificate embeds a RSA public key with an exponent of 0x100001 (see below). Along with a certificate, you need a private key (usually also PEM-encoded).

So basically the X509 system works with the following principle: CA are using their own private keys to sign components certificates, it is the CA role to sign only trusted component certificates. The trust is usually established out-of-bound of the signing request.

Then every component in the system knows the CA certificate (ie public key). If one component gets a message from another component, it checks the attached message signature with the CA certificate. If that validates, then the component is authenticated. Of course the component should also check the certificate validity, if the certificate has been revoked (from OCSP or a given CRL), and finally that the certificate subject matches who the component pretends to be (usually this is an hostname validation against some part of the certificate Subject)

RSA system

Most of X509 certificate are based on the RSA cryptosystem, so let’s see what it is.

The RSA cryptosystem is a public key pair system that works like this:

Key Generation

To generate a RSA key, we chose two prime number p and q.

We compute n=pq. We call n the modulus.

We compute φ(pq) = (p − 1)(q − 1).

We chose e so that e>1 and e<φ(pq) (e and φ(pq) must be coprime). e is called the exponent. It usually is 0x10001 because it greatly simplifies the computations later (and you know what I mean if you already implemented this :)).

Finally we compute d=e^-1 mod((p-1)(q-1)). This will be our secret key. Note that it is not possible to get d from only e (and since p and q are never kept after the computation this works).

In the end:

e and n form the public key
d is our private key

Encryption

So the usual actors when describing cryptosystems are Alice and Bob. Let’s use them.

Alice wants to send a message M to Bob. Alice knows Bob’s public key (e,n). She transform M in a number < n (this is called padding) that we’ll call m, then she computes: _c = m^e . mod(n) _

Decryption

When Bob wants to decrypt the message, he computes with his private key d: m = c^d . mod(n)

Signing message

Now if Alice wants to sign a message to Bob. She first computes a hash of her message called H, then she computes: s = H^{d mod n}. So she used her own private key. She sends both the message and the signature.

Bob, then gets the message computes H and computes h’ = H^{e mod n} with Alice’s public key. If h’ = h, then only Alice could have sent it.

Security

What makes this scheme work is the fundamental that finding p and q from n is a hard problem (understand for big values of n, it would take far longer than the validity of the message). This operation is called factorization. Current certificate are numbers containing 2048 bits, which roughly makes a 617 digits number to factor.

How does this fit in SSL?

So SSL (which BTW means Secure Socket Layer) and now TLS (SSL successor) is a protocol that aims to provide security of communications between two peers. It is above the transport protocol (usually TCP/IP) in the OSI model. It does this by using symmetric encryption and message authentication code (MAC for short). The standard is (now) described in RFC5246.

It works by first performing an handshake between peers. Then all the remaining communications are encrypted and tamperproof.

This handshake contains several phases (some are optional):

Client and server finds the best encryption scheme and MAC from the common list supported by both the server and the clients (in fact the server choses).
The server then sends its certificate and any intermediate CA that the client might need
The server may ask for the client certificate. The client may send its certificate.
Both peers may validate those certificates (against a common CA, from the CRL, etc…)
They then generate the session keys. The client generates a random number, encrypts it with the server public key. Only the server can decrypt it. From this random number, both peers generate the symmetric key that will be used for encryption and decryption.
The client may send a signed message of the previous handshake message. This way the server can verify the client knows his private key (this is the client validation). This phase is optional.

After that, each message is encrypted with the generated session keys using a symmetric cipher, and validated with an agreed on MAC. Usual symmetric ciphers range from RC4 to AES. A symmetric cipher is used because those are usually way faster than any asymmetric systems.

Application to Puppet

Puppet defines it’s own Certificate Authority that is usually running on the master (it is possible to run a CA only server, for instance if you have more than one master).

This CA can be used to:

generate new certificate for a given client out-of-bound
sign a new node that just sent his Certificate Signing Request
revoke any signed certificate
display certificate fingerprints

What is important to understand is the following:

Every node knows the CA certificate. This allows to check the validity of the master from a node
The master doesn’t need the node certificate, since it’s sent by the client when connecting. It just need to make sure the client knows the private key and this certificate has been signed by the master CA.

It is also important to understand that when your master is running behind an Apache proxy (for Passenger setups) or Nginx proxy (ie some mongrel setups):

The proxy is the SSL endpoint. It does all the validation and authentication of the node.
Traffic between the proxy and the master happens in clear
The master knows the client has been authenticated because the proxy adds an HTTP header that says so (usually X-Client-Verify for Apache/Passenger).

When running with webrick, webrick runs inside the puppetmaster process and does all this internally. Webrick tells the master internally if the node is authenticated or not.

When the master starts for the 1st time, it generates its own CA certificate and private key, initializes the CRL and generates a special certificate which I will call the server certificate. This certificate will be the one used in the SSL/TLS communication as the server certificate that is later sent to the client. This certificate subject will be the current master FQDN. If your master is also a client of itself (ie it runs a puppet agent), I recommend using this certificate as the client certificate.

The more important thing is that this server certificate advertises the following extension:

X509v3 Subject Alternative Name:

DNS:puppet, DNS:$fqdn, DNS:puppet.$domain

What this means is that this certificate will validate if the connection endpoint using it has any name matching puppet, the current fqdn or puppet in the current domain.

By default a client tries to connect to the “puppet” host (this can be changed with —server which I don’t recommend and is usually the source of most SSL trouble).

If your DNS system is well behaving, the client will connect to puppet.$domain. If your DNS contains a CNAME for puppet to your real master fqdn, then when the client will validate the server certificate it will succeed because it will compare “puppet” to one of those DNS: entries in the aforementioned certificate extension. BTW, if you need to change this list, you can use the —certdnsname option (note: this can be done afterward, but requires to re-generate the server certificate).

The whole client process is the following:

if the client runs for the 1st time, it generates a Certificate Signing Request and a private key. The former is an x509 certificate that is self-signed.
the client connects to the master (at this time the client is not authenticated) and sends its CSR, it will also receives the CA certificate and the CRL in return.
the master stores locally the CSR
the administrator checks the CSR and can eventually sign it (this process can be automated with autosigning). I strongly suggest verifying certificate fingerprint at this stage.
the client is then waiting for his signed certificate, which the master ultimately sends
All next communications will use this client certificate. Both the master and client will authenticate each others by virtue of sharing the same CA.

Troubleshooting SSL

Certificate content

First you can check any certificate content with this: openssl x509 -text -in /var/lib/puppet/ssl/private_keys/ip-172-31-38-148.us-west-2.compute.internal.pem

root@ip-172-31-38-148:~# openssl x509 -text -in /var/lib/puppet/ssl/ca/ca_crt.pem

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1 (0x1)

Signature Algorithm: sha1WithRSAEncryption

Issuer: CN=Puppet CA: ip-172-31-38-148.us-west-2.compute.internal

Validity

Not Before: Dec 2 02:18:15 2013 GMT

Not After : Dec 2 02:18:15 2018 GMT

Subject: CN=Puppet CA: ip-172-31-38-148.us-west-2.compute.internal

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (1024 bit)

Modulus:

00:b6:62:4d:ad:8d:da:60:a9:8a:0a:f4:08:88:4a:

e4:af:e4:d4:38:4e:71:0d:32:91:99:8b:7b:7a:76:

fb:b4:c2:0c:d5:ff:57:4d:19:1d:5f:ce:ce:70:1d:

f4:d9:ff:00:6e:e5:da:8d:ab:05:b0:24:ba:2a:0c:

f0:9c:1a:ec:74:a9:58:00:08:49:39:5b:69:c4:e0:

c5:0b:dc:73:74:f0:2e:c1:28:f4:16:df:59:58:d7:

b8:2e:1d:b9:15:af:26:b4:00:e5:96:25:98:07:d7:

86:95:cf:5a:63:1f:7b:e1:bf:ee:de:9e:f3:28:0a:

65:4b:54:16:da:78:e9:4b:35

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Certificate Sign, CRL Sign

X509v3 Subject Key Identifier:

88:1D:CB:89:9C:81:B9:45:E1:31:82:3E:59:F4:C2:D1:AA:8B:63:BC

X509v3 Basic Constraints: critical

CA:TRUE

Netscape Comment:

Puppet Ruby/OpenSSL Internal Certificate

Signature Algorithm: sha1WithRSAEncryption

7b:d3:e7:b2:53:b0:7d:37:95:d2:00:52:1b:74:06:36:37:97:

4c:d3:7a:3d:f2:b1:36:bd:7a:60:43:bd:f8:91:07:65:27:48:

f3:ae:2b:89:07:38:ce:f7:bc:37:c3:60:f8:c2:b3:e5:0f:97:

23:60:98:13:98:4c:08:6b:1e:86:9e:01:c3:d1:a8:9b:6a:c0:

e0:0a:26:b1:29:a2:70:b8:fd:c6:d2:2d:00:a8:9b:17:53:e3:

2b:03:68:25:cb:04:0e:86:07:b5:3d:79:bd:fe:cf:bd:6e:6b:

ad:f9:d9:83:99:b8:06:f6:48:43:84:99:c9:2e:d5:21:50:cf:

3b:ff

—–BEGIN CERTIFICATE—–

MIICczCCAdygAwIBAgIBATANBgkqhkiG9w0BAQUFADBBMT8wPQYDVQQDDDZQdXBw

ZXQgQ0E6IGlwLTE3Mi0zMS0zOC0xNDgudXMtd2VzdC0yLmNvbXB1dGUuaW50ZXJu

YWwwHhcNMTMxMjAyMDIxODE1WhcNMTgxMjAyMDIxODE1WjBBMT8wPQYDVQQDDDZQ

dXBwZXQgQ0E6IGlwLTE3Mi0zMS0zOC0xNDgudXMtd2VzdC0yLmNvbXB1dGUuaW50

ZXJuYWwwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALZiTa2N2mCpigr0CIhK

5K/k1DhOcQ0ykZmLe3p2+7TCDNX/V00ZHV/OznAd9Nn/AG7l2o2rBbAkuioM8Jwa

7HSpWAAISTlbacTgxQvcc3TwLsEo9BbfWVjXuC4duRWvJrQA5ZYlmAfXhpXPWmMf

e+G/7t6e8ygKZUtUFtp46Us1AgMBAAGjezB5MA4GA1UdDwEB/wQEAwIBBjAdBgNV

HQ4EFgQUiB3LiZyBuUXhMYI+WfTC0aqLY7wwDwYDVR0TAQH/BAUwAwEB/zA3Bglg

hkgBhvhCAQ0EKhYoUHVwcGV0IFJ1YnkvT3BlblNTTCBJbnRlcm5hbCBDZXJ0aWZp

Y2F0ZTANBgkqhkiG9w0BAQUFAAOBgQB70+eyU7B9N5XSAFIbdAY2N5dM03o98rE2

vXpgQ734kQdlJ0jzriuJBzjO97w3w2D4wrPlD5cjYJgTmEwIax6GngHD0aibasDg

CiaxKaJwuP3G0i0AqJsXU+MrA2glywQOhge1PXm9/s+9bmut+dmDmbgG9khDhJnJ

LtUhUM87/w==

—–END CERTIFICATE—–

Simulate a SSL connection

You can know more information about a SSL error by simulating a client connection. Log in the trouble node and:

openssl s_client -host puppet -port 8140 -cert /path/to/ssl/certs/node.domain.com.pem -key /path/to/ssl/private_keys/node.domain.com.pem -CAfile /path/to/ssl/certs/ca.pem

ssldump

Using ssldump or wireshark you can also learn more about ssl issues. For this to work, it is usually needed to force the cipher to use a simple cipher like RC4 (and also ssldump needs to know the private keys if you want it to decrypt the application data).

Some known issues

Also, in case of SSL troubles make sure your master isn’t using a different $ssldir than what you are thinking. If that happens, it’s possible your master is using a different dir and has regenerated its CA. If that happens no one node can connect to it anymore. This can happen if you upgrade a master from gem when it was installed first with a package (or the reverse).

If you regenerate a host, but forgot to remove its cert from the CA (with puppetca —clean), the master will refuse to sign it. If for any reason you need to fully re-install a given node without changing its fqdn, either use the previous certificate or clean this node certificate (which will automatically revoke the certificate for your own security).

Looking to the CRL content:

openssl crl -text -in /var/lib/puppet/ssl/ca/ca_crl.pem

Fingerprinting

Since puppet 2.6.0, it is possible to fingerprint certificates. If you manually sign your node, it is important to make sure you are signing the correct node and not a rogue system trying to pretend it is some genuine node. To do this you can get the certificate fingerprint of a node by running puppet agent —fingerprint, and when listing on the master the various CSR, you can make sure both fingerprint match

Client: puppet agent –test –fingerprint

Master Server: puppetca –list node.domain.com –fingerprint