Chef From The Start To The Beginning

published: 27th of April 2019

Intro

Chef is an infrastructure automation tool similar to Puppet and Salt. In this post I will setup a Chef infrastructure consisting of a Chef server, node and workstation to manage the infrastructure.

In April 2019 Chef announced that they are open sourcing all of their products under the Apache 2.0 license (A lot of Chef was already open source). This is fantastic and is the main reason I decided to dig deeper into Chef.

Chef has been around for a long time, it was initially released in 2009. There is a lot of information out there in which things have changed either slightly (or greatly) in Chef land and it was a bit of a journey to get all the pieces working together.

Chef Architecture

blog/chef-from-the-start-to-the-beginning/chef-infra.svg

Components

Chef Server - Central point of policy enforcement
Chef Node - Hosts that are managed by the Chef server
Chef Supermarket - Public repository for cookbooks
Cookbook - Collection of recipes
Recipe - List of tasks to apply to a node
Runlist - List of cookbooks to apply to a node

There are three on premises deployment models for the Chef server as well as a cloud hosted version. The on-premises deployments models are as follows.

Standalone
High availability
Tiered

The system requirements depend on how many Chef nodes are under management of the Chef server. I will install the standalone server on a Centos 7 VM with 2 CPUs and 4GB of RAM in this lab.

The following software versions will be installed.

Chef Server - 12.19.31
Chef Client - 14.12.9
Chef Workstation - 0.2.53

Chef Server Prerequisites

If Apache QPID is installed it will need to be disabled. You can check the install status with the rpm -qa command.

    cmd
  
    rpm -qa | grep qpid

If it is installed, follow the instructions here. to disable it.

Next up, an SELinux profile will not be configured by the Chef server installer. Chef advises to set SELinux to permissive and you need to figure out the polices yourself if you require enforcing to be enabled. Further info can be found here.

Set SELinux to permissive mode.

    cmd
  
    sudo setenforce Permissive
sudo sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

Now open up the required firewall ports.

    cmd
  
    sudo firewall-cmd --zone=public --add-service=http --permanent
sudo firewall-cmd --zone=public --add-service=https --permanent
sudo firewall-cmd --reload

Chef Server Installation

Download and install Chef server.

    cmd
  
    curl -O https://packages.chef.io/files/stable/chef-server/12.19.31/el/7/chef-server-core-12.19.31-1.el7.x86_64.rpm
sudo yum install -y chef-server-core-12.19.31-1.el7.x86_64.rpm

The web management console is free for upto 25 nodes (Possibly for unlimited nodes in the future). Although it is not strictly required, it gives you a nice visualisation of the infrastructure. Install it with the below command.

    cmd
  
    sudo chef-server-ctl install chef-manage

Now start up all the Chef services

    cmd
  
    sudo chef-server-ctl reconfigure

Start the Web management server and accept the license agreement.

    cmd
  

    sudo chef-manage-ctl reconfigure

# output
To use this software, you must agree to the terms of the software license agreement.
Press any key to continue.
Type 'yes' to accept the software license agreement, or anything else to cancel.
  

Chef Server Configuration

Users interact with the Chef server using certificates. Create a .chef/ directory in the users home directory to store the certificate.

    cmd
  
    sudo mkdir /home/bradmin/.chef/
sudo chown bradmin:bradmin /home/bradmin/.chef/
sudo chmod 0700 /home/bradmin/.chef/

Create a Chef server account and private key for the user.

    cmd
  
    # sudo chef-server-ctl user-create USERNAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename /path/to/username.pem

sudo chef-server-ctl user-create bradmin brad searle bradleysearle@lab.local 'SuperSecretPassword' --filename /home/bradmin/.chef/bradmin.pem

Note

Users can login to the web console and change their password.

Adjust the permissions of the .pem file

    cmd
  
    sudo chown bradmin:bradmin /home/bradmin/.chef/bradmin.pem
sudo chmod 0600 /home/bradmin/.chef/bradmin.pem

Finally create an Organisation.

    cmd
  
    sudo chef-server-ctl org-create testlab 'Test Lab' --association_user bradmin --filename /home/bradmin/.chef/testlab-validator.pem

That's it for the Chef server install. Browse to the url https://<hostname-or-ip>. Login with the user account created in the previous step and have a poke around.

Chef Workstation Installation

The Chef workstation package is a convenient way to get your workstation setup to communicate with the Chef server. The packages includes CLI utilities such as: chef, knife and berks.

A brief rundown of the utilities are as follows

chef - Helper commands for generating boilerplate
knife - Interact with Chef server and / or Nodes
berks - Manage cookbooks dependencies and installation

Download the workstation package for you system (I an using an Ubuntu workstation).

    cmd
  
    curl -O https://packages.chef.io/files/stable/chef-workstation/0.2.53/ubuntu/18.04/chef-workstation_0.2.53-1_amd64.deb

Install the workstation package.

    cmd
  
    sudo dpkg -i chef-workstation_0.2.53-1_amd64.deb

Chef Workstation Setup

Everything you need to do is done inside a repo directory called chef-repo. The chef command has a bunch of helper commands called generators. Generators help you to avoid writing boilerplate.

Create a chef-repo directory

    cmd
  
    chef generate repo chef-repo

This creates the following directory structure.

    cmd
  

    tree chef-repo/

# output
chef-repo/
├── chefignore
├── cookbooks
│   ├── example
│   │   ├── attributes
│   │   │   └── default.rb
│   │   ├── metadata.rb
│   │   ├── README.md
│   │   └── recipes
│   │       └── default.rb
│   └── README.md
├── data_bags
│   ├── example
│   │   └── example_item.json
│   └── README.md
├── environments
│   ├── example.json
│   └── README.md
├── LICENSE
├── README.md
└── roles
    ├── example.json
    └── README.md
  

Change into the chef-repo directory and create a .chef/ directory. The .chef/ will be used to store your private key.

    cmd
  
    cd chef-repo && mkdir .chef/

Add the .chef/ directory to the .gitignore file.

    cmd
  
    echo '.chef/' >> .gitignore

Copy the private key from the Chef server to your .chef/ directory.

    cmd
  
    scp bradmin@chefserver:~/.chef/bradmin.pem .chef/

Adjust the permissions to ensure only you have access.

    cmd
  
    chmod 0700 .chef/
chmod 0600 .chef/bradmin.pem

Chef uses the knife utility to communicate with the Chef server. Create a knife.rb in the .chef/ directory that defines the required parameters.

    cmd
  

    tee .chef/knife.rb > /dev/null << "EOF"
current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "bradmin"
client_key               "#{current_dir}/bradmin.pem"
chef_server_url          "https://chefserver01.lab.local/organizations/testlab"
syntax_check_cache_path  "#{ENV['HOME']}/.chef/syntaxcache"
cookbook_path            ["#{current_dir}/../cookbooks"]
EOF
  

To setup your connection to the Chef server you first need to download the servers SSL keys.

    cmd
  
    knife ssl fetch

# output
WARNING: Certificates from chefserver01.lab.local will be fetched and placed in your trusted_cert
directory (/home/bradmin/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for chefserver01_lab_local in /home/bradmin/chef-repo/.chef/trusted_certs/chefserver01_lab_local.crt

Confirm you can now communicate with the Chef server.

    cmd
  
    knife client list

# output
testlab-validator

Note

A successful connection will return <organisation>-validator.

Cookbooks

Cookbooks are a group of recipes that perform actions on a node.

The chef-client cookbook is developed by the Chef team and is used to manage the .... you guessed it chef-client utility. The chef-client is how managed nodes communicate with the Chef server.

A Cookbooks functionality can be customised by including it in your own cookbooks in a similar way to Class inheritance. Upstream cookbooks are not edited directly you create a wrapper cookbook that includes the upstream cookbook and the customisations you require.

Generate a wrapper cookbook for the base chef-client cookbook.

    cmd
  
    chef generate cookbook cookbooks/chef_client_wrapper

Add the chef-client cookbook as a dependency to the chef_client_wrapper cookbook metadata.rb file using the depends keyword.

    cmd
  

    # cookbooks/chef_client_wrapper/metadata.rb

name 'chef_client_wrapper'
maintainer 'The Authors'
maintainer_email 'you@example.com'
license 'All Rights Reserved'
description 'Installs/Configures chef_client_wrapper'
long_description 'Installs/Configures chef_client_wrapper'
version '0.1.0'
chef_version '>= 13.0'

# Add chef-client as a dependency
depends 'chef-client', '~> 11.1.3'
  

Note

It is good practice to pin the vrsion of your dependency cookbooks. ~> 11.1.3 in the example above.

As a minimum you need to include the chef-client cookbooks default recipe in the wrapper cookbooks recipes/default.rb file.

    file
  
    # cookbooks/chef_client_wrapper/recipes/default.rb

# Cookbook:: chef_client_wrapper
# Recipe:: default
#
# Copyright:: 2019, The Authors, All Rights Reserved.

include_recipe 'chef-client::default'

Use the berks utility to download the dependency cookbooks and upload them to the Chef server.

Note

The berks utility looks for a file called Berksfile in the currently directory. You will need to change into the directory containing the Berksfile whenever you use the berks utility.

    cmd
  
    cd cookbooks/chef_client_wrapper/

The berks install command fetches the dependency cookbook and also any dependency cookbooks it might have to you local workstation.

    cmd
  

    berks install

# output
Resolving cookbook dependencies...
Fetching 'chef_client_wrapper' from source at .
Fetching cookbook index from https://supermarket.chef.io...
Using chef-client (11.1.3)
Using logrotate (2.2.0)
Using cron (6.2.1)
Using chef_client_wrapper (0.1.0) from source at .
  

The berks upload command uploads the cookbook and all of the dependency cookbooks to the Chef server.

    cmd
  

    berks upload

# output
Uploaded cron (6.2.1) to: 'https://chefserver01.lab.local/organizations/testlab'
Uploaded logrotate (2.2.0) to: 'https://chefserver01.lab.local/organizations/testlab'
Uploaded chef-client (11.1.3) to: 'https://chefserver01.lab.local/organizations/testlab'
Uploaded chef_client_wrapper (0.1.0) to: 'https://chefserver01.lab.local/organizations/testlab'
  

Now that the workstation is all configured and the chef_client_wrapper cookbook and its dependencies are installed on the Chef server we are ready to bootstrap our Chef nodes.

Node Bootstrap

Bootstrapping a node will install the Chef client on the node and bring it under management of the Chef server. During the bootstrap process we apply the chef_client_wrapper cookbook which will start the chef-client as a daemon and configure the node to check-in with the Chef server for updates every 60 minutes.

The knife utility is used to bootstrap a chef node.

    cmd
  

    knife bootstrap chefnode01.lab.local -N chefnode01.lab.local -x bradmin --sudo -r 'recipe[chef_client_wrapper]'

# output
Creating new client for chefnode01.lab.local
Creating new node for chefnode01.lab.local
Connecting to chefnode01.lab.local
Warning: Permanently added '192.168.255.10' (ECDSA) to the list of known hosts.
chefnode01.lab.local knife sudo password:
Enter your password:
chefnode01.lab.local
chefnode01.lab.local -----> Installing Chef Omnibus (-v 14)
chefnode01.lab.local downloading https://omnitruck-direct.chef.io/chef/install.sh
chefnode01.lab.local   to file /tmp/install.sh.19787/install.sh
chefnode01.lab.local trying curl...
chefnode01.lab.local el 7 x86_64
chefnode01.lab.local Getting information for chef stable 14 for el...
chefnode01.lab.local downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=14&p=el&pv=7&m=x86_64
chefnode01.lab.local   to file /tmp/install.sh.19793/metadata.txt
chefnode01.lab.local trying curl...
chefnode01.lab.local sha1	0c987093a78f7a7c70e635301eb8f82f8f36c4a7
chefnode01.lab.local sha256	749a0550b220a2a50ce7744eb32ab8959840cb3d870cc19e58e10bd86726693a
chefnode01.lab.local url	https://packages.chef.io/files/stable/chef/14.12.9/el/7/chef-14.12.9-1.el7.x86_64.rpm
chefnode01.lab.local version	14.12.9
chefnode01.lab.local downloaded metadata file looks valid...
chefnode01.lab.local downloading https://packages.chef.io/files/stable/chef/14.12.9/el/7/chef-14.12.9-1.el7.x86_64.rpm
chefnode01.lab.local   to file /tmp/install.sh.19793/chef-14.12.9-1.el7.x86_64.rpm
chefnode01.lab.local trying curl...
chefnode01.lab.local Comparing checksum with sha256sum...
chefnode01.lab.local Installing chef 14
chefnode01.lab.local installing with rpm...
chefnode01.lab.local warning: /tmp/install.sh.19793/chef-14.12.9-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
chefnode01.lab.local Preparing...                          ################################# [100%]
chefnode01.lab.local Updating / installing...
chefnode01.lab.local    1:chef-14.12.9-1.el7               ################################# [100%]
chefnode01.lab.local Thank you for installing Chef!
chefnode01.lab.local Starting the first Chef Client run...
chefnode01.lab.local Starting Chef Client, version 14.12.9
chefnode01.lab.local resolving cookbooks for run list: ["chef_client_wrapper"]
chefnode01.lab.local Synchronizing Cookbooks:
chefnode01.lab.local   - chef_client_wrapper (0.1.0)
chefnode01.lab.local   - chef-client (11.1.3)
chefnode01.lab.local   - cron (6.2.1)
chefnode01.lab.local   - logrotate (2.2.0)
chefnode01.lab.local Installing Cookbook Gems:
chefnode01.lab.local Compiling Cookbooks...
chefnode01.lab.local Converging 10 resources
chefnode01.lab.local Recipe: chef-client::systemd_service
chefnode01.lab.local   * directory[/var/run/chef] action create
chefnode01.lab.local     - create new directory /var/run/chef
chefnode01.lab.local     - change owner from '' to 'root'
chefnode01.lab.local     - change group from '' to 'root'
chefnode01.lab.local     - restore selinux security context
chefnode01.lab.local   * directory[/var/cache/chef] action create
chefnode01.lab.local     - create new directory /var/cache/chef
chefnode01.lab.local     - change owner from '' to 'root'
chefnode01.lab.local     - change group from '' to 'root'
chefnode01.lab.local     - restore selinux security context
chefnode01.lab.local   * directory[/var/lib/chef] action create
chefnode01.lab.local     - create new directory /var/lib/chef
chefnode01.lab.local     - change owner from '' to 'root'
chefnode01.lab.local     - change group from '' to 'root'
chefnode01.lab.local     - restore selinux security context
chefnode01.lab.local   * directory[/var/log/chef] action create
chefnode01.lab.local     - create new directory /var/log/chef
chefnode01.lab.local     - change mode from '' to '0755'
chefnode01.lab.local     - change owner from '' to 'root'
chefnode01.lab.local     - change group from '' to 'root'
chefnode01.lab.local     - restore selinux security context
chefnode01.lab.local   * directory[/etc/chef] action create (up to date)
chefnode01.lab.local   * template[/etc/sysconfig/chef-client] action create
chefnode01.lab.local     - create new file /etc/sysconfig/chef-client
chefnode01.lab.local     - update content in file /etc/sysconfig/chef-client from none to ec7de1
chefnode01.lab.local     --- /etc/sysconfig/chef-client	2019-04-27 20:50:44.973462558 +1000
chefnode01.lab.local     +++ /etc/sysconfig/.chef-chef-client20190427-19787-47a021	2019-04-27 20:50:44.972462567 +1000
chefnode01.lab.local     @@ -1 +1,15 @@
chefnode01.lab.local     +# Configuration file for the chef-client service
chefnode01.lab.local     +
chefnode01.lab.local     +CONFIG=/etc/chef/client.rb
chefnode01.lab.local     +PIDFILE=/var/run/chef/client.pid
chefnode01.lab.local     +#LOCKFILE=/var/lock/subsys/chef-client
chefnode01.lab.local     +# Sleep interval between runs.
chefnode01.lab.local     +# This value is in seconds.
chefnode01.lab.local     +INTERVAL=1800
chefnode01.lab.local     +# Maximum amount of random delay before starting a run. Prevents every client
chefnode01.lab.local     +# from contacting the server at the exact same time.
chefnode01.lab.local     +# This value is in seconds.
chefnode01.lab.local     +SPLAY=300
chefnode01.lab.local     +# Any additional chef-client options.
chefnode01.lab.local     +OPTIONS=""
chefnode01.lab.local     - change mode from '' to '0644'
chefnode01.lab.local     - restore selinux security context
chefnode01.lab.local   * directory[/etc/systemd/system] action create (up to date)
chefnode01.lab.local   * systemd_unit[chef-client.service] action create
chefnode01.lab.local   Recipe: <Dynamically Defined Resource>
chefnode01.lab.local     * file[/etc/systemd/system/chef-client.service] action create
chefnode01.lab.local       - create new file /etc/systemd/system/chef-client.service
chefnode01.lab.local       - update content in file /etc/systemd/system/chef-client.service from none to 72baea
chefnode01.lab.local       --- /etc/systemd/system/chef-client.service	2019-04-27 20:50:45.096461385 +1000
chefnode01.lab.local       +++ /etc/systemd/system/.chef-chef-client20190427-19787-1tpdjg1.service	2019-04-27 20:50:45.063461699 +1000
chefnode01.lab.local       @@ -1 +1,15 @@
chefnode01.lab.local       +[Unit]
chefnode01.lab.local       +Description = Chef Client daemon
chefnode01.lab.local       +After = network.target auditd.service
chefnode01.lab.local       +
chefnode01.lab.local       +[Service]
chefnode01.lab.local       +Type = simple
chefnode01.lab.local       +EnvironmentFile = /etc/sysconfig/chef-client
chefnode01.lab.local       +ExecStart = /usr/bin/chef-client -c $CONFIG -i $INTERVAL -s $SPLAY $OPTIONS
chefnode01.lab.local       +ExecReload = /bin/kill -HUP $MAINPID
chefnode01.lab.local       +SuccessExitStatus = 3
chefnode01.lab.local       +Restart = always
chefnode01.lab.local       +
chefnode01.lab.local       +[Install]
chefnode01.lab.local       +WantedBy = multi-user.target
chefnode01.lab.local       - change mode from '' to '0644'
chefnode01.lab.local       - change owner from '' to 'root'
chefnode01.lab.local       - change group from '' to 'root'
chefnode01.lab.local       - restore selinux security context
chefnode01.lab.local     - creating unit: chef-client.service
chefnode01.lab.local Recipe: chef-client::systemd_service
chefnode01.lab.local   * service[chef-client] action enable
chefnode01.lab.local     - enable service service[chef-client]
chefnode01.lab.local   * service[chef-client] action start
chefnode01.lab.local     - start service service[chef-client]
chefnode01.lab.local   * systemd_unit[chef-client.timer] action stop (up to date)
chefnode01.lab.local   * systemd_unit[chef-client.timer] action disable (up to date)
chefnode01.lab.local   * systemd_unit[chef-client.timer] action delete (up to date)
chefnode01.lab.local   * service[chef-client] action restart
chefnode01.lab.local     - restart service service[chef-client]
chefnode01.lab.local
chefnode01.lab.local Running handlers:
chefnode01.lab.local Running handlers complete
chefnode01.lab.local Chef Client finished, 10/15 resources updated in 02 seconds
  

The options used in the knife bootstrap command are as follows.

-N - The name of the node when added to the Chef server
-x - The username to use to connect to the node
--sudo - Activate sudo on the node (Will prompt for sudo password)
-r - List of recipes to apply to the node

Confirm the node is connected to the Chef server.

    cmd
  
    knife node list

# output
chefnode01.lab.local

Get detailed information about the node.

    cmd
  

    knife node show

# output
Node Name:   chefnode01.lab.local
Environment: _default
FQDN:        chefnode01.lab.local
IP:          192.168.255.190
Run List:    recipe[chef_client_wrapper]
Roles:
Recipes:     chef_client_wrapper, chef_client_wrapper::default, chef-client::default, chef-client::service, chef-client::systemd_service
Platform:    centos 7.6.1810
Tags:
  

That's it. Rinse and repeat the bootstrap process for any other nodes you want to bring under management of the Chef server.

Outro

If you followed along and made it this far, you will have a working Chef infrastructure setup. There is a lot of moving parts and cooking based terminology to get your head around, but like most complex systems after a while things start to make sense. I have only just started using Chef and I am enjoying the experience so far and look forward to digging deeper.

Remember: This is not the end, it's merely the beginning!

The "from the start to the beginning" series aims to take you from nothing to getting you up and running. This is not meant to be a comprehensive guide, but should be enough to get you started on the journey.

Chef From The Start To The Beginning

Intro

Chef Architecture

Components

Chef Server Prerequisites

Chef Server Installation

Chef Server Configuration

Chef Workstation Installation

Chef Workstation Setup

Cookbooks

Node Bootstrap

Outro

Remember: This is not the end, it's merely the beginning!

Links

On this page