Installing Chef Delivery, SSH

This topic guides you through setting up Chef Delivery for SSH provisioning with the delivery-cluster cookbook and Chef provisioning, and validating the installation once Chef Delivery is set up. With the delivery-cluster cookbook, you install the Chef Delivery cluster on your own infrastructure using Chef provisioning and your license key.

Process Overview

The process for installing Chef Delivery using the delivery-cluster cookbook involves the following tasks:

  • Create and configure a dedicated provisioning node
  • Configure the delivery-cluster cookbook to use SSH provisioning
  • Run automation for delivery-cluster, which sets up a full Chef Delivery cluster and gives you login credentials
  • Verify the Chef Delivery provisioning
  • Set up your workstation to develop projects in Chef Delivery
  • Configure Chef Delivery
  • Validate the installation by creating a cookbook and kicking off a pipeline

Prerequisites

Chef Delivery requires a license from Chef to install.

Supported Platforms

Chef Delivery may be run on the following platforms. Do not mix platform versions within the Chef Delivery cluster.

OS / Versions
Ubuntu / 12.04, 14.04
Centos / 6.5, 6.6, 7
Redhat / 6.5, 6.6, 7

Infrastructure

Chef Delivery has the following infrastructure requirements:

Function / vCPU / RAM / Free disk space (in /var)
Provisioning Node / 1 (2.5GHz) / 1GB / 8GB
Delivery Server / 4 / 8GB / 80GB
Chef Server (must be v12). See additional information in note, below. / 4 / 8GB / 80GB
Supermarket Server / 4 / 8GB / 80GB
Build Nodes (at least 3, one for each Verify phase: Lint, Syntax, and Functional) / 2 / 4GB / 60GB
Infrastructure Nodes (at least 4, one for each environment stage: Acceptance, Union, Rehearsal, and Delivered) / 2 (for test purposes) / 4GB (for test purposes) / 60GB (for test purposes)

NOTE: If you want to use your existing Chef server in the Chef Delivery cluster, the version of the Chef server must be 12.2, it cannot have a high availability configuration.

Network and Ports

Chef Delivery has the following network and port requirements:

Ports / Description / Server / State
10000-10003 / TCP Push Jobs / Chef server / LISTEN
8989 / TCP Delivery Git (SCM) / Delivery server / LISTEN
443 / TCP HTTP Secure / Chef server, Delivery server / LISTEN
22 / TCP SSH / All / LISTEN
80 / TCP HTTP / Chef server, Delivery server / LISTEN
5672 / TCP Analytics MQ / n/a / n/a
10012-10013 / TCP Analytics Messages/Notifier / n/a / n/a

Install Chef Delivery

Begin installing Chef Delivery by creating a provisioning node to provision and maintain the cluster.

Provisioning Node

Chef Delivery uses Chef provisioning to create, upgrade, and manage the cluster. A dedicated provisioning node ensures a central place from which the Chef Delivery cluster is managed, including managing those allowed to control it. Use your existing means to provision a dedicated provisioning node with a supported platform. This machine does not need to be powerful because it runs only provisioning code.

On the provisioning node:

  1. Install a development environment:

For Debian-based (apt):

$ sudo apt-get install build-essential

For Red Hat Enterprise Linux-based (yum):

$ sudo yum groupinstall "Development Tools"

For Mac OS X:

$ sudoxcode-select --install

  1. Install Git and configure your GitHub username and email, for help see: Installing Git and First-Time Git Setup.
  2. Install the Chef Development Kit. Be sure to set the system Ruby; for details, see Add Ruby to $PATH.
  3. Install the Chef knife push utility:

$ chef gem install knife-push

  1. Move your Chef Delivery license key to ~/delivery.license.
  2. Clone the Git repo for the Chef Delivery cluster:

$ git clone ~/delivery-cluster

SSH Settings

The following section details settings for Chef Delivery with SSH provisioning.

Option / Description
Environment Name / This name becomes the default name for your Chef Delivery cluster environment, enterprise, and organization. The naming convention prohibits the use of capital letters, but should otherwise be alphanumeric.
Key File / The path to your .ssh key file; for example, /home/USERNAME/.ssh/USERNAME.pem. Note, you must set up the public/private SSH key-pair yourself before attempting to install Chef Delivery.
SSH Username / Enter the default SSH username for the system on which you are installing delivery-cluster. For example, for an Ubuntu machine, the default SSH username is ubuntu. You may need to look up what the default is for your system.
Chef Server Host, Delivery Server Host, and Build Nodes Host / For SSH, enter the IP addresses of the nodes that you have already provisioned. This is true also if you choose to enable a Chef Supermarket server.
Use existing chef-server / Accept the default, “no”, if you want delivery-cluster to configure a Chef server for you (you will be asked for the IP address of the server). Enter “yes” if you have a configured Chef server that you want to use and do not need delivery-cluster to configure one for you. If you do specify “yes”, please note that the version of the Chef server must be 12.2, it cannot have a high availability configuration. In addition, a special entry must be made to the environments configuration file (environments/NAME_OF_ENV.json) similar to:
"chef-server": {
"organization": "test",
"existing": true,
"host": "chef-server.example.com"
},
License File / The path to your Chef Delivery license file; the default path is your home directory.
Enable Supermarket / Enter “yes” if you intend to run cookbooks through your Chef Delivery pipeline.

SSH Provisioning

This section describes setting up a Chef Delivery cluster to use SSH for provisioning. This process creates at least five machines, depending on the selected configuration options. The typical (and recommended) starting setup is five nodes in addition to the provisioning node:

  • One to run the Chef server
  • One to run Chef Delivery
  • Three to act as build nodes
  • One to run the Chef Supermarket server; for cookbook dependency resolution, using Chef Supermarket server is required
  • For a test environment, use stand-in infrastructure nodes; we recommend four, one for each environment stage: Acceptance, Union, Rehearsal, and Delivered

These machines must be accessible via SSH and commands should be run using sudo without passwords.

On the provisioning node:

  1. Upload the private key (.pem file). For example:

$ scp -i ~/.ssh/USERNAME.pem USER@IP_ADDRESS:/home/USER/.ssh/USERNAME.pem

  1. Ensure that the appropriate ports are open. See Network and Ports for details.
  2. Move to the delivery-cluster directory: cd ~/delivery-cluster.
  3. Create a configuration file to drive Chef provisioning:

$ rakesetup:generate_env

This opens a wizard, the following code block shows the interactions. Variables you must enter are shown in ALL CAPs. Default values are shown in square brackets; pressing ENT sets the default value. This command creates an environments/ENV_NAME.json file with the specified settings that you can modify if need be.

$ rakesetup:generate_env

Gathering Cluster Information

Provide the following information to generate your environment.

Global Attributes

Environment Name [test]:

Cluster ID [test]:

Available Drivers: [ aws | ssh | vagrant ]

Driver Name [vagrant]: ssh

Driver Information [ssh]

SSH Username [vagrant]: DEFAULT_SSH_USERNAME_FOR_YOUR_SYSTEM

Key File [/home/mabrahms/.vagrant.d/insecure_private_key]: /home/USERNAME/.ssh/USERNAME.pem

Would you like to configure Proxy Settings? [no]:

Chef Server

Organization Name [test]:

Use existing chef-server? [no]:

Host [33.33.33.10]: IP_ADDR_FOR_CHEF_SERVER

Delivery Server

Package Version [latest]:

Enterprise Name [test]:

License File [/Users/salimafiune/delivery.license]: PATH_TO_DELIVERY_LICENSE

Host [33.33.33.10]: IP_ADDR_FOR_DELIVERY_SERVER

Analytics Server

Enable Analytics? [no]:

Supermarket Server

Enable Supermarket? [no]: yes

Host [33.33.33.10]: IP_ADDR_FOR_SUPERMARKET_SERVER

Build Nodes

Number of Build Nodes [1]: 3

Host for Build Node 1 [33.33.33.10]: IP_ADDR_FOR_BUILD_NODE_1

Host for Build Node 2 [33.33.33.10]: IP_ADDR_FOR_BUILD_NODE_2

Host for Build Node 3 [33.33.33.10]: IP_ADDR_FOR_BUILD_NODE_3

Specify a delivery-cli artifact? [no]:

Rendering Environment => environments/test.json

  1. Important! Export your new environment by executing the following: (example uses test as the environment name):

$ export CHEF_ENV=test

NOTE: You can name your environment file anything, but note the name for future reference.

Reset SSH Keys

When provisioning for SSH delivery, the placement of the keys is crucial. You may find yourself getting an error like this:

Error executing action 'converge' on resource 'machine[chef-server-test3]'

RuntimeError

Machine chef-server-test3 (on ssh:/home/mabrahms/delivery-cluster/.chef/provisioning/ssh)did not become ready within 120 seconds

Follow these steps to resolve the issue:

  1. Verify that your SSH keys are the problem by attempting to log into your chef server over SSH.

If you cannot do that, then your SSH keys need to be re-synced.

If you can, then your SSH configuration may be wrong. In some environments the problem may be that the Chef Delivery server’s SSH key has changed for the hostname or IP address your chef server uses. Local Vagrant clusters see this when the Chef serverbox file changes (or when switching between VMware and VirtualBox) because the SSH key is generated during the OS install.

  1. When the server SSH key is the problem, the fix is to delete the entry for that machine from $HOME/.ssh/known_hosts.
  2. To create an SSH key (without a passphrase):

$ ssh-keygen -t rsa -b 4096 -C ""

The output is similar to:

Generating public/private rsa key pair.

Enter file in which to save the key (/Users/username/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /Users/path/to/.ssh/id_rsa.

Your public key has been saved in /Users/path/to/.ssh/id_rsa.pub.

The key fingerprint is:

ac:8a:57:90:58:c1:cd:34:32:18:9d:f3:79:60:f3:41

The key's randomart image is:

+--[ RSA 4096]----+

| .==*o.E |

| . *o*.. |

| o + = . |

| . oo.o |

| . ..S |

| .. |

| .. |

| .*o*. |

| ... |

+------+

  1. Run the following:

$ cat .ssh/id_rsa.pub

The output is similar to:

ssh-rsa

AAAAB3NzaC1yc2EAAAADAQABAAACAQDa8BR/9bj5lVUfQP9Rsqon5qJMkiVm+JAtGi

wnhxqgyRhkYLIzm6+gcifDgMOMuwZA88Ib5WNRhxjlmTseapower4rH/jAAczdp1h1

7xLEEbUfQfkcqiy/Drp3k12345678ad234fgvdsasdfasdfR9ddNIeNvQ7OIpOCfLE

PCyFz3aRRuhpM/5cySFT7bl1O44bNgfiuqRzcXFscZb03WPlhaPwCvL2uxaRzdrAGQ

mE5jzCo6nORvKoGdVDa2++def33f3xPZCo3oJ08Q9XJ2CnfJlmyNe1hwI2NOQ3yRbc

nfSMona7ccSyHRWGs5bS//u6P0NK5AqH5jK8pg3XwtHZqLwUVy1wX0WnnJWg9IWXf3

2g3P4O4NJGVUeX33Czv32GK8YphuEweqFu/Ej7kQp1ppIxkEtrpBfMi3na0QqZlk6w

wghZLa++DUfWOhGsuuBgnsocAR5rLGy+gkypdie1Ydoe8qjLVZR/jKybQfQjuZOS30

fZnwJhl2ZaeraPfkEXlVhK02/8PIALGfeXdt9KvQN0p5c6lRoDxqBqslM+1KbKKcGd

lSGEsAIP9OOWBECRxlOwqlqGHtrgWKOr376dntMIy2+fFD/74tJMjRwbRzm8IGWmj6

OcF6EvTYYO4RmISD8G+6dm1m4MlxLS53aZQWgYWvRdfNB1DA

Zo3h9Q==

  1. Add the SSH public key to the user account.

Run Provisioning

To run provisioning issue this command from inside the delivery-cluster directory:

$ rakesetup:cluster

NOTE: If the first converge fails on the build nodes, try running the above step again. The Chef Delivery cluster is complicated and sometimes there are timeouts before the setup completes.

Verify Provisioning

To verify provisioning do the following, from inside the delivery-cluster directory:

  1. Run this command:

$ rakeinfo:list_core_services

  1. Navigate to your CHEF_SERVER_URL, and login with delivery:delivery.
  2. Click on Nodes. You’ll see your Chef Delivery server, your build nodes, and nodes for any other optional features that you installed.
  3. Get the credentials and URL with this command:

$ rakeinfo:delivery_creds

  1. Navigate to the Web Login and use the admin credentials to log in.
  2. Run this command:

$ knife node status

All build nodes should report available.

Configure Delivery

In Chef Delivery there are multiple levels of organization: enterprises, organizations, and projects. The provisioning step created the initial enterprise you specified in your environment file as the first name option. Enterprises are designed to provide units of multi-tenancy with separate sets of organizations and users. Begin configuring Chef Delivery by adding users and organizations.

NOTE: Chef Delivery, by default, hosts a Git server that you interact with through the delivery CLI commands. Additionally, you can integrate GitHub Enterprise or GitHub.com.

Add Users

Chef recommends that thedefault admin account not be used after Chef Delivery is installed. Instead, use the following web UI procedure to create a new user for yourself. Then log out as admin and log back in as the user you created.

  1. Log into the Chef Delivery web UI as an administrator.
  2. SelectUsersfrom the drop-down menu on the upper right.
    TheUserslist page opens. You can use the search filter in the upper right corner to make sure that the user is not already added.
  3. Click the plus sign (+) next toAdd a New User.
  4. In the Add a New User text area, select Internal,which means you are manually adding the user to the Chef Delivery database. The selection box is grey for the active selection.
    Options forName and Email, first name, last name, email address, andSecurity Information, a login name and password, appear.
  5. Enter the appropriate information.
    Select Admin in theRoles Within the Enterprise option list.
    ClickSave and Close.

TheUserlist page opens and a status message appears.

To check, clickEditand verify that the user details are present.

Create Organizations

Chef suggests creating a sandbox organization in which a test project can be run for verification and testing purposes. Additionally, it is recommended that each organization created corresponds to a working directory within which projects for that organization are managed.

To create an organization, do the following:

  1. Log into the Chef Delivery web UI as theAdmin user you created; use this command (run in your delivery-cluster directory) to find the credentials:

$ rakeinfo:delivery_creds

  1. Select Organizations from the drop-down menu on the upper right. The organizations list page opens.
  2. Click the plus sign (+) next to Add a New Organization.
  3. Enter an organization name in the text area, and then click Save & Close.

Setup a Workstation

After the Chef Delivery cluster has been created, team members need to install the Chef Delivery CLI on their workstations.

NOTE: If you reuse the provisioning node in the previous section of this installation guide as a workstation for using Chef Delivery, you can skip steps 1 through 4 below.

Setting up a workstation requires that you do the following:

  1. Install a development environment:
  2. Debian based (apt): apt-get install build-essential
  3. RHEL based (yum): yum groupinstall "Development Tools"
  4. OS X: xcode-select --install
  5. Install Git: Also, make sure that your SSH public key is added to your user profile in Chef Delivery.
  6. Install the Chef Development Kit: Be sure to follow all instructions; for example, to run delivery commands you must set the following:

echo 'eval "$(chef shell-init bash)"' > ~/.bash_profile

  1. Install the Chef knife push utility:chef gem install knife-push.
  2. Install and configure the Chef Delivery CLI, described next.

Install the Chef Delivery CLI

The delivery command line interface (CLI) is required for the workstation and for many Chef Delivery functions.

Mac OS X

For Mac OS X, run the following commands:

  1. Download the package
  2. Click the package and install.
  3. You may need to allow the package on the machine. To do this, open System Preferences -> Security & Privacy, and click Allow. You can also go to your Downloads folder and right-click the package to open the installer; simply accept the agreement and the command line interface is installed.
Ubuntu

For Ubuntu 14.04, run the following commands:

$ curl | sudo bash

$ sudo apt-get install delivery-cli

RHEL, SUSE

For Red Hat Enterprise Linux 6.5 and SUSE Enterprise Linux Server 6.5, run the following commands:

$ curl | sudo bash

$ sudo yum install delivery-cli.rpm

Microsoft Windows

For Microsoft Windows:

  1. Download the binary from
  2. Install Git (2.5) on Microsoft Windows:

WARNING: Git 2.6 won’t work.

  1. Modify the PATH environment variable to thepath to the location of the delivery command line interface. For example: C:\\chef\\delivery-cli\\bin

Configure theChef Delivery CLI

Before you use the Chef Delivery CLI from a workstation, you need to provide it with details such as the URL of the Delivery server, and the names of the relevant enterprise, organization, and user. The delivery setup subcommand creates a configuration file named .delivery/cli.toml with the required information.

The placement of the .delivery directory in your file hierarchy is significant. Like Git, Chef Delivery CLI commands search the current directory and parent directories to locate server settings. Because server settings are unique to an organization, we recommend that you create a directory for each organization as required and run the delivery setup command from each of those directories.

$ delivery setup --server=DELIVERY_SERVER_IP_ADDR --ent=ENTERPRISE --org=ORGANIZATION --user=USERNAME

Authenticate

You must authenticate access to the Chef Delivery Git server, which runs on port 8989 by default.

NOTE: This authentication step is for the Git server that comes with Chef Delivery. If you actually integrate with GitHub, you do not need to do this.

Run this SSH command; enter “Yes” when prompted.

$ ssh -l USER@ENTERPRISE -p 8989 DELIVERY_SERVER_IP_ADDRESS

You should get output similar to this:

$ ssh -l sally@chef -p 8989 10.194.11.99

The authenticity of host '[10.194.11.99]:8989 ([10.194.11.99]:8989)' can't be established.

RSA key fingerprint is ba:db:0c:97:f8:d4:6d:0f:0b:57:0d:0f:0e:a4:15:01.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '[10.194.11.99]:8989' (RSA) to the list of known hosts.

channel 0: protocol error: close rcvd twice

Hi sally@chef! You've successfully authenticated, but Chef Delivery does not provide shell access.

Connection to 10.194.11.99 closed.

Your workstation is now set up for use with Chef Delivery.

Validate the Installation

The surest way to validate your Chef Delivery installation is by creating a cookbook and submitting it to GitHub, which kicks off a pipeline allowing you to see how the process works.

If a project is a cookbook, we recommend starting with delivery-truck, an open source build cookbook created for driving cookbook pipelines in Chef Delivery. You can customize some aspects of delivery-truck through your project’s .delivery/config.json. To have more control, or to opt-out of some of the behavior of delivery-truck, create a wrapper build cookbook. To learn more about wrapper cookbooks, see Chef Supermarket -> Wrapper Cookbooks and the blog post Doing Wrapper Cookbooks Right.