Orchestration with Ansible

by HEIG-Cloud

Tags:

Posted on Fri, Dec 18, 2015


Orchestration tools

First of all, what is an orchestration tool? Well, as the name implies, it’s something that plays the rool of a chief conductor that is leading an orchestra. Its job is to manage the orchestra by giving orders and instructions.

Different tools

There are many orchestration tools that you can use and most of them are free. The famous ones are:

  • Ansible (the one we use)
  • Chef
  • Puppet
  • Salt

Usually, these tools require you to install a “supervisor” on a certain host that will act as the orchestrator. As for Ansible however, you only need to install Ansible and you are set. No need to install something else, configure or pay, you are all set.

As you can imagine we chose to use Ansible, though we could have used any of the others. The main point is to have an idempotent script that you can run to make your setup, update it and recover it in case of emergency.

Install Ansible

From now on, we will only be speaking about Ansible.

The main requirement for Ansible is python. To install Ansible you can use pip, like this:

$ sudo pip install ansible

If you don’t have pip, you can install it like this:

$ sudo easy_install pip

You can also get ansible from sources:

$ git clone git://github.com/ansible/ansible.git --recursive
$ cd ./ansible
$ source ./hacking/env-setup

And you can also install it on Mac with pip.

That’s all for the installation.

State scripts

There is a very cool concept with Ansible (and probably most of the orchestration tools), we don’t specify the commands we want to do, we specify a state. What does it mean? Let’s take an example, say we need to install the package called mysql-server. In Ansible we’ll say:

apt: 
    name: mysql-server
    state: latest
    update_cache: yes

So, what is up? Why not use apt-get and be done with it? First let’s see what we did in here:

  • apt, Ansible’s command to install a package
  • name, the name of the package we want to install
  • state, the state we want. Latest means we want the last version available.
  • update_cache, will launch apt-get update before the operation.

There are more options, you can check them here.

So why is this powerful? Well, we never used commands! It means that the Ansible script can work on different OS. For instance, if I wanted to install mysql-server on Ubuntu:

$ sudo apt-get install mysql-server

And on RH or CentOS?

$ yum install mysql-server

But on Ansible it’s always the same. We just say, I want to have mysql’s latest version installed, I don’t care how you do it. That’s the power of Ansible. That’s also why these scripts can be idempotents. If the state we want to reach is already ok, for instance, we are trying to install a package that is already installed, nothing will be done.

What if we wanted to uninstall mysql-server with Ansible?

apt: 
    name: mysql-server
    state: absent

The new state is “absent”, meaning that it must be deleted if it exists or nothing will be done.

YAML

If you’ve ever worked on configuration files, maybe for an application or a website, you’ve probably already know a little about YAML. This is a language with a very human friendly syntax. It means that it is quite easy to understand at first glance, unline XML or JSON for example.

With Ansible, we are going to create playbooks, these “books” contain instructions that Ansible will execute on the remote hosts.

This here is an example of YAML code:

---
- name: Install cinder-api
  hosts: cinder-api
  sudo: True
  gather_facts: True
  vars: 
    packages:
      - cinder-api
      - cinder-scheduler
      - python-cinderclient
    services:
      - cinder-scheduler
      - cinder-api

  tasks: 

  - name: Install packages
    apt:
      pkg: "{{ item }}"
      state: latest
    with_items: packages

You can already see what a playbook looks like, in here we indicate on which hosts we want to run this, if we need sudo permissions, if we want to gather_facts, we set a few variables and have a task to install a package. We will see this in detail later ;-)

Architecture of the project

This is an example of a project, we’ll use this example as reference.

Project/ 
    - playbooks/
        - book1.yaml
        - book2.yaml
    - inventory
    - ansible.cfg
    - group_vars/
        - all.yaml

Templating

The other functionnality that we use a lot with Ansible is templating. This is very powerful, it allows us to use variables in files. For instance, we need to have an IP address in a conf file, instead of writing it ourselves, we put a variable. It means that if one day we need to change the IP, we don’t need to change the conf file itself. It gives us a lot of options and we are going to see a few examples, so you can grasp the amazing possibilities if offers you.

Ansible uses jinja2 templates, it’s quite easy to understand how they work. Variables will be inside {{ }}, when starting a line, you’ll need to use “{{ }}”, that’s pretty much all you need to know.

Let’s create a directory for variables. If you create a directory called group_vars, the variables defined inside will be usable automatically. That’s quite useful so we will do this. Inside, you can create various files but again, if you create a file called all.yaml, the content will be available automatically. So in the end:

$ touch group_vars/all.yaml

Once you have your all.yaml file created, you can start creating variables like this:

instance_tunnel_network: 172.20.0.0
instance_tunnel_netmask: 255.255.0.0
instance_tunnel_broadcast: 172.20.255.255

instance_tunnel_address_network: 172.20.0.1
instance_tunnel_address_compute1: 172.20.0.11

#You can also use python methods, to get a specifi IP or to get the content of a file for example
controller_host: "{{ hostvars['controller']|find_ip(management_network) }}"
keystone_admin_token : "{{ lookup('password', inventory_dir + '/credentials/keystone-admin-token') }}"

The cool thing is, you can have all your passwords inside files and in the conf files, you get the content of these pasword. Very useful if you want to share your implementation but don’t want to share your passwords (which would probably be a bad idea).

We can now use these vars in playbooks or in text files, like conf files. Let’s see an example of both:

# Inside a playbook
  - name: Create the service project 
    keystone_user: 
      endpoint: "{{ keystone_admin_url }}"
      token: "{{ keystone_admin_token }}" 
      tenant: service
      tenant_description: "Service Project"   

# Inside a conf file
  bind-address      = {{ controller_host }}

Ansible template instruction

When you want to replace a conf file by one that has been “templated”, you need to use ansible’s template command. Let’s say we want to change the my.cnf file of mysql on a certain host with one we templated ourselves (changing the bind-address value with {{ controller_host }} for instance).

- name: install mysql config file that binds to management network interface
  template: 
    src: templates/etc/mysql/my.cnf 
    dest: /etc/mysql/my.cnf 
    owner: root 
    group: root 
    mode: 0644
    backup: yes

You can also specify the mode, group and user of the file once it has been copied on the remote host. The backup line will copy the original file and leave it as name.backup, in our case, we’ll have a my.cnf.backup file present on the remote host.

Project

Once you have Ansible ready, you can test if you have this command in your path for instance:

ansible-playbook 

This is the command we’ll be using to launch playbooks. Now let’s see what we need for an Ansible project.

  • Ansible installed (obviously)
  • SSH on all remote hosts
  • An inventory file
  • A playbook
  • A file containing vars (optionnal)
  • A config file ansible.cfg (optionnal)

This would be the minimal setting as we see it. You need to know that Ansible has a lot of modules, some of them are in the “core”, some are extra and you can also create your own modules.

There are also a few good practices about the playbooks that we will see later.

SSH on remote

Ansible needs to connect to the remote hosts to run the commands. To do so, it needs to be able to ssh on the remote hosts.

If you don’t know how to create a ssh key pair, check this link.

You can also edit the ansible.cfg file to match the ssh user, like this:

[defaults]
ssh_user = sshuser
ssh_pass = pa$$word

Once you can ssh on the remote hosts without typing a password and with the user “ssh_user”, it should work fine.

Inventory file

The inventory file is very important, it contains all the information about the hosts. You must know that with Ansible, you can create groups of hosts. Let’s say you have 10 nodes:

  • 8 web servers
  • 1 mysql server
  • 1 load balancer

You could create 3 groups, like:

  • [web]
  • [mysql]
  • [loadbalancing]

Why do that? Well, chances are, you probably will be doing the same operations on all the web servers, so why not just say, do this operation on all the web servers instead of doing this indivudally? Fair point right? :)

Let’s see an example of an inventory file:

controller ansible_ssh_host=xxx.xxx.xxx.xxx
network    ansible_ssh_host=xxx.xxx.xxx.xxx

compute1   ansible_ssh_host=xxx.xxx.xxx.xxx
compute2   ansible_ssh_host=xxx.xxx.xxx.xxx
compute3   ansible_ssh_host=xxx.xxx.xxx.xxx

[mysql]
controller

[computenodes]
compute1
compute2
compute3

There you go, 5 hosts, 2 groups. When writing a playbook, you can use any of the following hosts:

  • controller
  • network
  • compute1
  • compute2
  • compute3
  • mysql
  • computenodes

This is of course a very basic inventory file. There are other options too, you can check them here. You also probably noticed the ansible_ssh_host=xxx.xxx.xxx.xxx part, it’s the IP address for each host that will be accessed by Ansible. Specifying it once on top of the file is enough, when creating groups you only need to specify the name of the host.That’s enough about the inventory.

Playbooks

Now that we can access all our remote hosts and that we have our groups, all we need to do is to actually give them instructions, that’s what playbooks are for.

There are a few good practices when it comes to playbooks, since we need to install many services we chose to do it like so :

We take the example architecture from before:

Project/ 
    - services/
        - mysql/
            - main.yaml
            - install.yaml
            - setup.yaml
        - keystone/
            - main.yaml
            - install.yaml 
            - setup.yaml
    - templates/
        - etc/
            - mysql/
                - my.cnf
            - keystone/
                - keystone.conf
        - home/
    - inventory
    - ansible.cfg
    - group_vars/
        - all.yaml
    - openstack.yaml

So, the main thing is, we have a directory for each “service” and a directory containing the templates. What is a service? MySQL, PHP, Apache, Keystone, Nova, Neutron, whatever. For us, we decided that each package or group of packages giving us a specific functionnality would be one. In each service directory you will find the playbooks and a main.yaml file. This file only contains include lines to call the other files. The same thing happens with the openstack.yaml file. It only includes the mains in each services directories, like this:

$ cat openstack.yaml
---
- include: services/mysql/main.yaml
- include: services/keystone/main.yaml
- include: services/nova/main.yaml

$ cat services/mysql/main.yaml
---
- include: install_mysql.yaml
- include: configure_mysql.yaml

This is a very basic example but should be enough for you to get the idea.

Once all is said and done, we can run the openstack.yaml playbook like this:

$ ansible-playbook -i inventory openstack.yaml # -i is for specifying an inventory file

Conclusion

Ansible is a very powerful tool that is quite easy to use. The official documentation is complete and there are a lot of examples on the web. Also, it is important to always think about idempotency, because you can also use shell commands with ansible, sometimes you don’t have a choice. If you have to, make sure it won’t act redundantly (like adding the same line many times in a file instead of just once).