Ansible: stop / start services on random hosts


Warning: count(): Parameter must be an array or an object that implements Countable in /home/fbsqlcom/public_html/youdidwhatwithtsql.com/wp-content/plugins/wp-codebox/main.php on line 31

In the coming weeks I’m performing some testing of a new application on a Cassandra cluster. To add a little randomness into some of the tests I thought it would be interesting to give the Cassandra service a little kick. I created a simple Ansible playbook this afternoon that does this. A simple Chaos Monkey if you like. Here’s the basic flow of the playbook…

1. Select random host from play_hosts.
2. Stop service.
3. Wait for interval.
4. Start service.
5. Wait for Service back up, Port?
6. Wait for second defined interval.

These tasks are repeated n number of times as specified by the user. This playbook is interesting because it uses a couple of nifty tricks…

The first is a method to dynamically generate a list to iterate over. This is how we control the number of times we restart the service.

  - name: Generate a list we will iterate over
    set_fact:
      restart_iterations: "{{ restart_iterations | default([]) + [item | int] }}"
    with_sequence: start=1 end="{{ max_iterations }}"
    run_once: yes

The second uses the loop construct. The current_iteration variable is made available in the tasks.yml file

  - name: Run main tasks file
    include_tasks: tasks.yml
    loop:  "{{ restart_iterations }}"
    loop_control:
      loop_var: current_iteration

The playbook can optionally produce a log to make it a little easier to see what nodes it’s executing on…

tail -f /tmp/service_restarter.log 
2019-01-27 18:38:51 CET - Current iteration is 1 and is executing on cnode2
2019-01-27 18:41:18 CET - Current iteration is 2 and is executing on cnode4
2019-01-27 18:43:47 CET - Current iteration is 3 and is executing on cnode3
2019-01-27 18:46:15 CET - Current iteration is 4 and is executing on cnode3
2019-01-27 18:48:42 CET - Current iteration is 5 and is executing on cnode5
2019-01-27 18:51:07 CET - Current iteration is 6 and is executing on cnode2
2019-01-27 18:53:34 CET - Current iteration is 7 and is executing on cnode1
2019-01-27 18:56:03 CET - Current iteration is 8 and is executing on cnode4
2019-01-27 18:58:29 CET - Current iteration is 9 and is executing on cnode4
2019-01-27 19:00:55 CET - Current iteration is 10 and is executing on cnode2

Here’s how you might execute the playbook…

ansible-playbook -l cassandra -i inventory service_restarter.yml

The playbook is available over on my github/ServiceRestarter.

While this playbook was designed with Cassandra in mind it should work for any service that listens on a TCP port. See the README file for an explanation of the variables.

Use restview to to make the Ansible rst documentation browsable

The ansible-doc package not only installs the command line tool but also some quite detailed Ansible documentation in rst format. It would be nice if it was browsable in a html format. Here’s how that can happen (Redhat/CentOS)

First install pip and restview

sudo yum install python-pip
sudo pip install restview

This will allow all hosts on your network to access the documentation at http://hostname:33333 if your firewall allows it.

restview /usr/share/doc/ansible-doc-2.7.5/rst/ --listen 33333 --allowed-hosts * &

Alternatively, if you’re on a desktop computer, use the following to launch a browser…

restview /usr/share/doc/ansible-doc-2.7.5/rst/ --browser

Hint: This might be useful for those taking the EX407 Ansible Exam assuming you can install these packages. Having this at your fingertips could prove to be very useful should something like how to structure a jinja2 template slip your mind.

There are a few rendering issues, resulting in broken links, but nevertheless there’s a lot of very useful information. I’ll update this if I get around to finding a solution for that (probably a restview alternative).  Here’s a few screenshots showing what’s provided…

ansible documentation ansible documentation ansible documentation ansible documentation

UPDATE: I took the Ansible exam and I now know this isn’t really needed. Ample easily used documentation is provided.

Create a space-separated list of play_hosts in Ansible

Sometimes I need a list of hosts as a string when working with Ansible. Pacemaker clustering is one example. Here’s a snippet of Ansible that does this..

    - name: Setup list of cluster hosts
      set_fact:
        host_list: "{{ host_list }}{{ (play_hosts.index(item) == 0) | ternary('',' ') }}{{ item }}"
      loop: "{{ play_hosts }}"
      run_once: yes

This play will produce the following output;

ok: [cnode1] => {
    "host_list": "cnode1 cnode2 cnode4 cnode3"
}

If you need a different separator just change the second parameter in the ternary function. The below example produces a comma-separated list of play_hosts…

    - name: Setup list of cluster hosts
      set_fact:
        host_list: "{{ host_list }}{{ (play_hosts.index(item) == 0) | ternary('',',') }}{{ item }}"
      loop: "{{ play_hosts }}"
      run_once: yes
ok: [cnode1] => {
    "host_list": "cnode1,cnode2,cnode4,cnode3"
}

Offset cron jobs with Ansible

Sometimes I want to run the same cronjob on a few hosts but I might want to offset them slightly if I’m accessing any shared resources. Here’s an easy way to do that, for a small number of hosts, using Ansible

- name: Ensure cron exists
  cron:
    name: Test Job
    minute: "{{ play_hosts.index(inventory_hostname) }}-59/5"
    job: /usr/local/bin/myscript.sh >> /var/log/log.log
    user: web

This would create a job, every 5 minutes, with a one minute offset compared to the previous host. So assuming four hosts we’d end up with the following cron job schedules;

hostname cron schedule Will run at minutes past the hour
host1 0-59/5 * * * * 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55
host2 1-59/5 * * * * 1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56
host2 2-59/5 * * * * 2, 7, 12, 17, 22, 27, 32, 37, 42, 47, 52, 57
host4 3-59/5 * * * * 3, 8, 13, 18, 23, 28, 33, 38, 43, 48, 53, 58

For larger number of hosts it would probably be better to group hosts and run the offset via that.

ssh-copy-id automation with a list of hosts

Here’s another version of my ssh-copy-id script this time using a text file containing a list of hosts. The hosts file should contain a single host per line.

#!/bin/bash
 
export SSH_USER="user"
read -s PASSWORD
export PASSWORD
 
 
while read HOST; do
            export HOST;
            expect -c '
            set SSH_USER $env(SSH_USER)
            set HOST $env(HOST)
            set PASSWORD $env(PASSWORD)
            spawn ssh-copy-id $SSH_USER@$HOST
            expect {
                        "continue" {
                                    send "yes\n";
                                    exp_continue
                        }
                        "assword:" {
                                    send "$PASSWORD\n";
                        }
            }
            expect eof'
            echo "Done $HOST"
done < "$1"

Execute the script and pass the path to the text file as a parameter. i.e.

./auto_ssh.sh /path/to/host/list.txt;