Jenkins test instance with Vagrant & Ansible

Here’s yet another project using Vagrant and Ansible. This time it’s for a Jenkins instance. To get started head on over to the Jenkins test instance github page. Consult the readme for setup instructions.


AWX installation using Vagrant and Ansible

Over on my github is a new project firing up a test instance of AWX. This is based on the following awx installation notes. The project is AWX-on-CentOS-7. You’ll need Virtualbox, Vagrant and Ansible installed to get this up and running. Getting started is simple;

1. From a shell

ansible-galaxy install geerlingguy.repo-epel
git clone https://github.com/rhysmeister/AWX-on-CentOS-7
cd AWX-on-CentOS-7
vagrant up

2. Access url http://http://192.168.4.111/

3. Login with;

u: admin
p: password

You’ll be presented with the main screen of AWX.

Main page of Ansible AWX

Main page of Ansible AWX


Staged service restart with Ansible

I’ve been working on a small project to create a Cassandra Cluster for Development purposes. I’m using Vagrant and Ansible to deploy a 5-node Cassandra Cluster and node #5 would always fail to join the cluster.

I checked /var/log/cassandra/cassandra.log and this is what I found;

INFO  [InternalResponseStage:1] 2017-09-09 18:49:07,673 ColumnFamilyStore.java:406 - Initializing system_auth.roles
INFO  [main] 2017-09-09 18:49:08,666 StorageService.java:1439 - JOINING: waiting for schema information to complete
ERROR [main] 2017-09-09 18:49:09,687 MigrationManager.java:172 - Migration task failed to complete
ERROR [main] 2017-09-09 18:49:10,688 MigrationManager.java:172 - Migration task failed to complete
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: schema complete, ready to bootstrap
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: waiting for pending range calculation
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: calculation complete, ready to bootstrap
Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
ERROR [main] 2017-09-09 18:49:12,960 CassandraDaemon.java:706 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,988 HintsService.java:220 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 Gossiper.java:1538 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 MessagingService.java:984 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/192.168.44.105] 2017-09-09 18:49:13,002 MessagingService.java:1338 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:13,360 HintsService.java:220 - Paused hints dispatch

With the section of interest being;

Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true

When I manually started the service it would join the cluster with no issues. There was clearly a timing issue here preventing the final node from joining the cassandra ring. I thought the solution might lie in using the serial ansible keyword but this is only applicable to the play, not the task level, and it didn’t have the level of control I wanted.

I found some discussion of the issue, on the ansible github, and adapted a workaround to include a sleep between each cassandra service start.

  - name: Staged Cassandra Service Start
    run_once: true
    with_items: "{{ play_hosts }}"
    delegate_to: "{{ item }}"
    shell: "sleep 60 && /usr/sbin/service cassandra start"
    when: deploy_mode == True

This makes clever use of the delegate_to to execute a sleep and service restart on each host. This staged execution of the cassandra service start allowed all nodes to join the ring successfully.


MySQL 5.7: root password is not in mysqld.log

I came across this issue today when working on an ansible playbook with MySQL 5.7. Old habits die hard and I was still trying to use mysql_install_db to initialise my instance. It seems a few others have been doing the same. The effect of using mysql_install_db in more recent version of MySQL is that we end up not knowing the root password. This is now set to a random value rather than being blank/unset. Nothing is logged to the mysqld.log file unless you use mysqld –initialize first;

Instead of using mysql_install_db we should be doing something like this;

  - name: Init MySQL
    command: mysqld --initialize --datadir=/var/lib/mysql
    args:
      creates: /var/lib/mysql/mysql/user.frm
    become_user: mysql

Now when searching for the root password we will find something in the error log;

sudo cat /var/log/mysqld.log | grep "temporary password"
2017-09-02T15:16:32.318530Z 1 [Note] A temporary password is generated for root@localhost: XXXXXXXX

We can login to the instance with the root user using this password;

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

But we are clearly limited in what we can do. We are unable to read any tables or even view the databases. We must reset the password first. This bash one-liner will do that;

mysql -u root -p$(cat /var/log/mysqld.log | grep "temporary password" | rev | cut -d " " -f 1 | rev) -e "SET PASSWORD FOR root@localhost = 'BigSecret'" --connect-expired-password;

We can put this into an ansible task to continue with our automation;

  - name: Reset the root@localhost password
    shell: mysql -u root -p$(cat /var/log/mysqld.log | grep "temporary password" | rev | cut -d " " -f 1 | rev) -e "SET PASSWORD FOR root@localhost = 'BigSecret'" --connect-expired-password && touch /home/vagrant/root_pw_reset.success;
    args:
      creates: /home/vagrant/root_pw_reset.success

I’d recommend you put the bash line into a script and use the copy module to copy it to your host before executing it. It looks a whole lot tidier that way. Happy automating!


A Cassandra Cluster using Vagrant and Ansible

I’ve started a new project to create a Cassandra Cluster for development purposes. It’s available on my github and uses Vagrant, Ansible, and VirtualBox.

Assuming everything is installed it’s quite easy to get started;

git clone https://github.com/rhysmeister/CassandraCluster.git
cd CassandraCluster
vagrant up

Check the status of the machines;

vagrant status;
Current machine states:

cnode1                    running (virtualbox)
cnode2                    running (virtualbox)
cnode3                    running (virtualbox)
cnode4                    running (virtualbox)
cnode5                    running (virtualbox)

To access a node via ssh;

vagrant ssh cnode1;

One inside the host we can view the status of the Cassandra Cluster with nodetool;

[vagrant@cnode1 ~]$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
UN  192.168.44.104  106.51 KiB  256          40.0%             b191d49f-822c-40d3-bde4-926c4494a707  rack1
UN  192.168.44.105  84.39 KiB  256          39.4%             2b7d5381-7121-46f4-8800-dad9fadc4c85  rack1
UN  192.168.44.101  104.06 KiB  256          39.2%             cd6d8ed2-d0c0-4c90-90a1-bda096b422e1  rack1
UN  192.168.44.102  69.98 KiB  256          41.4%             303c762c-351d-43a6-a910-9a2afa3ec2be  rack1
UN  192.168.44.103  109.04 KiB  256          40.1%             0023da19-7b3f-420b-a6b8-ace8b5118b0d  rack1

The Administrator credentials for Cassandra are set in the cassandra.yml file and can be modified.

See the following variables;

cassandra_admin_user
cassandra_admin_user_pwd