Setting up a Ansible Module Test Environment

I’ve begun developing some Ansible modules and have created a Vagrant environment to help with testing. You can check it over over on my github. The environment has been created to test some MongoDB modules but can easily be repurposed to another use. It’s quite simple to get started;

git clone https://github.com/rhysmeister/AnsibleTest.git
cd AnsibleTest
vagrant up

This will fire up a VM and install two versions of Python; 2.6 and 3.5. These are the two versions of python that all Ansible modules should be tested against. These version can be activated with the following bash aliases; py26 and py35. These aliases will set the version of python to the appropriate one as well as setting up the ansible environment so your custom modules can be found (those that are in the git repo cloned during setup).

The tasks/main.yml file does contain a bunch of other tasks, to deploy some bash scripts and ansible playbooks, to help with module testing. These can be removed or modified to suit your own purposes.

Getting JConsole working over ssh with Cassandra

I’ve been working with Cassandra recently and wanted to start using JConsole. JConsole exposes some pretty useful information about Cassandra and the Java environment in which it runs. It operates over JMX and by default it only accepts connections from the localhost. Since we don’t run GUIs on our servers and we didn’t want to open up the JMX service to remote connections we needed to do a little wizardry to connect. First I thought of tunneling over ssh;

ssh -N -L 7199:localhost:7199 remotehost

This is a pretty standard way to connect a service that only listens for connections from the localhost. However this would not work at all. After much troubleshooting I came across a solution.

The solution was to connect via a SOCKS proxy;

ssh -fN -D 7199 remotehost

This essentially setup up a proxy, running on port 7199 (this can be anything, it does not have to match the JMX port), to the remote node (cassandra in this case). Then we can connect with JConsole like so…

jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7199 service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi -J-DsocksNonProxyHosts=

This allows JConsole to connect to the JMX service running on the remote Cassandra node;

JConsole connectted via JMX to a remote Cassandra Node with a SOCKS proxy

Jenkins test instance with Vagrant & Ansible

Here’s yet another project using Vagrant and Ansible. This time it’s for a Jenkins instance. To get started head on over to the Jenkins test instance github page. Consult the readme for setup instructions.

AWX installation using Vagrant and Ansible

Over on my github is a new project firing up a test instance of AWX. This is based on the following awx installation notes. The project is AWX-on-CentOS-7. You’ll need Virtualbox, Vagrant and Ansible installed to get this up and running. Getting started is simple;

1. From a shell

ansible-galaxy install geerlingguy.repo-epel
git clone https://github.com/rhysmeister/AWX-on-CentOS-7
cd AWX-on-CentOS-7
vagrant up

2. Access url http://http://192.168.4.111/

3. Login with;

u: admin
p: password

You’ll be presented with the main screen of AWX.

Main page of Ansible AWX

Main page of Ansible AWX

Staged service restart with Ansible

I’ve been working on a small project to create a Cassandra Cluster for Development purposes. I’m using Vagrant and Ansible to deploy a 5-node Cassandra Cluster and node #5 would always fail to join the cluster.

I checked /var/log/cassandra/cassandra.log and this is what I found;

INFO  [InternalResponseStage:1] 2017-09-09 18:49:07,673 ColumnFamilyStore.java:406 - Initializing system_auth.roles
INFO  [main] 2017-09-09 18:49:08,666 StorageService.java:1439 - JOINING: waiting for schema information to complete
ERROR [main] 2017-09-09 18:49:09,687 MigrationManager.java:172 - Migration task failed to complete
ERROR [main] 2017-09-09 18:49:10,688 MigrationManager.java:172 - Migration task failed to complete
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: schema complete, ready to bootstrap
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: waiting for pending range calculation
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: calculation complete, ready to bootstrap
Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
ERROR [main] 2017-09-09 18:49:12,960 CassandraDaemon.java:706 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,988 HintsService.java:220 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 Gossiper.java:1538 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 MessagingService.java:984 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/192.168.44.105] 2017-09-09 18:49:13,002 MessagingService.java:1338 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:13,360 HintsService.java:220 - Paused hints dispatch

With the section of interest being;

Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true

When I manually started the service it would join the cluster with no issues. There was clearly a timing issue here preventing the final node from joining the cassandra ring. I thought the solution might lie in using the serial ansible keyword but this is only applicable to the play, not the task level, and it didn’t have the level of control I wanted.

I found some discussion of the issue, on the ansible github, and adapted a workaround to include a sleep between each cassandra service start.

  - name: Staged Cassandra Service Start
    run_once: true
    with_items: "{{ play_hosts }}"
    delegate_to: "{{ item }}"
    shell: "sleep 60 && /usr/sbin/service cassandra start"
    when: deploy_mode == True

This makes clever use of the delegate_to to execute a sleep and service restart on each host. This staged execution of the cassandra service start allowed all nodes to join the ring successfully.