Getting JConsole working over ssh with Cassandra

I’ve been working with Cassandra recently and wanted to start using JConsole. JConsole exposes some pretty useful information about Cassandra and the Java environment in which it runs. It operates over JMX and by default it only accepts connections from the localhost. Since we don’t run GUIs on our servers and we didn’t want to open up the JMX service to remote connections we needed to do a little wizardry to connect. First I thought of tunneling over ssh;

ssh -N -L 7199:localhost:7199 remotehost

This is a pretty standard way to connect a service that only listens for connections from the localhost. However this would not work at all. After much troubleshooting I came across a solution.

The solution was to connect via a SOCKS proxy;

ssh -fN -D 7199 remotehost

This essentially setup up a proxy, running on port 7199 (this can be anything, it does not have to match the JMX port), to the remote node (cassandra in this case). Then we can connect with JConsole like so…

jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7199 service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi -J-DsocksNonProxyHosts=

This allows JConsole to connect to the JMX service running on the remote Cassandra node;

JConsole connectted via JMX to a remote Cassandra Node with a SOCKS proxy

Jenkins test instance with Vagrant & Ansible

Here’s yet another project using Vagrant and Ansible. This time it’s for a Jenkins instance. To get started head on over to the Jenkins test instance github page. Consult the readme for setup instructions.

AWX installation using Vagrant and Ansible

Over on my github is a new project firing up a test instance of AWX. This is based on the following awx installation notes. The project is AWX-on-CentOS-7. You’ll need Virtualbox, Vagrant and Ansible installed to get this up and running. Getting started is simple;

1. From a shell

ansible-galaxy install geerlingguy.repo-epel
git clone https://github.com/rhysmeister/AWX-on-CentOS-7
cd AWX-on-CentOS-7
vagrant up

2. Access url http://http://192.168.4.111/

3. Login with;

u: admin
p: password

You’ll be presented with the main screen of AWX.

Main page of Ansible AWX

Main page of Ansible AWX

Staged service restart with Ansible


Warning: count(): Parameter must be an array or an object that implements Countable in /home/fbsqlcom/public_html/youdidwhatwithtsql.com/wp-content/plugins/wp-codebox/main.php on line 31

I’ve been working on a small project to create a Cassandra Cluster for Development purposes. I’m using Vagrant and Ansible to deploy a 5-node Cassandra Cluster and node #5 would always fail to join the cluster.

I checked /var/log/cassandra/cassandra.log and this is what I found;

INFO  [InternalResponseStage:1] 2017-09-09 18:49:07,673 ColumnFamilyStore.java:406 - Initializing system_auth.roles
INFO  [main] 2017-09-09 18:49:08,666 StorageService.java:1439 - JOINING: waiting for schema information to complete
ERROR [main] 2017-09-09 18:49:09,687 MigrationManager.java:172 - Migration task failed to complete
ERROR [main] 2017-09-09 18:49:10,688 MigrationManager.java:172 - Migration task failed to complete
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: schema complete, ready to bootstrap
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: waiting for pending range calculation
INFO  [main] 2017-09-09 18:49:12,952 StorageService.java:1439 - JOINING: calculation complete, ready to bootstrap
Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
ERROR [main] 2017-09-09 18:49:12,960 CassandraDaemon.java:706 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:902) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,988 HintsService.java:220 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 Gossiper.java:1538 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:12,989 MessagingService.java:984 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/192.168.44.105] 2017-09-09 18:49:13,002 MessagingService.java:1338 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2017-09-09 18:49:13,360 HintsService.java:220 - Paused hints dispatch

With the section of interest being;

Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true

When I manually started the service it would join the cluster with no issues. There was clearly a timing issue here preventing the final node from joining the cassandra ring. I thought the solution might lie in using the serial ansible keyword but this is only applicable to the play, not the task level, and it didn’t have the level of control I wanted.

I found some discussion of the issue, on the ansible github, and adapted a workaround to include a sleep between each cassandra service start.

  - name: Staged Cassandra Service Start
    run_once: true
    with_items: "{{ play_hosts }}"
    delegate_to: "{{ item }}"
    shell: "sleep 60 && /usr/sbin/service cassandra start"
    when: deploy_mode == True

This makes clever use of the delegate_to to execute a sleep and service restart on each host. This staged execution of the cassandra service start allowed all nodes to join the ring successfully.

MySQL 5.7: root password is not in mysqld.log

I came across this issue today when working on an ansible playbook with MySQL 5.7. Old habits die hard and I was still trying to use mysql_install_db to initialise my instance. It seems a few others have been doing the same. The effect of using mysql_install_db in more recent version of MySQL is that we end up not knowing the root password. This is now set to a random value rather than being blank/unset. Nothing is logged to the mysqld.log file unless you use mysqld –initialize first;

Instead of using mysql_install_db we should be doing something like this;

  - name: Init MySQL
    command: mysqld --initialize --datadir=/var/lib/mysql
    args:
      creates: /var/lib/mysql/mysql/user.frm
    become_user: mysql

Now when searching for the root password we will find something in the error log;

sudo cat /var/log/mysqld.log | grep "temporary password"
2017-09-02T15:16:32.318530Z 1 [Note] A temporary password is generated for root@localhost: XXXXXXXX

We can login to the instance with the root user using this password;

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

But we are clearly limited in what we can do. We are unable to read any tables or even view the databases. We must reset the password first. This bash one-liner will do that;

mysql -u root -p$(cat /var/log/mysqld.log | grep "temporary password" | rev | cut -d " " -f 1 | rev) -e "SET PASSWORD FOR root@localhost = 'BigSecret'" --connect-expired-password;

We can put this into an ansible task to continue with our automation;

  - name: Reset the root@localhost password
    shell: mysql -u root -p$(cat /var/log/mysqld.log | grep "temporary password" | rev | cut -d " " -f 1 | rev) -e "SET PASSWORD FOR root@localhost = 'BigSecret'" --connect-expired-password && touch /home/vagrant/root_pw_reset.success;
    args:
      creates: /home/vagrant/root_pw_reset.success

I’d recommend you put the bash line into a script and use the copy module to copy it to your host before executing it. It looks a whole lot tidier that way. Happy automating!