Linux: Reclaim disk space used by “deleted” files

I had a misbehaving application consuming a large amount of space in /tmp. The files were visible in the /tmp volume itself but lsof allowed me to identify them.

lsof -a +L1 -c s3fs /tmp
s3fs    59614 root   28u   REG  253,3 584056832     0   22 /tmp/tmpfMIMLU4 (deleted)
s3fs    59614 root   29u   REG  253,3 584056832     0   15 /tmp/tmpfC3KN7h (deleted)
s3fs    59614 root   31u   REG  253,3 584056832     0   24 /tmp/tmpfkA6wcj (deleted)
s3fs    59614 root   32u   REG  253,3 584056832     0   23 /tmp/tmpfJxs04J (deleted)
s3fs    59614 root   34u   REG  253,3 584056832     0   12 /tmp/tmpfgg8Ifr (deleted)
s3fs    59614 root   35u   REG  253,3 584056832     0   27 /tmp/tmpfbR2pji (deleted)

The best way to reclaim this disk space would be to restart the application, in this case s3fs. Sadly I wasn’t in the position to be able to do this. So a little skulldugery was in need…

It’s possible to truncate the file in the proc filesystem with the pid and fd. Example below…

: > /proc/59614/fd/31 # Yes the command starts with a colon

The above example truncates the file /tmp/tmpfkA6wcj to zero bytes and releases the space to the operating system. This should be safe to use but, as always with stuff you read on the Internet, make sure you do your own testing, due diligence, keep out of reach of children and so on.

ansible-vault unexpected exception on Ubuntu

When attempting to edit an ansible-vault file…

ansible-vault edit roles/cassandra_backup/vars/test_s3_cfg.yaml 

The following error was received…

ERROR! Unexpected Exception, this is probably a bug: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

Encountered on this version of Ubuntu…

Linux xxxxxxxxx 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The full python stacktrace can be viewed as follows…

ansible-vault edit roles/cassandra_backup/vars/test_s3_cfg.yaml -vvv
Traceback (most recent call last):

  File "/usr/local/bin/ansible-vault", line 118, in 

    exit_code =

  File "/usr/local/lib/python2.7/dist-packages/ansible/cli/", line 255, in run


  File "/usr/local/lib/python2.7/dist-packages/ansible/cli/", line 155, in execute


  File "/usr/local/lib/python2.7/dist-packages/ansible/cli/", line 446, in execute_edit


  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/", line 953, in edit_file

    plaintext, vault_id_used, vault_secret_used = self.vault.decrypt_and_get_vault_id(vaulttext)

  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/", line 736, in decrypt_and_get_vault_id

    b_plaintext = this_cipher.decrypt(b_vaulttext, vault_secret)

  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/", line 1316, in decrypt

    b_key1, b_key2, b_iv = cls._gen_key_initctr(b_password, b_salt)

  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/", line 1158, in _gen_key_initctr

    b_derivedkey = cls._create_key_cryptography(b_password, b_salt, key_length, iv_length)

  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/", line 1131, in _create_key_cryptography

    b_derivedkey = kdf.derive(b_password)

  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/kdf/", line 50, in derive


  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/", line 307, in derive_pbkdf2_hmac

    key_material_ptr = self._ffi.from_buffer(key_material)

TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

This is due to a problem with packages instakll via apt and pip. It can be fixed with the following procedure…

sudo -E pip uninstall cryptography -y
sudo -E apt-get purge python3-cryptography
sudo -E apt-get autoremove
sudo -E pip3 install --upgrade cryptography

“could not open session” error in docker container

I received the following error, attempting to cat a log file, inside a docker contain when troubleshooting another issue…

TASK [setup_cassandra : shell] *************************************************
changed: [testhost] => {"changed": true, "cmd": "cat /var/log/cassandra/*", "delta": "0:00:00.004140", "end": "2019-03-16 18:48:28.684133", "rc": 0, "start": "2019-03-16 18:48:28.679993", "stderr": "", "stderr_lines": [], "stdout": "could not open session", "stdout_lines": ["could not open session"]}

A bit of googling suggested the use of the –privileged flag of the docker command. I was using the ansible-test tool, which invokes docker, so the equivalent flag was –docker-privileged

test/runner/ansible-test integration -v cassandra_backup --docker centos6 --docker-privileged

This flag then allowed me to continue with my troubleshooting.

Ansible: stop / start services on random hosts

In the coming weeks I’m performing some testing of a new application on a Cassandra cluster. To add a little randomness into some of the tests I thought it would be interesting to give the Cassandra service a little kick. I created a simple Ansible playbook this afternoon that does this. A simple Chaos Monkey if you like. Here’s the basic flow of the playbook…

1. Select random host from play_hosts.
2. Stop service.
3. Wait for interval.
4. Start service.
5. Wait for Service back up, Port?
6. Wait for second defined interval.

These tasks are repeated n number of times as specified by the user. This playbook is interesting because it uses a couple of nifty tricks…

The first is a method to dynamically generate a list to iterate over. This is how we control the number of times we restart the service.

  - name: Generate a list we will iterate over
      restart_iterations: "{{ restart_iterations | default([]) + [item | int] }}"
    with_sequence: start=1 end="{{ max_iterations }}"
    run_once: yes

The second uses the loop construct. The current_iteration variable is made available in the tasks.yml file

  - name: Run main tasks file
    include_tasks: tasks.yml
    loop:  "{{ restart_iterations }}"
      loop_var: current_iteration

The playbook can optionally produce a log to make it a little easier to see what nodes it’s executing on…

tail -f /tmp/service_restarter.log 
2019-01-27 18:38:51 CET - Current iteration is 1 and is executing on cnode2
2019-01-27 18:41:18 CET - Current iteration is 2 and is executing on cnode4
2019-01-27 18:43:47 CET - Current iteration is 3 and is executing on cnode3
2019-01-27 18:46:15 CET - Current iteration is 4 and is executing on cnode3
2019-01-27 18:48:42 CET - Current iteration is 5 and is executing on cnode5
2019-01-27 18:51:07 CET - Current iteration is 6 and is executing on cnode2
2019-01-27 18:53:34 CET - Current iteration is 7 and is executing on cnode1
2019-01-27 18:56:03 CET - Current iteration is 8 and is executing on cnode4
2019-01-27 18:58:29 CET - Current iteration is 9 and is executing on cnode4
2019-01-27 19:00:55 CET - Current iteration is 10 and is executing on cnode2

Here’s how you might execute the playbook…

ansible-playbook -l cassandra -i inventory service_restarter.yml

The playbook is available over on my github/ServiceRestarter.

While this playbook was designed with Cassandra in mind it should work for any service that listens on a TCP port. See the README file for an explanation of the variables.

Use restview to to make the Ansible rst documentation browsable

The ansible-doc package not only installs the command line tool but also some quite detailed Ansible documentation in rst format. It would be nice if it was browsable in a html format. Here’s how that can happen (Redhat/CentOS)

First install pip and restview

sudo yum install python-pip
sudo pip install restview

This will allow all hosts on your network to access the documentation at http://hostname:33333 if your firewall allows it.

restview /usr/share/doc/ansible-doc-2.7.5/rst/ --listen 33333 --allowed-hosts * &

Alternatively, if you’re on a desktop computer, use the following to launch a browser…

restview /usr/share/doc/ansible-doc-2.7.5/rst/ --browser

Hint: This might be useful for those taking the EX407 Ansible Exam assuming you can install these packages. Having this at your fingertips could prove to be very useful should something like how to structure a jinja2 template slip your mind.

There are a few rendering issues, resulting in broken links, but nevertheless there’s a lot of very useful information. I’ll update this if I get around to finding a solution for that (probably a restview alternative).  Here’s a few screenshots showing what’s provided…

ansible documentation ansible documentation ansible documentation ansible documentation

UPDATE: I took the Ansible exam and I now know this isn’t really needed. Ample easily used documentation is provided.