A quick intro to Ansible

[rlkoshak (Rich Koshak)]

I’ve been asked to provide a tutorial for using Ansible. Ansible has very good docs. This isn’t going to be a thorough tutorial. My goal is to provide just enough information that you will be able to use the Ansible docs to find out the details you need. I’m also not an expert and there are many anti-patterns and violations of best practice in the examples below.

The way I think about Ansible is as a bunch of scripts.

You have three main things to worry about with Ansible. The inventory, the playbook, and roles.

Inventory

This is a list of all your machines tagged with their purpose. For example, here is my inventory.

[desktops]
huginn

[media]
medusa

[homeauto]
argus

[nightscout]
fenrir

[cctv]
argus

[pis]
manticore
cerberos
hydra
norns-vpn
sibyl ansible_user=pi
sirens

[nas]
fafnir

[sensorReporter]
cerberos
hydra
manticore

[banana]
nidhogg

[tripwire]
manticore
cerberos
hydra
norns-vpn
sibyl ansible_user=pi
nidhogg
sirens
fenrir
argus
medusa
huginn
fafnir

[camera]
cerberos

[aiy]
sibyl ansible_user=pi

[music]
sirens

[remoteoh]
norns-vpn

[computers]
fafnir
fenrir
medusa
argus
manticore
cerberos
hydra
norns-vpn
sibyl ansible_user=pi
huginn
nidhogg
sirens

[nutclients]
fafnir
fenrir
medusa
argus
huginn

[shinobi]
argus

[docker]
argus
medusa

Yes, I have a Greek and Norse myth creature theme going. The main thing to notice is that machines can and do appear under more than one category. This is where I can, for example, put a RPi cam on one of my other RPis (let’s say hydra) and install and configure all the software just by adding hydra under the [camera] category. The next time I run those playbooks hydra will be set up and configured just like cerberos is. There are lots of places you can define this. I like to check this into git with the rest of my configs so I put this in a separate file. It can go under /etc/ansible as well.

The host names need to be resolvable from the Ansible machine (machine where you are running ansible-playbook). You may need to us FQDN or IP address depending on your network and local DNS configuration.

Playbook

The playbooks are where you define what configuration needs to be done on each of those categories. Here is my playbook for homeauto.

---

# Prerequisites:
# - install ssh
# - set up ssh for certificate only logins
# - install python
# - configure login account with sudo permissions

- hosts: homeauto
  roles:
    - common
    - fish
#    - mount-home
    - vm
    - docker
    - ssmtp
    - openzwave
    - mosquitto
    - influxdb
    - openhab
#    - grafana # not working for some reason
    - screengrab
    - share-ohconf
    - multitail
    - glances
    - sensorreporter
    - tripwire

Put in English, this file says “for all the hosts tagged as homeauto in the inventory, execute the following roles, common, fish, …”

At the top of the file you will see I’ve documented some prerequisites. Ansible requires ssh and python on the hosts you are trying to configure. To avoid having to type in my password a bunch of times or to provide a password to ansible-playbook, I configure all of my machines with ssh certificate login.

The way that Ansible executes a playbook is:

1. ssh to the remote machine
2. if required, sudo to another user
3. download the Ansible python modules necessary to execute the roles
4. execute those python modules in order with the defined parameters
5. clean up

So you don’t need to have Ansible installed on the remote machine, just on the machine you are running Ansible on. However, some of the python modules will require python libraries to be installed on the remote machine. Luckily you can add a task to your playbooks to install the prerequisites just before they are needed.

Roles

Roles is where we get down to the fine detail of defining all of the steps necessary to configure a machine with a given role. I’ve split my roles up by application (i.e. each application has their own role). This is not the only approach but it works for me. You see the following roles in the playbook above:

• common: perform an apt-get update, install common apps like vim, htop, and curl, add the machine’s hostname to /etc/hosts, and install the nut client (this last one should probably be it’s own separate role.
• fish: install fish, make it my default shell, and download my fish configs from git
• mount-home: configure fstab to mount my home directory from my NAS (I only do this on my desktop now)
• vm: install OpenVMTools
• docker: all of the services I install are running in Docker, install the latest Docker and other ansillary stuff like py-docker so the later roles can download and run Images.
• ssmtp: install and configure ssmtp so I can get emails from the host (e.g. results of cron jobs)
• openzwave: on occasion I’ve found it useful to administer my controller
• openhab: download and configure the openHAB Docker image; roles can depend on other roles and this one depends on influxdb and mosquitto so both of those get downloaded and configured too; in addition to just installing openHAB, it also configures InfluxDB with the openhab database; OH configs are checked out of git
• sreengrab: installs and configures screengrab
• share-ohconf: sets up a samba share of the OH configs so I can edit them using VSCode (with the SSH Remote extension this is no longer needed and I plan on removing it)
• multitail: installs and configures it with openHAB log highlighting
• glances: installs and configures glances which is kind of like htop only with more information
• sensorReporter: my own python script
• tripwire: I find it useful to keep track of what my machines are doing when I’m not looking. This role requires you to manually install tripwire first because I’ve not figured out a way to do this automatically because tripwire uses a curses interface during installation.

Ansible has a defined file structure it likes you to have roles defined in. See https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html#role-directory-structure for details.

My structure is as follows:

ansible
    koshak.net-hosts: Inventory
    update: It is not best practice, but I have separate build and update playbooks
    build
        group_vars/all: place to define variables used across roles
        home-auto.yml: file included above, also includes a bunch of other playbooks like media, nightscout, pis, etc.
        roles: this is where the roles go
            openhab: the openHAB role.
                see the docs for the folders under this

Luckily you don’t have to create all these folders by hand. Ansible provides a tool called ansible-galaxy that can create these folders for you. From the build folder I’d run:

ansible-galaxy init roles/openhab

and the openhab folder and all the special Ansible subfolders will be created with a default (i.e. empty) yml file in each.

Let’s look at my openhab role:

--
- name: Change openhab group to 9001
  group:
    gid: 9001
    name: openhab
    state: present
    system: yes
  become: yes

- name: Create openhab user
  user:
    comment: 'openHAB'
    createhome: no
    name: openhab
    shell: /bin/false
    state: present
    system: yes
    uid: 9001 # uid of openhab user inside the official container
    group: openhab
  become: yes

- name: Add the openhab user to the dialout group
  command: usermod -a -G dialout openhab
  become: yes

- name: Add {{ share_user }} to the openhab group
  command: usermod -a -G openhab {{ share_user }}
  become: yes

- name: Set permissions on openhab data folder so we can check out into it
  file:
    path: "{{ openhab_data }}"
    state: directory
    owner: openhab
    group: openhab
    mode: u=rwx,g=rwx,o=rx
  become: yes

- name: Checkout openhab config
  git:
    repo: "{{ openhab_conf_repo }}"
    dest: "{{ openhab_data }}"
    accept_hostkey: yes
  become: yes

- name: Change ownership of openhab config
  file:
    path: "{{ openhab_data }}"
    owner: openhab
    group: openhab
    recurse: yes
  become: yes

- name: Create expected folders if they don't already exist
  file:
    path: "{{ item }}"
    state: directory
    owner: openhab
    group: openhab
  become: yes
  become_user: openhab
  with_items:
    - "{{ openhab_data }}/conf"
    - "{{ openhab_data }}/userdata"
    - "{{ openhab_data }}/addons"
    - "{{ openhab_data }}/.java"

- name: Create database
  influxdb_database:
    hostname: "{{ influxdb_ip_address }}"
    database_name: "{{ openhab_influxdb_database_name }}"
    state: present
    username: admin
    password: "{{ influxdb_admin_password }}"

# TODO there is currently a bug which prevents us from using influx in the container
- name: Create openhab user
#  command: influx -username admin -password {{ influxdb_admin_password }} -database '{{ openhab_influxdb_database_name }}' -execute "CREATE USER {{ influx_openhab_user }} WITH PASSWORD '{{ influx_openhab_password }}'"
  command: curl -XPOST http://localhost:8086/query?db={{ openhab_influxdb_database_name }}&u=admin&p={{ influxdb_admin_password }} --data-urlencode "q=CREATE USER {{ influx_openhab_user }} WITH PASSWORD '{{ influx_openhab_password }}'"

- name: Give openhab permissions on openhab db
#  command: influx -username admin -password {{ influxdb_admin_password }} -database '{{ openhab_influxdb_database_name }}' -execute "GRANT ALL ON {{ openhab_influxdb_database_name }} TO {{ influx_openhab_user }}"
  command: curl -XPOST http://localhost:8086/query?db={{ openhab_influxdb_database_name }}&u=admin&p={{ influxdb_admin_password }} --data-urlencode "q=GRANT ALL ON {{ openhab_influxdb_database_name }} TO {{ influx_openhab_user }}"

- name: Create grafana user
#  command: influx -username admin -password {{ influxdb_admin_password }} -database '{{ openhab_influxdb_database_name }}' -execute "CREATE USER {{ influx_grafana_user }} WITH PASSWORD '{{ influx_grafana_password }}'"
  command: curl -XPOST http://localhost:8086/query?db={{ openhab_influxdb_database_name }}&u=admin&p={{ influxdb_admin_password }} --data-urlencode "q=CREATE USER {{ influx_grafana_user }} WITH PASSWORD '{{ influx_grafana_password }}'"

- name: Give grafana read permissions on openhab_db
##  command: influx -username admin -password {{ influxdb_admin_password }} -database '{{ openhab_influxdb_database_name }}' -execute "GRANT READ ON {{ openhab_influxdb_database_name }} TO {{ influx_grafana_user }}"
  command: curl -XPOST http://localhost:8086/query?db={{ openhab_influxdb_database_name }}&u=admin&p={{ influxdb_admin_password }} --data-urlencode "q=GRANT READ ON {{ openhab_influxdb_database_name }} TO {{ influx_grafana_user }}"

- name: Install/update openHAB docker
  docker_container:
    detach: True
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0:rwm"
      - "/dev/ttyUSB1:/dev/ttyUSB1:rwm"
    env:
      EXTRA_JAVA_OPTS: "-Xbootclasspath/a:/openhab/conf/automation/jython/jython-standalone-2.7.0.jar -Dpython.home=/ope
nhab/conf/automation/jython -Dpython.path=/openhab/conf/automation/lib/python"
      CRYPTO_POLICY: unlimited
    hostname: argus.koshak.net
    image: "{{ openhab_version }}"
    log_driver: syslog
    name: openhab
    network_mode: host
    pull: True
    recreate: True
    restart: True
    restart_policy: always
    tty: yes
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - "{{ openhab_data }}/conf:/openhab/conf"
      - "{{ openhab_data }}/userdata:/openhab/userdata"
      - "{{ openhab_data }}/addons:/openhab/addons"

As you can see a role, just like a playbook, is in YAML format. Each section is a task. We give each task a name so we can more easily follow the execution when it runs. Then we call an Ansible module to perform some task. https://docs.ansible.com/ansible/latest/modules/modules_by_category.html lists all the hundreds of available modules. Just about any administration task you can think of has a module, and if no there is command which lets you execute an arbitrary command line command.

So, from the top of the above playbook we have the following tasks.

• group: creates an openhab group and sets the gid to 9001 which matches the gid inside the container. The become:yes means run the command as another user, in this case root.

• user: create an openhab user with uid 9001 which matches the uid inside the container.

• command: execute the command usermod to add the openhab user to the dialout group

• command: execute the command usermod to add my login ({{ }} denotes a variable, in this case {{ shared_user }} is my login and it’s defined in global_vars/all) to the openhab group

• file: create a folder and set it’s permissions to allow group write access. {{ openhab_data }} is a variable defined in the ansible/build/roles/openhab/vars/main.yml.

• git: checks out my OH config from git into the data directory.

• file: change ownership of the checked out files to openhab:openhab

• file: created required folders in case there was nothing in git to check out and this is a brand new install. The with_items is a deprecated way to run the same task on more than once with different values. There is a newer way to do it, see the Ansible docs or the warning that pops up when you run it for details.

• influxdb_database: create the openhab database on InfluxDB

• command: create the openhab user on InfluxDB

• command: give the openhab user permissions on the openhab database on InfluxDB

• command: create the grafan user on InfluxDB

• command: give the grafana user read permissions on the openhab database on InfluxDB

• docker_container: download the official openHAB Docker image and start a container running

One further note about roles. Roles should be idempotent. This means that you can run it repeatedly and if there is nothing to do, Ansible won’t do anything. Only if there is something to do does it run. This is where I’ve gone wrong in a few areas and why I have two different sets of Ansible roles and playbooks, one for building and another for updating. Eventually I’ll correct this problem.

Ansible Galaxy

One important thing to know is that you probably don’t even have to write most of your Ansible roles yourself. Ansible provides a service similar to dockerhub where users can check in and distribute their developed roles. Github is also another great place to look. There is at least one developed for openHAB (besides the one I posted above).

I’d be happy to answer any questions if there is anything not clear or provide further examples below.

[marcel_erkel]

I think the topic from @holger_hees is worth to mention here as well for more inspiration:

Complete automated IoT Server Setup

Hello, maybe some people already knowing my complex openHAB Setup. During the last years, also the underlying server setup becomes more and more complex with a lot of services and their configuration. Now, I rebuild everything from scratch using ansible and vagrant. It is a fully automated solution to install and configure my server. Just to give you a quick impression. It is based on ~340 ansible tasks. If you want to know more about it, I tried to document it here. http://www.intranet-der…

[fex (Felix)]

A question on which I’m still a bit undecided, not only in the context of ansible, is the use of a dedicated, nologin openhab user in combination with git. As far as I see it I would have to run chown -R openhab:openhab after every git pull, is that correct?

How do you handle updates to your openhab config? I see that you (@rlkoshak) initially create an dedicated openhab user with u/gid 9001, check out your config from git with sudo and than change ownership of these files to your user openhab

- name: Checkout openhab config
  git:
    repo: "{{ openhab_conf_repo }}"
    dest: "{{ openhab_data }}"
    accept_hostkey: yes
  become: yes

- name: Change ownership of openhab config
  file:
    path: "{{ openhab_data }}"
    owner: openhab
    group: openhab
    recurse: yes
  become: yes

But how do you handle updates to your running instance? Do you use sudo git pull & sudo chown -R openhab:openhab for every update?
Or always via ansible? Do you maybe even run the whole openhab role whenever you want to pull an update to your config?? Or is that where your separation between build and update roles comes into play and you have a separate role for pulling changes via git?

[rlkoshak (Rich Koshak)]

@fex

As far as I see it I would have to run chown -R openhab:openhab after every git pull , is that correct?

Not necessarily. Were I to rewrite/update my role in the OP I would probable configure the openhab user with the SSH certs needed to clone/pull from git as itself. Even if you can’t log into openhab it still has a home directory. On an installed openHAB system that home is /var/lib/openhab2. You can specify the home to where ever you deem appropriate (I’d choose /opt/userdata for symmetry) and configure a .ssh folder with the right certs in there. Then you don’t need to change the permissions.

As it is written above, I’ve done all of that stuff with root instead of the openhab user so I have to chown/chmod the files after I check them out.

@fex

But how do you handle updates to your running instance? Do you use sudo git pull & sudo chown -R openhab:openhab for every update?

The way I work, I don’t. I make the edits in place and push them to my git server. I almost never pull them. And in those rare cases where I do need to pull them (e.g. back out of a change) I do the chown/chmod. I’ve also since configured my openhab user with the right ssh certs so now I do the operations as the openhab user, but I’ve not gone back to change my Ansible playbook yet (and I did that after posting the above).

Another thing I do to make things easier is to add my login to the openhab group and make sure all the files have group write permission. Then I don’t have to sudo -u openhab to change things.

@fex

Or always via ansible? Do you maybe even run the whole openhab role whenever you want to pull an update to your config?? Or is that where your separation between build and update roles comes into play and you have a separate role for pulling changes via git?

This is a case where my update roles are separate from my build roles but it’s largely driven by other factors. In general, the configs that are running in my OH instance are the latest and greatest. The configs that are in git will either be the same or slightly behind those so doing a git pull would never be performed as part of an update. But that’s just the way I work. For others having a git pull part of the update would make sense. For me it doesn’t because there would never be something to pull.

NOTE: this is just the case with openHAB. In other cases like my fish functions and sensorReporter configs I do in fact include a git pull as part of the update roles, largely because these get deployed to many machines and the pull helps keep them all in sync.

Ultimately, set things up in a way that makes sense for you and the way that you work. And don’t be afraid to change things as you go along. But, if you were being an Ansible purist, there would be no difference between the upgrade and installation roles and the roles would be very careful not to change things if there is nothing to change. My roles are pretty sloppy in that regard, which is perfectly fine for my purposes. But it’s also why I never submitted any of them to Ansible Galaxy.

[fex (Felix)]

Did use additional spare time during the last few months to familiarize myself with Ansible - and to create a ‘Getting Started’ project:

fex01/ansible-gitserver

Next steps: creating a playbook for openHAB