Long story short - Backup often, backup soon and write tutorials

You have to manually create the scripts, or download them from Ansible Galaxy or GitHub. Most common software has had some developer already create a Role for you so all you have to do is download it and run it. Once you get some experience, I find it takes less time to code it than to do it from the command line, especially for things that need to be done to more than one machine.

For example, I recently discovered that ssmpt is no longer supported on Raspbian Buster and msmtp is recommended. So I figured out how to install it and configure it from Google. Coded up the role to do all the steps (including uninstalling ssmtp). Then I tested it on one machine, fixed the bugs if there are any, and verify it worked. Then I run it again on all the machines. It took about 20 minutes to switch all 11 of my machines from ssmtp to msmtp. And now when I need to set up a new machine or VM, msmtp will just get installed and configured and I don’t even need to remember that I’m using msmtp.

It’s not a backup system. It’s not intended to be a backup system and I didn’t bring it up to be used as a backup system. I only brought it up to address the “tutorials” mention in the OP. You don’t need tutorials if everything you’ve done is already captured in an ansible playbook.

It absolutely does touch the core OS. But as I’ve said before, the whole idea of the OS being this separate thing on Linux is not correct. The “core OS” is really just the version of the Linux kernel + the thousands of base libraries + a pre-selected set of software that have been tested and known to work well together. All of that, including the kernel, can be updated by apt-get. But only within a given release of the distro. When they release a new version of the OS it’s a much bigger deal. There can be many breaking changes between the versions and the software may be wired together differently. Therefore they treat it as a wholly new and separate thing. Running apt-get update will not migrate you from Ubuntu 18.4 to Ubuntu 20.4.

and it is not always possible and pretty much never advisable to upgrade a distro like that in place. They always recommend a fresh install of the OS because an in place upgrade leaves a lot of cruft, always takes longer (for an RPi we are talking hours), and it often fails leaving you with a broken OS that needs to be reinstalled from scratch anyway.

So in the upgrade case using Ansible, you would install a fresh Ubuntu 20.4 (when it comes out), create your user (which is usually part of the installation process) and configure ssh (Ansible works through ssh). Then run the ansible playbook and it rebuilds the machine on the new OS.

It’s like openHABian only it supports anything, not just openHAB stuff.

If you only have a disk image, you are potentially stuck forever on that old and potentially unsupported version of Raspbian (or whatever) if for some reason the in place upgrade fails to work.

There is a little bit of up front learning of the syntax. Once you have that, it really takes no longer to code it in Ansible than it does to just type it by hand.

For example, here is my msmtp role discussed about above:

---
- name: Install msmtp
  apt:
    name: [msmtp, msmtp-mta, mailutils]
  become: yes

- name: Create config
  template:
    dest: /etc/msmtprc
    mode: u=rw,g=r
    src: msmtprc
  become: yes

The first task installs msmtp and related packages. The second task copies the config file I have as part of my role (I’m not going to show it as it has me password in it) to the destination machine.

That’s it.

Even something really complex like cloning a git repo and building the source code isn’t much more work than that.

Far less time than setting it up by hand on all of those machines and, in my experience, once you learn Ansible, less time than doing it by hand the first time. It’s really not that complex.

Probably my most complex role is my Mosquitto role, but that’s because I have it generate the self signed certificates and set up a user and password as part of the role. It combines around three different tutorials from three different websites. But as you can see, it’s a little long but very straight forward to understand what it’s doing.

---

- name: Create mosquitto user
  user:
    comment: 'Mosquitto'
    createhome: no
    name: mosquitto
    shell: /bin/false
    state: present
    system: yes
  become: yes

- name: Mount mosquitto from file share
  include_role:
    name: mount-cifs
  vars:
    mount_mode: '0660'
    cifs_user: "{{ share_user }}"
    cifs_pass: "{{ share_pass }}"
    cifs_domain: "{{ workgroup }}"
    mount_user: "mosquitto"
    mount_path: "{{ mosquitto_data }}"
    mount_src: "{{ mosquitto_mount }}"

- name: Create mosquitto directories
  file:
    path: "{{ item }}"
    state: directory
    mode: u=rwx,g=rwx,o=rx
  become: yes
  become_user: mosquitto
  with_items:
    - "{{ mosquitto_data }}/config"
    - "{{ mosquitto_data }}/data"
    - "{{ mosquitto_data }}/log"

- name: Copy the prepared mosquitto.conf
  copy:
    src: mosquitto.conf
    dest: "{{ mosquitto_data }}/config/mosquitto.conf"
    mode: u=rw,g=rw,o=r
  become: yes
  become_user: mosquitto

- name: Install mosquitto cients, temporarily install mosquitto
  apt:
    name: "{{ item }}"
    update_cache: no
  become: yes
  with_items:
    - mosquitto
    - mosquitto-clients
    - openssl

- name: Install pexpect
  pip:
    name: pexpect
  become: yes

- name: Generate passwd file
  expect:
    command: mosquitto_passwd -c {{ mosquitto_data }}/config/passwd {{ mosquitto_user }}
    responses:
      'Password\:' : "{{ mosquitto_passwd }}"
      'Reenter password\:' : "{{mosquitto_passwd }}"
  become: yes
  become_user: mosquitto

- name: Remove mosquitto
  apt:
    name: mosquitto
    update_cache: no
    purge: yes
    state: absent
  become: yes

- name: Recreate mosquitto user
  user:
    comment: 'Mosquitto'
    createhome: no
    name: mosquitto
    shell: /bin/false
    state: present
    system: yes
  become: yes

- name: Does the CA already exist?
  stat:
    path: "{{ mosquitto_data }}/config/ca.crt"
  register: mosquitto_ca

- name: Create mosquitto certificate authority
  expect:
    command: openssl req -new -x509 -days 1461 -extensions v3_ca -keyout {{ mosquitto_data }}/config/ca.key -out {{ mosquitto_data }}/config/ca.crt
    responses:
      'Enter PEM pass phrase\:' : "{{ mosquitto_ca_passwd }}"
      'Verifying \- Enter PEM pass phrase\:' : "{{ mosquitto_ca_passwd }}"
      'Country Name \(2 letter code\) \[AU\]\:' : "{{ mosquitto_ca_country }}"
      'State or Province Name \(full name\) \[Some\-State\]\:' : "{{ mosquitto_ca_state }}"
      'Locality Name \(eg, city\) \[\]\:' : "{{ mosquitto_ca_city }}"
      'Organization Name \(eg, company\) \[Internet Widgits Pty Ltd\]\:' : "{{ mosquitto_ca_org }}"
      'Organizational Unit Name \(eg, section\) \[\]\:' : "{{ mosquitto_ca_unit }}"
      'Common Name \(e\.g\. server FQDN or YOUR name\) \[\]\:' : "{{ mosquitto_ca_fqdn }}"
      'Email Address \[\]\:' : "{{ mosquitto_ca_email }}"
  when: mosquitto_ca.stat.islnk is not defined
  become: yes
  become_user: mosquitto

- name: Create server key and cert
  include_role:
    name: create-cert
  vars:
    key: "{{ mosquitto_data }}/config/server.key"
    csr: "{{ mosquitto_data }}/config/server.csr"
    crt: "{{ mosquitto_data }}/config/server.crt"
    cert_country: "{{ mosquitto_ca_country }}"
    cert_state: "{{ mosquitto_ca_state }}"
    cert_city: "{{ mosquitto_ca_city }}"
    cert_org: "{{ mosquitto_ca_org }}"
    cert_unit: "{{ mosquitto_ca_unit }}"
    cert_fqdn: "{{ mosquitto_ca_fqdn }}"
    cert_email: "{{ mosquitto_ca_email }}"
    ca_crt: "{{ mosquitto_data }}/config/ca.crt"
    ca_passwd: "{{ mosquitto_ca_passwd }}"
    ca_key: "{{ mosquitto_data }}/config/ca.key"
  become: yes
  become_user: mosquitto

- name: Get uid
  command: id -u mosquitto
  register: mosquitto_uid

- name: Get gid
  command: id -g mosquitto
  register: mosquitto_gid


- name: Run the eclipse-mosquitto docker container
  docker_container:
    detach: True
    exposed_ports:
      - 1883
      - 9001
      - 8883
    hostname: mosquitto.koshak.net
    image: eclipse-mosquitto
    log_driver: syslog
    name: mosquitto
    published_ports:
      - "1883:1883"
      - "9001:9001"
      - "8883:8883"
    recreate: True
    restart: True
    restart_policy: always
    state: started
    user: 999 # TODO why can't I use the variable created above?
    volumes:
      - /etc/passwd:/etc/passwd:ro
      - /etc/localtime:/etc/localtime:ro
      - /usr/share/zoneinfo:/usr/share/zoneinfo:ro
      - "{{ mosquitto_data }}/config:/mosquitto/config"
      - "{{ mosquitto_data }}/log:/mosquitto/log"
      - "{{ mosquitto_data }}/data:/mosquitto/data"

NOTE: There is some deprecated syntax in the above I’ve not bothered correcting yet.

Stuff in {{ }} are variables defined in another file. The only think unusual here is that I mount the mosquitto folder from a samba share using another role called mount-cifs. Over time I’m slowly replacing samba with NFS. When I do, I’ll have an mount-nfs role and call it instead.

Anyway, you should be able to scroll through that role and figure out what each one of the roles does. In fact, you could probably figure out the exact command line(s) that you would execute by hand for each task. It’s not that hard. In fact it’s far easier than developing openHAB Rules.

It does take some discipline though. No manual changes allowed. All changes, tweaks, config updates, etc must be done through Ansible. If not, then they will be lost when you need to rebuild that machine. That can be frustrating if you only want to change one word in a config file or the like. So you either need to do the change in Ansible or after you make the change remember to update the Ansible roles.

No. But does looking up the user task in the Ansible reference and typing:

- name: Create mosquitto user
  user:
    comment: 'Mosquitto'
    createhome: no
    name: mosquitto
    shell: /bin/false
    state: present
    system: yes
  become: yes

take more time than looking up a tutorial for creating a new user (I forget, which is better adduser or useradd), looking at the man page to remember the arguments to make a system user with no shell or home directory and typing:

adduser --system --no-create-home --disabled-password --disabled-login mosquitto

And if you add in the work of writing that down somewhere so you don’t have to look it up again and remember what you’ve done then you’ve actually created more work for yourself in not using Ansible.

Yes.

Here’s the general flow if you want to install something new.

  1. Create a new role directory structure of your Ansible folder. I use ansible-galaxy init roles/newprogram for that. This is usually on some other machine.
  2. Research the new program so you understand what needs to be done to install it (you have to do this anyway).
  3. Edit roles/newprogram/tasks/main.yml.
  4. Look at the Ansible reference docs to the Ansible equivalent for all of the commands that you need to run as discovered in step 1. Write the tasks as necessary into main.yml. Ansible is usually idempotent meaning you can rerun the same tasks over and over again and nothing will happen if there is nothing to change. This will let you build up the role step by step if necessary, letting you check that everything is working before moving on to the next lines.
2 Likes

Speaking to the original post, I back up my text configuration files and my org.ops4j.pax.logging.cfg whenever I make changes to them by copying them to Google Drive. Only takes a minute to do that.

When my SD card crashed in August, it was a simple task of restoring the last full backup and then copying the most recent text files over to get up and running.

2 Likes

Even though I´m only able to understand part of the scripts (I guess it would be easier if I was used to Linux). I cant help thinking, that this may not seem like a huge job, but insted it looks more like doing a setup twice. Ofcouse there is the benefit of only doing this once and remember to add changes. And I do understand the smart task in just having a bunch of scripts to set up a machine from. But as you said, for one machine, this would be way overkill.

Ansible is based on Python which is OS agnostic just like Java . It uses standard YAML syntax which is used by other software too.

Yep, exactly the files that I also backup regulary. Put them in a freh install and you are fine again :wink:

Why? How? You code it once in the playbook and then execute it. You don’t go and make the changes by hand and then update the playbook. The whole point is that all changes to all the machines under Ansible management are only ever made through Ansible. You only do it once.

I cheat a little bit for very minor changes but that’s bad and I shouldn’t do it.

Please don’t think that you log into a machine and start hacking on the keyboard to, for example install and configure Mosquitto and then go back to Ansible and convert what you did to a role. No no no. You start with the Ansible Role. You may never need to log into the other machine directly at all.

1 Like

I thought you were setting up your “real” machine, and afterwards was coding the Ansible scripts?

Okay, thats what I thought, which is why I said it looks like doing the same twice.

I’m confused. It’s not clear that you understand so I’ll state it again.

I do not do anything twice. I code it in Ansible first and only ever code it in Ansible.

2 Likes

Sorry, I do understand, after you´d explain it in the previous message :slight_smile: Before that, I thought you were doing the setup twice.

1 Like

He’s just confused about your former confusion.:thinking::rofl:

No wonder why this is confusing :slight_smile:

1 Like

All this confusion is very confusing :stuck_out_tongue:

2 Likes

OK I’m going to post all my config files to the forums every day then :wink:

I personally use VMs for openhab, so I can snapshot, test, tear out my hair then restore. Works great for when you want to try out that latest openhab snapshot build!

1 Like

On that topic, I keep all my configuration in git (as does @rlkoshak I believe) and bake things into docker images.

So, if hardware fails, or I break things (as it might have happened once or twice or 3000 times) I am literally one “docker run” away from having things up and running again…

1 Like