I would recommend looking very hard to isolate that Qbus part so that you would never be editing anything that will affect it unless you are actually wanting to. Editing iptables for something else should not have impacted that.
Easier said then done, I know. But something like qbusMQTT could help provide that separation. All the Qbus stuff can be isolated off on it’s own, never touched. The only complicated thing is making sure that whatever is hosting qbusMQTT can see both the Qbus network and the MQTT broker. After that, the only way you’ll break Qbus accidentally is if you break one of those connections.
Just an idea. I’ve been the person everyone is staring at more than once. For many years no new device could join my LAN without being given a static lease in DHCP. If they did, they could not get to the internet (I think DNS was ultimately what didn’t work for them). It’s kind of embarrasing to have to say “oh, you want on the wifi? I need to disable random mac addresses on your phone and then add you to the list of static addresses.”
Now I only have to do that if they want to be exempted from the parental controls filtering. 
Way back in the time before Google when you actually had to register your website with the search engines I ran one of the largest mythology related link aggregation sites on the Internet. Google pretty much killed that whole concept and the forum I hosted was used too little to keep it up. So I let it go and it lives on only in the vaults of the wayback machine now.
For the interested, here’s my role so far.
Note: an Ansible role is separated into a folder structure.
Prerequisites: an account on Backblaze and create a key to provide access. Some local file storage somewhere accessible over sftp and all the ssh certificates deployed to allow what ever user is running the backup (root in my case) can sftp to the destination.
The rule of thumb is to choose a Backblaze region far from your home so region wide disaster scenarios do not wipe out both your home and your backups.
generic_backup/tasks/main.yml
---
# roles/generic-backup/tasks/main.yml
- name: Check if Restic is installed
ansible.builtin.command: which restic
register: restic_check
failed_when: false
changed_when: false
- name: Install Restic from official binary (if missing)
when: restic_check.rc != 0
block:
- name: Get latest release version
ansible.builtin.uri:
url: https://api.github.com/repos/restic/restic/releases/latest
method: GET
return_content: true
register: restic_latest_release
- name: Set Restic version fact
ansible.builtin.set_fact:
restic_version: "{{ restic_latest_release.json.tag_name | replace('v', '') }}"
# yamllint disable rule:line-length
- name: Set Restic download URL
ansible.builtin.set_fact:
restic_url: https://github.com/restic/restic/releases/download/v{{ restic_version }}/restic_{{ restic_version }}_linux_{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}.bz2
checksum_url: "https://github.com/restic/restic/releases/download/v{{ restic_version }}/SHA256SUMS"
# yamllint enable rule:colons
- name: Download Restic binary and checksum file
ansible.builtin.get_url:
url: "{{ item }}"
dest: "/tmp/{{ item | basename }}"
mode: '0644'
loop:
- "{{ restic_url }}"
- "{{ checksum_url }}"
- name: Get expected checksum from file
ansible.builtin.command: >-
awk '/restic_{{ restic_version }}_linux_{{ "arm64" if ansible_architecture == "aarch64" else "amd64" }}.bz2/ {print $1}' /tmp/SHA256SUMS
register: expected_checksum
changed_when: false
- name: Calculate local binary checksum
ansible.builtin.stat:
path: "/tmp/restic_{{ restic_version }}_linux_{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}.bz2"
checksum_algorithm: sha256
register: local_binary
- name: Verify checksum matches
ansible.builtin.fail:
msg: "Checksum mismatch! Expected {{ expected_checksum.stdout }}, got {{ local_binary.stat.checksum }}"
when: local_binary.stat.checksum != expected_checksum.stdout
- name: Extract Restic binary
ansible.builtin.command: "bzip2 -d -f /tmp/restic_{{ restic_version }}_linux_{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}.bz2"
changed_when: false
- name: Move Restic to /usr/local/bin
ansible.builtin.copy:
src: "/tmp/restic_{{ restic_version }}_linux_{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}"
dest: /usr/local/bin/restic
mode: '0755'
remote_src: true
become: true
- name: Update Restic if already installed
ansible.builtin.command: restic self-update
register: restic_update
changed_when: "'is up to date' not in restic_update.stdout"
when: restic_check.rc == 0
become: true
- name: Create secure Restic configuration directory
ansible.builtin.file:
path: /etc/restic
state: directory
mode: '0700'
become: true
- name: Deploy password file for {{ service_name }}
ansible.builtin.copy:
content: "{{ restic_repo_password }}"
dest: "/etc/restic/{{ service_name }}.pass"
mode: '0600'
become: true
- name: Set Restic repo location with port for {{ service_name }}
ansible.builtin.set_fact:
restic_repo: "sftp://{{ restic_user }}@{{ restic_server }}:{{ restic_port | default('22') }}/{{ restic_vault_path }}/{{ ansible_hostname }}/{{ service_name }}"
- name: Set remote Restic repo location for {{ service_name }}
ansible.builtin.set_fact:
remote_restic_repo: "s3:{{ restic_backblaze_bucket_endpoint }}/{{ restic_backblaze_bucket }}/{{ ansible_hostname }}/{{ service_name }}"
- name: Install PostgreSQL client for {{ service_name }}
become: true
when: (db_name | default('')) | length > 0
block:
- name: Add PostgreSQL GPG key
ansible.builtin.get_url:
url: https://www.postgresql.org/media/keys/ACCC4CF8.asc
dest: /usr/share/keyrings/postgresql-archive-keyring.asc
mode: '0644'
become: true
- name: Add PostgreSQL repository
ansible.builtin.apt_repository:
# Note the 'signed-by' pointing to the exact file we just downloaded
repo: "deb [arch=amd64 signed-by=/usr/share/keyrings/postgresql-archive-keyring.asc] http://apt.postgresql.org/pub/repos/apt {{ ansible_distribution_release }}-pgdg main"
state: present
filename: pgdg
update_cache: true
become: true
- name: Ensure PostgreSQL client is installed if DB dump is required for {{ service_name }}
ansible.builtin.package:
name: postgresql-client
state: present
become: true
- name: Configure log rotation for restic backups for {{ service_name }} # noqa: risky-file-permissions
ansible.builtin.copy:
dest: "/etc/logrotate.d/restic-{{ service_name }}"
content: |
{{ restic_log_dir }}/{{ service_name }}.log {
rotate 7
daily
compress
missingok
create 0644 root root
}
become: true
- name: Install execution script for {{ service_name }}
ansible.builtin.template:
src: restic-backup.sh.j2
dest: "/usr/local/bin/restic-backup-{{ service_name }}.sh"
mode: '0700'
become: true
- name: Deploy systemd service unit for {{ service_name }}
ansible.builtin.template:
src: restic-backup.service.j2
dest: "/etc/systemd/system/restic-backup-{{ service_name }}.service"
mode: '0644'
become: true
register: service_unit
- name: Deploy systemd timer unit for {{ service_name }}
ansible.builtin.template:
src: restic-backup.timer.j2
dest: "/etc/systemd/system/restic-backup-{{ service_name }}.timer"
mode: '0644'
become: true
register: timer_unit
- name: Reload systemd daemon for {{ service_name }}
ansible.builtin.systemd:
daemon_reload: true
when: service_unit.changed or timer_unit.changed
become: true
- name: Enable and start systemd timer for {{ service_name }}
ansible.builtin.systemd:
name: "restic-backup-{{ service_name }}.timer"
state: started
enabled: true
become: true
- name: Check if local Restic repository exists for {{ service_name }}
ansible.builtin.command: >-
restic list keys -r {{ restic_repo }}
register: restic_check_repo
failed_when: false
changed_when: false
environment:
RESTIC_PASSWORD_FILE: "/etc/restic/{{ service_name }}.pass"
become: true
- name: Initialize local Restic repository if it does not exist for {{ service_name }}
ansible.builtin.command: >-
restic init -r {{ restic_repo }}
when: restic_check_repo.rc != 0
changed_when: true
environment:
RESTIC_PASSWORD_FILE: "/etc/restic/{{ service_name }}.pass"
become: true
- name: Check if remote Restic repository exists for {{ service_name }}
ansible.builtin.command: >-
restic list keys -r {{ remote_restic_repo }}
register: restic_check_b2_repo
failed_when: false
changed_when: false
environment:
RESTIC_PASSWORD_FILE: "/etc/restic/{{ service_name }}.pass"
AWS_ACCESS_KEY_ID: "{{ backblaze_restic_key_id }}"
AWS_SECRET_ACCESS_KEY: "{{ backblaze_restic_api_key }}"
become: true
- name: Initialize remote Restic repository if it does not exist for {{ service_name }}
ansible.builtin.command: >-
restic init -r {{ remote_restic_repo }}
when: restic_check_b2_repo.rc != 0
changed_when: true
environment:
RESTIC_PASSWORD_FILE: "/etc/restic/{{ service_name }}.pass"
AWS_ACCESS_KEY_ID: "{{ backblaze_restic_key_id }}"
AWS_SECRET_ACCESS_KEY: "{{ backblaze_restic_api_key }}"
become: true
- name: Create local repo configuration fragment on backrest host for {{ service_name }}
delegate_to: "{{ backrest_host }}"
ansible.builtin.template:
src: repo_fragment.json.j2
dest: "{{ backrest_home }}/fragments/local-{{ service_name }}.json"
mode: '0600'
owner: "{{ backrest_uid }}"
group: "{{ backrest_uid }}"
vars:
# Pass all variables required by the template here
repo_type: "sftp"
repo_id: "local-{{ service_name }}"
repo_uri: "sftp:{{ restic_user }}@{{ restic_server }}:/{{ restic_vault_path }}/{{ inventory_hostname }}/{{ service_name }}"
become: true
register: local_fragment
- name: Create remote repo configuration fragment on backrest host for {{ service_name }}
delegate_to: "{{ backrest_host }}"
ansible.builtin.template:
src: repo_fragment.json.j2
dest: "{{ backrest_home }}/fragments/remote-{{ service_name }}.json"
mode: '0600'
owner: "{{ backrest_uid }}"
group: "{{ backrest_uid }}"
vars:
# Pass all variables required by the template here
repo_type: "s3"
repo_id: "remote-{{ service_name }}"
repo_uri: "{{ remote_restic_repo }}"
become: true
register: remote_fragment
- name: Rebuild the Backrest config and restart
when: local_fragment is changed or remote_fragment is changed
block:
- name: Rebuild Backrest Config for {{ service_name }}
become: true
delegate_to: "{{ backrest_host }}"
ansible.builtin.shell: |
set -o pipefail
# Assemble and overwrite the config
jq --slurpfile new_repos <(jq -s '{repos: .}' {{ backrest_home }}/fragments/*.json) \
'.repos = $new_repos[0].repos' {{ backrest_home }}/config/config.json > {{ backrest_home }}/fragments/config.json.tmp && \
mv {{ backrest_home }}/fragments/config.json.tmp {{ backrest_home }}/config/config.json
args:
executable: /bin/bash
changed_when: true
- name: Restart Backrest container for {{ service_name }}
become: true
delegate_to: "{{ backrest_host }}"
community.docker.docker_container:
name: backrest
state: started
restart: true
Stuff in {{ }} are variabled defined elsewhere.
generic_backup/templates/repo_fragment.json.j2
Note: templates are Jinja.
{
"id": "{{ repo_id }}",
"uri": "{{ repo_uri }}",
"guid": "{{ ( inventory_hostname + service_name + repo_uri) | hash('sha256') }}",
"password": "{{ restic_repo_password }}",
{% if repo_type == 'sftp' %}
"flags": ["--option=sftp.args='-oBatchMode=yes -p {{ restic_port | default('22') }}'"],
{% elif repo_type == 's3' %}
"env": [
"AWS_ACCESS_KEY_ID={{ backblaze_restic_key_id }}",
"AWS_SECRET_ACCESS_KEY={{ backblaze_restic_api_key }}"
],
{% endif %}
"prunePolicy": {
"schedule": { "disabled": true, "clock": "CLOCK_LAST_RUN_TIME" },
"maxUnusedPercent": 10
},
"checkPolicy": {
"schedule": { "cron": "0 0 1 * *", "clock": "CLOCK_LAST_RUN_TIME" }
},
"commandPrefix": {},
"forgetPolicy": {
"schedule": { "disabled": true, "clock": "CLOCK_LAST_RUN_TIME" },
"retention": {
"policyTimeBucketed": { "hourly": 24, "daily": 30, "monthly": 12 }
}
}
}
generic_backup/templates/restic_backup.service.j2
[Unit]
Description=Restic backup for {{ service_name }}
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/restic-backup-{{ service_name }}.sh
# Low priority to avoid impacting service performance
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
User=root
Environment=LANG=en_US.UTF-8
Environment=LC_ALL=en_US.UTF-8
generic_backup/templates/restic_backup.sh.j2
#!/bin/bash
set -o pipefail
export LANG=en_US.UTF-8
export LANGUAGE=en_US:en
export LC_ALL=en_US.UTF-8
# Define constants
SERVICE_NAME="{{ service_name }}"
CACHE_DIR="/var/cache/restic/${SERVICE_NAME}"
SENTINEL_FILE="${CACHE_DIR}/last_run_failed"
LOG_FILE="{{ restic_log_dir }}/{{ service_name }}.log"
export RESTIC_PASSWORD_FILE="/etc/restic/${SERVICE_NAME}.pass"
export RESTIC_CACHE_DIR="${CACHE_DIR}"
# --- Create needed directories and files
mkdir -p "${CACHE_DIR}"
mkdir -p "{{ restic_log_dir }}"
[ ! -f "$LOG_FILE" ] && touch "$LOG_FILE" && chmod 0644 "$LOG_FILE"
# --- Helper Functions ---
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
send_alert() {
local REASON="$1"
log "ALERT: $REASON"
if [ ! -f "${SENTINEL_FILE}" ]; then
echo -e "Subject: Restic Backup Failed: ${SERVICE_NAME}\n\nBackup failed at $(date).\nReason: ${REASON}" | msmtp {{ email_login }}
touch "${SENTINEL_FILE}"
fi
}
run_restic() {
# $1 = command (e.g., backup)
# $2 = repository (e.g., sftp:user@host:/path or s3:...)
# ${@:3} = remaining arguments (e.g., source_dir)
local CMD="$1"
local REPO="$2"
shift 2
local CMD_OUT
log "Running restic $CMD on repo ${REPO}..."
CMD_OUT=$(/usr/local/bin/restic -r "$REPO" "$CMD" "$@" 2>&1)
local EXIT_CODE=$?
# Exit codes 0 and 3 are acceptable for "successful enough" backups
if [ $EXIT_CODE -eq 0 ] || [ $EXIT_CODE -eq 3 ]; then
log "Restic $CMD completed successfully (Exit $EXIT_CODE)."
else
send_alert "Restic $CMD failed with critical error (Exit $EXIT_CODE): ${CMD_OUT}"
exit $EXIT_CODE
fi
log "Restic $CMD successful."
}
cleanup() {
{% if db_name is defined and db_name | length > 0 %}
# Clean up DB dump
[ -f "$DUMP_FILE" ] && rm -f "$DUMP_FILE"
{% endif %}
# Clean up credentials
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
}
trap cleanup EXIT
# --- Database Logic ---
{% if db_name is defined and db_name | length > 0 %}
DUMP_FILE="{{ source_dir }}/{{ service_name }}_db_dump.sql"
DB_HOST="{{ db_host | default(postgresql_host | default('localhost')) }}"
DB_PORT="{{ db_port | default(postgresql_port | default('5432')) }}"
DB_USER="{{ db_user | default(postgresql_user) }}"
DB_PASS="{{ db_password | default(postgresql_password) }}"
log "Starting database dump for {{ db_name }} to $DUMP_FILE..."
export PGPASSWORD="${DB_PASS}"
{% if db_name == 'all' %}
ERROR_MSG=$(pg_dumpall -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" > "$DUMP_FILE" 2>> "$LOG_FILE")
{% else %}
ERROR_MSG=$(pg_dump -Fc -h "${DB_HOST}" -p "${DB_PORT}" -U "${DB_USER}" {{ db_name }} > "$DUMP_FILE" 2>> "$LOG_FILE")
{% endif %}
if [ $? -ne 0 ]; then
send_alert "Database dump failed: $ERROR_MSG"
exit 1
fi
log "Database dump created successfully."
log "Validating dump integrity..."
if [ ! -s "$DUMP_FILE" ]; then
send_alert "Dump file is empty."
exit 1
fi
{% if db_name == 'all' %}
# Verify the SQL file is complete
if ! tail -n 5 "$DUMP_FILE" | grep -qEi "PostgreSQL.*database.*cluster.*dump.*complete"; then
log "Debug: Last 5 lines of dump:"
tail -n 5 "$DUMP_FILE" >> "$LOG_FILE"
send_alert "Dump integrity check FAILED: 'dump complete' marker missing or malformed."
exit 1
fi
{% else %}
# Validate binary dump integrity
ERROR_MSG=$(pg_restore -l "$DUMP_FILE" 2>&1 > /dev/null)
if [ $? -ne 0 ]; then
send_alert "Invalid DB dump file format: ${ERROR_MSG}"
exit 1
fi
{% endif %}
{% endif %}
# --- Execution ---
# 1. Run Backup (Primary)
log "Starting primary backup of {{ source_dir }}..."
run_restic "backup" "{{ restic_repo }}" "{{ source_dir }}"
# 2. Run Backup (Secondary - Backblaze B2)
# Ensure these environment variables are set either here or in your system environment
export AWS_ACCESS_KEY_ID="{{ backblaze_restic_key_id }}"
export AWS_SECRET_ACCESS_KEY="{{ backblaze_restic_api_key }}"
log "Starting secondary backup to Backblaze..."
# Note: We use a different repository path/variables here
run_restic "backup" "{{ remote_restic_repo }}" "{{ source_dir }}"
if [ $? -ne 0 ]; then
send_alert "Secondary backup to Backblaze failed."
fi
# 3. If we get here, the backup succeeded
rm -f "${SENTINEL_FILE}"
# 4. Maintenance & Integrity
log "Backup successful. Starting maintenance..."
# Maintenance on Primary
run_restic "forget" "{{ restic_repo }}" --keep-last 7 --prune
run_restic "check" "{{ restic_repo }}"
# Maintenance on Secondary (Backblaze)
# Note: Ensure the environment variables above are still set for this!
run_restic "forget" "{{ remote_restic_repo }}" --keep-last 7 --prune
run_restic "check" "{{ remote_restic_repo }}"
log "Backup cycle completed successfully."
generic_backup/templates/restic-backup.timer.j2
[Unit]
Description=Timer for {{ service_name }} backup
[Timer]
# Use the passed variable, default to 03:00:00 if not provided
OnCalendar={{ backup_schedule | default('*-*-* 03:00:00') }}
Persistent=true
# Randomly delay start by up to 15 minutes to stagger traffic
RandomizedDelaySec=900
[Install]
WantedBy=timers.target
In roles that deploy a service that I want to have backed up, in the <rolename>/meta/main.yml add
dependencies:
- role: generic-backup
vars:
service_name: "openhab"
source_dir: "{{ openhab_home }}"
backup_schedule: "*-*-* 01:00:00"
If the backup failed, I what to know why it failed and figure out if there is a way I can avoid it in the future. I don’t just want to blindly re-run it and frankly doing so may destroy the evidence I need to diagnose the original problem.
That’s the same reason I don’t automatically restart stuff.
If I’m in a hurry, I can ssh to a machine and run a script, or bring up Semaphore and run the needed Ansible role(s) to get everything back up and returned to a known good state. Both can be done from my phone and with tailscale both can be done from anywhere in the world (ish, I do use geoblocking on my firewall which I need to disable when I go overseas).
By focusing on diagnosing and mitigating problems as they occur the number of times I need to do either of the above is about once a quarter, maybe less. It’s just not worth the extra complexity to add a button to the notification when I wouldn’t push the button anyway.
I’m not arguing against the feature over all, it’s just not for me. A convenience that I’d never use is no convenience at all.
Healthchecks is just a convenient way to get an alert when a backup completelty failed to run at all. If it ran or failed I already get an alert.
Just to be clear, while I do have two open ports (port 80 and port 443), the only thing on those ports are a few services that I’ve found no way to make accessible to my wife’s students (Nextcloud shares) and my inlaws (Vaultwarden) from the internet. I cannot require my wife to install Tailscale on each of her student’s machines and asking my inlaws to do anything on the computer is a recipie for disaster.
But if it were not for these I’d not have any open ports. Yet I would still keep my leased DNS name and use my reverse proxy. Why? Because with a DNS from a service that has a plug-in to ACME I can get a trusted certificate for *.mydomain.com. With that certificate, internal to my LAN, my family can go to https://openhab.mydomain.com and not only does it resolve but the certificate is trusted so I don’t have to mess with setting up my own CA. The DNS server is set up so even those DNS requests do not leave the LAN.
I always approach everything as local first, cloud only if required. If I go down the API route, it will be with LLMs. Only if I exhaust what they can do will I consider a subscription to Claude or one of the others.
