[Solved] Something completely killed my OH3.2

after migrating from textual config to UI based config yesterday, everything runs just fine after restart.

after that I tried to convert my first rule from rulesDSL to JSRule today…

to test the functionality of my new rule, I’ve deactivated the DSL rule, and saved the new rule in UI.

then I restarted openHAB to see if the rule is working like expected, but to my surprice openhab is starting up without logging anything, no signs of life in android app, webUI…

systemctl status openhab returns

● openhab.service - openHAB - empowering the smart home
   Loaded: loaded (/lib/systemd/system/openhab.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-03-12 10:12:04 CET; 13min ago
     Docs: https://www.openhab.org/docs/
           https://community.openhab.org
 Main PID: 77 (java)
    Tasks: 32 (limit: 4915)
   Memory: 806.8M
   CGroup: /system.slice/openhab.service
           └─77 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab

Mar 12 10:12:12 openhab3 karaf[77]:         at org.eclipse.osgi.storage.Storage.saveGenerations(Storage.java:1389)
Mar 12 10:12:12 openhab3 karaf[77]:         at org.eclipse.osgi.storage.Storage.save0(Storage.java:1339)
Mar 12 10:12:12 openhab3 karaf[77]:         at org.eclipse.osgi.storage.Storage.save(Storage.java:1308)
Mar 12 10:12:12 openhab3 karaf[77]:         at org.eclipse.osgi.internal.framework.StorageSaver$StorageSaverTask.run(StorageSaver.java:31)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar 12 10:12:12 openhab3 karaf[77]:         at java.base/java.lang.Thread.run(Thread.java:829)
lines 1-21/21 (END)

any idea, how to revert this (deleting the rule from db or something?!?)

I would have posted the ?malicious? JSRule, but I think while OH not running there’s no chance to see it without webUI :wink:

seems like karaf is also dead…

ssh -p 8101 openhab@localhost returns

ssh: connect to host localhost port 8101: Connection refused

The UI defined things/items/rules/… are stored in jsondb located in /var/lib/openhab/jsondb/.
Backup files of these are stored in /var/lib/openhab/jsondb/backup/.
You should be able to use these files to go back to a state before doing your change.

1 Like

Thanks a lot Wolfgang! I’ll try and report back if I could encourage OH to start normally again :wink:

The top part of the exception is cut off but it looks like it’s having some sort of file access problem. And it’s coming from karaf so that means it’s probably not OH specific and more fundamental. A rule or corrupt JSONDB file can’t prevent karaf from coming up all by itself.

How did you do the restart?

Are you running openHABian?

I don’t think it’s the rule that’s the problem but something weird has gone wrong with the files.

I’ve seen similar behavior in the past when the logs folder had permissions that wouldn’t allow the openhab user to write, for example.

A full file system could cause a problem like this too.

When there is a corrupt jsondb file, usually that won’t stop OH from starting so I’m really skeptical that your rule caused the problem and it was the restart/reboot or something else external.

@Wolfgang_S points to the backup of the jsondb files but also look at file and directiry permissions and look for oddness in other config files in userdata/etc and the like.

If you can, try to start OH using the start.sh script or use journalctl to see the to of that exception which might tell us more.

1 Like

:see_no_evil: :see_no_evil: :see_no_evil:
yes that’s it… even with logrotate something spammed my hdd on the server.

I found the error 2 hours ago… for the moment I deleted 2 unused lxc-containers that gave me about 18GB so I think I’m fine for now…

I realized this because I had a look at the logs and wondered why there’s data missing from the last hours.

but funny that it occured while creating the new rule - I restarted openhab 2 hours before without any error. In this small window of time this happened to the hdd…

It runs on a NUC with 100GB ssd & 8GB ram

…I was totally shocked for a moment… you know I changed my config in general just yesterday :wink:

by the way… this is the rule where I was a little unsure with the syntax…

const display = items.getItem("DisplayState");
const harmony = items.getItem("HQHub_CurrentActivity");

rules.JSRule({
  name: "DisplayState",
  description: "see if TV is turned on or off",
  triggers: [triggers.ChannelEventTrigger('harmonyhub:hub:HarmonyHub:activityStarted'),triggers.ItemCommandTrigger('LightScene')],
  execute: data => {
    console.log("Display State", "Display State: "+display.state);
    if(harmony.state == 'PowerOff' || harmony.state == 'VinylPlay' || harmony.state == 'GuitarAmp' || harmony.state == 'Squeeze' || harmony.state == 'CDPlayer'){
      if( display.state !== "OFF" ){
        display.sendCommand("OFF");
        console.log("Display State", "Switching Display State "+display.state);
      }
    }
    if(harmony.state == 'Fernsehen' || harmony.state == 'ShieldTV'){
      if( display.state !== "ON" ){
        display.sendCommand("ON");
        console.log("Display State", "Switching Display State "+display.state);
      }
    }
  },
  tags: ["Display"],
  id: "DisplayState"
});

…just to be complete :see_no_evil:

I’m glad it wasn’t OH that caused the problem. You’d have uncovered a completely new bug if that were the case. I’m glad you’re up and running again!

1 Like

Indeed, that’s right :grinning: :wink:

I created a little bash script to remind me if the filesystem runs out of space…

I have set the limit to 75% to test the script… normally it is set to 90%…

Here is the script

#!/bin/bash
EXCLUDE_LIST="<define excluded directories here>|<seperated with | >"
df -H | grep -vE "^Filesystem|tmpfs|cdrom|${EXCLUDE_LIST}" | awk '{print $5 " " $6}' | while read output;
do
  echo $output
  usage=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usage -ge 90 ]; then
    printf "Subject: <your subject here>\n\n$(hostname) is running out of space!\n\nThe actual disk-usage on <your partition name> partition is $usage%%\n\nPlease consider freeing up some diskspace!\n\n$(date)" |
    msmtp -d <me@my mailaddress.com>
  fi
done

open /etc/cron.daily/ and save a new file with the name memwatchdog.sh:

cd /etc/cron.daily/ && sudo nano memwatchdog.sh

This will open an empty file, paste the script here and edit it!

You can easily customize the script to your needs (edit subject and body)
note:

if [ $usage -ge 90 ]; then

The “90” is the percent of diskusage to start sending alert mails…

After that you will have to setup linux to send mails over smtp, you need your smtp server credentials…

Install mailservice and editor from linux console:

sudo apt install msmtp msmtp-mta mailutils nano

setup msmtp mail:

sudo nano /etc/msmtprc

This will open an empty file (if not confiured)
paste the following and change the settings with your smtp credentials:

# Set default values for all following accounts.
defaults
# Use the mail submission port 587 instead of the SMTP port 25.
port 587
# Always use TLS.
tls on
# Set a list of trusted CAs for TLS. The default is to use system settings, but
# you can select your own file.
tls_trust_file /etc/ssl/certs/ca-certificates.crt
# The SMTP server of your ISP
account <your isp name>
host <your smtp server address e.g. mail.myisp.com>
from <the mailaddress to send mails from e.g. me@mymail.com>
auth on
user <your username>
password <your password>
# Set default account to your isp
account default: <your isp name (must be the same as account name above)>
# Map local users to mail addresses
aliases /etc/aliases

In order to be able to use the mail command you need to install mailx additionally:

sudo apt install bsd-mailx

After that configure mailx to use msmtp

sudo nano /etc/mail.rc

scroll down to the last line and append this:

set mta=/usr/bin/msmtp

now you should be able to send mail from your headless linux system.

Test your configuration with something like:

echo "Subject:Hello World/nThis is my first mail send from my server" | msmtp -d <your email address e.g. me@mymail.com>

or with mail instead of msmtp

echo "This is my first mail send from my server" | mail -s "Hello World" <your email address e.g. me@mymail.com> 

Now open crontab

sudo crontab -e

to automatically start the script add this line at the end:

0 8 * * * /etc/cron.daily/memwatchdog.sh

this will start the script at 8 a.m. every day…

have fun!