Grafana not available after OH2 update to the latest components

wiewior · November 11, 2022, 12:18pm

Hello,
on the OH2 Release 2.5.12 I made RaspberryPi system update. After that Grafana is not starting anymore. I tried to restart grafana manually and made few hard resets as well. Nothing helped.
My lovely OH2 system is not showing my graphs anymore displaying the ERR_CONNECTION_REFUSED in the browser

How can I resolve it?

● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2022-11-11 13:06:17 CET; 5min ago
     Docs: http://docs.grafana.org
  Process: 642 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=deb cfg:default.paths.logs=${LOG_DIR} c
 Main PID: 642 (code=exited, status=1/FAILURE)

Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Unit entered failed state.
Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Nov 11 13:06:17 OH2 systemd[1]: Stopped Grafana instance.
Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Start request repeated too quickly.
Nov 11 13:06:17 OH2 systemd[1]: Failed to start Grafana instance.
Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Unit entered failed state.
Nov 11 13:06:17 OH2 systemd[1]: grafana-server.service: Failed with result 'exit-code'.

sudo journalctl | grep grafana shows

Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:24] Unknown lvalue 'LockPersonality' in section 'Service'
Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:29] Unknown lvalue 'ProtectClock' in section 'Service'
Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:32] Unknown lvalue 'ProtectHostname' in section 'Service'
Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:33] Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:36] Unknown lvalue 'ProtectProc' in section 'Service'
Nov 11 13:06:04 OH2 systemd[1]: [/usr/lib/systemd/system/grafana-server.service:42] Unknown lvalue 'RestrictSUIDSGID' in section 'Service'
Nov 11 13:06:13 OH2 grafana-server[524]: /usr/sbin/grafana-server: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28' not found (required by /usr/sbin/grafana-server)

Wolfgang_S · November 11, 2022, 1:05pm

What exactly does

mean ?

You just did an update of all the OH / system components using apt upgrade ?
What OS do you use ?
Is it a version before Bullseye ? Which one ?
Which version of Grafana is installed ?
What it updated to a more recent one that may not be supported under your ‘old’ OS ?

wiewior · November 11, 2022, 1:17pm

I executed this update:

I assume grafana or other dependencies were updated what cause the issue

Which version of Grafana is installed ?

To be honest, I do not know. I do not know how to check the curretn version of grafana now

Wolfgang_S · November 11, 2022, 1:26pm

You can do:

dpkg -l | grep grafana

in a linux console/shell.

I assume that you are using Debian Stretch which uses glibc 2.24 while Debian Buster uses 2.28 and Bullseye uses 2.31.

Run following command in a shell to check which OS version you use: lsb_release -a

wiewior · November 11, 2022, 1:29pm

grafana 9.2.4

lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.13 (stretch)
Release:        9.13
Codename:       stretch

Means I need to revert to older version, right? How to do this without violating my grafana graphs and settings?

rlkoshak · November 11, 2022, 3:59pm

If you are going to keep running deprecated and no longer supported software like OH 2, please please don’t upgrade like this. It’s only a matter of time before something outside of your control will change and you’ll either be forced to upgrade or abandon the system entirely.

If you are on OH 2, you should never upgrade the system, OS, or anything else. You should also limit how much you change to your OH configs because each and every change could be the thing that breaks your system and there probably won’t be a fix for it.

Your problems are compounded by the fact that you are running on a no longer supported operating system to begin with. So any updates that do get installed are highly likely to break.

In this case, the problem has something to do with Grafana which to some extent makes it outside the scope of this forum. Please do check the Grafana support pages, forum, and github for similar problems running Graphana on an RPi.

Hopefully downgrading Grafana to a version that is supported on Stretch will work. If it does, don’t change anything. And consider what you will do when the system finally breaks due to things outside your control.

If it doesn’t, create a new SD card from a fresh openHABian image which still has support to install OH 2.5.12. You can restore a backup of your configs on to that and at least you’ll be running with a newer and supported operating system which should support newer Grafana versions.

wiewior · November 11, 2022, 5:32pm

Hello Rich,
this is very valuable hint. Thank you for this. My plan is to switch to Oh3. I was planing to do this after bringing some improvemetns. It seems this was/is not a good strategy. I made sd image and backup but if I revert to this one i will lose some small updates i made on my HAB Panel. This will be the last move if i am not going to fix it. I will try to revert to older version of the Grafana if I new how to do this without losing my settings.

Would you recomendat at this stage switching to Oh3? Would this fixt the problem with Grafana?

Wolfgang_S · November 11, 2022, 5:39pm

have a look into directory /var/cache/apt/archives if the older grafana version is still available on your SD card.

such things can happen if third party software/repositories are in use.

wiewior · November 11, 2022, 5:42pm

openhabian@OH2:/var/cache/apt/archives $ ls | grep grafana
grafana_6.2.5_armhf.deb
grafana_6.3.5_armhf.deb
grafana_6.4.4_armhf.deb
grafana_6.5.2_armhf.deb
grafana_6.6.0_armhf.deb
grafana_6.6.2_armhf.deb
grafana_7.0.3_armhf.deb
grafana_7.1.5_armhf.deb
grafana_7.3.1_armhf.deb
grafana_7.3.4_armhf.deb
grafana_7.4.3_armhf.deb
grafana_7.5.3_armhf.deb
grafana_8.1.1_armhf.deb
grafana_8.3.3_armhf.deb
grafana_9.2.4_armhf.deb

It seems there is sth there but I have no ifder how to revert to the older version.

rlkoshak · November 11, 2022, 5:51pm

Yes, OH 2 is over two years past it’s support date. Search the forum, there are some steps you need to take to migrate your HABPanels but they can be migrated from one OH 2 to another or from one OH 2 to an OH 3. But there are other breaking changes in rules and such you’ll have to adopt to.

No, for that you need to update the whole operating system, not just OH 3. Grafana’s ability to run has nothing to do with OH itself.

wiewior · November 11, 2022, 6:02pm

If I would make: sudo apt-get upgarade
and than update to OH3

would it work and fix my problem with grafana?

wiewior · November 11, 2022, 6:13pm

Reverting to older Grafana Version works! Uff!!!

sudo apt install grafana=8.4.7

I wll not touch it in the future and try to switch to OH3

rlkoshak · November 11, 2022, 6:13pm

No, that won’t upgrade the way you need.

openHABian is based on Debian Linux. Each version of Debian Linux has a name (taken from characters from the Toy Story movies). A version of Debian is only supported for a few years before support is dropped. The end of support means no more updates to the OS are offered. It’s end of life.
When you run apt-get upgrade it’s going to look for upgrades to that version of the OS, in your case stretch. Support for stretch ended in 2020 so there are no upgrades to be had.

While it’s theoretically possible to upgrade from stretch to buster and from there to bullseye (you can’t jump straight to `bullseye, that is highly recommended against. Each step takes hours to complete and a lot can go wrong. It takes less time in the long run and safer to just start over from scratch with a fresh SD card image.from openHABian and migrate your configs to that. Note that openHABian will still let you install and use OH 2.5. It’s the operating system that needs to be upgraded.

Or, see if you can go back to a pervious version of Grafana, and then don’t touch it, ever.

wiewior · November 11, 2022, 8:15pm

I understand precisely. Thank you.
I will be more careful with the update and read more before doing that. I am sure I can find some posts in this forum on the porting procedure.