Grafana doesn't start after reboot in fresh OpenHabian installation due to /var/log permissions

In installed InfluxDB+Grafana from the OpenHabian-config tool, which seemed to work fine. Both the DB and grafana were working great. I just rebooted for the first time since then and Grafana failed to start.

sudo journalctl | grep grafana made it pretty obvious what the problem was… missing permissions to write log files:
Failed to start grafana. error: failed to create log directory "/var/log/grafana": mkdir /var/log/grafana: permission denied

I fixed it with chmod ugo+rwx /var/log. This was probably more permissions that needed, but it needed it working and it’s working. If there is more appropriate set of permissions, someone please do tell and I will change it. Then sudo systemctl start grafana-server successfully started the service.

So, bug in the installer or mystery malfunction?

This is how the permisions should look like:

drwxr-xr-x 2 grafana           grafana        4096 Jan  8 09:17 grafana

Directory here was created during the installation.

The postinstall script of the deb package contains following lines:
|

postinst:# Set user permissions on /var/log/grafana, /var/lib/grafana
postinst:mkdir -p /var/log/grafana /var/lib/grafana
postinst:chown -R $GRAFANA_USER:$GRAFANA_GROUP /var/log/grafana /var/lib/grafana
postinst:chmod 755 /var/log/grafana /var/lib/grafana

In case during installation there was no error and grafana was started during the installation I would assume that ZRAM was active during the install and the new created directory was not synced back to the SD card.

After making this post, I found NGINX also didn’t start. The /var/log/nginx directory and log file was gone. So this further eludes to the directories and everything having existed and then got deleted. I don’t know anything about zram but it sounds like that’s what happened.

Un- and reinstall ZRAM from the menu then try again.

1 Like

I found the same thing happened when I rebooted again. The NGINX and Grafana log directories were gone and permissions gone.

I uninstalled zram from the menu. Ran the installer for InfluxDB+Grafana again without zram. Everything is fine now and worked upon rebooting.

I’m not familiar with zram. What are the benefits of reinstalling it? I’m concerned it will create more problems if there’s something wrong with it. Is leaving it off detrimental to anything?

Yes, see Corrupt FileSystems every 2-3 month? - Setup, Configuration and Use / Beginners - openHAB Community

Alright, zram it is. Should I expect to see this kind of problem with zram happening over and over then? Or should I expect reinstalling zram will fix whatever was wrong with it before?

the latter

OK. Would power loss vs a soft reboot command be a culprit here?

I would cause data loss yes but nothing dramatic, worst case your historic measurements are lost but OH should continue to work.

Same problem here after a powerloss.
Grafana doesn’t start.

chown -R grafana:grafana /var/log/grafana /var/lib/grafana

solves it.

EDIT
Redid a powerloss. The /var/log/grafana directory actually disappears.

sudo mkdir -p /var/log/grafana /var/lib/grafana
sudo chown -R grafana:grafana /var/log/grafana /var/lib/grafana

These two lines are needed to restore grafana. I didn’t touch anything related to zram.
Hope that helps.

BUT, does it stay that way after the next power loss? Prior to uninstalling and reinstalling zram, the entire directories and all permissions associated with them were lost when power was dropped. I would recreate the directories and fix the permissions. Then on power loss, it would all go away again. Like nothing was ever leaving the zram.

I haven’t tried cutting the power since uninstalling/reinstalling zram.

Had the same Issue and it took me the last 2 days to get it fixed. After I got Grafana running again, I had to reinstall my Deconz-Binding to which made me restart my RPi several times.

Just to find out, my Grafana wasn’t working again.

So it’s not just power loss but seems to be on every reboot, which is kind of annoying (and I do not remember having to do this on any previous reboots).

Is there no way around?

Hello,

Same problem here.
Grafana and Nginx log directories are deleted on reboot of the RPI.
Manually adding the directories solves it, but is not a permanent fix.

I still have no idea what is causing this or how to solve it.

Any suggestions are welcome.

In case you use ZRAM did you try to stop/deinstall ZRAM; create the Grafana, Nginx log directory and install ZRAM again ?

I searched for zram, but couldn’t find it, so I guess this isn’t installed.
I use Openhab RPI Distro v1.6.2.

In case you do not have enabled ZRAM using openhabian-config it is not used.
Please check if you use a tmpfs ( especially for the /var/log directory ).

mount | grep tmpfs

I’m not using zram, doubt if I should. But thats for an other time.

Regarding this issue.
I reïnstalled the system with the latest RPI Distro 1.6.3.
This resolved the issue for me.

Thanks.