[SOLVED] Influx doesn't start after openhabian-config as it wants to start as root

I lately installed influxdb via openhabian-config which ran for sometime.
Then at a certain point in time I had to restart my PI and I had to manually recreate the database and then it worked fine until a few days ago but then even with all the things I tried (even uninstalling and reinstalling it and now since today being on the 3.4 release) there is no way I am able to get it to run again.
(Michael: I followed you advice, read all the docs thoroughly and put the logging to debugmode=maximum which is how I got the messages below)

Note that I just did a new from scratch install via openhabian-config

runtimeInfo:
  version: 3.4.0.RC1
  buildString: Milestone Build
locale: en-DE
systemInfo:
  configFolder: /etc/openhab
  userdataFolder: /var/lib/openhab
  logFolder: /var/log/openhab
  javaVersion: 11.0.16
  javaVendor: Raspbian
  osName: Linux
  osVersion: 5.10.103-v7l+
  osArchitecture: arm
  availableProcessors: 4
  freeMemory: 264709600
  totalMemory: 559939584
  startLevel: 100
Dec 24 14:00:56 ourhome systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Dec 24 14:00:56 ourhome influxd-systemd-start.sh[19335]: Merging with configuration at: /etc/influxdb/influxdb.conf
Dec 24 14:00:56 ourhome influxd-systemd-start.sh[19335]: ts=2022-12-24T13:00:56.988806Z lvl=info msg="InfluxDB starting" log_id=0ey3P770000 version=1.8.10 branch=1.8 com
Dec 24 14:00:56 ourhome influxd-systemd-start.sh[19335]: ts=2022-12-24T13:00:56.989009Z lvl=info msg="Go runtime" log_id=0ey3P770000 version=go1.13.8 maxprocs=4
Dec 24 14:00:56 ourhome influxd-systemd-start.sh[19335]: run: create server: mkdir all: mkdir /root/.influxdb: permission denied
Dec 24 14:00:57 ourhome influxd-systemd-start.sh[19335]: Merging with configuration at: /etc/influxdb/influxdb.conf

This one is striking

run: create server: mkdir all: mkdir /root/.influxdb: permission denied

though I don’t know why it wants to use the root-user
even though in /etc/influxdb/influxdb.conf the user is set to influxdb

# If you modify this, please also make sure to edit init.sh

[Unit]
Description=InfluxDB is an open-source, distributed, time series database
Documentation=https://docs.influxdata.com/influxdb/
After=network-online.target

[Service]
User=influxdb
Group=influxdb
LimitNOFILE=65536
EnvironmentFile=-/etc/default/influxdb
ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh
KillMode=control-group
Restart=on-failure
Type=forking
PIDFile=/var/lib/influxdb/influxd.pid
[Install]
WantedBy=multi-user.target
Alias=influxd.service

Does anyone have an idea how to track down this issue or what the root cause could be? (I already looked around in several influxdb forums and also checked the threads here but couldn’t find anything that hints to that).

Did you entered any settings in /etc/default/influxdb that change the default behavior ?
The messages in your log seem to be generated by the influxd binary being called in the very first line of
/usr/lib/influxdb/scripts/influxd-systemd-start.sh.

The code you posted does not show the content ( as described

) of /etc/influxdb/influxdb.conf but the content of /etc/systemd/system/multi-user.target.wants/influxdb.service.
In case your /etc/influxdb/influxdb.conf really contains what you posted that might be the problem.

No I didn’t.

You are right, not sure how that happened :worried:

Here are contents of the two files:

/etc/systemd/system/multi-user.target.wants/influxdb.service

# If you modify this, please also make sure to edit init.sh

[Unit]
Description=InfluxDB is an open-source, distributed, time series database
Documentation=https://docs.influxdata.com/influxdb/
After=network-online.target

[Service]
User=influxdb
Group=influxdb
LimitNOFILE=65536
EnvironmentFile=-/etc/default/influxdb
ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh
KillMode=control-group
Restart=on-failure
Type=forking
PIDFile=/var/lib/influxdb/influxd.pid

[Install]
WantedBy=multi-user.target
Alias=influxd.service

and here is the content of /etc/influxdb/influxdb.conf

reporting-disabled = false
bind-address = "127.0.0.1:8088"

[meta]
  dir = "/root/.influxdb/meta"
  retention-autocreate = true
  logging-enabled = true

[data]
  dir = "/root/.influxdb/data"
  index-version = "inmem"
  wal-dir = "/root/.influxdb/wal"
  wal-fsync-delay = "0s"
  validate-keys = false
  strict-error-handling = false
  query-log-enabled = true
  cache-max-memory-size = 1073741824
  cache-snapshot-memory-size = 26214400
  cache-snapshot-write-cold-duration = "10m0s"
  compact-full-write-cold-duration = "4h0m0s"
  compact-throughput = 50331648
  compact-throughput-burst = 50331648
  max-series-per-database = 1000000
  max-values-per-tag = 100000
  max-concurrent-compactions = 0
  max-index-log-file-size = 1048576
  series-id-set-cache-size = 100
  series-file-max-concurrent-snapshot-compactions = 0
  trace-logging-enabled = false
  tsm-use-madv-willneed = false

[coordinator]
  write-timeout = "10s"
  max-concurrent-queries = 0
  query-timeout = "0s"
  log-queries-after = "0s"
  max-select-point = 0
  max-select-series = 0
  max-select-buckets = 0

[retention]
  enabled = true
  check-interval = "30m0s"

[shard-precreation]
  enabled = true
  check-interval = "10m0s"
  advance-period = "30m0s"

[monitor]
  store-enabled = true
  store-database = "_internal"
  store-interval = "10s"

[subscriber]
  enabled = true
  http-timeout = "30s"
  insecure-skip-verify = false
  ca-certs = ""
  write-concurrency = 40
  write-buffer-size = 1000

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  suppress-write-log = false
  write-tracing = false
  flux-enabled = false
  flux-log-enabled = false
  pprof-enabled = false
  pprof-auth-enabled = false
  debug-pprof-enabled = false
  ping-auth-enabled = false
  prom-read-auth-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
  https-private-key = ""
  max-row-limit = 10000
  max-connection-limit = 0
  shared-secret = ""
  realm = "InfluxDB"
  unix-socket-enabled = false
  unix-socket-permissions = "0777"
  bind-socket = "/var/run/influxdb.sock"
  max-body-size = 25000000
  access-log-path = ""
  max-concurrent-write-limit = 0
  max-enqueued-write-limit = 0
  enqueued-write-timeout = 30000000000

[logging]
  format = "auto"
  level = "info"
  suppress-logo = false

[[graphite]]
  enabled = false
  bind-address = ":2003"
  database = "graphite"
  retention-policy = ""
  protocol = "tcp"
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "1s"
  consistency-level = "one"
  separator = "."
  udp-read-buffer = 0

[[collectd]]
  enabled = false
  bind-address = ":25826"
  database = "collectd"
  retention-policy = ""
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "10s"
  read-buffer = 0
  typesdb = "/usr/share/collectd/types.db"
  security-level = "none"
  auth-file = "/etc/collectd/auth_file"
  parse-multivalue-plugin = "split"

[[opentsdb]]
  enabled = false
  bind-address = ":4242"
  database = "opentsdb"
  retention-policy = ""
  consistency-level = "one"
  tls-enabled = false
  certificate = "/etc/ssl/influxdb.pem"
  batch-size = 1000
  batch-pending = 5
  batch-timeout = "1s"
  log-point-errors = true

[[udp]]
  enabled = false
  bind-address = ":8089"
  database = "udp"
  retention-policy = ""
  batch-size = 5000
  batch-pending = 10
  read-buffer = 0
  batch-timeout = "1s"
  precision = ""

[continuous_queries]
  log-enabled = true
  enabled = true
  query-stats-enabled = false
  run-interval = "1s"

[tls]
  min-version = ""
  max-version = ""

and you are right, it points to

/root/.influxdb/meta

Where would be the right location to point it to?

This is the content of my influxdb.conf file.
influxdb.conf.txt (21.1 KB)

1 Like

:pray: @Wolfgang_S
That seems to do the trick. I just used your config and started up. I will now install the persistence addon again and see it if works in openHAB.

Though openHAB still has an “authentication problem”. I am restarting OH now.

Unfortunately now I get:

==> /var/log/openhab/openhab.log <==

2022-12-24 17:24:27.375 [ERROR] [org.influxdb.impl.BatchProcessor    ] - Batch could not be sent. Data will be lost

org.influxdb.InfluxDBException$AuthorizationFailedException: authorization failed

even though the right credentials are in /etc/openhab/services/influxdb.cfg

and the exception appears as soon as I installed the persistence binding (without activating it as default persistence)

Is it possible to connect using the command line client?

Yes, though I cannot connect to the database that should have been created. I am gönne try create oh db tomorrow manually as it seems not to be existing. What is weird that cannot even do an execute show database

Whatever I do I get

I start it

influx -username admin -password mypasswd

Connected to http://localhost:8086 version 1.8.10
InfluxDB shell version: 1.8.10
> show databases
ERR: authorization failed
Warning: It is possible this error is due to not setting a database.
Please set a database with the command "use <database>".
> show users
ERR: authorization failed
Warning: It is possible this error is due to not setting a database.
Please set a database with the command "use <database>".

Now I changed the config-file you gave me (located at /etc/influxdb/influxdb.conf) and

uncommented the bind to be

bind-address = “127.0.0.1:8088”

commented auth-enabled!

 # Determines whether user authentication is enabled over HTTP/HTTPS.
 # auth-enabled = true

At least I can now login and see the database

Connected to http://localhost:8086 version 1.8.10
InfluxDB shell version: 1.8.10
> show users
user    admin
----    -----
admin   true
openhab false
grafana false

Now everything works again.

1 Like

Hi Wolfgang,

and the story goes on. Similar like what happened before I started this thread that influxdb stopped working happened this morning. What is even weirder is that I have two openHAB instances running (dev and prod) that have both their own influxdb services running and both(!) stopped working this morning. I have no clue what the cause was but it happened. Now I cannot get them to run anymore.

When I start it with

sudo systemctl start influxdb

I get

Job for influxdb.service failed because the control process exited with error code.
See "systemctl status influxdb.service" and "journalctl -xe" for details.

journalctl --unit=influxdb -n 100 --no-pager

hardly tells anything

Jan 07 10:14:32 openhabian influxd-systemd-start.sh[31117]: /usr/lib/influxdb/scripts/influxd-systemd-start.sh: line 5: /var/lib/influxdb/influxd.pid: Permission denied
Jan 07 10:14:32 openhabian systemd[1]: influxdb.service: Control process exited, code=exited, status=1/FAILURE
Jan 07 10:14:32 openhabian systemd[1]: influxdb.service: Failed with result 'exit-code'.
Jan 07 10:14:32 openhabian systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
Jan 07 10:14:32 openhabian systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 5.
Jan 07 10:14:32 openhabian systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Jan 07 10:14:32 openhabian systemd[1]: influxdb.service: Start request repeated too quickly.
Jan 07 10:14:32 openhabian systemd[1]: influxdb.service: Failed with result 'exit-code'.
Jan 07 10:14:32 openhabian systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.

except permission denied of /var/lib/influxdb/influxd.pid

ls -l /var/lib/influxdb

says

drwxr-xr-x 2 root root 4096 Jan 7 09:56 data
drwxr-xr-x 2 root root 4096 Jan 7 09:56 meta

less /var/log/influxdb/influxdb.log
/var/log/influxdb/influxdb.log: No such file or directory

mount | grep influxdb
/dev/mmcblk0p2 on /opt/zram/influxdb.bind type ext4 (rw,noatime)
overlay2 on /var/lib/influxdb type overlay (rw,relatime,lowerdir=/opt/zram/influxdb.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on)

I deactivated zram, reinstalled influx, it did’t start, reactivated zram, it is still not starting…

sudo chown -R influxdb:influxdb /var/lib/influxdb/*

after doing all these things I ended up with the following

influxd

2023-01-07T10:04:54.600923Z	info	InfluxDB starting	{"log_id": "0fEvtZn0000", "version": "1.8.10", "branch": "1.8", "commit": "688e697c51fd"}
2023-01-07T10:04:54.601032Z	info	Go runtime	{"log_id": "0fEvtZn0000", "version": "go1.13.8", "maxprocs": 4}
/var/lib/influxdb/meta/meta.dbtmp: permission denied

and

sudo systemctl restart influxdb
sudo journalctl -u influxdb.service

Jan 07 11:08:12 openhabian systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Jan 07 11:08:12 openhabian influxd-systemd-start.sh[39012]: /usr/lib/influxdb/scripts/influxd-systemd-start.sh: line 5: /var/lib/influxdb/influxd.pid: Permission denied

UPDATE - SOLUTION

Based on what I found here InfluxDB doesn't work after reboot, problem with rights - #5 by ostap

what did help was

sudo chown -R 1000:1000 /var/lib/influxdb 
sudo chown -R influxdb:influxdb /var/lib/influxdb

sudo systemctl daemon-reload
sudo systemctl enable influxdb.service
sudo systemctl start influxdb.service

Though what I still don’t understand why it happened in the first place and not for the first time? :worried:

1 Like

I - as far as I remember - only saw this when I was installing a new service while zram was enabled.
My explanation at that time was - might not be correct - that the new directories that were created while installing the service did not get the correct permissions in the underlying file system. After a reboot the service was not starting. Only stopping / disabling zram and installing the service again helped at that time. I did not look into the details at that time it might have worked to change the ownership/permissions in the underlying file system.

1 Like

I can actually second this. This is what mstormi explained my once. During installing zram it does “remember” its permissions for the folders. Folders under control of zram keep their permission information like it was during the that installation which is then used on reboot. Hence, as Wolfgang said, if you install something new that persists in an area which lies in zram, the only way I know is to first uninstall zram, install the service and then reinstall zram.

Why both services failed at the same time probably falls under the category “strange” :wink:

1 Like