InfluxDB losing data after reboot since migrating to SSD

Hello community,

I have a strange problem going on since a few days, let me first give some background:

I migrated my storage for my Raspberry Pi4b from a HDD to SSD, using raspiBackup. Actually it was no problem at all, made a backup from HDD, and restored it on SSD. On the first look, everything works fine (and OH runs fast as hell by the way).

But now I found out that influxdb is “loosing” it’s data after every reboot. Before reboot, graph is fine and populated with actual data, e.g. for my energy counter daily consumption:

2020-06-06 15_10_11-Zählerstand allgemein - Grafana

But if I reboot now, for today it shows “No data”, and if I look back 7 days, it last updated on 3rd of June (before switching to the SSD):

2020-06-06 15_27_22-Zählerstand allgemein - Grafana

Logs only show Info, no error related to InfluxDB:

2020-06-01 21:21:27.728 [INFO ] [el.core.internal.ModelRepositoryImpl] - Loading model 'influxdb.persist'

For reference, here are my configs for persistence:

influxdb.cfg

# The database URL, e.g. http://127.0.0.1:8086 or https://127.0.0.1:8084 .
# Defaults to: http://127.0.0.1:8086
url=http://127.0.0.1:8086

# The name of the database user, e.g. openhab.
# Defaults to: openhab
user=openhab

# The password of the database user.
password=somepasswort

# The name of the database, e.g. openhab.
# Defaults to: openhab
db=openhab_db

# The retention policy to be used, needs to configured in InfluxDB
# Till v0.13: 'default', since v1.0: 'autogen'
retentionPolicy=autogen

influxdb.persist

Strategies {
    everyMinute : "0 * * * * ?"
    everyHour   : "0 0 * * * ?"
    everyDay    : "0 0 0 * * ?"
}

Items {
    *   : strategy = everyChange, everyHour

}


mapdb.persist

Strategies {

}

Items {

   *: strategy = everyChange, restoreOnStartup, everyHour

}

From the logs, influxdb shows no errors, and I can see openhab writing to it:

Jun 06 16:25:16 openhab systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.343205Z lvl=info msg="InfluxDB starting" log_id=0NEWBf7l000 version=1.8.0 branch=1.8 commit=781490de48220d7695a05c29e5a36f550a4568f5
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.346023Z lvl=info msg="Go runtime" log_id=0NEWBf7l000 version=go1.13.8 maxprocs=4
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.460620Z lvl=info msg="Using data dir" log_id=0NEWBf7l000 service=store path=/var/lib/influxdb/data
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.460788Z lvl=info msg="Compaction settings" log_id=0NEWBf7l000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.462250Z lvl=info msg="Open store (start)" log_id=0NEWBf7l000 service=store trace_id=0NEWBfaW000 op_name=tsdb_open op_event=start
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.539891Z lvl=info msg="Opened file" log_id=0NEWBf7l000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/openhab_db/autogen/1/000000001-000000001.tsm id=0 duration=8.246ms
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.549301Z lvl=info msg="Reading file" log_id=0NEWBf7l000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/openhab_db/autogen/2/_00001.wal size=4780730
Jun 06 16:25:17 openhab influxd[597]: ts=2020-06-06T14:25:17.604234Z lvl=info msg="Opened shard" log_id=0NEWBf7l000 service=store trace_id=0NEWBfaW000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/openhab_db/autogen/1 duration=105.089ms
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.331140Z lvl=info msg="File corrupt" log_id=0NEWBf7l000 engine=tsm1 service=cacheloader error="snappy: corrupt input" path=/var/lib/influxdb/wal/openhab_db/autogen/2/_00001.wal pos=2097099
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.352057Z lvl=info msg="Opened shard" log_id=0NEWBf7l000 service=store trace_id=0NEWBfaW000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/openhab_db/autogen/2 duration=854.699ms
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.352508Z lvl=info msg="Open store (end)" log_id=0NEWBf7l000 service=store trace_id=0NEWBfaW000 op_name=tsdb_open op_event=end op_elapsed=890.255ms
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.353859Z lvl=info msg="Opened service" log_id=0NEWBf7l000 service=subscriber
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.353960Z lvl=info msg="Starting monitor service" log_id=0NEWBf7l000 service=monitor
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.353989Z lvl=info msg="Registered diagnostics client" log_id=0NEWBf7l000 service=monitor name=build
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354010Z lvl=info msg="Registered diagnostics client" log_id=0NEWBf7l000 service=monitor name=runtime
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354029Z lvl=info msg="Registered diagnostics client" log_id=0NEWBf7l000 service=monitor name=network
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354057Z lvl=info msg="Registered diagnostics client" log_id=0NEWBf7l000 service=monitor name=system
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354099Z lvl=info msg="Starting precreation service" log_id=0NEWBf7l000 service=shard-precreation check_interval=10m advance_period=30m
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354148Z lvl=info msg="Starting snapshot service" log_id=0NEWBf7l000 service=snapshot
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.354187Z lvl=info msg="Starting continuous query service" log_id=0NEWBf7l000 service=continuous_querier
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.355327Z lvl=info msg="Starting HTTP service" log_id=0NEWBf7l000 service=httpd authentication=true
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.355382Z lvl=info msg="opened HTTP access log" log_id=0NEWBf7l000 service=httpd path=stderr
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.355401Z lvl=info msg="Auth is enabled but shared-secret is blank. BearerAuthentication is disabled." log_id=0NEWBf7l000 service=httpd
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.360727Z lvl=info msg="Listening on HTTP" log_id=0NEWBf7l000 service=httpd addr=127.0.0.1:8086 https=false
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.360813Z lvl=info msg="Starting retention policy enforcement service" log_id=0NEWBf7l000 service=retention check_interval=30m
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.361018Z lvl=info msg="Listening for signals" log_id=0NEWBf7l000
Jun 06 16:25:18 openhab influxd[597]: ts=2020-06-06T14:25:18.514181Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0NEWBf7l000
Jun 06 17:23:11 openhab influxd[597]: [httpd] 127.0.0.1 - - [06/Jun/2020:17:23:11 +0200] "GET /ping HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a1cdb5bc-a809-11ea-8001-dca632acf7d1 1611
Jun 06 17:23:11 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:11 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a234ff31-a809-11ea-8002-dca632acf7d1 163084
Jun 06 17:23:12 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:12 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a2e2eeab-a809-11ea-8003-dca632acf7d1 6947
Jun 06 17:23:13 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:13 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a2f1b21c-a809-11ea-8004-dca632acf7d1 5218
Jun 06 17:23:13 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:13 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a33df711-a809-11ea-8005-dca632acf7d1 5378
Jun 06 17:23:13 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:13 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a36bc0ed-a809-11ea-8006-dca632acf7d1 5185
Jun 06 17:23:15 openhab influxd[597]: [httpd] 127.0.0.1 - openhab [06/Jun/2020:17:23:15 +0200] "POST /write?consistency=one&db=openhab_db&p=%5BREDACTED%5D&precision=n&rp=autogen&u=openhab HTTP/1.1" 204 0 "-" "okhttp/2.4.0" a4513a12-a809-11ea-8007-dca632acf7d1 6536

Any ideas why influxdb is “loosing” data after reboot?

Seems I found the problem, I oversaw this log entry at first sight. I removed this corrupt file, restarted everything, and the file was created again. Now values are stored correct and “survive” a reboot. However, the missing data seems lost.

I also suspect that the problem occured while backup and recovery when migrating to SSD, usually I don’t stop services when running raspiBackup, perhaps file was in use when dd’ed and thus not restored correctly.