Broken influx persistence - Fixed!

Add-ons Persistence Services
,
1

My /var partition ran out of space today…and its broken Influxdb persistance of some metrics… not all, just some…

After i cleared down some rubbish files by

sudo apt-get autoclean
sudo apt-get autoremove

then a reboot i noticed grafana had missing data for some metrics…others were fine and continue to be.
So checking my log i see this error message:

2017-09-26 22:15:40.568 [ERROR] [g.eclipse.smarthome.core.persistence] - [org.eclipse.smarthome.core.persistence.persistencemanager(56)] The addPersistenceService  an exception
java.io.IOError: java.io.IOException: Zero Header, data corrupted
        at org.mapdb.SerializerBase.deserialize(SerializerBase.java:881)
        at org.mapdb.SerializerBase.deserialize(SerializerBase.java:867)
        at org.mapdb.SerializerPojo.deserialize(SerializerPojo.java:701)
        at org.mapdb.BTreeKeySerializer$BasicKeySerializer.deserialize(BTreeKeySerializer.java:89)
        at org.mapdb.BTreeMap$NodeSerializer.deserialize(BTreeMap.java:465)
        at org.mapdb.BTreeMap$NodeSerializer.deserialize(BTreeMap.java:288)
        at org.mapdb.Store.deserialize(Store.java:297)
        at org.mapdb.StoreDirect.get2(StoreDirect.java:475)
        at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
        at org.mapdb.StoreWAL.get(StoreWAL.java:320)
        at org.mapdb.Caches$HashTable.get(Caches.java:246)
        at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
        at org.mapdb.BTreeMap.get(BTreeMap.java:602)
        at org.mapdb.BTreeMap.get(BTreeMap.java:589)
        at org.openhab.persistence.mapdb.internal.MapDBPersistenceService.query(MapDBPersistenceService.java:175)
        at org.openhab.core.persistence.internal.QueryablePersistenceServiceDelegate.query(QueryablePersistenceServiceDelegate.java:47)
        at org.eclipse.smarthome.core.persistence.internal.PersistenceManagerImpl.initialize(PersistenceManagerImpl.java:250)
        at org.eclipse.smarthome.core.persistence.internal.PersistenceManagerImpl.startEventHandling(PersistenceManagerImpl.java:365)
        at org.eclipse.smarthome.core.persistence.internal.PersistenceManagerImpl.addPersistenceService(PersistenceManagerImpl.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_144]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_144]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_144]
        at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:224)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:617)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:501)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BindMethod.invoke(BindMethod.java:655)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1722)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager.invokeBindMethod(DependencyManager.java:1699)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeBindMethod(SingleComponentManager.java:371)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:320)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:291)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)[31:org.apache.felix.scr:2.0.6]
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)[org.eclipse.osgi-3.10.101.v20150820-1
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:914)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)[org.eclipse.osgi-3.10.101.v20150820-14
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)[org.eclipse.osgi-3.10.101.v20150820-1432.ja
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:482)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:998)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.openhab.core.persistence.internal.PersistenceServiceFactory.registerDelegateService(PersistenceServiceFactory.java:72)
        at org.openhab.core.persistence.internal.PersistenceServiceFactory.addPersistenceService(PersistenceServiceFactory.java:53)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_144]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_144]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_144]
        at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:224)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:617)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:501)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.inject.BindMethod.invoke(BindMethod.java:655)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1722)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager.invokeBindMethod(DependencyManager.java:1699)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeBindMethod(SingleComponentManager.java:371)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:320)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:291)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)[31:org.apache.felix.scr:2.0.6]
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)[org.eclipse.osgi-3.10.101.v20150820-1
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:914)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)[org.eclipse.osgi-3.10.101.v20150820-14
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)[org.eclipse.osgi-3.10.101.v20150820-1432.ja
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)[31:org.apache.felix.scr:2.0.6]
        at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)[31:org.apache.felix.scr:2.0.6]
        at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)[org.osgi.core-6.0.0.jar:]
        at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:415)[org.osgi.core-6.0.0.jar:]
        at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)[org.osgi.core-6.0.0.jar:]
        at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)[org.osgi.core-6.0.0.jar:]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:902)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)[org.eclipse.osgi-3.10.101.v201508
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)[org.eclipse.osgi-3.10.101.v20150820-1432.jar
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)[org.eclipse.osgi-3.10.101.v20150820-1432.jar
        at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)[org.eclipse.osgi-3.10.101.v20150820-143
        at org.eclipse.osgi.container.Module.publishEvent(Module.java:466)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.container.Module.start(Module.java:457)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1582)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1562)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1533)[org.eclipse.osgi-3.10.101.v20150820-1432
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1476)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)[org.eclipse.osgi-3.10.101.v20150820-1432.jar:]
Caused by: java.io.IOException: Zero Header, data corrupted
        ... 115 more
2017-09-26 22:15:40.919 [INFO ] [basic.internal.servlet.WebAppServlet] - Started Basic UI at /basicui/app

So i thought perhaps the influxdata measurement has gone bad…so i dropped one of the measurements and again restarted openhab…same error!..and the measurement hasnt been recreated.

So im now thinking it must be some data file on my openhab instance…which makes sense as the userdata is in /var/lib/openhab2 …the mount which ran out of space before.

So anyone have any ideas where this “data corrupted” may be??

2

I’ve no idea how to help but ask when you do figure it out come back and let us know how you fixed it.

However, I do notice that the error is coming from MapDB, not InfluxDB. Maybe you are looking in the wrong place?

3

Thanks @rlkoshak for your reply and for picking up that its MapDB - doh!

I think i can probably safely reset the MapDB as pretty much all my config is in the txt files ( items/sitemaps etc)

But before i do - can the MapDB be fixed? edited?

Cheers,
Grge

4

Not that I am aware. But since it only keeps the absolutely most recent value the amount of data lost is relatively low, though I do recognize the loss of the data is not zero.

If you are really concerned, you might switch restoreOnStartup config to some other DB, test that it works, then wipe the MapDB, then move it back. You might not lose as much, if any, data that way.

5

Yeah i just looked at my mapdb.persist:

Items {
	* : strategy = everyChange, everyDay, restoreOnStartup	
}

Id forgotten we now use jsondb for config…so its only the restoreOnStartup that i lose…and thats OK.
So i renamed the mapdb data directory and ill see what happens.

Cheers!

6

And now my influx is back and reporting correctly!

So it was the corruption of the mapdb that was my problem - deleting/recreating it has fixed my issue caused by the directory running out of space.

Good lesson learned!
I should probably look into some system monitor binding to alert on low space!

1 Like
7

If you find one write a tutorial. I discovered the machine I run OH on ran out of room too, just the other day. My persistence is stored on a NAS shared folder so all that “broke” was my ability to apt-get update. Its a VM and I clearly need to increase the storage on that VM.

Glad you got it working!

8

this fixed my issue -

I know this is an old thread, but should anyone have mapdb+influxdb and find their influxdb no longer working you may need to delete your mapdb files.