I have checked my events.log and it has ca 160k lines per day .
I think it is definitely too much, this also cause large persistence db (160MB /day). Question is simple how to reduce it ?
I donât need updates all events every minute. All date/ stages coming from:
GPIO, Satel/alarm binding, 1wire and Modbus.
1wire is fine, becuase there is (default) option to update to bus values which was updated.
In Satel there is no such option, every 20s - which I set in config data are polling to Openhab, even it is some sensor which change status very rare. Also I can not config longer time (default is 10s) because for other items I will need to wait for reaction
Same problem with GPIO or even worst because I canât see how to set refresh time, so each items are refresh every few secunds, even with same value.
Also Modbus produce a lot of events.
Any ideas how to reduce it ?
Is it any method to disable events reporting in a log file ?
This will disable the event.log entirely. The event.logâs purpose is to show everything that gets posted to the event bus. All command and updates get posted there so this is an all or nothing log. If there are some things you need to track in a log, do so through rules after turning off the event.log.
In your .persist file make sure you are using an everyChange strategy and not an everyUpdate strategy. Only persist those Items you need to restoreOnStartup, chart, or access historic data. You can also use MapDB which only saves the most recent state for those Items that you only need restoreOnStartup and you other persistence only for those Items where you need historical data and charting.
In persistence (db4o) I use Everychange + Restore on Startup for all and everyminute for 10 items which I used for charts. I think it is not much.
Problem is with events, if there are many then persistance is also large, becuase of frequent changes.
What do you think 160k events per 24h is not much ?
The only way to change the number of events is to change whatever is producing the events to not generate so many. Youâve already tried to so all you can do from within OH is be more choosy about what does and does not get persisted.
NOTE: you only need to do everyMinute if you are using rrd4j. You donât need to do it for db4o as far as I know.
I would use MapDB with restoreOnStartup for all and db4o only for those Items you chart or compare a previous value to the current value in rules.
Does anybody knows what is optimum quantity of events and size of persistence for Raspberry Pi2 ?
Currently I have 28k events per 24h, I donât know is it a much or not.
I use just db4o persistence and it makes ca 250MB file per day. I have limited persistence for such data which I really needs. I use everychange / restore on start-up strategy on all (maybe this is mistake ?) items (+/- 140) and just on 9 with everyminute strategy (I will delete it for tests).
Result of that is I need to wait for a chart at least few minutes , as I wrote above when I would delete persistence, charts will work fine, but for first 1-2 days.
I have also noticed, that web GUI (and aps on mobile) has quite large LAG, especially in setpoints - usually it takes few seconds to increase value
Maybe it is also caused strange timing, when I have one by one instruction in rule and both of them change item I see even 30sec delay.
When I check memory on Pi it is fine with about 30% usage, procesor workload jumps up to 100% (on core) when I use GUI, if not it is much lower.
It is not easy, because of bindings config. Problem is that refresh time is set once per each binding, when I pull data from alarm system (Satel binding) I have data which I can update once per hour but some of items I must refresh more frequent :(. Same thing with Modbus, 1wire, etc.
For GPIO even I have not found such parameter as refresh time, so I dont based on what Openhab is keep asking it.
Generally I have change timing from default miliseconds to thousands of ms, so I have refresh every few minutes, but again still system has a lag and it influence on cooperation via binding (ie I must wait for change)
25k per day is HUGE! Are you sure you are doing every change and not every update?
Iâm on my phone and have been traveling since 3 am so I apologize for being terseb and not explaining why.
Use MapDB for restore on startup. It only saves the last value so the db never grows so the size wonât impact performance. But MapDB is worthless for Charting or historic values.
Use rrd4j for charts and historic values of numeric values in rules. Like MapDB, rrd4j never grows and therefore the size of the db will not impact performance.
Only use db4o for those items you want to keep exact values for long time for Charting or analyais. (Months or years).
There is no reason to save every value for every item forever when all you care about is the most recent one to restore on startup. There is no reason to store every value forever for items where you only care about data from the last week or so, it donât need exact values for older data.
Unless you are really into analyzing your data, you should only have a few items stored to db4o or rrd4j.
I hope you mean it is huge for persistence
I have cutted most of them, by changing refresh time, so now I have ca 10k per 24h, but I have plan to add more items so this probably will grow up. Also now I am unable to add some rules because information about state is too late. This is big disadvantage of bindings that refresh time is set globally for all items. In only 1wire I have found refresh parameter that I can set per item, so I can set diff refresh times based on importance of item / events
I have reconfigure persistence as you suggest, I agree that there are no need to keep and remember all of events. I have used mapDB as well. Also I have not found persistence option how to control how long particular event will be keep in db, so probably my ex Db40 keeps date since it was initiated.
Now it works fine I hope for next days it will be no change.
Thanks a lot !
That is correct. Anything saved to db4o is saved forever or until you manually delete it. And I donât think it is designed to work well with dbs of many gigabytes in size.
Thatâs interesting problem. However I wouldnât try to fix it on bindingsâ side (like in OneWire binding), but it should be handled on OH side IMO. However in some cases this might be useful to get information about each update. But the question is: should we bother at all? You can disable event logging since this is rather for debuging purposes, and for persistence you have appropriate options (everyChange), the same for rules. All of them have been already mentioned by @rlkoshak.
Regarding Satel binding I have 1sec refresh time configured and this is not a problem at all. The good thing is that Satel gives you the information what has changed since last check, but bad thing is that the data is organized as large bitmaps of all zones/partitions/outputs, so for zone violation you have one bitmap of all zones and you get the info that something has just changed there - so all defined items for zone violation need to be updated. But still - it should not be a problem.
If I will change for 1sec then I will have more then 100k events per day . I read about 60 items from Satel. Thatâs the problem I dont need to update every item such frequently , but there is genereal, one refresh parameter. Satel just give every refresh time info about items stage, if it will update just on event it will be great, but I think it is not possible. So all the time I receive events with same items, even their stage have not been changed.
BTW. Do you have broken pipe error from time to time ? When I restart OH service every 4-7 time I got such error from the binding. After that I am unable to connect automatically so I need to restart Satel. Tech support from Satel told me that there is incorrect reconnection mechanism in OH.
This is the one thing I do not like about the persistence solution. There is no way to persist real data (i.e. not averages like in rrd4j) with a defined time (or size) limit. I am using db4o (for only a few items) because I want to work with proper historic values, however, I only need the history to go back so far in time (letâs say last 24 hours, last week, etc.).
Currently there is no way to do this, and my db4o store keeps growing,
Do you know of a way (a tool) to purge old data from db4o? I donât want to delete the entire store since I then loose data that I care about.
Side note: From the documentation it looks like mapdb supports setting a time to live (max time, max count or max size) of a database. It would have been cool if this was supported in openHAB. I guess the mapdb support that we have now, that is mostly for restore on startup, could simply be implemented as âmapdb with a max count of 1â in this case.
There are some tools that claim to allow you to manually view and edit db4o. Iâve not used them so canât say whether Iâm correct on that.
With rrd4j, keep in mind it only starts to compress the data after a week or so so if you only need exact Vanessa fur such a short period of time rrd4j would probably work.
But I do agree, OH could really use a mechanism to clear out old data in the dbs that just continue to grow. Be it a setting like MapDB, or an action.
I like the idea of an action because you can write rules to find time which Items are cleared out and for how long.
@sherif I am not telling you to switch to 1sec refresh time, but such configuration works well for me. Regarding your problem, why donât you disable event logging at all? Itâs easiest and reasonable solution in your case.
Regarding broken pipe error I havenât seen antyhing like that on my installation yet and also havenât heard from other people. Does this happen at start or after some time? In my case it sometimes (over week or so) disconnects on a timeout and the immediately reconnects again.
How to disable event logging ?
Can I disable only Satelâs part, sometimes I use it for troubleshooting, so I need the rest ?
Maybe I will test it, now when I reconfigured persistence I can back to many events, if it will not impact on system stability and speed.
What do you mean by"
The good thing is that Satel gives you the information what has changed since last check
Is it any other method to update items then frequently asked all items to refresh ?
Broken pipe I have received after some Openhab restart or system restarts (during initialization), it looks like follows:
2016-05-23 14:39:43.117 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Connecting to ETHM-1 module at 192.168.1.105:7094
2016-05-23 14:39:43.130 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - ETHM-1 module connected successfully
2016-05-23 14:39:43.169 [ERROR] [o.o.b.s.i.protocol.SatelModule] - Unexpected exception occurred during writing a message
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0]
at java.net.SocketOutputStream.write(SocketOutputStream.java:132) ~[na:1.8.0]
at org.openhab.binding.satel.internal.protocol.SatelModule.writeMessage(SatelModule.java:310) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.communicationLoop(SatelModule.java:371) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.access$1(SatelModule.java:338) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule$CommunicationWatchdog$2.run(SatelModule.java:465) [bundlefile:na]
at java.lang.Thread.run(Thread.java:744) [na:1.8.0]
2016-05-23 14:39:43.172 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Closing connection to ETHM-1 module
2016-05-23 14:39:44.113 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Connecting to ETHM-1 module at 192.168.1.105:7094
and so on, then OH will never connect to Satel without Satel restart.
Comment out EVENTFILE appender ref in âruntime.buseventsâ logger (logback.xml). But you cannot disable specific events in the log, just enable/disable the log completely.
Regarding refresh: the bindig sends command to check what has changed, there is such thing. But this is for all kind of devicesâ status. It is hard to explain if youâre not familiar with the protocol. Basically you for example some device has âviolationâ status changed, but you donât know which, so you must update all of items with that status (zone:violation:xxx), which helps somehow but not much.
Regarding broken pipe: it looks like your ethm hangs for some reason. I would upgrade the FW to latest version (unless you already did this). Do you use encrypted connection or plain?
After while I think the reason could be also in disconection. If you restart so frequently it might happen that the module does not close the connection and since it allows only one client it will not allow you to connect again. However this happened only once to me when I was replacing a switch, never in regular use.
I understand that binding just refresh all items, but it will be nice if refresh time can be diff for diff items. Then I dont need to refresh so often all items.
Yesterday I got next broken pipe. I have recognized that roller shutters didnât closed after dark, so I have check a log:
2016-06-06 19:16:25.448 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - ETHM-1 module connected successfully
2016-06-06 19:16:25.466 [INFO ] [.service.AbstractActiveService] - Exec Refresh Service has been started
2016-06-06 19:16:25.503 [ERROR] [o.o.b.s.i.protocol.SatelModule] - Unexpected exception occurred during writing a message
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0]
at java.net.SocketOutputStream.write(SocketOutputStream.java:132) ~[na:1.8.0]
at org.openhab.binding.satel.internal.protocol.SatelModule.writeMessage(SatelModule.java:310) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.communicationLoop(SatelModule.java:371) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.access$1(SatelModule.java:338) [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule$CommunicationWatchdog$2.run(SatelModule.java:465) [bundlefile:na]
at java.lang.Thread.run(Thread.java:744) [na:1.8.0]
It was after PI restart. I can try to update ETHM, but according support it will not help, becuase it wasnât such fix on this kind of issues. they told me that it looks like OH keeps connection and donât connect properly. ETHM after connection broke is ready for next connection. It is not so frequently, because I do restarts after some changes, ca once per week.
I dont use encryption for integration.
.
, as I wrote above when I would delete persistence, charts will work fine, but for first 1-2 days.
. I read about 60 items from Satel. Thatâs the problem I dont need to update every item such frequently , but there is genereal, one refresh parameter. Satel just give every refresh time info about items stage, if it will update just on event it will be great, but I think it is not possible. So all the time I receive events with same items, even their stage have not been changed.