OutOfMemoryError (Java heap space) after a few days of running openHAB2

nspo · August 25, 2018, 12:00pm

Platform information:
- Hardware: Intel Core i7 x64, 1 GiB RAM
- OS: Ubuntu 18.04.1 x64
- Java Runtime Environment: OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.18.04.1-b13)
- openHAB version: 2.3.0
Issue of the topic: After running for a few days, openHAB crashes with an OutOfMemoryError. It might have to to something with charts and/or sitemap generation.
Memory dump after crash:
- 199.14mb (84.0%) is used by Java Static org.eclipse.smarthome.io.rest.internal.RESTActivator.context
- Contents of object above:

 org.eclipse.osgi.internal.framework.BundleContextImpl.servicesInUse ↘ 203,915K (84.0%), 1 reference(s)
 {j.u.HashMap}.values ↘ 203,778K (84.0%), 21 reference(s)
 org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.cachedService ↘ 203,778K (84.0%), 20 reference(s)
 org.eclipse.smarthome.io.rest.sitemap.internal.SitemapResource.eventOutputs ↘ 201,736K (83.2%), 1 reference(s)
 com.google.common.collect.MapMakerInternalMap.segments ↘ 201,736K (83.2%), 1 reference(s)
 com.google.common.collect.MapMakerInternalMap$Segment[] ↘ 201,736K (83.2%), 4 reference(s)
 com.google.common.collect.MapMakerInternalMap$Segment.table ↘ 201,735K (83.2%), 4 reference(s)
 java.util.concurrent.atomic.AtomicReferenceArray.array ↘ 201,735K (83.2%), 4 reference(s)
 Object[] ↘ 201,671K (83.1%), 8,178 reference(s)
 com.google.common.collect.MapMakerInternalMap$StrongEntry.valueReference ↘ 150,471K (62.0%), 8,178 reference(s)
 com.google.common.collect.MapMakerInternalMap$WeakValueReference.referent ↘ 150,215K (61.9%), 8,174 reference(s)
 com.google.common.collect.MapMakerInternalMap$WeakValueReference self 255K (0.1%), 8,178 object(s)
 com.google.common.collect.MapMakerInternalMap$StrongEntry.{next} ↘ 50,944K (21.0%), 3,142 reference(s)
 com.google.common.collect.MapMakerInternalMap$StrongEntry.valueReference ↘ 50,846K (21.0%), 3,142 reference(s)
 com.google.common.collect.MapMakerInternalMap$StrongEntry self 98K (<0.1%), 3,142 object(s)

Top Memory Headers: java.util.concurrent.LinkedBlockingDeque$Node (24%, 57.2mb, Count 4,997,841)
openhab> bundle:list

START LEVEL 100 , List Threshold: 50
 ID │ State    │ Lvl │ Version                │ Name
────┼──────────┼─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 15 │ Active   │  80 │ 5.3.1.201602281253     │ OSGi JAX-RS Connector
 16 │ Active   │  80 │ 2.4.5                  │ Jackson-annotations
 17 │ Active   │  80 │ 2.4.5                  │ Jackson-core
 18 │ Active   │  80 │ 2.4.5                  │ jackson-databind
 19 │ Active   │  80 │ 2.4.5                  │ Jackson-dataformat-XML
 20 │ Active   │  80 │ 2.4.5                  │ Jackson-dataformat-YAML
 21 │ Active   │  80 │ 2.4.5                  │ Jackson-module-JAXB-annotations
 22 │ Active   │  80 │ 2.7.0                  │ Gson
 23 │ Active   │  80 │ 18.0.0                 │ Guava: Google Core Libraries for Java
 24 │ Active   │  80 │ 3.0.0.v201312141243    │ Google Guice (No AOP)
 25 │ Active   │  80 │ 1.5.8                  │ swagger-annotations
 26 │ Active   │  80 │ 3.19.0.GA              │ Javassist
 28 │ Active   │  80 │ 3.5.2                  │ JmDNS
 30 │ Active   │  80 │ 1.0.0                  │ Units of Measurement API
 32 │ Active   │  80 │ 1.1.0.Final            │ Bean Validation API
 33 │ Active   │  80 │ 2.0.1                  │ javax.ws.rs-api
 34 │ Active   │  80 │ 3.2.0.v201101311130    │ ANTLR Runtime
 35 │ Active   │  80 │ 3.2.1                  │ Commons Collections
 36 │ Active   │  80 │ 1.1                    │ Commons Exec
 37 │ Active   │  80 │ 2.2.0                  │ Commons IO
 38 │ Active   │  80 │ 2.6                    │ Commons Lang
 48 │ Active   │  80 │ 4.1.5                  │ Apache Karaf :: OSGi Services :: Event
 67 │ Active   │  80 │ 2.11.0.v20150805-0538  │ EMF Common
 68 │ Active   │  80 │ 2.11.1.v20150805-0538  │ EMF Ecore
 69 │ Active   │  80 │ 2.11.1.v20150805-0538  │ EMF XML/XMI Persistence
 70 │ Active   │  80 │ 3.8.0.v20160509-1230   │ Common Eclipse Runtime
 71 │ Active   │  80 │ 3.6.100.v20160223-2218 │ Extension Registry Support
 80 │ Active   │  80 │ 9.3.21.v20170918       │ Jetty :: Proxy
 94 │ Active   │  80 │ 0.2.1.v20170706-0855   │ org.eclipse.lsp4j
 95 │ Active   │  80 │ 0.2.1.v20170706-0855   │ org.eclipse.lsp4j.jsonrpc
 96 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Config Core
 97 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Configuration Discovery
 98 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Configuration mDNS Discovery
 99 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Config Dispatcher
100 │ Active   │  75 │ 0.10.0.oh230           │ Eclipse SmartHome Config XML
101 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core
102 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core Audio
103 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome AutoUpdate Binding
104 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core Binding XML
105 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core ID
106 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core Persistence
107 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Scheduler Service
108 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core Thing
109 │ Active   │  75 │ 0.10.0.oh230           │ Eclipse SmartHome Core Thing XML
110 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Transformation Service
111 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core Voice
112 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Console
113 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Console for OSGi runtime Karaf
114 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome JavaSound I/O, Fragments: 180
115 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Monitor
116 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Net I/O Bundle
117 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome REST Interface Bundle
118 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Core REST API
119 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome REST mDNS Announcer
120 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome REST Interface JAX-RS optimization Bundle
121 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Sitemap REST API
122 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome SSE REST API
123 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Voice REST API
124 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Bonjour/MDS Service Discovery Bundle
125 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Web Audio Support
126 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Model Core
127 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Item Model
128 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Item Model IDE
129 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Item Model Runtime
130 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Language Server
131 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Persistence Model
132 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Persistence Model IDE
133 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Persistence Runtime
134 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Rule Model
135 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Rule Model IDE
136 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Rule Runtime
137 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Script
138 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Script Model IDE
139 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Script Runtime
140 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Sitemap Model
141 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Sitemap Model IDE
142 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Sitemap Runtime
143 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Thing Model
144 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Thing Model IDE
145 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Thing Model Runtime
146 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Json Storage Service
147 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome UI
148 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome UI Icons
149 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Classic IconSet
150 │ Active   │  80 │ 2.12.0.v20170518-0757  │ Xtend Runtime Library
151 │ Active   │  80 │ 2.12.0.v20170518-0757  │ Xtend Macro Interfaces
152 │ Active   │  80 │ 2.12.0.v20170518-0959  │ Xtext
153 │ Active   │  80 │ 2.12.0.v20170519-0752  │ Xtext Common Types
154 │ Active   │  80 │ 2.12.0.v20171017-1700  │ Xtext IDE Core
155 │ Active   │  80 │ 2.12.0.v20170518-0959  │ Xtext Utility
156 │ Active   │  80 │ 2.12.0.v20170519-0752  │ Xbase Model
157 │ Active   │  80 │ 2.12.0.v20170519-0752  │ Xbase Generic IDE Services
158 │ Active   │  80 │ 2.12.0.v20170518-0757  │ Xbase Runtime Library
173 │ Active   │  80 │ 2.3.0                  │ JUPnP Library
174 │ Active   │  80 │ 1.9.6                  │ MIME streaming extension
175 │ Active   │  80 │ 5.0.2                  │ Require-Bundle Fix :: org.objectweb.asm
177 │ Active   │  90 │ 2.3.0                  │ openHAB Core
178 │ Active   │  80 │ 2.3.0                  │ openHAB Karaf Integration
180 │ Resolved │  80 │ 2.3.0                  │ openHAB Sound Support, Hosts: 114
181 │ Active   │  80 │ 2.3.0                  │ openHAB Dashboard UI
186 │ Active   │  80 │ 3.1.4                  │ Stax2 API
187 │ Active   │  80 │ 1.0.2                  │ Units of Measurement Common Library
188 │ Active   │  80 │ 1.0.8                  │ Units of Measurement Implementation for Java SE
189 │ Active   │  80 │ 3.2.0                  │ Commons Net
190 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Basic UI, Fragments: 192
191 │ Active   │  80 │ 0.10.0.oh230           │ Eclipse SmartHome Paper UI, Fragments: 195
192 │ Resolved │  75 │ 2.3.0                  │ openHAB Basic UI Fragment, Hosts: 190
193 │ Active   │  80 │ 2.3.0                  │ HABPanel User Interface
194 │ Active   │  80 │ 2.3.0                  │ HomeBuilder User Interface
195 │ Resolved │  75 │ 2.3.0                  │ openHAB Paper UI Theme Fragment, Hosts: 191
196 │ Active   │  80 │ 1.6.0                  │ Commons Codec
197 │ Active   │  80 │ 4.2.3                  │ Apache HttpClient OSGi bundle
198 │ Active   │  80 │ 4.2.3                  │ Apache HttpCore OSGi bundle
199 │ Active   │  80 │ 3.1.0.7                │ Apache ServiceMix :: Bundles :: commons-httpclient
200 │ Active   │  80 │ 1.12.0                 │ openHAB MQTT Binding
201 │ Active   │  80 │ 2.3.0                  │ openHAB 1.x Compatibility Layer
202 │ Active   │  80 │ 1.12.0                 │ openHAB MQTT Transport Bundle
203 │ Active   │  80 │ 1.12.0                 │ openHAB RRD4j Persistence Bundle
205 │ Active   │  80 │ 1.12.0                 │ openHAB Expire Binding
206 │ Active   │  80 │ 1.12.0                 │ openHAB Weather Binding

openhab.log: Redirecting...
services/weather.cfg

apikey.ForecastIo=foobar

location.karlsruhe.latitude=49.006889
location.karlsruhe.longitude=8.403653
location.karlsruhe.provider=ForecastIo
location.karlsruhe.language=de
location.karlsruhe.updateInterval=5

Items: only two few temperature/humidity/pressure sensors and the related values from the weather service
MQTT broker runinng
RRD4J installed:
- persistence/rrd4j.persist

Strategies {
	everyMinute : "0 * * * * ?"
	everyHour 	: "0 0 * * * ?"
	everyDay	: "0 0 0 * * ?"
	default = everyChange
}
Items {
	// persist everything when the value is updated, just a default, and restore them from database on startup
	* : strategy = everyHour, restoreOnStartup

	gChart* : strategy = everyMinute

	// alternatively you can add specific items here, such as
	//Bedroom_Humidity,JamesInOffice : strategy = everyMinute
}

sitemaps/charts_bar.sitemap (+ one other nearly identical sitemap)

sitemap charts_bar label="Bar Charts" {
	Frame label="Temperature & humidity" {
		Text item=bar_temp1
		Text item=bar_temp2
		Text item=outside_temp
		Text item=bar_humid
		Text item=outside_humid
		Text item=bar_pressure
		Text item=outside_pressure
		Switch item=Temp_Chart_Period label="Chart period" mappings=[0="Hour", 1="Day", 2="Week"]
		Chart item=gTempBar period=h refresh=600 visibility=[Temp_Chart_Period==0] legend=true
		Chart item=gTempBar period=D refresh=900 visibility=[Temp_Chart_Period==1, Temp_Chart_Period=="NULL"] legend=true
		Chart item=gTempBar period=W refresh=600 visibility=[Temp_Chart_Period==2] legend=true
		Chart item=gHumidBar period=h refresh=900 visibility=[Temp_Chart_Period==0] legend=true
		Chart item=gHumidBar period=D refresh=3600 visibility=[Temp_Chart_Period==1, Temp_Chart_Period=="NULL"] legend=true
		Chart item=gHumidBar period=W refresh=3600 visibility=[Temp_Chart_Period==2] legend=true
		Chart item=gPressBar period=h refresh=900 visibility=[Temp_Chart_Period==0] legend=true
		Chart item=gPressBar period=D refresh=3600 visibility=[Temp_Chart_Period==1, Temp_Chart_Period=="NULL"] legend=true
		Chart item=gPressBar period=W refresh=3600 visibility=[Temp_Chart_Period==2] legend=true

	}
}

H102 · August 25, 2018, 2:34pm

Looking at your log, I see several messages about service warnings and taking more than 5000ms. Could this be an issue with the sitemap settings?

You can try changing your sitemap refresh to 60000 (every minute) for all chart items, especially if you have multiple charts, and see if that helps.

nspo · August 25, 2018, 2:58pm

I did indeed have very high refresh rates (600-3600 ms apparently). I changed it to 60000 and will report back if the error occurs again. Thanks!

diyha · August 25, 2018, 4:01pm

I had mem leak issues with RandomAccessFileAppender that is now default in OH dist.

Try my loging cfg to see if that improves anything

https://github.com/diyhomeauto/smarthome/blob/master/org.ops4j.pax.logging.cfg

You need to backup your old cfg USERDATA/etc/org.ops4j.pax.logging.cfg and then drop above file at USERDATA/etc/

nspo · August 29, 2018, 10:17pm

The error occurred again with the same error message although I set the refresh rates much lower. I will try diyha’s suggestion now

diyha · August 29, 2018, 10:18pm

How many hours/days before it happens?

nspo · August 29, 2018, 10:21pm

About 4 days.

diyha · August 29, 2018, 10:25pm

Ok. BTW I noticed you are running openJDK on x86 platform. Generally it shouldn’t be a problem. My Eclipse IDE runs fine on that, but I am not sure openHAB runtime will run on it reliably. Just a guess. If it is a personal non-commercial setup, may I suggest you try Oracle JDK or if its commercial then openJDK build from Azul Systems? I am not entirely sure if Azul Systems openJDK build is better than openJDK build available by default on Ubuntu x86 systems.

Oracle JVM is the best out there but if you are distributing the solution to a customer, you have to pay $12 to $40 per year depending on CPU rank in Oracle’s chart. On rasppi and similar quad core ARMs it comes to around $12-$15 per year. X86 even higher.

And according to rumors, from 2019 you might not even be allowed to use Oracle JDK 8+ for any commercial purpose, even if you are not distributing it to anyone. You must get license from them and renew every year. Freelance devs, you may need to pay for Oracle JDK updates, if you are using it in your IDE.

nspo · September 3, 2018, 11:03am

So far, so good! openhab runs with the same memory usage as when I started it 5 days ago. I am still using the OpenJDK (that would be my next try if the bug appears again). I basically only replaced RandomAccessFileAppender with RollingFile, as you recommended. Thanks!

diyha · September 3, 2018, 5:11pm

I am hoping OpenJDK works stable. I don’t like Zulu taking over Linux community maintained OpenJDK build.

nspo · September 10, 2018, 2:17pm

I had a crash again - but not because of an OutOfMemoryError.

2018-09-04 22:25:31.155 [ERROR] [ersey.server.ServerRuntime$Responder] - An I/O error has occurred while writing a response message entity chunk to the container output stream.

Log: https://nopaste.xyz/?7da6596d5a626adf#0TGbJBkB3jlLYjoodaQ4Fyzyir/paMTdZ6e0kgDYCyo=

Edit: The long error above might not be a serious problem - maybe a symptom of low resources. But openhab still completely crashed, apparently about 2 days later (also in the log) - at least I could not connect anymore. After restarting only openhab, it worked again.

nspo · September 22, 2018, 9:15am

Now using Oracle JDK and not using RandomAccessFileAppender, I still got an OutOfMemoryError again

https://nopaste.linux-dev.org/?1185108

diyha · September 28, 2018, 9:48pm

199MB static mem means you got a lot of classes loaded. Is it possible you allocate 256MB to classes and another 128-256MB for heap? If overall 512MB is not enough then there is some genuine bug in one of the bindings or extensions, unless you are running a very large setup. In that case, you need to buy more ram.

nspo · September 29, 2018, 6:16am

One thing to mention is maybe that I had a few (2 sensors, 6 values total) sensors that sent new values every 2 seconds. But that’s about all the system was receiving externally.

wborn · September 29, 2018, 4:01pm

Looks like all your memory is being eaten by SSE Sitemap subscriptions. I suggest updating to the latest 2.4.0 Milestone or SNAPSHOT build.

It has a fix for releasing sitemap subscriptions:

Do you think your fix will also fix this memory leak @Lolodomo?

There’s also another ESH issue about this:

github.com/eclipse-archived/smarthome

Leak (or excessive memory usage) in SitemapResource

opened 01:34AM - 31 Jan 18 UTC

closed 06:38AM - 12 Jun 18 UTC

sebmarchand

bug UI

(This is a continuation of the discussion on this OpenHab issue: https://communi…ty.openhab.org/t/sitemap-causing-oom/38936/10) **Summary**: It looks like there's no restriction on how many objects [SitemapResource::eventOutputs](https://github.com/eclipse/smarthome/blob/dcef6131b70c231a604b4beec5dc9e1769fbb88f/bundles/io/org.eclipse.smarthome.io.rest.sitemap/src/main/java/org/eclipse/smarthome/io/rest/sitemap/internal/SitemapResource.java#L130) can contain and this can easily lead to an OOM. **Details**: If I keep OpenHab2's BasicUI open in my browser for a long period of time (e.g. overnight) this will likely lead to an OOM crash, I've investigated a bit on this and the heap dump I got seem to point to [SitemapResource::eventOutputs](https://github.com/eclipse/smarthome/blob/dcef6131b70c231a604b4beec5dc9e1769fbb88f/bundles/io/org.eclipse.smarthome.io.rest.sitemap/src/main/java/org/eclipse/smarthome/io/rest/sitemap/internal/SitemapResource.java#L130). Here's some information from the heap dump analysis: Problem Suspect (as provided by Memory Analyzer): > One instance of "org.eclipse.smarthome.io.rest.sitemap.internal.SitemapResource" loaded by "org.eclipse.smarthome.io.rest.sitemap" occupies 893,758,768 (89.71%) bytes. The memory is accumulated in one instance of "org.eclipse.smarthome.io.rest.sitemap.internal.SitemapResource" loaded by "org.eclipse.smarthome.io.rest.sitemap". And here's the "Path to GC root" for this SitemapResource object: ``` Class Name | Shallow Heap | Retained Heap --------------------------------------------------------------------------------------------------------------------------------------------------- org.eclipse.smarthome.io.rest.sitemap.internal.SitemapResource @ 0xc45e1a88 | 48 | 893,758,768 '- m_implementationObject org.apache.felix.scr.impl.manager.ComponentContextImpl @ 0xc45e18a0 | 56 | 720 '- m_componentContext org.apache.felix.scr.impl.manager.SingleComponentManager @ 0xc45e0680 | 104 | 1,816 |- service org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl @ 0xc45e16d0 | 64 | 408 | '- [232] java.lang.Object[567] @ 0xc5fa49e8 | 2,288 | 2,288 | '- elementData java.util.ArrayList @ 0xc48317c0 | 24 | 2,312 | '- allPublishedServices org.eclipse.osgi.internal.serviceregistry.ServiceRegistry @ 0xc482d9b8 | 48 | 107,192 | |- serviceRegistry org.eclipse.osgi.internal.framework.EquinoxContainer @ 0xc43feec8 | 72 | 4,544 | | |- container org.eclipse.osgi.internal.url.URLStreamHandlerFactoryImpl @ 0xc44ced18 | 40 | 200 | | | '- factory class java.net.URL @ 0xc4306878 System Class | 32 | 864 | | |- equinoxContainer org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle @ 0xc464bd88| 40 | 80 | | |- container org.eclipse.osgi.internal.url.ContentHandlerFactoryImpl @ 0xc48329c8 | 40 | 128 | | |- container org.eclipse.osgi.internal.framework.BundleContextImpl @ 0xc4ca9718 | 40 | 424 | | |- threadFactory java.util.concurrent.ScheduledThreadPoolExecutor @ 0xc464bcf8 | 80 | 528 | | |- equinoxContainer org.eclipse.osgi.internal.framework.EquinoxBundle @ 0xc4796458 | 32 | 48 | | |- equinoxContainer org.eclipse.osgi.internal.framework.EquinoxBundle @ 0xc47eef80 | 32 | 48 | | |- container org.eclipse.osgi.internal.framework.BundleContextImpl @ 0xc4f919d8 | 40 | 904 | | |- container org.eclipse.osgi.internal.loader.BundleLoader @ 0xd6457650 | 72 | 1,040 | | '- Total: 9 entries | | | |- registry org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl @ 0xc4cb3480 | 64 | 360 | |- registry org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl @ 0xc94d6988 | 64 | 376 | '- Total: 3 entries | | |- value java.util.HashMap$Node @ 0xc5c61730 | 32 | 56 '- Total: 2 entries | | --------------------------------------------------------------------------------------------------------------------------------------------------- ``` I think that the root cause of my problem has nothing to do with the SitemapResource , there shouldn't be so many event sent to it in the first place and I'll investigate on this separately, but it'd be nice if there was a way to make sure that this class can't cause this excessive memory usage. Thanks! Please let me know if there's anything I can share here to make it easier to debug this issue. I've tried looking at it but it's beyond my Java skills :)

With its own community thread:

Bredmich · June 14, 2020, 11:24am

Hi diyha,
could you please re-upload your logging cfg?

Thanks!

diyha · August 24, 2020, 1:56pm

github.com/asvilabs/esh-oh

org.ops4j.pax.logging.cfg

master

# Common pattern layout for appenders
#log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n

# Root logger
log4j2.rootLogger.level = INFO
log4j2.rootLogger.appenderRefs = out, osgi
log4j2.rootLogger.appenderRef.out.ref = LOGFILE
log4j2.rootLogger.appenderRef.osgi.ref = OSGI

# Karaf Shell logger
log4j2.logger.shell.name = org.apache.karaf.shell.support
log4j2.logger.shell.level = OFF
log4j2.logger.shell.appenderRefs = stdout
log4j2.logger.shell.appenderRef.stdout.ref = STDOUT

# Security audit logger
log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit
log4j2.logger.audit.level = INFO
log4j2.logger.audit.additivity = false
log4j2.logger.audit.appenderRefs = audit

This file has been truncated. show original

diyha · August 24, 2020, 2:09pm

the appender, RollingFile, I am using is reliable. The random access appender had memory leak issues 1-2 years back, I don’t know if thats’s been fixed.

matt1 · August 25, 2020, 12:53am

Are you using any image items in the sitemap that are updating? Since I stopped using image items to display camera feeds I have not seen any OOME in over a year. Since I have not seen anyone else suggest it yet try disabling bindings one by one until rule them out as the cause. Change things to work out the cause is by process of elimination.

diyha · August 26, 2020, 3:46pm

No I don’t handle cameras through openhab. There might be many reasons for memory leakage, its a pain to figure it out, try visualvm, a free tool, it might give you some leads. Best luck!