Restoring from Backup failed

Hello,
OH3.2 on openhabian with all the updates, RPi 4, 4GB RAM.

I wanted to revert some changes of my OH, so I restored OH from my pre-last backup. After that, OH is not working properly, even if I have reverted to original latest-today’s backup.

myIP:9001 is not working, myIP:8080 showed Error 404
then I have made

sudo systemctl stop openhab .service
openhab-cli clean-cache
openhab-cli start

myIP:8080 was back again, but most of the bindings are UNINITIALIZED,
I got tons of error messages, at the beginning I found this:

[ERROR] [.core.karaf.internal.FeatureInstaller] - Failed installing ‘openhab-misc-openhabcloud, openhab-ui-habot, openhab-voice-picotts, openhab-binding-network, openhab-binding-darksky, openhab-ui-habpanel, openhab-binding-zigbee, openhab-binding-shelly, openhab-binding-feed, openhab-binding-mqtt, openhab-transformation-map, openhab-binding-solaredge, openhab-binding-ipcamera, openhab-binding-kodi, openhab-binding-tradfri, openhab-binding-telegram, openhab-binding-systeminfo, openhab-binding-luftdateninfo, openhab-transformation-jinja, openhab-binding-icalendar, openhab-binding-networkupstools, openhab-voice-voicerss, openhab-binding-mikrotik, openhab-binding-mail, openhab-binding-mpd, openhab-binding-spotify, openhab-transformation-jsonpath, openhab-binding-miio, openhab-binding-mihome, openhab-persistence-rrd4j, openhab-ui-basic, openhab-binding-ntp, openhab-binding-openweathermap, openhab-binding-http, openhab-binding-astro, openhab-binding-gpstracker, openhab-binding-remoteopenhab, openhab-binding-weathercompany’: null

after a while the karaf console stopped to send new messages and WebInterface becomes very slow.

What can I do to save the system from reinstalling? I spend months to tune up my system and wrote thousands lines of .rules code. I hope that there is a fix to save me from catastrophic scenario :open_mouth:

Thanks,
Michal

11:13:09.554 [WARN ] [ommon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception:
java.lang.OutOfMemoryError: Java heap space
11:13:04.909 [WARN ] [ommon.WrappedScheduledExecutorService] - Scheduled runnable ended with an exception:
java.lang.OutOfMemoryError: Java heap space
11:13:06.732 [ERROR] [org.apache.cxf.jaxrs.utils.JAXRSUtils] - Problem with writing the data, class org.openhab.core.io.rest.Stream2JSONInputStream, ContentType: application/json
11:13:31.358 [WARN ] [pache.cxf.phase.PhaseInterceptorChain] - Interceptor for {http://internal.id.core.openhab.org/}UUIDResource has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleWriteException(JAXRSOutInterceptor.java:399) ~[?:?]
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:275) ~[?:?]
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:122) ~[?:?]
at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84) ~[?:?]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:90) ~[bundleFile:3.4.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222) ~[bundleFile:3.4.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:13:23.760 [ERROR] [r.AbstractFaultChainInitiatorObserver] - An unexpected error occurred during error handling. No further error processing will occur.
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[bundleFile:3.4.5]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222) ~[bundleFile:3.4.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:16:37.531 [WARN ] [org.eclipse.jetty.server.HttpChannel ] - /rest/things
java.lang.OutOfMemoryError: Java heap space
11:14:45.870 [WARN ] [pache.cxf.phase.PhaseInterceptorChain] - Application {http://internal.id.core.openhab.org/}UUIDResource has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[bundleFile:3.4.5]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[bundleFile:3.4.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
org.ops4j.pax.logging.pax-logging-api [log4j2] ERROR : An exception occurred processing Appender LOGFILE
11:17:25.736 [WARN ] [pache.cxf.phase.PhaseInterceptorChain] - Exception in handleFault on interceptor org.apache.cxf.jaxrs.interceptor.JAXRSDefaultFaultOutInterceptor@8040be
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[bundleFile:3.4.5]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[bundleFile:3.4.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:16:21.794 [WARN ] [org.eclipse.jetty.server.HttpChannel ] - /rest/addons/types
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[?:?]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[?:?]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[?:?]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[?:?]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[?:?]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[?:?]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[?:?]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[?:?]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[?:?]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[?:?]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[?:?]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222) ~[?:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[?:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[?:?]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[?:?]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[?:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[?:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[?:?]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[?:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:20:06.863 [ERROR] [r.AbstractFaultChainInitiatorObserver] - An unexpected error occurred during error handling. No further error processing will occur.
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[bundleFile:3.4.5]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[bundleFile:3.4.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.5]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:22:38.429 [WARN ] [org.eclipse.jetty.server.HttpChannel ] - /rest/auth/token
org.apache.cxf.interceptor.Fault: Java heap space
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) ~[?:?]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) ~[?:?]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[?:?]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[?:?]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[?:?]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[?:?]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[?:?]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[?:?]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[?:?]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[?:?]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[?:?]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[?:?]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[?:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[?:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[?:?]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[?:?]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[?:?]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[?:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[?:?]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[?:?]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[?:?]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) ~[bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
11:30:51.319 [ERROR] [.xtext.validation.CompositeEValidator] - Error executing EValidator
java.lang.OutOfMemoryError: Java heap space
11:40:50.479 [ERROR] [.xtext.validation.CompositeEValidator] - Error executing EValidator
java.lang.OutOfMemoryError: Java heap space

Was this backup from a 3.2 as well or an earlier version?

There have been other reports of this error but it usually resolves itself on it’s own according to reports. However, for some reason on some systems all the add-ons need to be reinstalled on every restart. It’s not clear why that’s happening but it’s going to be something related to your system and not directly caused by OH itself.

There might be a related problem if you are running OpenJDK instead of Zulu Java.

The exception you are seeing after that one is an out of memory error which is not caused by your config or restored backup directly. I can’t say what is causing it but I would do the following:

  1. Save your backup off the system.
  2. Reinstall the system (new openHABian SD card image). Make sure to install the same version of OH as your backup.
  3. Restore your backup.
  4. If it’s an older OH version, only then upgrade to the latest OH.
1 Like

Thanks, Rich, again.
After I have started to uninstall and install first of the add-ons, every other addon has started working again. I have rebooted the system and everything looks like its working again!

My Java is

openhabian@openHABianDevice:~ $ java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK Client VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

so it looks like it is the correct Zulu one.

The problem was probably I have updated from 3.1 to 3.2 and loaded the 3.1 backup.
Well, my raspbian is outdated now, so one day I will try to do the complete reinstall.

Thanks!
Michal

So is this either or?
in my case it’s:

openhabian@nukem:~$ java --version
openjdk 11.0.11 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
OpenJDK Server VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)

I have the same problems after upgrading to 3.2 and using an older backup from 3.1

If it says Zulu you have Zulu. There are other implementations of OpenJDK as well as one just called OpenJDK. You have the right one as far as I can tell.

2 Likes

Awesome - thank you.

Hi everyone,

I have a similar problem with OH 3.2

I wanted to migrate my openHAB installation from one machine to another. Both machines are in the same network and running on the the current raspbian installation with:

java -version
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK Client VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

and

openhab-cli info

Version:     3.2.0 (Build)

User:        openhab (Active Process 5126)
User Groups: openhab tty dialout audio bluetooth

Directories: Folder Name      | Path                        | User:Group
             -----------      | ----                        | ----------
             OPENHAB_HOME     | /usr/share/openhab          | openhab:openhab
             OPENHAB_RUNTIME  | /usr/share/openhab/runtime  | openhab:openhab
             OPENHAB_USERDATA | /var/lib/openhab            | openhab:openhab
             OPENHAB_CONF     | /etc/openhab                | openhab:openhab
             OPENHAB_LOGDIR   | /var/log/openhab            | openhab:openhab

… ok, the active process number differs :wink:.

On my first machine openhab works fine. I did two backup versions on this machine with “openhab-cli backup” and “openhab-cli backup --full”.
After that I created a clean openHAB installation on my second machine and tried to restore the backup of my first machine. I didn’t made any configurations to the clean openHAB installation of the second machine. No user creation, no binding or addon installation.
The restore process with “sudo openhab-cli restore mybackup.zip” works well with both types of my backup, but after the restore finished, openHAB isn’t starting anymore. In my syslog file I can see a lot of java exceptions.

Do I have to do further steps to migrate to another machine or restore my installation? I can’t get a clue to this java exceptions, maybe anyone can help?

This is a part of my syslog:

Dec 28 09:48:37 liese systemd[1]: Started openHAB - empowering the smart home.
Dec 28 09:48:47 liese karaf[5126]: org.apache.karaf.features.internal.util.MultiException: Error:
Dec 28 09:48:47 liese karaf[5126]: #011Error downloading mvn:org.ops4j.pax.logging/pax-logging-log4j2/2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011Error downloading mvn:org.ops4j.pax.logging/pax-logging-api/2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:91)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:457)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:452)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:224)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:399)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1069)
Dec 28 09:48:47 liese karaf[5126]: #011at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:1004)
Dec 28 09:48:47 liese karaf[5126]: #011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 28 09:48:47 liese karaf[5126]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 28 09:48:47 liese karaf[5126]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 28 09:48:47 liese karaf[5126]: #011at java.base/java.lang.Thread.run(Thread.java:829)
Dec 28 09:48:47 liese karaf[5126]: #011Suppressed: java.io.IOException: Error downloading mvn:org.ops4j.pax.logging/pax-logging-log4j2/2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:77)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 3 more
Dec 28 09:48:47 liese karaf[5126]: #011Caused by: java.io.IOException: Error resolving artifact org.ops4j.pax.logging:pax-logging-log4j2:jar:2.0.12: [Could not find artifact org.ops4j.pax.logging:pax-logging-log4j2:jar:2.0.12 in openhab (https://openhab.jfrog.io/openhab/libs-release/)]
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 6 more
Dec 28 09:48:47 liese karaf[5126]: #011#011Suppressed: shaded.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.ops4j.pax.logging:pax-logging-log4j2:jar:2.0.12 in openhab (https://openhab.jfrog.io/openhab/libs-release/)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:48)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:368)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:642)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:489)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:390)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011... 11 more
Dec 28 09:48:47 liese karaf[5126]: #011Caused by: shaded.org.eclipse.aether.resolution.ArtifactResolutionException: Error resolving artifact org.ops4j.pax.logging:pax-logging-log4j2:jar:2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:413)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 11 more
Dec 28 09:48:47 liese karaf[5126]: #011Suppressed: java.io.IOException: Error downloading mvn:org.ops4j.pax.logging/pax-logging-api/2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:77)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 28 09:48:47 liese karaf[5126]: #011#011at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 3 more
Dec 28 09:48:47 liese karaf[5126]: #011Caused by: java.io.IOException: Error resolving artifact org.ops4j.pax.logging:pax-logging-api:jar:2.0.12: [Could not find artifact org.ops4j.pax.logging:pax-logging-api:jar:2.0.12 in openhab (https://openhab.jfrog.io/openhab/libs-release/)]
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.configureIOException(AetherBasedResolver.java:803)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:774)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:657)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:598)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:565)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.MavenDownloadTask.download(MavenDownloadTask.java:52)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 6 more
Dec 28 09:48:47 liese karaf[5126]: #011#011Suppressed: shaded.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.ops4j.pax.logging:pax-logging-api:jar:2.0.12 in openhab (https://openhab.jfrog.io/openhab/libs-release/)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:48)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:368)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:642)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:489)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:390)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767)
Dec 28 09:48:47 liese karaf[5126]: #011#011#011... 11 more
Dec 28 09:48:47 liese karaf[5126]: #011Caused by: shaded.org.eclipse.aether.resolution.ArtifactResolutionException: Error resolving artifact org.ops4j.pax.logging:pax-logging-api:jar:2.0.12
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:413)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:215)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:192)
Dec 28 09:48:47 liese karaf[5126]: #011#011at shaded.org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:247)
Dec 28 09:48:47 liese karaf[5126]: #011#011at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:767)
Dec 28 09:48:47 liese karaf[5126]: #011#011... 11 more

This seems to be similar to openHAB 3.2 Release discussion - #6 by timbms .
May be that helps to solve the issue.

Thanks, but I decided to go back to a ssd image before the issue.
I did not try to follow @Kai recommendation though.

It would have been interesting to find out if this would have solved the issue.

HI @Wolfgang_S

yes, that did the trick, BUT ONLY if I restore with a normal backup, not with a full backup. If I do a full backup the error messages are different:

Dec 28 13:41:37 liese systemd[1]: Started openHAB - empowering the smart home.
Dec 28 13:41:43 liese karaf[10672]: org.ops4j.pax.logging.pax-logging-log4j2 [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:43 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.ops4j.pax.logging.pax-logging-log4j2_2.0.9 [6]
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:43 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.ops4j.pax.logging.log4j2.internal.Activator cannot be found by org.ops4j.pax.logging.pax-logging-log4j2_2.0.9
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:796)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:749)
Dec 28 13:41:43 liese karaf[10672]: #011... 13 more
Dec 28 13:41:43 liese karaf[10672]: org.apache.felix.coordinator [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:43 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.apache.felix.coordinator_1.0.2 [9]
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:43 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.apache.felix.coordinator.impl.Activator cannot be found by org.apache.felix.coordinator_1.0.2
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:796)
Dec 28 13:41:43 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:749)
Dec 28 13:41:43 liese karaf[10672]: #011... 13 more
Dec 28 13:41:44 liese karaf[10672]: org.apache.felix.configadmin.plugin.interpolation [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:44 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.apache.felix.configadmin.plugin.interpolation_1.1.2 [13]
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:44 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.apache.felix.configadmin.plugin.interpolation.Activator cannot be found by org.apache.felix.configadmin.plugin.interpolation_1.1.2
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:796)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:749)
Dec 28 13:41:44 liese karaf[10672]: #011... 13 more
Dec 28 13:41:44 liese karaf[10672]: org.apache.felix.configurator [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:44 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.apache.felix.configurator_1.0.14 [14]
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:44 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.apache.felix.configurator.impl.Activator cannot be found by org.apache.felix.configurator_1.0.14
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:796)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:749)
Dec 28 13:41:44 liese karaf[10672]: #011... 13 more
Dec 28 13:41:44 liese karaf[10672]: org.glassfish.hk2.osgi-resource-locator [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:44 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.glassfish.hk2.osgi-resource-locator_1.0.3 [147]
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:44 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.glassfish.hk2.osgiresourcelocator.Activator cannot be found by org.glassfish.hk2.osgi-resource-locator_1.0.3
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:796)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:749)
Dec 28 13:41:44 liese karaf[10672]: #011... 13 more
Dec 28 13:41:44 liese karaf[10672]: org.apache.karaf.deployer.kar [Events.Framework] ERROR : FrameworkEvent ERROR
Dec 28 13:41:44 liese karaf[10672]: org.osgi.framework.BundleException: Error loading bundle activator. org.apache.karaf.deployer.kar_4.3.2 [78]
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:757)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1028)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.doStart(Module.java:605)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.Module.start(Module.java:468)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1849)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1842)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1785)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1747)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1669)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Dec 28 13:41:44 liese karaf[10672]: Caused by: java.lang.ClassNotFoundException: org.apache.karaf.deployer.kar.osgi.Activator cannot be found by org.apache.karaf.deployer.kar_4.3.2
Dec 28 13:41:44 liese karaf[10672]: #011at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
....

Anyway, I never understood the difference between backup and full backup. Backup is working and that’s fine.

But I still think, there is a bug in the backup or restore script. After the clean installation “/var/lib/openhab/etc/org.apache.karaf.features.xml” is like it should be on machine2. The file is also right on machine1, where I do the backup. After restoring the backup on machine2 the file is empty on machine2 (only filled with comments), as mentioned in the linked thread.

Full backup includes cache and tmp folder.

Okay thanks. Didn’t even know, that openHAB makes use of a cache or has temporary files.