I have to leave office right now
I’ll be back
Monday
I found a smoking gun for your case, your logging shows this:
2017-10-05 16:32:31.123 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.328 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.430 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.470 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.611 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.690 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.168 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:31.969 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.032 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.315 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.387 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.394 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.578 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.615 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.723 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.692 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.646 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.677 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.895 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.937 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:32.949 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.024 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.040 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.048 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.493 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.503 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.552 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.568 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.896 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.815 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.600 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.599 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.598 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.584 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.583 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:33.570 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.120 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.185 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.192 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.217 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.227 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.238 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.244 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.426 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.619 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.687 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.748 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.810 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.890 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.598 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.588 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.586 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.548 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.533 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.907 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:36.935 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.035 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.307 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.078 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.567 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.564 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.535 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.480 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.348 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.332 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.882 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.935 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.008 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.019 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.040 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.808 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.116 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.138 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.168 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.177 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.671 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.650 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.645 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.644 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:37.643 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.352 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.259 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.243 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.635 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.646 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.663 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.693 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.702 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.713 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.728 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.605 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.743 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.520 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.460 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.435 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.373 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.358 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.867 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.918 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.958 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.964 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:38.998 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.270 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.291 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.254 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.213 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.408 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.451 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.548 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.210 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.141 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.137 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.136 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.114 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.086 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.057 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.040 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.038 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.664 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.678 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.688 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.698 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.718 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.767 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.527 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.488 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.459 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.348 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.293 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:40.187 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:40.268 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:40.299 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:40.309 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:40.138 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.948 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.921 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.919 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.913 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.912 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.882 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
2017-10-05 16:32:39.839 [WARN ] [thome.ui.internal.chart.ChartServlet] - Illegal argument in chart: No Persistence service found.
...
And if you look at: https://github.com/eclipse/smarthome/blob/master/bundles/ui/org.eclipse.smarthome.ui/src/main/java/org/eclipse/smarthome/ui/internal/chart/ChartServlet.java#L278
It looks like a plausible place for those image to be left open
Can you say in detail where you see a resource being left open? BufferedImage is not a Closable object, so I do not see where a resource leak can originate from that code…
Then the train stopped, so I had to stop looking, I’m expecting mildly that it could be somewhere inside the provider. I didn’t mean that I found it, just the combination of “imageio5104763224528198381.tmp” files and these log look like an interesting combination to me, I was about look into the builder used inside the provider but could not find the source quickly, so I will be looking at the decompiled one first
Edit: some typos
From https://github.com/eclipse/smarthome/blob/master/bundles/ui/org.eclipse.smarthome.ui/src/main/java/org/eclipse/smarthome/ui/internal/chart/ChartServlet.java#L280 I end up at the following code fragment in javax.imageio:
public FileCacheImageOutputStream(OutputStream stream, File cacheDir)
throws IOException {
if (stream == null) {
throw new IllegalArgumentException("stream == null!");
}
if ((cacheDir != null) && !(cacheDir.isDirectory())) {
throw new IllegalArgumentException("Not a directory!");
}
this.stream = stream;
if (cacheDir == null)
this.cacheFile = Files.createTempFile("imageio", ".tmp").toFile();
else
this.cacheFile = Files.createTempFile(cacheDir.toPath(), "imageio", ".tmp")
.toFile();
this.cache = new RandomAccessFile(cacheFile, "rw");
this.closeAction = StreamCloser.createCloseAction(this);
StreamCloser.addToQueue(closeAction);
}
Looking at it, I would say that it could maybe also be a consequence of an almost out-of-memory or other system problems. The code which I copied is part of the JDK so it should be rather unique if it gives problems, although I do not really like the part where the implementation states that it closes the images when the vm closes, so I prepared a PR: https://github.com/eclipse/smarthome/pull/4380
I think this is not the only problem which you are facing so maybe a heap-dump will give more insights.
Please also not limit yourself to a lot of these lines, but give me/us a text file with all of them, because to me they do not look the same
I can provide now a clean ‘bug shot’, which can be downloaded here:
http://kulturnetz-drensteinfurt.de/Medien/Log4HABDebug_02.zip
It includes a readme describing how the error has been reproduced.
If you need additional information please give me some instructions what has to be dumped from the raspberry.
That is interesting so its all about the non closed connections, because the number of graphs being left behind is really limited.
And you are running openHABian with the Zulu jdk?
Can others reproduce this easily?
I would really be interested in a heap dump but can’t find what tools are supported by zulu for that, can you validate whether this works: https://stackoverflow.com/questions/407612/how-to-get-a-thread-and-heap-dump-of-a-java-process-on-windows-thats-not-runnin
Can you also run:
lsof -P
lsof -n
Whatever gets the real values of the ports instead http-alt
I assume its just 8080 but its good to know for sure
I’m Running a Zulu for PIZeroW
pi@MeteringPiZeroW:~ $ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu Embedded 8.20.0.42-linux-aarch32hf) (build 1.8.0_121-b42)
OpenJDK Client VM (Zulu Embedded 8.20.0.42-linux-aarch32hf) (build 25.121-b42, mixed mode, Evaluation)
and a HotSpot for PI3
pi@MeteringPiRed:~ $ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) Client VM (build 25.65-b01, mixed mode)
On both PIs the same behaviour. As soon as the Chrome Refresh is pressed the problem occurs.
Here are the Download Links for the PI3 Heaps:
http://kulturnetz-drensteinfurt.de/Medien/heap_01.bin
http://kulturnetz-drensteinfurt.de/Medien/heap_02.bin
Here are the requested outputs:
http://kulturnetz-drensteinfurt.de/Medien/lsof_n.txt
http://kulturnetz-drensteinfurt.de/Medien/lsof_p.txt
I’m having an hard time opening them, they also look really small. Can you validate that the processes uses only 10 MB?
I got an exception during generating the heap dump:
Dumping heap to heap_03.bin ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jmap.JMap.runTool(JMap.java:201)
at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: java.lang.InternalError: Metadata does not appear to be polymorphic
at sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:278)
at sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102)
at sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:68)
at sun.jvm.hotspot.memory.DictionaryEntry.klass(DictionaryEntry.java:71)
at sun.jvm.hotspot.memory.Dictionary.classesDo(Dictionary.java:66)
at sun.jvm.hotspot.memory.SystemDictionary.classesDo(SystemDictionary.java:190)
at sun.jvm.hotspot.memory.SystemDictionary.allClassesDo(SystemDictionary.java:183)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClasses(HeapHprofBinWriter.java:954)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:427)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83)
First result on Google:
Check the user which runs openhab vs the user you try to create the dump with
Here is a snap shot of what happened during a ‘Chrome Refresh’ on the PI3. The system is loosing the control of processes and IO
pi@MeteringPiRed:~/bin $ sudo ./startPerformanceMonitoring.sh
13:10:27 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 917767 0 38124 899040 13354 0 3210 0 0 0
13:10:37 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 917787 0 38139 903092 13356 0 3210 0 0 0
13:10:47 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 917979 0 38167 906956 13357 0 3210 0 0 0
13:10:58 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 917995 0 38181 911011 13357 0 3210 0 0 0
13:10:08 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 918351 0 38229 914761 13364 0 3215 0 0 0
13:10:18 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 918362 0 38257 918942 13364 0 3215 0 0 0
13:10:29 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 918405 0 38294 922963 13365 0 3217 0 0 0
13:10:39 --- CLOSE_WAIT: 0 --- CPU_STAT: cpu 918428 0 38323 927005 13367 0 3217 0 0 0
13:10:50 --- CLOSE_WAIT: 154 --- CPU_STAT: cpu 921434 0 38434 928022 13367 0 3226 0 0 0
13:10:00 --- CLOSE_WAIT: 316 --- CPU_STAT: cpu 924840 0 38534 928621 13367 0 3244 0 0 0
13:10:10 --- CLOSE_WAIT: 511 --- CPU_STAT: cpu 928567 0 38665 928999 13367 0 3264 0 0 0
13:10:21 --- CLOSE_WAIT: 657 --- CPU_STAT: cpu 932107 0 38779 929488 13367 0 3280 0 0 0
13:10:31 --- CLOSE_WAIT: 869 --- CPU_STAT: cpu 935709 0 38881 930004 13367 0 3300 0 0 0
13:10:42 --- CLOSE_WAIT: 934 --- CPU_STAT: cpu 939342 0 38980 930467 13367 0 3319 0 0 0
13:10:53 --- CLOSE_WAIT: 826 --- CPU_STAT: cpu 943116 0 39041 930885 13367 0 3320 0 0 0
13:10:03 --- CLOSE_WAIT: 721 --- CPU_STAT: cpu 946881 0 39144 931312 13367 0 3323 0 0 0
13:10:14 --- CLOSE_WAIT: 605 --- CPU_STAT: cpu 950653 0 39221 931733 13367 0 3325 0 0 0
13:10:25 --- CLOSE_WAIT: 493 --- CPU_STAT: cpu 954365 0 39293 932175 13368 0 3328 0 0 0
13:10:35 --- CLOSE_WAIT: 386 --- CPU_STAT: cpu 958126 0 39357 932601 13368 0 3330 0 0 0
13:10:46 --- CLOSE_WAIT: 283 --- CPU_STAT: cpu 961826 0 39418 933061 13368 0 3335 0 0 0
13:10:56 --- CLOSE_WAIT: 172 --- CPU_STAT: cpu 965586 0 39467 933498 13368 0 3336 0 0 0
13:10:07 --- CLOSE_WAIT: 79 --- CPU_STAT: cpu 969328 0 39549 933872 13368 0 3340 0 0 0
13:10:18 --- CLOSE_WAIT: 2 --- CPU_STAT: cpu 973039 0 39619 934281 13368 0 3343 0 0 0
13:10:28 --- CLOSE_WAIT: 206 --- CPU_STAT: cpu 976940 0 39720 934473 13368 0 3370 0 0 0
13:10:39 --- CLOSE_WAIT: 376 --- CPU_STAT: cpu 980638 0 39801 934909 13368 0 3385 0 0 0
13:10:49 --- CLOSE_WAIT: 594 --- CPU_STAT: cpu 984336 0 39899 935341 13368 0 3410 0 0 0
13:10:00 --- CLOSE_WAIT: 811 --- CPU_STAT: cpu 987988 0 40007 935814 13368 0 3431 0 0 0
13:10:11 --- CLOSE_WAIT: 1031 --- CPU_STAT: cpu 991954 0 40148 935983 13368 0 3457 0 0 0
13:10:21 --- CLOSE_WAIT: 1232 --- CPU_STAT: cpu 997565 0 40283 936520 13368 0 3493 0 0 0
13:10:37 --- CLOSE_WAIT: 1556 --- CPU_STAT: cpu 1001307 0 40402 936993 13368 0 3525 0 0 0
13:10:48 --- CLOSE_WAIT: 1769 --- CPU_STAT: cpu 1004952 0 40574 937417 13368 0 3574 0 0 0
13:10:59 --- CLOSE_WAIT: 1980 --- CPU_STAT: cpu 1008984 0 40739 937594 13369 0 3609 0 0 0
13:10:10 --- CLOSE_WAIT: 2198 --- CPU_STAT: cpu 1012765 0 40902 937995 13369 0 3644 0 0 0
13:10:21 --- CLOSE_WAIT: 2428 --- CPU_STAT: cpu 1016923 0 41053 938171 13369 0 3665 0 0 0
13:10:32 --- CLOSE_WAIT: 2662 --- CPU_STAT: cpu 1020760 0 41213 938628 13370 0 3686 0 0 0
13:10:44 --- CLOSE_WAIT: 2812 --- CPU_STAT: cpu 1024514 0 41353 939093 13370 0 3707 0 0 0
13:10:54 --- CLOSE_WAIT: 2703 --- CPU_STAT: cpu 1028728 0 41490 939249 13370 0 3717 0 0 0
Were you able to create the heap-dump using the user under which openHAB is running?
Yes that one works, it has 2751 connections ‘open’/
- can you enable some debug logging:
http://docs.openhab.org/administration/logging.html
- Can you check the Chrome developer tools whether the browser
serverreally fired so many requests, especially the network tab is relevant, just check what requests are being fired at the server, most likely requests fail and are somehow retried and fail again, at least that is what I hope
maybe a wireshark file for the traffic between http client and host?
Great