Grafana rendered image fails some time

Apps & Services HABPanel
,
1

I am using a rendered image from Grafana in my HABPanel. The image widget is set to refresh every 60s. Maybe 1 out of 5 times it shows an error.

Image URL is:
http://server:3000/render/d-solo/94Xmp88Zk/temperaturen?orgId=1&from=now-24h&to=now&panelId=5&width=800&height=150&tz=Europe%2FBerlin&fullscreen

To start tracking the root cause, I need to know where this error comes from. Is it Chromium, HABPanel, openHAB, Grafana… ?
I did capture a screenshot: https://imgur.com/a/wx6SwUw

Can anybody tell me which component creates this error?

I am running openHAB 2.5 on a RPi 3 B+ (openhabian), Grafana + influxDB on another host via Docker.

2

When you right-click and save the image and open it in an editor, what does it show? An HTML page?
What does the Grafana log show?

1 Like
3

I did try right click and “open image in new tab” - the desired graph is shown then.

Logs from grafana’s docker container do not show and error. I didn’t increase log level yet.

4

So as you’re saying that it basically works and you get an error “1 out of 5 times” I’d say it’s something during the image-rendering process or that the rendered image isn’t ready within a specific amount of time when accessed by HABpanel.

How long does it take for Grafana to render that image if you open it in via URL?

1 Like
5

It takes less than 2 seconds, the docker host is pretty decent (Xeon CPU E3-1240 v3 @ 3.40GHz)

6

OK, then I’ve got the following questionnaire;

Information on it here.

  • the image-URL (Grafana) is HTTP, no HTTPs (in relation to " Certificate signed by internal certificate authorities")?
  • which Grafana-version do you have? (in relation to “PhantomJS” - there’s now a “Grafana Image Renderer plugin or remote rendering service”)

If I remember correctly there was a topic on this here from end of last year…
EDIT: here is @rlkoshak post

7

I assume OP already uses that given the reference to Chromium:

I suggest to increase the logging level of Grafana and see what comes out of it. And also have a good look at the logging of the rendering container (if you are using that setup…).

(For the record: I am running a Grafana container and a separate rendering container on Linux and it works flawlessly, so it should be possible. :sunglasses:)

8

I am using Grafana Version 6.6.2 (3fa63cfc34). The Webserver is used via unencrypted HTTP.

9

OK, and have you already tried what @noppes123 was saying?

10

Not yet. Planning to do so.

I am still hoping someone recognizes the error pic I’ve linked in the first post. :slight_smile:

11

Don’t hold your breath :smiley:. That only shows it is not a valid image… nothing more.

12

Today I did some digging.

Although Grafana logs the attemp to render the image as successfully:

t=2020-03-08T12:54:05+0000 lvl=info msg=Rendering logger=rendering renderer=phantomJS path="d-solo/94Xmp88Zk/temperaturen?orgId=1&from=now-24h&to=now&panelId=5&width=800&height=150&tz=Europe%2FBerlin&fullscreen&_t=2020-03-08T12:43:24.702Z",
t=2020-03-08T12:54:05+0000 lvl=dbug msg="executing Phantomjs" logger=rendering renderer=phantomJS binPath=/usr/share/grafana/tools/phantomjs/phantomjs cmdArgs="[--ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=false /usr/share/grafana/tools/phantomjs/render.js url=http://localhost:3000/d-solo/94Xmp88Zk/temperaturen?orgId=1&from=now-24h&to=now&panelId=5&width=800&height=150&tz=Europe%2FBerlin&fullscreen&_t=2020-03-08T12:43:24.702Z&render=1 width=800 height=150 png=/var/lib/grafana/png/dFaFkcwddB0Z2HC0jm9K.png domain=localhost timeout=60 renderKey=RM8sPVPWHqkgdVuem0ggdGlgdnuB7Eut]" timezone=Europe/Berlin,
t=2020-03-08T12:54:06+0000 lvl=dbug msg="Phantomjs output" logger=rendering renderer=phantomJS out="Loading a web page: http://localhost:3000/d-solo/94Xmp88Zk/temperaturen?orgId=1&from=now-24h&to=now&panelId=5&width=800&height=150&tz=Europe%2FBerlin&fullscreen&_t=2020-03-08T12:43:24.702Z&render=1 status: success 60000\n",
t=2020-03-08T12:54:06+0000 lvl=dbug msg="Image rendered" logger=rendering renderer=phantomJS path=/var/lib/grafana/png/dFaFkcwddB0Z2HC0jm9K.png,

the file has a size of 0:

bash-5.0$ ls -la dFaFkcwddB0Z2HC0jm9K.png
-rw-r–r-- 1 grafana grafana 0 Mar 8 12:54 dFaFkcwddB0Z2HC0jm9K.png

it happens from time to time, but I didn’t figure out what is causing it. not yet :slight_smile:

bash-5.0$ ls -latr | tail -20
-rw-r–r-- 1 grafana grafana 7819 Mar 8 12:53 Gmlz3PulmZSkEQvgOV9z.png
-rw-r–r-- 1 grafana grafana 7815 Mar 8 12:53 khk5Sm5MWskJ632qQ0L4.png
-rw-r–r-- 1 grafana grafana 7815 Mar 8 12:53 gOsFb64HJoLxXE5wdnmq.png
-rw-r–r-- 1 grafana grafana 7808 Mar 8 12:53 qtwPftsqRPY14B1Qd0mg.png
-rw-r–r-- 1 grafana grafana 7809 Mar 8 12:53 iZpzAayobiTHYxLD1zDP.png
-rw-r–r-- 1 grafana grafana 7829 Mar 8 12:53 LayC5D6DhQ6t4cnn232j.png
-rw-r–r-- 1 grafana grafana 0 Mar 8 12:54 dFaFkcwddB0Z2HC0jm9K.png
-rw-r–r-- 1 grafana grafana 7820 Mar 8 12:54 caWApVQaZwYr5iE5Hzqk.png
-rw-r–r-- 1 grafana grafana 7824 Mar 8 12:54 5N2FAunQ5cC3vNVjgngi.png
-rw-r–r-- 1 grafana grafana 7825 Mar 8 12:55 rSKfEEFUE4tMjJ3DM2ir.png
-rw-r–r-- 1 grafana grafana 7826 Mar 8 12:55 Y766judbm4FLo2BRCoHk.png
-rw-r–r-- 1 grafana grafana 7843 Mar 8 12:57 lvNa8MEX4XBetsxodOFV.png
-rw-r–r-- 1 grafana grafana 0 Mar 8 12:57 Dde2NFKJ4dqjDXRf3hyo.png
-rw-r–r-- 1 grafana grafana 7848 Mar 8 12:58 Qw2wNZHirc36u6JJwnLE.png
-rw-r–r-- 1 grafana grafana 7843 Mar 8 12:59 ElBuPlU17LIEpf5uMgWK.png
-rw-r–r-- 1 grafana grafana 7856 Mar 8 13:00 83Pzi156dQuUtFwWVKS7.png
drwxrwxrwx 4 1000 1000 4096 Mar 8 13:00 …
-rw-r–r-- 1 grafana grafana 11108 Mar 8 13:00 mfTla36Hziir1cIV9tRt.png
-rw-r–r-- 1 grafana grafana 9051 Mar 8 13:01 Dnh9R6F2OPoZ6vb7m4Pd.png
drwx------ 2 grafana grafana 901120 Mar 8 13:01 .

13

seems solved:
I had still running ‘cadvisor’ from my first try with Grafana and influxDB. This did eat up much HDD space and after that it put much load on influxDB and on the host. After stopping that service und removing its data I did not have a any empty files again.

1 Like