REST items list truncated

Setup, Configuration and Use Runtime
1

So I only identified this by playing around with NodeRed, but it seems as though a request to http:///rest/items is randomly truncated which is causing problems with NodeRed and also I would assume anything that is using REST to fetch item data.

I do have quite a few items, and I can’t seem to make any sense of things as it’s basically a random truncation. I have not once had a request truncate in the same place. I tried to add the whole response but the forums won’t let me(too much text), so I’ll copy the last little bit here :

{"link":"http://10.0.0.249:8080/rest/items/AddyMotion_SensorLuminance","state":"23","stateDescription":{"pattern":"%.1f","readOnly":true,"options":[]},"editable":true,"type":"Number","name":"AddyMotion_SensorLuminance","label":"Sensor (luminance)","category":"Temperature","tags":[],"groupNames":[]},{"members":[],"link":"http://10.0.0.249:8080/rest/items/AutoOffTimes","state":"NULL","editable":false,"type":"Group","name":"AutoOffTimes","tags":[],"groupNames":[]},{"link":"http://10.0.0.249:8080/rest/items/DiningRoomSonosFetchQueue","state":"1","stateDescription":{"pattern":"%s","readOnly":false,"options":[]},"editable":false,"type":"String","name":"DiningRoomSonosFetchQueue","label":"Dining Room Sonos FIXME FetchQueue","tags":[],"groupNames":["GDevices"]},{"link":"http://10.0.0.249:8080/rest/items/BRB_AutoOff","state":"ON","editable":false,"type":"Switch","name":"BRB_AutoOff","label":"Downstairs Bathroom Auto-Off","tags":[],"groupNames":["AutomationControl","AutoOffDevices"]},{"link":"http://10.0.0.249:8080/rest/items/LR_SonosPlayPause","state":"OFF","editable":false,"type":"Switch","name":"LR_SonosPlayPause","label":"LR Play/Pause","tags":[],"groupNames"

It’s cutoff right there at the end, if I were to refresh the page it would cutoff somewhere else.

I have had it anywhere from 131,058 characters to 49,149 characters. It’s all over the place, but never complete.

Any ideas?

  • Platform information:
    • Hardware: 64bit/196G/92TB
    • OS: 4.15.0-45-generic #48-Ubuntu SMP
    • Java Runtime Environment: openjdk version “1.8.0_191”
    • openHAB version:2.5.0~S1573-1
  • Issue of the topic: Requesting rest/items returns truncated results
2

Just checked the results from “smarthome:items list” in the console, and I have 617 items total. Last 2 times I refresh, I get to item 155 first time, and 512 the second time. It’s really all over the place.

3

Assuming you’re using the REST API to do this…

http://myhost:8080/rest/items

One thing I’ve noticed… If you do this in a browser, it can take quite a few seconds for the browser to load the complete list. In my case, with over 1600 items, it takes about 8-10 seconds before I can scroll to the end of the list in the browser window. This is on my mac connecting to the linux host running openHAB, so there’s a network connection involved.

OTOH, using wget from the command prompt, it takes about 0.09 seconds to download the entire list. Also from my mac across the network.

mark@macbook:~/tmp$ wget http://myhost:8080/rest/items
--2019-04-18 18:00:18--  http://myhost:8080/rest/items
Resolving myhost (myhost)... 192.168.10.34
Connecting to myhost (myhost)|192.168.10.34|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘items’

items                                                      [ <=>                                                                                                                         ] 950.85K  --.-KB/s    in 0.09s   

2019-04-18 18:00:18 (10.2 MB/s) - ‘items’ saved [973674]
4

Yea, it does and I give it time to complete. Even did the same thing you did and here is what I get :

wget http://10.0.0.249:8080/rest/items
--2019-04-19 08:57:51--  http://wget/
Resolving wget (wget)... failed: Name or service not known.
wget: unable to resolve host address ‘wget’
--2019-04-19 08:57:51--  http://10.0.0.249:8080/rest/items
Connecting to 10.0.0.249:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘items’

items                                                 [  <=>                                                                                                      ]  48.00K  26.8KB/s    in 1.8s

2019-04-19 08:57:54 (26.8 KB/s) - ‘items’ saved [49152]

FINISHED --2019-04-19 08:57:54--
Total wall clock time: 3.2s
Downloaded: 1 files, 48K in 1.8s (26.8 KB/s)

And the resulting file is also truncated.

Here is the second time I run the same line and you can see I get a different size response.

wget http://10.0.0.249:8080/rest/items
--2019-04-19 08:59:49--  http://wget/
Resolving wget (wget)... failed: Name or service not known.
wget: unable to resolve host address ‘wget’
--2019-04-19 08:59:49--  http://10.0.0.249:8080/rest/items
Connecting to 10.0.0.249:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘items.1’

items.1                                               [   <=>                                                                                                     ] 128.00K  41.5KB/s    in 3.1s

2019-04-19 08:59:53 (41.5 KB/s) - ‘items.1’ saved [131072]

FINISHED --2019-04-19 08:59:53--
Total wall clock time: 3.8s
Downloaded: 1 files, 128K in 3.1s (41.5 KB/s)
5

And just to clarify, this was done from the machine that OH is running on so no network involved. Doing it over the network does not make a difference, though the network between my workstation to the server this is on is a 10gb network so it should not make a difference at all.

6 
--2019-04-19 08:57:51--  http://wget/
Resolving wget (wget)... failed: Name or service not known.
wget: unable to resolve host address ‘wget’

That looks strange.

7

dumb question… is wget installed?

8

Yea, that was my mistake, I originally typed “wget wget http…”… Nothing weird there.

9

openHAB “streams” on the items and things endpoint.
Every item/thing is serialized individually and send to the wire.

At the same time openHAB does not send a http length header.

My guess is that either clients are prematurely close the connection,
or the very old jetty we use has bugs and is overwhelmed by the many
small write requests.

10

So nothing can really be done here? Posted above noted they had way more items than I do, so I’m not sure how it’s working in their case but not mine.

11

I remember this is a powerful machine running the host, so maybe to further troubleshoot this issue, try a test set up (running in another docker maybe?) or remove all items files, place only one simple items file, check results, add more items, check again. Make new items files, clear cache and tmp in between rounds. I think you did a great job tracking this issue down when we thought it was node red. Try to eliminate possibilities until the problem shows itself or we can get repeatable results. Sorry not a real solution

12

I have a total of 4 instances, one main one (with most of the items), a remote instance for a detached garage, one staging and one test. 3 of them seem fine when it comes to this only the one “production” is failing. Though it certainly has a majority of the items. (Also noticed that the REST Api page is broken as well, but I think that’s an issue w/ the snapshot.

I’m considering rolling it back to 2.4 to see if the issue is there also. It used to be so much easier to do this when it was item files, now with the JsonDB it’s not so simple so I’ll have to see how I go about troubleshooting this one. I guess my problem is most of the items are not through files, they were mostly added through PaperUI or Habmin, so I’m sort of stuck in regards to “removing some” as it’ll likely break other things.

I’ll see what I can do here while still keeping the wife happy that things work…

13

hmmmm… that is pointing at this one instance having something slightly buggered huh?

yeah… that is mission critical task for sure :grinning:

14

Yup, it’s only the one, but the others have at maximum 10 things, maybe 25 items, so a far smaller data set.

The good news for me is the wife and kids are gone for the day, haha…

15

Just wanted to finish up this post and let anyone know(who may have the same problem some day?) that the issue disappeared after a completely fresh re-install. I did have to literally start from scratch though(adding all my things again and linking the items) as the second I tried to start w/ a backup copy of the jsondb things would go horribly south again.

No idea what was in the old json, but it made things very unhappy.

1 Like
16

FYI: I have reworked the REST interface code and modernized it (3000 lines less code). That work is hopefully in for the final OH 2.5 release, we’ll see. But I do understand why it would only send half of the items from time to time. There was a lot of patch work going on in that 10 years old code.

1 Like
17

Thanks David, greatly appreciated!