Parsing JSON to a web page

OK, so this is a bit off-topic on this forum, but since this place is filled with talented people and the end goal is data integration to OpenHAB, I’ll ask here.
First a bit of background: The city I live in published an event calendar via an API that produces JSON output. I’m fully aware that the individual data could be imported to items, but as I’m only planning on displaying the calendar entries, not actually planning on triggering anything, I’d like to just display a web page in HABpanel. Now, I do confess I know absolutely nothing of javascript so my questions might seem stupid to somebody more knowledgeable. By repeatedly bashing my head to the wall, I’ve come up with the following only partially working solution:

<!DOCTYPE html>
<html>
<body>

<h2 id="demo1"></h2>
<img id="myImg" style="float:right;width:200px;">
<p id="demo3"></p>
<p id="demo2"></p>

<script>
var json = new XMLHttpRequest();
json.open("GET", "https://api.turku.fi/linkedevents/v1/event/?keyword=childfamilies&end=today&page=1&start=today&include=location&format=json", true);
json.responseType = 'json';
json.send();

	json.onload = function() {
    	var pages = parseInt(json.response.meta.count/20)+1;
        console.log(pages);
		var myObj = json.response['data'].filter(function (data) {
	    	var start = new Date(data.start_time)
	        var today = new Date()
	        if (start - today <= 0) {
		        data.start_in_past = 1;
	        } else {
		        data.start_in_past = 0;
	        }
	        if (typeof data.name.fi == 'undefined') {
	        	data.name.fi = data.name.sv;
	        }
	        if (typeof data.description.fi == 'undefined') {
	        	data.description.fi = data.description.sv;
	        }
		    return data.start_in_past==0 && data.sub_events==0;
	    });
    
		console.log(myObj)
		for (var i = 0; i < myObj.length; i++) {
			document.getElementById("demo1").innerHTML = myObj[i].name.fi;
	        document.getElementById("demo2").innerHTML = myObj[i].description.fi;
			document.getElementById("demo3").innerHTML = new Date(myObj[i].start_time);
		    document.getElementById("myImg").src = myObj[i].images[0].url;
	    }
	}


</script>

</body>
</html>

What this manages to do, is read the first (of many pages worth) of data from the API. It then filters the entries to only include single events and then displays one of these events on the web page. I must say I’m pretty damn proud of even getting this far, but this is as far as I can get on my own I’m afraid.
What I need help with is:

  1. as data is spread across multiple pages, they would all have to be read. There’s a meta-section in the data that has the url of the following page (on the last page it is null). There’s also the total number of items and always 20 items per page, so the number of the last page can also be calculated from that (that’s what var pages is for in the code).
    I however have no idea of how to load the multiple pages, other than hard code, which isn’t a good solution as the number of pages isn’t fixed.
  2. When finally all the pages are loaded, the JSON would need to merged/concatenated or done some other magic to so that the data would be in a single array.
  3. When that would finally be done, the page elements would need to be dynamically created so that all the calendar entries are shown and not just the one as it is now.
  4. Then finally I suppose I can just throw the html anywhere where it’s accessible to OpenHAB and render it in a frame on HABpanel?

I’m greatful for any and all help you can provide.

Mikael

Fresh air always makes the brain work better, so I got an idea possibly solving 1 and 2. The idea is, that since page 1 is always loaded anyway, I’ll load it, count the number of pages and store it as the first value in an array. Then load pages 2 to whatever and store them in the same array in a for loop. They data could then be read via nested for loops iterating over the main array and the sub arrays. Does that sound plausible? I won’t have time to test it until tomorrow unfortunately.

I’ve moved this posting to the Habpanel section of the forum. I think it is more likely to get the attention of the right people over there.

This wasn’t necessarily the wrong place to post it, I just think there is better.

Once you get this working I’m sure the community would love a posting to the Habmin widget gallery.

Good luck and thanks for posting!

1 Like

There is a good example of what you can do with HABPanel and a specially designed external controller.
Doing it directly in HABPanel rather than a separate web page means you can directly use AngularJS and other frameworks and controls at your disposal.
Fetching the results of a REST API and displaying a list of results with pagination suddenly becomes not so hard :wink:

Your best resources to learn would be:

Since it’s a good textbook case that could serve as an example I’ll guide you on the right track:
This would be your AngularJS controller for your widget - put it in conf/html/turku.controller.js. It loads the data and puts it in its scope along with some pagination data:

angular
    .module('app.widgets')
    .controller('TurkuEventsController', function ($http, $scope, $filter) {

        var baseUrl = 'https://api.turku.fi/linkedevents/v1/event/?keyword=childfamilies&end=today&start=today&include=location&format=json';

        // this scope function retrieves the current page (in $scope.currentPage) from the API
        $scope.loadData = function () {
            // append the page parameter to the base url
            var url = baseUrl + '&page=' + $scope.pagination.currentPage;

            // get the data and put in the scope
            $http.get(url).then(function (response) {
                $scope.events = response.data.data;

                // also store the number of items for the pagination control
                $scope.pagination.totalItems = response.data.meta.count;
            });
        };

        // put the current time in the scope for easy comparison in the template
        $scope.now = $filter('date')(new Date(), 'yyyy-MM-ddTHH:mm:ssZ');

        // initialize the pagination object
        $scope.pagination = {
            totalItems: 0,
            currentPage: 1
        };

        // fetch the first page
        $scope.loadData();
    });

Next create a template widget and put this code:

<div oc-lazy-load="'/static/turku.controller.js'" style="position: absolute; top: 0; bottom: 0; left: 0; right: 0;">
  <div ng-controller="TurkuEventsController">
    <ul ng-if="pagination.totalItems" uib-pagination
        total-items="pagination.totalItems" items-per-page="20"
        ng-model="pagination.currentPage" ng-change="loadData()"></ul>
    
    <uib-accordion close-others="true"
                   style="text-align: left; color: black;
                          position: absolute; bottom: 10px; right: 10px; top: 60px; left: 10px;
                          overflow: auto">
      <div uib-accordion-group
           class="panel-default"
           ng-repeat="event in events"
           ng-if="event.end_time >= now && event.sub_events.length == 0"
           heading="{{event.start_time | date:'short'}} - {{ event.end_time | date:'short'}}: {{event.name.fi || event.name.sv}}">
        
        <img ng-if="event.images.length > 0" ng-src="{{event.images[0].url}}" class="pull-right" style="height: 200px"></img>
        
        <div ng-bind-html="event.description.fi || event.description.sv"></div>
      </div>
           
    </uib-accordion>
  </div>
</div>

Note how I lazy-loaded the controller on the first line, assigned it to the inner part of the widget using ng-controller, and leveraged ui-bootstrap’s pagination control to do the heavy lifting for the paging stuff and accordions with ng-repeat="event in events" to present the data in a fancy list. The ng-change="loadData()" on the pagination control ensures the data for the appropriate page is retrieved from the API when the page is changed.

The end result after less than one hour is what you’d expect and may not be exactly what you want, but it’s a start:

You can iterate upon this, bonus points if you create a separate post for your widget and include the widgetgallery tag to display it in the widget gallery :slight_smile:

Good luck!

3 Likes

Wow! Just plain wow!
It took me five days to get to how far I got. I’ll definitely read this and the linked threads through thourougly and try to learn as much as I can. Thank you so much for the help!

Mikael

I forgot to thank Rich for moving this post, it didn’t take long for it to get the needed attention after you did so, so thank you!

I think I basically understand everything that’s happening with the code well enough to see what’s happening where and I really like the use of the accordion.
I, not surprisingly, have a further question though. As due to the filtering the different pages in the source have differing numbers of events (could range from 0 to 20), i was thinking if it’s possible to omit the pagination control and instead just load all of the pages to a single accordion? I suppose it really isn’t a question of if it’s possible but rather how to do it really…
Should it be done in the angular or within the template? The only solution I can think of is to ng-repeat over the total number of pages (calculated in a similar way to my cruse js example).

Mikael

If you’re sure you’ll never have more than a handful of pages (<10) I guess you can but it’s not generally considered good practice :slight_smile:

In that case since you always have the URL of the next page in the response you will simply change the processing of a page’s response like this:

[...]
$http.get(url).then(function (response) {
    $scope.currentPage += 1;
    $scope.events = $scope.events.concat(response.data.data);
    $scope.nextPageUrl = response.data.meta.next;
    if ($scope.nextPageUrl) {
          // allow a 250ms delay between the loads
          // (add $timeout to the controller's function arguments after $http)
         $timeout($scope.loadData, 250);
    }
});
[...]

This will call loadData recursively, concatenating the results together until there’s no more pages to load.
You’ll have to initialize the array before the first load with: $scope.events = [], and also eventually get rid of the pagination.

Your case is also a good candidate for so-called “infinite scrolling”, it’s a little more involved but also more elegant. See for instance http://sroze.github.io/ngInfiniteScroll/ (you can lazy-load it along with your own code) or http://jsfiddle.net/vojtajina/U7Bz9/

Once again, thank you so much! In theory the list of events could be very long, but I only plan to list small subsets, so I don’t think I’ll get to more than 10 pages ever. Just checked that e.g. my example query extended for the next week only produces 3 pages more and the others are even shorter.
I’ll definitely have a play with this as soon as I have some more spare time and will post to the widget gallery when I have something to show!

Mikael

Here’s the cleaned up solution with infinite scrolling - it just felt like a good exercise :slight_smile:
It could also be rather easily adapted to other APIs with pointers to the next page:

Controller:

angular
    .module('app.widgets')
    .controller('TurkuEventsController', function ($http, $scope, $filter) {

        // this scope function retrieves the next page from the API
        $scope.loadData = function () {
            if (!$scope.nextPageUrl) return;

            // get the data and concatenate it to the existing array
            $http.get($scope.nextPageUrl).then(function (response) {
                $scope.events = $scope.events.concat(response.data.data);
                $scope.nextPageUrl = response.data.meta.next;
            });
        };

        // put the current time in the scope for easy comparisons in the template
        $scope.now = $filter('date')(new Date(), 'yyyy-MM-ddTHH:mm:ssZ');

        // initialize scope variables and fetch the first page
        $scope.events = [];
        $scope.nextPageUrl = 'https://api.turku.fi/linkedevents/v1/event/?keyword=childfamilies&end=today&start=today&include=location&format=json';
        $scope.loadData();
    })
    // simple directive for infinite scrolling. Credits: http://jsfiddle.net/vojtajina/U7Bz9/
    .directive('whenScrolled', function() {
        return function(scope, elm, attr) {
            var raw = elm[0];
            
            elm.bind('scroll', function() {
                if (raw.scrollTop + raw.offsetHeight >= raw.scrollHeight) {
                    scope.$apply(attr.whenScrolled);
                }
            });
        };
    });

Template:

<div oc-lazy-load="'/static/turku.controller.js'" style="position: absolute; top: 0; bottom: 0; left: 0; right: 0;">
  <div ng-controller="TurkuEventsController">
    <uib-accordion close-others="true"
                   style="text-align: left; color: black;
                          position: absolute; bottom: 10px; right: 10px; top: 10px; left: 10px;
                          overflow: auto" when-scrolled="loadData()">
      <div uib-accordion-group
           class="panel-default"
           ng-repeat="event in events"
           ng-if="event.end_time >= now && event.sub_events.length == 0"
           heading="{{event.start_time | date:'short'}} - {{ event.end_time | date:'short'}}: {{event.name.fi || event.name.sv}}">
        
        <img ng-if="event.images.length > 0" ng-src="{{event.images[0].url}}" class="pull-right" style="height: 200px"></img>
        
        <div ng-bind-html="event.description.fi || event.description.sv"></div>
      </div>
           
    </uib-accordion>
  </div>
</div>
3 Likes

In order to make the infinite scrolling work, one has to put ng-infinite-scroll.min.js into conf/html and lazy-load said file as well, correct?
You know, there isn’t much for me to add to this really, you’ve already perfected it…
Oh and I suppose this needs OpenHAB 2.2 and won’t work on 2.1?

Finally had the time to try this. Downloaded and lazy-loaded the infinite-scroll and it works perfect if the template is small enough that there is scrolling to begin with. Not an issue though, the template will be such a size, that the first page probably always fills it. So thank you once again! Now onto customization!

Mikael

Hello! I can’t make parsing work, I’m trying for the second day and no way :frowning :frowning:
my JSON:
{"events":[{"Event":{"Id":"1344"}},{"Event":{"Id":"1344}}...
already as soon as I did not change the scope

`$http.get(url).then(function (response) {
        $scope.events = response.data.events.Event
    });

           ng-repeat="event in events"
           heading="{{event.Event.Id}}" `