Astro:sun:night#start channel UNDEF and others too

Add-ons Bindings
,
23

Hi,

Great that you wrote fix. But the Travis CI build failed.

I think that is good idea. UNDEF for me, mean something wrong :wink: NOT_PRESENT is more natural.

I don’t think it is a bug.
Look at https://www.timeanddate.com/sun/poland/warsaw on e.g. 26 of May. The dusk start at evening of 25 and it doesn’t stop till midnight (so it can stop at 24:00). And on 26 of May the Dawn stop at the morning, but never start (so it can start at 00:00). Astronomical Twilight is from: 00:00 - 02:35 and from 22:32 - 00:00.
It also should be NOT_PRESENT, not UNDEF.

24

Build fails because there is something wrong with Smartmeter Binding 2.5.0-SNAPSHOT. You need to take a closer look into build log to find out. It looks like Travis builds all bindings together.

Ok, I will take a look at this and go back to you later.

25

UNDEF is explicitly supposed to be used in this sort of situation. UNDEF essentially means NOT_PRESENT. I think a sentence added to the README to explain what it means when one sees UNDEF would be more appropriate (and easier to get approved) than adding a new special State type to the core, especially since that new State type is already supposed to be covered by the existing UNDEF. Adding NOT_PRESENT or replacing NOT_PRESENT with UNDEF will require changes to the OH core and will ripple out to several other bindings, so the change cannot be taken lightly.

The main reason why UNDEF is confusing to some users is it isn’t used all that much by the bindings. But I’m happy to see more and more bindings use it just like Astro does, to indicate that there is no State or the State cannot be known (e.g. when the MQTT Broker Thing goes offline, all of the Channels that use that Broker go to UNDEF to indicate we have no way to know the current states).

1 Like
26

That’s just a matter of interpreting the language used. UNDEF stands for undefined, not for error. It can mean error, but it can also mean can’t calculate that, as in “what weekday is 29th Feb this year?” and it can mean “I can’t get that data at this time” because a device is busy.

27

Do you mean astro docs?

28

Yes, that is what I mean.

29

The definition of UNDEF comes from the OpenHAB Core library. Here is how it is commented in the code:

/**
 * There are situations when item states do not have any defined value.
 * This might be because they have not been initialized yet (never
 * received an state update so far) or because their state is ambiguous
 * (e.g. a dimmed light that is treated as a switch (ON/OFF) will have
 * an undefined state if it is dimmed to 50%).
 */

My impression about UNDEF - in general - is that the value is undefined. But we do not know what is the root caus of this. It may be as stated in the astro docs:

But it also may be a result of a bug inside of astro - just like in case of reported #5006 issue. Another thing is when OpenHAB service is during startup and the state of thing is really undefined yet.

I have the feeling that Astro should never deliver UNDEF, because every status/phase/range can be well known by doing a correct calculations, so the state that NIGHT is not present could be indicated by something different than UNDEF.
For now we can also live with UNDEF but it would be good eliminate as much of those as we can.

@majherek, thanks for the link to this online datetime calulator. I have took a look at th 19th and 20th of May in Warsaw and I have prepared the table of what states/ranges should generate Astro binding to coved those two days correctly (in my opinion):

   ASTRO_DUSK = 19th May 00:12 -> 19th May 00:12		
        NIGHT = 19th May 00:12 -> 19th May 00:51
EVENING_NIGHT = 19th May 00:12 -> 19th May 00:32 (NOON + 12h)
MORNING_NIGHT = 19th May 00:32 -> 19th May 00:51
   ASTRO_DAWN = 19th May 00:51 -> 19th May 02:51
  NAUTIC_DAWN = 19th May 02:51 -> 19th May 03:52
   CIVIL_DAWN = 19th May 03:52 -> 19th May 04:35
     SUN_RISE = 19th May 04:35 -> 19th May 04:35
     DAYLIGHT = 19th May 04:35 -> 19th May 20:29
	 NOON = 19th May 12:32
      SUN_SET = 19th May 20:29 -> 19th May 20:29
   CIVIL_DUSK = 19th May 20:29 -> 19th May 21:13
  NAUTIC_DUSK = 19th May 21:13 -> 19th May 22:15
   ASTRO_DUSK = 19th May 22:15 -> 20th May 00:00
   
   ASTRO_DUSK = 20th May 00:00 -> 20th May 00:32 (NOON + 12h)  
        NIGHT =    NOT_PRESENT -> NOT_PRESENT
EVENING_NIGHT =    NOT_PRESENT -> NOT_PRESENT
MORNING_NIGHT =    NOT_PRESENT -> NOT_PRESENT 
   ASTRO_DAWN = 20th May 00:32 -> 20th May 02:49
  NAUTIC_DAWN = 20th May 02:49 -> 20th May 03:50
   CIVIL_DAWN = 20th May 03:50 -> 20th May 04:34
     SUN_RISE = 20th May 04:34 -> 20th May 04:34
     DAYLIGHT = 20th May 04:34 -> 20th May 20:31	 
	 NOON = 20th May 12:32	 
      SUN_SET = 20th May 20:31 -> 20th May 20:31
   CIVIL_DUSK = 20th May 20:31 -> 20th May 21:15	  
  NAUTIC_DUSK = 20th May 21:15 -> 20th May 22:17
   ASTRO_DUSK = 20th May 22:17 -> 21th May 00:00

Astro calculates a sun rise/set as a range with about 4 minutes duration (not included above). I have also use NOT_PRESENT in the example. With this representation all of the ranges will be consistent with the astronomical events. Unfortunately Astro binding is not able to deliver two ASTRO_DUSK events/ranges for the specific date (there a two ASTRO_DUSKs at 20th May 2019 in Warsaw). I do not know how can it be “fixed”. One of the option could be to deliver the closesd range/event, so between midnight and noon astro will devliver

   ASTRO_DUSK = 20th May 00:00 -> 20th May 00:32 (NOON + 12h)

and after the noon it will deliver

   ASTRO_DUSK = 20th May 22:17 -> 21th May 00:00

What do you think guys?

30

Opposite opinion from me. If there is no sunrise today then today’s sunrise time is not defined. UNDEF is entirely appropriate,

Barring genuine bugs, there is no other reason for Astro to set UNDEF than this kind of situation. Adding some other new state (which would have to affect all Items of DateTime state, right across openHAB), some new state like NOT_PRESENT doesn’t add any information. Whether it says UNDEF or NOT_PRESENT you know what it means - there’s no answer for that question today.

If it’s not been calculated yet after start up, it will be NULL. The description from core is a bit misleading here - Items are created NULL and stay that way until updated by some action.

The fact that something may get set to UNDEF due to some bug is not I think relevant. A bug could just as well cause unexpected setting of NOT_PRESENT or 23:59 1/1./2099. A bug is a bug, an erroneous value is an erroneous value.

31

Thanks for clarification around this NULL value. However I have read this topic again and I see that @rlkoshak has also explained this before. Sorry - my bad.

Yes, I need to agree with this.

@rossko57, so you propose something like this: in Astro binding UNDEF means that the specific event/phase/range/date/time is not present at the specific date and location. Other words if night_startTime is UNDEF, then it means that there is no night at the specific time and place. This would be also backward compatible so there will be no need to update the rules in the openHAB system. I can understand this point of view.
So now - with this aproach - we confirmed that Astro binding can return UNDEF for some specific channel, which - under some specific conditions (like high latitude locationas at the June doesn’t have NIGHT) - is an expected behavior. Unless somebody says, that for his location at this specific date the date of event is wrongly calculated, which should be treated as a bug (then it should be fixed in Astro binding).

1 Like
32

:+1: I completely agree. If there is no such time today, then UNDEF is completely appropriate. There is nothing about the state UNDEF that implies the Item is UNDEF because of some sort of problem. Nor should there be IMHO. If we don’t know the state of a Channel, the Channel is in an undefined state. There are other ways to determine if there is something wrong (e.g. the Offline state for Channels).

If an Item gets set to UNDEF erroneously has no bearing on the argument for the inclusion of a new NOT_PRESENT state.

There is already enough confusion on the difference between UNDEF and NULL where the distinction is pretty clear cut. The distinction between UNDEF and NOT_PRESENT is going to be much too subtle and it will cause a ton of confusion for users.

I do agree, if a Channel is calculable by Astro that Channel should have a valid state. But if it isn’t then UNDEF is the proper state.

1 Like
33

Don’t apologise, it’s useful to air these things.
I think the NULL / UNDEF distinction was not nearly as clear cut in OH1 (where we had Undefined instead) and there may be woolly comments like the one you found that come from that historical background.

34

Great question Witold and great discussion gentlemen

And of course you can test for UNDEF in a rule with an if statement and have your interface of choice display whatever you want under those conditions

35

Ok, so the UNDEF variable is for now clarified.

I have just one question to clarify. Please go back to my previous post where I have put an example of all phases on 19th and 20th May in Warsaw. There are two ASTRO_DUSKs (one right after the midnight on 19th May, second right before midnight of the same day) ranges at those days. Astro binding can only provide one ASTRO_DUSK range at the time. Which one should be provided?
And second thing: if some specific range overlaps the midnight, then what should provide Astro? For example there is the following Astro Dusk

19th May 22:15 -> 20th May 00:32

So for example on the 19th May 2019, what should Astro provide:
the full range
ASTRO_DUSK = 19th May 22:15 -> 20th May 00:32
or truncated to midnight range
ASTRO_DUSK = 19th May 22:15 -> 20th May 00:00
or UNDEF as the astro dusk doesn’t end at th 19th May
ASTRO_DUSK = 19th May 22:15 -> UNDEF
?
Looks like https://www.timeanddate.com always truncates the date/time to midnight.

36

Good point. No idea what can be done about that.
There’s only one of each static channel type. So let’s populate that with the next event, just after midnight.
When Astro next recalculates at midnight - it will have already missed the second one for what is now the previous day.

A dirty fix might be to have it recalculate at noon as well as midnight.

Not sure what you mean here; do you mean the phase name string channel? That really just tells you what the last event was out of a set of particular events. e.g. ASTRO_DAWN, NAUTIC_DAWN, CIVIL_DAWN …

I would say sun phase name would change to ASTRO_DUSK at 19th May 22:15, there would be no NIGHT phase today I think?, then change to ASTRO_DAWN at 20th May 00:32

37

Let’s take as an example a simple ASTRO_DUSK channel; actually there are three channels:

astroDusk#start
astroDusk#end
astroDusk#duration

In more cases the night starts before midnight and ends after midnight. Let’s assume that at some specific date we have astroDusk#start = 16 Jun 23:10 astroDusk#end = 17 Jun 00:36
What should say Astro in those two channels (for now let’s forget what is Astro doing now):
astroDusk#start = 16 Jun 23:10 astroDusk#end = 17 Jun 00:36
or
astroDusk#start = 16 Jun 23:10 astroDusk#end = 17 Jun 00:00
or
astroDusk#start = 16 Jun 23:10 astroDusk#end = UNDEF
?
The https://www.timeanddate.com shows the data always truncated to midnight, but I’m not quite sure now if this is a good way to go for Astro. If Astro shows on 16 Jun something like:
astroDusk#start = 16 Jun 23:10 astroDusk#end = 17 Jun 00:00
then on 17 Jun it needs to theoretically show:
astroDusk#start = 17 Jun 23:10 astroDusk#end = 17 Jun 00:36
which will lead to a false astroDusk#start event to be triggered at the midnight, which is incorrect. Astro should show a reall times and not truncate them to midnight and also it should not set UNDEF.

For example, my test openHAB installation shows now ASTRO_DUSK as below:

23:24:12.654 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_start updated to 2019-06-16T23:10:00.000+0200
23:24:12.670 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_end updated to UNDEF
23:24:12.670 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_duration updated to UNDEF

Those UNDEFS could be replaced with a real dates, however those dates are tomorrow days. I do not know - maybe I’m chasing a ghost here?

38

Are not all times Astro calculates for today? As in the actual day that it is?
And then at midnight it recalculates everything for that next about to start day?
maybe I don’t understand

39

Can’t comment on what you want, but so far as I can see current behaviour is that at midnight or boot time, events are calculated for today’s calendar day.
Any particular event may not happen today, as we know (UNDEFs)

Your astroDusk examples are essentially the same case as “ordinary” sun:night channels at low latitudes.
Does #duration work for that, night spanning across midnight? Which of the two night periods in today’s date does it refer to? Never looked at it.
If that does work, I would say your astroDusk duration UNDEF observation is a genuine bug.

Curious what happens in the Arctic … does #duration give months? Would you want next sunrise to be set for months ahead?

I wonder if “solving” the odd but real case of what to do about two of the same kind of events in one calendar day would umm, throw some light on possible improvements.

I do not think we want to discard the notion of providing events for today?
Example, we boot up and want to look at sunrise and sunset to determine if it is daylight. We want today’s events, even if they are in the past “now”.

Equally, there are uses for knowing tomorrow’s events, perhaps we want to schedule something for next dawn. Perhaps thinking of the Arctic case helps here … maybe not tomorrow’s sunrise, but just the next sunrise after “now”?

Not thought through, but perhaps each event could provide an extra DateTime channel #next.
Sometimes that would be the same as today’s #start because that hasn’t happened yet, sometimes that would be in the future, maybe tomorrow or next month.
There is a recalculation cost to that, which should not be high because few will use this stuff .
But as Astro generates events anyway, maybe it is not a large burden to have each event fire a recalculation of it’s own #next.

40

In Astro there is something like PositionalJob and this is executed as how often as configured in the thing refresh interval. I have chenged it in my openHAB test environment to 15 seconds, so the channel values are updated every 15 seconds.
There is also a DailyJobSun which is executed 30 seconds after the midnight and registers the assynchronous triggers for all of the astro events (so it seems to be possible to set a thing refresh interval to some high value - like every 6 hours - and those asynchronous triggers will publish events at the correct time anyway).

I have made a short tests around passing the midnight from 16th Jun to 17th Jun, for city Warsaw in Poland.
Before midnight I had something like below in the log:

23:59:57.285 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_start updated to 2019-06-16T23:10:00.000+0200
23:59:57.349 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_end updated to UNDEF
23:59:57.355 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_duration updated to UNDEF
23:59:57.460 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_phase_name updated to ASTRO_DUSK

It says UNDEF for astroDusk_end and astroDusk_duration, however the phase_name phase name is calculated correctly to ASTRO_DUSK. In the source code - I see - that to calculate the phase name it silently assumes that ASTRO_DUSK ends at midnight however it says that the end time is UNDEF.
Interesting thing is when the date changes, right after the midnight:

00:00:12.293 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_start updated to 2019-06-17T23:11:00.000+0200
00:00:12.297 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_end updated to UNDEF
00:00:12.302 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDusk_duration updated to UNDEF
00:00:12.247 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDawn_start updated to UNDEF
00:00:12.252 [safeCall-1] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDawn_end updated to 2019-06-17T02:06:00.000+0200
00:00:12.257 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_astroDawn_duration updated to UNDEF
00:00:12.494 [safeCall-3] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_phase_name updated to ASTRO_DAWN

Right after the midnight Astro calculates the DUSK to be at the end of the current date but it wrongly calculates that we are in ASTRO_DAWN since the midnight, and the current sun phase is wrongly calculated to ASTRO_DAWN. The correct result should be:

ASTRO_DUSK till 17 May 00:36
ASTRO_DAWN from 17 May 00:36 till 02:04

NAUTIC_DAWN in my case is calculated correctly.

Another interesting thing. I have changed the date in my computer to 20 Apr 2019, to see how Astro calculates NIGHT. Here it is:

09:44:27.416 [safeCall-3] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_night_start updated to 2019-04-20T21:59:00.000+0200
09:44:27.422 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_night_end updated to 2019-04-21T03:10:00.000+0200
09:44:27.429 [safeCall-2] INFO  smarthome.event.ItemStateEvent:53 - astro_sun_8c2f8c23_night_duration updated to 311 min

Its startTime is on 20th Apr but endTime is on 21th Apr. The duration is calculated correctly (as a difference of endTime and startTime). So for NIGHT it desn’t truncate to midnight but it shows the real night end on the next day.
Could Astro do the same with ASTRO_DUSK, NAUTIC_DUSK and CIVIL_DUSK?

I have the feeling that Astro was at first designed to provide events related to the specific date but we observe a weired behavior around midnight for a high latitude locations. I also feel that Astro should deliver events related to the current day but there are some special cases when we have two the same events at the same date (like two ASTRO_DUSK or two NAUTIC_DUSK and so on) but Astro is able to deliver only one of them at the time. So maybe it could provide events for the current date but - in case of doubled events - it should provide the closesd one? This means that Astro should also reschedule the event triggering after the NOON?

Also the Sun’s noon is not alwasy at 12:00. In the north latitude on earth the Sun’s noon is around 12:30 on Jun and around 11:30 on Dec. The same relates the Sun’s midnight: it varies from 23:30 to 00:30, so the ASTRO_DUSK may end between 23:30 and 00:30.

This is a good question. At least the current sun phase name should be correctly calculated to DAYLIGHT or NIGHT.
From one side - we take again an example of ASTRO_DUSK - if Astro truncates the ASTRO_DUSK to midnight, then it should generate two ASTRO_DUSKs:

ASTRO_DUSK = 16th Jun 23:10 -> 17th Jun 00:00
ASTRO_DUSK = 17th Jun 00:00 -> 17th Jun 00:36

then astroDusk#start will be generated twice. What if I have a rule in my openHAB, to switch on some lights for 30 minutes on every astroDusk#start? Then I expect the lights to be on only between 23:10 and 23:40, but not on 00:00 and 00:30.

41

Thank you very much for your effort in the research for a solution @wmarkow . I tried to follow this thread during my holidays. A lot of things have been mentioned and taken into account for a proper solution. From my point of view we should not truncate the times to midnight for any phase. We should do it similar to the NIGHT phase. Most important reason are the false events. I do not think it is a good solution to suppress events which could lead to confusion or false positives too. But IIRC even when choosing this approach it could be possible to see two ASTRO_DUSKs per day. Couldn’t it? Thus handling two similar phases per day must be possible. Which matches your first proposal.

2 Likes
42

I have a small update on the issue. Indeed a midnight truncate is some kind of artifical border. It well good to understand for humans, because the date changes at midnight, however this midnight border is not consistent with Earth natural rotation. A better “truncate moment” would be a “true midnight”, which is moment where the Sun has the lowest elevation below horizont. However on some places at some specific time the Sun never goes under horizon, however at true midnight it will have the lowest elevation.
For example - if night is present - then an evening night ends always at true midnight and a morning night begins always at true midnight. Another example: if a night is not present but astro twilight is present, then astro dusk always ends at true midnight, and astro dawn starts always at true midnight, and so on.

Astro has one special range which is called NIGHT. The specific property of this range is that it overlaps a two dates (before midnight and after midnight).

To overcome all of those events Astro library need to make a special care for calculation of events:

  • if the current time is between midnight and true midnight then it mean that the current sun phase is EVENING_NIGHT of the previous day (if night is present at this specific day) and Astro shouold deliver the NIGHT as a night which passes between those two dates. Also the EVENING_NIGHT should be delivered as from yesterday date, and MORNING_NIGHT should be delivered as from today date
  • if the current time is after true midnight then we are currently in MORNING_NIGHT of current date, however we are still in the NIGHT between yesterday and today
  • if the current time is after current NIGHT finishes, then it mean that we are in ASTRO_DAWN but astro should recalculate the NIGHT range to bo the next night available (so between current date and tomorrow). MORNING_NIGHT should be from current date and EVENING_DATE should be also from current date.
  • a special care must be taken if we currently are in a full daylight like in north subpolar locations at June. Then Astro could calculate the real daylight start time and end time and also the duration for this can give months, like @rossko57 has previously asked.

I have managed to rework Astro library to support those cases mentioned above. Some kind of refactoring needed to be done. Previously everything was calculated by a SunCalc calculator, which was hard for testing. After refactoring and reimplementing the following calculators are available in Astro:

  • SunPositionCalc refactored from SunCalc - it calculates the sun position
  • SunRadiationCalc refactored from SunCalc - it calculates sun radiation info
  • SunDailyEventsCalc refactored from SunCalc - it calculates a basic daily sun events, like sun rise, sun set, astro dawn start, nautic dawn start, civil dawn start, noon, true midnight, and so on
  • SunDailyRangesCalc - implemented by me. Basing on SunDailyEventsCalc it creates a sun daily ranges, so it is able to deliver correct values for NIGHT, MORNING_NIGHT, EVENING_NIGHT, ASTR_DAWN, and so on. It takes care for midnigh, true midnight transitions, full daylight, etc.
  • SunCalc I left for backward compatibility but it is refactored and it is more or less an aggregation of all other calculators.

Moreover I have implemented unit tests to cover the special cases:

  • test for Murmansk location Russia north subpolar for a full daylight in June
  • test for Alert location in Canada (also subpolar) for a full daylight in June and night and astro twilight in December
  • test for McMurdo location in Antarctica (south subpolar) for a full daylight in December

Those changes are available in my astro-fixes branch.
One thing is still missing, the astro events must be recalculated not only after midnight, but also shortly after true midnight and shortly after night finishes (if night is present).

Please let me know what you think. If would be good if you guys could build the astro binding at test it in your environments.