Sending strings to Number-item is not locale independent!

Spaceman_Spiff · February 29, 2016, 7:48pm

Hello,

in the following rule I try to post a string update to a number item:

        logDebug( "Keller", String::format( "%15s | ", "Get"))
        var min1 = Keller_Waschmaschine_Energie.averageSince(now.minusMinutes(1)) as DecimalType
        
        logDebug( "Keller", String::format( "%15s | ", "Cast"))
        var String test = String::format( "%.1f", min1.floatValue)
        
        logDebug( "Keller", String::format( "%15s | -> '%s'", "Post", test))
        postUpdate( Keller_Waschmaschine_Energie_1min, test)
        
        test = test.replace(",", ".")
        logDebug( "Keller", String::format( "%15s | -> '%s'", "Post replace", test))
        postUpdate( Keller_Waschmaschine_Energie_1min, test)
        
        logDebug( "Keller", String::format( "%15s | ", "Old"))
        Keller_Waschmaschine_Energie_1min.postUpdate( Keller_Waschmaschine_Energie.averageSince(now.minusMinutes(1)))
        Keller_Waschmaschine_Energie_5min.postUpdate( Keller_Waschmaschine_Energie.averageSince(now.minusMinutes(5)))
        logDebug( "Keller", String::format( "%15s | ", "done"))

This produces the following output:

2016-02-29 20:39:05.691 [DEBUG] [rg.openhab.model.script.Keller] -             Get |
2016-02-29 20:39:05.701 [DEBUG] [rg.openhab.model.script.Keller] -            Cast |
2016-02-29 20:39:05.707 [DEBUG] [rg.openhab.model.script.Keller] -            Post | -> '0,2'
2016-02-29 20:39:05.713 [WARN ] [.c.i.events.EventPublisherImpl] - given new state is NULL, couldn't post update for 'Keller_Waschmaschine_Energie_1min'
2016-02-29 20:39:05.717 [DEBUG] [rg.openhab.model.script.Keller] -    Post replace | -> '0.2'
2016-02-29 20:39:05.726 [DEBUG] [rg.openhab.model.script.Keller] -             Old |
2016-02-29 20:39:05.746 [DEBUG] [rg.openhab.model.script.Keller] -            done |

I am running a pi with a german locale.
Note how the postUpdate with the comma fails, which is the german standard for a floating point number.
When I change it to the english format with the string.replace() everything works fine!
This hast confused other users already and is most likely a bug!

Spaceman_Spiff · March 2, 2016, 7:39am

Can anybody reproduce this?

watou · March 2, 2016, 8:44am

You would have to use this version of String::format:

https://docs.oracle.com/javase/6/docs/api/java/lang/String.html#format(java.util.Locale, java.lang.String, java.lang.Object…) Passing a Locale that formats the number in the expected format.

Or better yet, don’t pass it through a string at all, just

var DecimalType test = new DecimalType(min1.floatValue);

and then just postUpdate of test directly.

Spaceman_Spiff · March 2, 2016, 4:07pm

This doen’t work for the same reasons:

Test-Rule:

    logDebug( "Bad", "test dot")

    var DecimalType test = new DecimalType( "1.2")
    Keller_Waschmaschine_Energie.postUpdate(test)
    
    logDebug( "Bad", "test comma")

    var DecimalType test = new DecimalType( "1,2")
    Keller_Waschmaschine_Energie.postUpdate(test)
    
    logDebug( "Bad", "done")

Output:

2016-03-02 17:01:19.009 [DEBUG] [org.openhab.model.script.Bad  ] - test dot
2016-03-02 17:01:19.039 [DEBUG] [org.openhab.model.script.Bad  ] - test comma
2016-03-02 17:01:19.043 [ERROR] [m.r.internal.engine.RuleEngine] - Error during the execution of startup rule 'Cast test':

Why should I use a different version of String::format when it works just fine?
The problem is the cast from a string to a number-item.
This cast is - as I have written - not locale independent.

watou · March 2, 2016, 6:25pm

You’ve given a test of something different from my advice. It is clear to me why

new DecimalType("1,2")

would fail, as that is not an accepted decimal number format.

Why not try my advice of

Keller_Waschmaschine_Energie.postUpdate(new DecimalType(min1.floatValue))

?

Or more simply:

Keller_Waschmaschine_Energie.postUpdate(min1)

Spaceman_Spiff · March 2, 2016, 7:26pm

Hi watou,

thank you for your quick reply! It is clear to me, too. But why is the String::format depending on the locale and the constructor of DecimalType is not? Why can’t I round to two digits with String::format when the postUpdate clearly accepts a string?

I am trying to prevent this:
2016-03-02 20:25:08.797 [INFO ] [runtime.busevents ] - Keller_Waschmaschine_Energie_1min state updated to 3.666666666666666518636930049979127943515777587890625

watou · March 2, 2016, 7:46pm

Using floatValue could be a cause, because native float and double can’t represent exact decimal numbers like BigDecimal does (which holds the value in a DecimalType). Does averageSince(now.minusMinutes(1)) return that long number, or is it only visible after using floatValue?

Spaceman_Spiff · March 2, 2016, 7:53pm

It is returning a long number. I am using it on a signal which is changing a lot.

watou · March 2, 2016, 7:57pm

The long numbers are coming from the use of double in the persistence layer:

    static public DecimalType averageSince(Item item, AbstractInstant timestamp, String serviceName) {
        Iterable<HistoricItem> result = getAllStatesSince(item, timestamp, serviceName);
        Iterator<HistoricItem> it = result.iterator();

        DecimalType value = (DecimalType) item.getStateAs(DecimalType.class);
        if (value == null) {
            value = DecimalType.ZERO;
        }

        double average = value.doubleValue();
        int quantity = 1;
        while (it.hasNext()) {
            State state = it.next().getState();
            if (state instanceof DecimalType) {
                value = (DecimalType) state;
                average += value.doubleValue();
                quantity++;
            }
        }
        average /= quantity;

        return new DecimalType(average);

Perhaps you could round the BigDecimal that returns from min1.toBigDecimal?

Spaceman_Spiff · March 2, 2016, 8:05pm

This is what I am trying to do.
I am still not getting why String::format is locale dependend and the constructor of DecimalType is not! Either all are or none are.

watou · March 2, 2016, 10:26pm

openHAB has to have a locale-independent format for decimal numbers. Java gives you the ability to format strings in the default, or a specific, locale.

Spaceman_Spiff · March 3, 2016, 7:22am

Storing of values can of course be locale independent. But the constructor when parsing from a string should not be.
Because “1,2” is a valid number.
I am using the String::format with the default locale. I did not pass any more arguments.
Therefore I am expecting the constructor for DecimalType to work in the same way.

watou · March 3, 2016, 11:05am

So you’re saying that the constructor for DecimalType should know every possible locale, and have the ability to correctly guess which locale is being implied by the number’s format? Or that the constructor should only understand the number formatting of the default locale, even when numbers are formatted by systems that use a different locale? Or that the constructor should only know how to parse the canonical locale format and the default locale format, and be able to guess correctly that “1.200” is either one point two hundred, or twelve hundred? As you can see, this would be its own nightmare, when all you have to do is construct a DecimalType from a built-in numeric type.

Or, knowing that your numeric data is arriving in the non-canonical locale format, parse it accordingly. for example, there is a binding (the wired connection for Daikin heatpumps/aircons) that receives numeric data in the German format (“12,5”) and all it has to do is use the correct Locale to parse them – no problem!

Spaceman_Spiff · March 4, 2016, 10:27am

Let’s take a look form the users-point of view.

If I send “1,200” I get a nullpointer exception and my whole rule breaks.
I am confused and can’t figure out why, because the DecimalTypes work just fine.
If I send “1,200” and it gets interpreted as 1200 I quickly realize I am off by factor thousand and just divide the value before passing it to the rule. It is not a pain for me as a user, because everything works fine.
Following your argumentation I should be allowed to pass “1,000,000,123.456” to the constructor, which of course doesn’t work.

So I am suggesting that both the dot and the comma is allowed if it only appears once.

watou · March 4, 2016, 10:54am

Thousands separators are not supported at all because this is a “pretty-printing” feature that is locale specific, so parsing it would require knowledge of the locale in which it is valid. Better documentation of openHAB formats might be in order, to reduce/eliminate confusion.
“I quickly realize” == code that is guessing what you mean, which would be a nightmare for reliability.
I don’t think I argued that that would be a good idea. See #1 above.

Write an algorithm that attempts to reliably interpret formatted numbers in multiple locales, without possibly guessing incorrectly. I can’t. I believe that DecimalType ought to have the single canonical format it does, so there is no guessing as to what the string represents, as does the underlying BigDecimal class.

Spaceman_Spiff · March 4, 2016, 12:08pm

Seriously - it is not that hard:
If there is exactly one comma in it, replace it with a dot. If a dot is in it, do nothing.
This works because - as you said - thounds separators are not allowed.
Simple solution.
At least printing a proper error message would be helpful.

To be honest this is getting quite frustrating for me.
I was just trying to make stuff easier for other people which definitely have (or will have) the same problem as I had. But if nobody sees this as a problem - I know what to do from now on.

watou · March 4, 2016, 5:25pm

You are so right. I hope that the new ways to write rules in future versions will focus on meaningful feedback on errors in syntax, runtime error messages, so users aren’t left to stab in the dark. That, plus really sharp documentation, could make a world of difference.