JSR223: Accented characters åäöüéÅÄÖÜÉ doesn't work in my OH installation

Setup, Configuration and Use Beginners
1
  • Platform information:
    • OS: ubuntu 16.04LTS
    • openHAB version: 2.2

Accented characters åäöüéÅÄÖÜÉ doesn’t display properly in the log when using Jython scripts. My scripts are all UTF8 encoded of course.

The string åäöüéÅÄÖÜÉ shows up as åäöüéà ÄÖÜÉ and the same problem when passing them to os.system which is a larger problem for me.

Using an Xtend script, I don’t see the same issues.

Any ideas?

2

“the log”. Which log? Opened in what application?

3

I use the logging functionality of openHAB 2 and log files are written to /var/log/openhab2/openhab.log
Normally I can use the Karaf console to monitor the log in realtime but a simple tail /var/log/openhab2/openhab.log will also do the job.

But this is not a logging issue. Consider the following statement:

    events.postUpdate("MyStringItem", "åäöüéÅÄÖÜÉåäöüéÅÄÖÜÉ")

It will store garbled characters into the MyStringItem item.

In case it’s of importance …

I use Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Vendor is: Oracle Corporation

Output from running os.system('set > /tmp/test.txt') within a jython script:

EXTRA_JAVA_OPTS='-Djava.awt.headless=true -XX:+UseG1GC -Xbootclasspath/a:/home/someuser/jython2.7.0/jython.jar -Dpython.home=/home/someuser/jython2.7.0 -Dpython.path=/etc/openhab2/automation/lib/python'
HOME='/var/lib/openhab2'
IFS=' 	
'
JAVA_HOME='/usr/lib/jvm/java-8-oracle/jre'
JAVA_MAX_MEM='512M'
JAVA_MIN_MEM='128M'
JAVA_OPTS='
  -Dopenhab.home=/usr/share/openhab2
  -Dopenhab.conf=/etc/openhab2
  -Dopenhab.runtime=/usr/share/openhab2/runtime
  -Dopenhab.userdata=/var/lib/openhab2
  -Dopenhab.logdir=/var/log/openhab2
  -Dfelix.cm.dir=/var/lib/openhab2/config
  -Djetty.host=0.0.0.0
  -Djetty.http.compliance=RFC2616
  -Dorg.ops4j.pax.web.listening.addresses=0.0.0.0
  -Dorg.osgi.service.http.port=8080
  -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -XX:+UseG1GC -Xbootclasspath/a:/home/someuser/jython2.7.0/jython.jar -Dpython.home=/home/someuser/jython2.7.0 -Dpython.path=/etc/openhab2/automation/lib/python'
KARAF_BASE='/var/lib/openhab2'
KARAF_DATA='/var/lib/openhab2'
KARAF_ETC='/var/lib/openhab2/etc'
KARAF_SCRIPT='karaf'
LANG='en_US.UTF-8'
LC_ADDRESS='sv_SE.UTF-8'
LC_IDENTIFICATION='sv_SE.UTF-8'
LC_MEASUREMENT='sv_SE.UTF-8'
LC_MONETARY='sv_SE.UTF-8'
LC_NAME='sv_SE.UTF-8'
LC_NUMERIC='sv_SE.UTF-8'
LC_PAPER='sv_SE.UTF-8'
LC_TELEPHONE='sv_SE.UTF-8'
LC_TIME='sv_SE.UTF-8'
LD_LIBRARY_PATH=':/var/lib/openhab2/lib:/usr/share/openhab2/runtime/lib'
LOGNAME='openhab'
NLSPATH='/usr/dt/lib/nls/msg/%L/%N.cat'
OLDPWD='/usr/share/openhab2'
OPENHAB_BACKUPS='/usr/share/openhab2/backups'
OPENHAB_CONF='/etc/openhab2'
OPENHAB_HOME='/usr/share/openhab2'
OPENHAB_LOGDIR='/var/log/openhab2'
OPENHAB_RUNTIME='/usr/share/openhab2/runtime'
OPENHAB_STARTMODE='daemon'
OPENHAB_USERDATA='/var/lib/openhab2'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='9432'
PS1='$ '
PS2='> '
PS4='+ '
PWD='/var/lib/openhab2'
SHELL='/bin/false'
USER='openhab'
XFILESEARCHPATH='/usr/dt/app-defaults/%L/Dt'

Thanks for helping. :smile:

4

Make sure that the file containing the characters is UTF-8 encoded. I have seen similar problems with danish characters.

5

Thanks for your kind help Martin

Unfortunately it’s not as easy as that. I’m using VSC for editing jython scripts and encoding UTF-8 is selected for the file. I use the same editor for Xtend rules to test and they do work in this aspect.

I double checked the file encoding:

$ file -bi myscript.py
text/x-python; charset=utf-8

I read somewhere that openhab is converting the JSR223 jython scripts into javascript. It should be interesting to see if those javascripts look OK or not. I don’t know where they reside.

6

Hmmm, strange, I some danish characters in my jython files, and they are displayed correct whenever I remeber to UTF-8 encode the files.

7

@mr_eskildsen, are you also using jsr223 scripts (jython) as I do?

@namraccr I’m running out of ideas … I’ve tried the following

  1. Installed OH 2.2 on a clean new server with Ubuntu 16.04
  2. This time I’ve installed the Zulu Java platform.

The problem persists on the new installation too.

Even though I know that my files are indeed UTF-8 encoded I’ve tried the Eclipse editor but nothing changes.

When I try the same thing using an Xtend script, these things works. It’s just a problem when I use Jython scripts. The problem is not specifically related to logging or os calls. Also if I do a events.postUpdate like this:

events.postUpdate("Sonos_CurrentTrack", "Ärlige Åkes ökända låt")

The item Sonos_CurrentTrack will hold garbled characters.

Any ideas someone? Your kind help is highly appreciated.

Cheers!

8

It seems that when scripts are loaded the encoding is not specified to be UTF-8 so it will fallback to some default encoding.

This page has some additional info on how to get the Java defaults and how to change them.

9

I use jython script. My setup is a raspberry pi using Oracle java.

10

@mr_eskildsen, thanks

I’m such an idiot. Python version is 2.7 and the default python character set is ‘ascii’ on my system! I feel like I’m back in the 80’s again :smile:

This example shows how to do it:

    logger.info(u'smørrebrød')
    events.postUpdate("Sonos_CurrentTrack", u'smørrebrød')

I put on the “shame hat” and apologize for using your time.

Cheers

11

You could watch Back to the future…
I would like to see you play “smørrebrød” on your Sonos

12

I’m so happy now @mr_eskildsen when this is solved. I guess I will just eat the smørrebrød and play Rabalderstræde on my Sonos!
:rofl:

13

Sounds like an excellent idea :slight_smile: