SEPIA Open Assistant - Privacy respecting, self-hosted voice-control

pa1 · February 23, 2020, 4:10pm

I ran into your really cool assistant when looking around after using snips, rhasspy, rasa and so on. With some tweaks I got it running in my environment. But there is a big drawback, which is a show stopper for me. I have plenty of devices of the same type in each room. Typically there are about 6 to 10 lights, each with different names in every room. Multiple main lights and multiple ambient lights…
With snips or rhasspy I simply trained a custom model for ASR and NLU/INTENTS which works with my lights (“Deckenlicht, Wandlicht, Treppenlicht, Bodenlicht, Ambiente Schräge, Spiegellicht, …”). I’ve found no way to do this with SEPIA. Am I simply missing the point?
I am able to train the Kaldi asr, which I did already, but NLU won’t get it. Ok, I could click “Teach it” on every sentence, but this would be a very very lengthy process.

You already implemented tags in your openhab adapter (sepia-type, …), so why not implement another tag “sepia-synonym” which automatically trains the NLU? This is the way I did slot injection on snips or training rasa in rhasspy.

Another really cool idea would be an mqtt adapter for NLU (like hermes). Publishing a text to it and receive the intent with slots and propabilities. I use an telegram bot for chatting with my “Home”, which would attach quite easy to such an connector.

At the moment I am using custom RPi3a/Respeaker4 devices in each room for voice interaction, while using a Sonos speaker in each room as Speaker-Device for my assistant. The sonos service subscribes to mqtt-protocol and parses the action to get the room to speak the text to. This service also does the tts part.
Where could I attach this sonos service to you sepia server? Probably I need to modify the tts module… but how to get it persistent to updates?

sepia-assistant · February 23, 2020, 7:15pm

Hello @pa1

Currently there is no easy way to train a complete custom NLU model for SEPIA and I understand that the Teach-UI might not be sufficient for your task. The non-easy methods would be to use the Python bridge module integrated into SEPIA’s NLU chain to actually use the model you’ve trained with other tools (e.g. Rasa), but that would require some Python coding and I should probably build a demo first. The other non-easy option would be to write some custom rules via the SEPIA Java SDK.
I’m planning to improve this for one of the next versions and started some discussions with Michael from Rhasspy to find out if we can make things more compatible

Something that might not completely satisfy you but could be a quick workaround for now is to use numbered devices, e.g. “Lamp 1 in the living room”. In the SEPIA Control-HUB you can give each device a name and add a number in brackets like “Deckenlampe (1)” (if you don’t use brackets SEPIA will use the number in the answer text which may or may not be desired).

I’ve thought a lot about this issue with device names and the problem here is that names of devices can be arbitrary and cross-talk with other SEPIA services can happen easily the more devices you have. That said it might actually be possible to implement your idea partially if the user already entered the smart home service. From there I can use the devices list obtained from the smart home HUB and check the “sepia-name” tag against the user input … … I will try that
Looking at your device names (“Deckenlampe” etc.) there might be another quick fix I can apply to make those devices recognizable immediately

There is a MQTT demo service in the SEPIA extensions repo that might be interesting for you. The new DIY SEPIA clients (currently in beta testing) have the feature to broadcast all kinds of events to the underlying WebSocket server (CLEXI). Those events could be rerouted to a MQTT broker with a little plugin.
I’m happy to help building a small demo if you want to try one of these options

cu,
Florian

openhabgs · May 19, 2020, 4:23pm

Hi,
I set up SEPIA according to the documentaion. All fine.
Connection to OpenHAB is working, things are downloaded to SEPIA and cen be switched.
Now if I log in with a user having these roles: smarthomeguest,smarthomeadmin,
and enter the command “Switch lights…:”,
I always receive the answer:
“Sorry openhabgs, but you need a permission to use this smart home service.”
Any idea where this “permission thing” has to be set?

There are no related entries to be found in the SEPIA log or in Openhab.log

sepia-assistant · May 19, 2020, 8:59pm

Hi,
Can you please double-check that the given roles are really existing for this user. The only place where the message is used is right after the roles check. I’m assuming you are not using the admin account but created a fresh user? ^^.

openhabgs · May 20, 2020, 6:38am

Hi Florian,
newly created user (of course :-):

Thanks for looking into it!

Log-Einträge:

2020-05-20 08:34:04 LOG - User ‘uid1003’ requested roles of user: sepiaopenhab@sepia.localhost
2020-05-20 08:36:03 LOG - USER uid1007 used SERVICE events_personal - TS: 1589956563328 - LANGUAGE: de - CLIENT: wstation_browser_v0.21.0 - API: v2.4.1
2020-05-20 08:36:03 LOG - USER uid1007 used SERVICE timer - TS: 1589956563330 - LANGUAGE: de - CLIENT: wstation_browser_v0.21.0 - API: v2.4.1
2020-05-20 08:36:03 LOG - USER uid1007 used SERVICE timer - TS: 1589956563330 - LANGUAGE: de - CLIENT: wstation_browser_v0.21.0 - API: v2.4.1
2020-05-20 08:36:11 LOG - USER uid1007 used SERVICE smartdevice - TS: 1589956571862 - LANGUAGE: de - CLIENT: wstation_browser_v0.21.0 - API: v2.4.1

Spaceman_Spiff · May 20, 2020, 12:02pm

Where do you store the item settings? Is it possible to manually link the items to SEPIA?

sepia-assistant · May 20, 2020, 7:52pm

This looks correct … very confusing
I feel almost stupid to ask but are you using ‘sepiaopenhab@sepia.localhost’ to login? Or ‘uid1007’? Basically my question is: are you absolutely sure that ‘sepiaopenhab@sepia.localhost’ and ‘uid1007’ are identical?
Right now I have only two possible explanations:

Either you somehow have 2 instances of the server running and the client app is accessing the wrong one or
sepiaopenhab@sepia.localhost is not uid1007

Both explanations don’t seem very likely but lets be 100% sure before we continue the search ^^.

Do you mean where inside openHAB? If you look into your openHAB items you should see “sepia-XY” tags, e.g. “sepia-type”, “sepia-name”, “sepia-room”. You can edit those by hand if you want. Possible values can be found here in the SEPIA docs.

openhabgs · May 20, 2020, 9:15pm

Hi Florian,
please see above in the screenshot of the chat-protocol: I logged in as “sepiaopenhab@sepia.localhost” and it shows that “(uid1007) joined the chat”.

grafik

I also created a new user:

grafik

Assigned the role:

grafik

Same result:
grafik

How could I check for 2 server instances?

sepia-assistant · May 20, 2020, 10:24pm

ok this is really really strange!
Since you could login with the newly created user we can also rule out that there is some rouge server instance running I guess.

I’ve double-checked twice now and there is only one exact location in the code where this answer can be triggered. It looks like this:

if (!nluResult.input.user.hasRole(Role.smarthomeguest)){ 
    service.setCustomAnswer(notAllowed);
    return
}

Can you try the following:

Log in with uid1007 or uid1011 to the Control Hub instead of the client app
Try to edit and toggle the smart home devices
Use the “Assistant Testing” page of the Control HUB to send the text “Licht einschalten/Switch on lights” (Button: “answer”)

Oh and one more thing to check would be the LOG file of the chat-server which is ~/SEPIA/sepia-websocket-server-java/log.out. It should mention the user roles of uid1007 and uid1011 as well.

Spaceman_Spiff · May 21, 2020, 6:28am

Do you mean where inside openHAB? If you look into your openHAB items you should see “sepia-XY” tags, e.g. “sepia-type”, “sepia-name”, “sepia-room”. You can edit those by hand if you want. Possible values can be found here in the SEPIA docs.

I see you use tags, wouldn’t metadata be much easier?

openhabgs · May 21, 2020, 7:59pm

Hi Florian, thanks for following up…

Result on “ANSWER” is:

{
“result”: “still_ok”,
“hasAction”: false,
“cardInfo”: [],
“answer”: “Sorry openhab g s, aber du brauchst erst eine Erlaubnis, um diesen Smart Home Service zu nutzen.”,
“more”: {
“certainty_lvl”: 1,
“cmd_summary”: “smartdevice;;smart_device={“value_local”:“das Licht”,“device_tag”:“Licht”,“value”:“light”};;action={“value_local”:“anschalten”,“value”:”"};;smart_device_value=;;room=;;",
“context”: “smartdevice;;default”,
“language”: “de”,
“user”: “uid1007”
},
“hasCard”: false,
“actionInfo”: [],
“answer_clean”: “Sorry openhab g s, aber du brauchst erst eine Erlaubnis, um diesen Smart Home Service zu nutzen.”,
“htmlInfo”: “”,
“hasInfo”: false,
“resultInfo”: {
“cmd”: “smartdevice”,
“state”: “”,
“device”: “”,
“room”: “”
}
}

~/SEPIA/sepia-websocket-server-java/log.out . shows:

2020-05-20 08:28:54 LOG - 2020-05-20 08:28:54 Welcome to the SEPIA webSocket server - port: 20723
2020-05-20 08:28:54 LOG - Initialization complete, lets go!
2020-05-20 08:28:59 [qtp13557592-16] INFO SepiaAuthenticationHandler - Authenticated: uid1005, roles: [“user”,“assistant”], deviceId: serv1
2020-05-20 09:57:22 [qtp13557592-16] INFO SepiaAuthenticationHandler - Authenticated: uid1007, roles: [“user”], deviceId: wstation
2020-05-20 23:14:26 [qtp13557592-19] INFO SepiaAuthenticationHandler - Authenticated: uid1011, roles: [“user”], deviceId: wstation
2020-05-20 23:14:26 [qtp13557592-19] INFO SocketChannelPool - New channel has been created by ‘uid1011’ with ID: uid1011
2020-05-20 23:14:27 ERROR - getDocument - ElasticSearch - error in ‘chat-users/all’: {“code”:404,“HTTP_REST_SUCCESS”:false}
2020-05-21 03:19:45 [Connector-Scheduler-1bb4c77] ERROR SepiaSocketHandler - SepiaSocketHandler reported: java.util.concurrent.TimeoutException: Idle timeout expired: 14400000/14400000 ms
2020-05-21 03:19:46 [qtp13557592-19] INFO SepiaAuthenticationHandler - Authenticated: uid1005, roles: [“user”,“assistant”], deviceId: serv1
2020-05-21 07:19:46 [Connector-Scheduler-1bb4c77] ERROR SepiaSocketHandler - SepiaSocketHandler reported: java.util.concurrent.TimeoutException: Idle timeout expired: 14400001/14400000 ms

sepia-assistant · May 22, 2020, 9:08am

Here we have the problem! For some reason the role changes you made are not stored. The chat server sees only the default role “user”. Can you confirm the following steps once more:

Enter ‘uid1007’ in the ‘UserID’ field of the roles section (user management page in Control HUB). Actually if you’ve previously used the email here instead of the UID (as seen on the screenshot) you should have gotten an error
Press ‘Get roles’ button to check the current settings
If you see ‘smarthomeguest’ this can only mean that the control HUB and the client use different databases … which would be extremely confusing since you could login with the newly created user uid1011
If you DON’T see the role add it once more e.g. like this: ‘user, smarthomeguest’ and then press ‘Set roles’
Finally you should see “result”: “success” in the debug field
Just to double-check one final time press ‘Get roles’ again to see if the database remembers your settings

The decision to use ‘tags’ was made way back in 2018 when the first openHAB integration was released … I’m not really sure anymore why but I remember I tried different things an this one worked best. From the description given for ‘metadata’ it looks like a suited field. Maybe I overlooked it then because I just tried it with the my current setup and the field doesn’t show up in my items (probably because its empty).
Anyway handling ‘tags’ is already pretty easy so I’m not sure about the “much easier” part … though I agree it would be “a bit” easier

openhabgs · May 22, 2020, 9:33am

Great help - now it works! ( logging in with uid1007 OR sepiaopenhab…)
Next questions

what is my problem with the email user management?
The English sentence “Switch on light in Study” is understood and executed
If I switch to German - the program understands my Sentence "Licht einschalten, but not the (English) Room specification:

grafik580×666 37.8 KB
Is there a translation of the “Rooms list” and or can this list be modified?

sepia-assistant · May 22, 2020, 9:42am

Happy to hear it finally works
Can you say what the problem was so maybe I can implement something to prevent it in the future?

About your question: German and English have two separate language models, in German the name for study is e.g. “Arbeitszimmer”, “Studier-Zimmer”, “Arbeitsraum”, “Herren-Zimmer” etc.

openhabgs · May 22, 2020, 12:03pm

Hi,
don’t know where the problem originated. I set up a new user (e.g. like in above screenshot “test@test.test” - SEPIA automtically assigned the UID (“uid1011”) - I assigned the role to this email-user , did NOT get an error message (screenshot above) - and ended up with the problem you solved.
May the documentation does not specify that th erole has to be assigned to the UID (or I overlooked it?)

Where is the language model choosen/selected? Currently I only get the English room labels displayed.

sepia-assistant · May 22, 2020, 12:55pm

This is the strange part I could not reproduce since it’s giving me an error message directly when I try set the role using an email. Maybe I’ve already improved this at some point after the v2.4.1 release and forgot about it ^^. I’ll keep an eye on it though

The labels are just generalized names representing a type of device or room. Unfortunately the Control HUB views are only available in English because its very time consuming to translate everything but if you see for example “Living Room” in the room selector that maps to the value “livingroom” and depending on your active app language corresponds to a number of names like “living-room”, “living rooms”, “Wohnzimmer” etc… … so the short answer: just select your app language and imagine how you would call this room in the given language. If you find something that doesn’t work feel free to report it for example in the GitHub issues section

openhabgs · May 22, 2020, 1:38pm

Still don’t get the room-types topic…

I have to set the tag in Openhab (not editable by SEPIA because created as text item):

Switch Sofalampe “Sofalampe” [“sepia-type=power_outlet”, “sepia-room=Wohnzimmer”]
Switch SEPIAtest “SEPIAtest” [“sepia-type=power_outlet”, “sepia-room=livingroom”]

shows in SEPIA as

So I CAN only select from and work with the English room types documented here.

How and where to activate the German room tags (“Wohnzimmer”…) please?

sepia-assistant · May 22, 2020, 2:32pm

Oh I see the problem now. The thing is the language does not matter for any of the ‘sepia-xy’ tags. The same way you used ‘sepia-type’ you can simply assign ‘sepia-room=livingroom’ and it will work in any language. Its an abstraction layer of the SEPIA server.

If you have a room that does not fit to any of the given room types (as seen in the list from the docs you’ve linked) you can use type “other” and index 1, 2, … . This is not perfect but due to the tight coupling of types to the NLU its currently not possible to create new types on-the-fly.
[EDIT] Another way is to give the device a name that involves the unknown room type or makes it unique somehow. This will work in v2.5.0.
Of cause you can always use the Teach-UI to assign for example the command “lights in room 202 to on” to you custom sentence “lights on in the party room” or “lights on floor 1” etc.

openhabgs · May 22, 2020, 5:10pm

Thanks again, but I still miss the point.

What is the way to issue a GERMAN command (e.g. “Schalte den (“Power Outlet”?) im Wohnzimmer / in der Garage aus” for the below items?

To switch the language - (EN/DE) - do I just have to choose this in the user settings - or somewhere else?

grafik

sepia-assistant · May 22, 2020, 5:48pm

For “SEPIAtest”:
“Schalte die Steckdose in Garage 10 aus”
Here I’d suggest to remove the room index unless you really have 10 garages . The room index always refers directly to the room type. “Garage” without index will only match empty index or index 1.

For “Sofalampe”
“Schalte die Steckdose im Wohnzimmer aus”
Here I’d suggest to change the type to ‘light’ to avoid conflicts with with the sentence “Sofalampe im Wohnzimmer ausschalten” (this would look for type ‘light’). Since openHAB knows how to control the device SEPIA does not need to know that it is actually a plug switch.

I hope that clarifies some things

Correct, simply choose it in the user settings. It is also possible to say “speak German”