[OUTDATED Snips is dead!] Voice Control - How to use Snips AI and MQTT 2.4

Tutorials & Examples Solutions
1

Hi Folks,

SnipsAI is a nice solution to talk to your Openhab. Other than Alexa or Google etc. it uses offline service and isn’t connected to any cloud service to manage ASR/TTS. But you can use an external TTS cloud service if the built in voice isn’t good enough for your purposes.

My Snips is installed on a raspberry pi. I connect it via LAN to my openhab server.

And here is how to manually configure a MQTT broker and thing for Snips:

First you need to add a snips.things file

Bridge mqtt:broker:SnipsAI "MQTT - SnipsAI Broker" @ "MQTT" [ 
  host="<IP of your Snips service>",
  secure=false,
  port=1883,
  qos=0,
  retainMessages=false,
  clientID="Openhab2",
  keep_alive_time=30000,
  reconnect_time=60000,
  username="<username if you added one in Snips config>",
  password="<password if you added one in Snips config>"
]
{
    Thing mqtt:topic:SnipsAI "MQTT - SnipsAI Thing" @ "MQTT"  {
        Channels:
            Type string : Snips_Intent "Snips Intent" [ 
                stateTopic="hermes/intent/#:state:default",
                commandTopic="hermes/intent/#:state:default"      
            ]
    }
}

now you should be able to receive mqtt messages from Snips

You also have to create a snips.items file

String  Snips_Intent    "Snips Intent"  (gSnips)    { channel="mqtt:topic:SnipsAI:Snips_Intent" }

Now you can use this Item to add rules to your openhab. create snips.rules file like this

rule "Snips"
when
    Item Snips_Intent received update
then
    var String intentName = transform("JSONPATH", "$.intent.intentName", Snips_Intent.state.toString)
    var String onlyIntent = intentName.split(":").get(1);
    var String device_val = transform("JSONPATH", "$.slots[0].value.value", Snips_Intent.state.toString)
    
    logInfo ("Snips Intent: ", Snips_Intent.state.toString)
    
    logInfo ("Snips Gerät: ", device_val)
    
    onlyIntent = onlyIntent.toLowerCase
    
    logInfo ("Snips device value: ", onlyIntent)

    logInfo("Snips device value: ", device_val)

Now you can add intents to your Snips installation and expand your rule with something like this

    if (onlyIntent == 'LampsOn') {
        SwitchOnOff = "ON"
    }
    else if (onlyIntent == 'LampsOff') {
        SwitchOnOff = "OFF"
    }
    
    if (device_val == 'Livingroom') {
        sendCommand(<mylightItem>, SwitchOnOff)
    }

How you can configure your Snips is not part of this tutorial but you can use this explaination video to get started.

Hope this will help… Have fun

5 Likes
2

Great tutorial! Thanks for posting!

3

Do you have special microphone hardware (multi-array or similar)?

By the way, openHAB has a full Natural Language Processor as well (not merged yet: https://github.com/openhab/openhab-core/pull/734) including intents for item control. But that doesn’t help with Snips I guess, because they need to manage intents themselves I guess?

4

You can use the old Playstation eye webcam which has a 4 mic array and works great on a RaspberryPi. You can get it for around 10€ on Ebay. Or you can use a respeaker hat. Snips did a great post comparing mics here https://medium.com/snips-ai/benchmarking-microphone-arrays-respeaker-conexant-microsemi-acuedge-matrix-creator-minidsp-950de8876fda
You are right snips handles intents themselves as you train them on their online console and than only download the trained model to your pi for offline use.

5

Great to see other people use snips too. You could add the session Id from the snips json so that you can send back a session end to snips Hermes over Mqtt. Otherwise snips will always end the session on an error.

6

Hi Guys,

i have a Problem with my Thing config.
I tried so much but nothing helps.

After i set the stateTopic at the channel the whole MQTT Bridge und the Thing goes offline.

2019-06-04 10:26:59.658 [INFO ] [.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to '192.168.0.140' with clientid openhab2 and file store '/var/lib/openhab2/mqtt/192.168.0.140'
2019-06-04 10:26:59.757 [INFO ] [ansport.mqtt.internal.ClientCallback] - MQTT connection to '192.168.0.140' was lost

With the 2.5 snapshot binding i have the same Problem, with cleared Cache.

Here is my Thing config:

Bridge mqtt:broker:SnipsAI "MQTT SnipsAI Broker" @ "MQTT" [ 
  host="192.168.0.140",
  secure=false,
  port=1883,
  qos=0,
  retainMessages=false,
  clientID="openhab2",
  keep_alive_time=30000,
  reconnect_time=60000,
  username="",
  password=""
]
{
    Thing topic SnipsAI "MQTT SnipsAI" @ "MQTT"  {
        Channels:
            Type string : Snips_Intent "Snips Intent" [ 
                stateTopic="hermes/intent/#:state:default",
                commandTopic="hermes/intent/#:state:default" 
            ]
    }
}

Hope you have any advices - i have no idea anymore what that could be.

thx
vaillan

7

Hello,
I see one thing, the intent topic should be configured as read only. Your replies to snips have to be published to the dialogueManager topics so you should remove the command topic for the intent channel.
Unfortunately I can’t give you any further hints as I don’t use the thing files for the mqtt binding and do all my configuration through Ui.
Have you tested if the IP and stuff are right using a tool like mqtt explorer? Can you connect to the Snips broker see all the Snips topics when you monitor this connection?
Best regards Johannes

8

Yes that will not work, because the topics are invalid. “#” must be the last character of a topic name. Whatever you have added after it will error out and the binding has only one way to tell you: By going offline.

9

i also tried that.

But now i deleted the .Things file and made it with UI.
The channel stays online now, i´ll give Feedback, if it is working now.

thx
vaillan

10

it is working now, thank you.

@Dan which snips app are you using for that configuration?

best regards
Vaillan

11

He probably made his own. But for the structure that he is working with I would guess that for this basic example he made two intents and one custom slottype in his assistant in the Snips console. The custom slottype is for the device names and contains the openhab item names and than aliases for each. He uses this custom slot type as a the only slot in both intents. He has one intent for switching on and one for switching of and trains each one with an appropriate number of example sentences where he marks the device slot in each. That way he gets the structure seen above. Correct me if I’m wrong @Dan
You could also create a second custom slot type which contains on and of and combine both intents in one and have a more complicated rule on the openhab side.
Using a available app you would always have to go through and insert all your own items so it’s easier to just build your own as its so simple.
Best regards Johannes

12

once the javascript helper libraries are all done I might rewrite my nodered code i use in conjunction with the openhab mqtt2 binding to work straight in openhab. This is my code in nodered for handling an intent from snips that contains any number of on off and dimming commands strung together in one intent and sends it on to the publish trigger topic of the mqtt binding. Its tried with strings of up to ten commands but in theory you could go on forever:

const alleitems = [
    ["item1", "openhabitem1"],
    ["item2", "openhabitem2"],
    ["item3", "openhabitem3"],
    ["item4", "openhabitem4"],
    ["item5", "openhabitem5"]
]
const dimitems = [
    ["item1", "openhabdimitem1"],
    ["item3", "openhabdimitem3"],
    ["item5", "openhabdimitem5"]
]
var id = msg.payload.sessionId;
flow.set("Id",id);
var slotsarrraw = msg.payload.slots;
var slotarr = [];
var col1 = [];
var col2 = [];
var dim1 = [];
var dim2 = [];
var stopic;
var spayload;
var output;
var outputdim;
var check;
var incnum = function(element) {
    return element.match(/^[0-9]*$/);
}
function colsplit(input) {
    for(var i = 0; i < input.length; i++) {
        col1.push(input[i][0]);
        col2.push(input[i][1]);
    }
}
function dimsplit(input) {
    for(var i = 0; i < input.length; i++) {
        dim1.push(input[i][0]);
        dim2.push(input[i][1]);
    }
}
if (slotsarrraw.length > 1) {
    for(i = 0; i < slotsarrraw.length; i++) {
        slotarr.push(slotsarrraw[i].value.value.toString());
    }
    if (slotarr.includes("einschalten") === false && slotarr.includes("ausschalten") === false && slotarr.some(incnum) === false) return;
    colsplit(alleitems);
    dimsplit(dimitems);
    for(var a = 0; a < slotarr.length; a++) {
        if (col1.includes(slotarr[a])) {
            output = col1.indexOf(slotarr[a]);
            output = col2[output];
            outputdim = dim1.indexOf(slotarr[a]);
            outputdim = dim2[outputdim];
            check = 0;
            for(var b = a + 1; check === 0; b++) {
                if (slotarr[b] === "ausschalten") {
                    stopic = "allItemsin/" + output;
                    spayload = "OFF";
                    node.send({topic:stopic, payload:spayload});
                    check = 1;
                } else if (slotarr[b] === "einschalten") {
                    stopic = "allItemsin/" + output;
                    spayload = "ON";
                    node.send({topic:stopic, payload:spayload});
                    check = 1;
                } else if (slotarr[b].match(/^[0-9]*$/) && dim1.includes(slotarr[a])) {
                    stopic = "allItemsin/" + outputdim;
                    spayload = slotarr[b];
                    node.send({topic:stopic, payload:spayload});
                    check = 1;
                } else {
                    if(b < slotarr.length) {
                        check = 0;
                    } else {
                        check = 0;
                        b = -1;
                    }
                }
            }
        }
    }
} else return;
13

Hi Vaillan,
right now I’m trying to create my own snips-app to communicate with openhab2…
but my progress is pending cause I got other things to do right now… But I saw that there are few openhab apps for snips in progress, so maybe there’s some useful stuff for you…

If I get any further with my own app I post a link…

and like @JGKK mentioned you can create your own otherwise… I think he gave a good starting point for doing that :wink:

1 Like
14

Great tutorial, @Dan! I plan to take my very first steps with it this weekend and I am looking forward to it! :smiley:

One remark: Although I have not tried it yet but I believe that something like

if (onlyIntent == 'LampsOn') { ... }

will not work because of

onlyIntent = onlyIntent.toLowerCase

converting the intent’s characters to lowercase letters before. So it should read

if (onlyIntent == 'lampson') { ... }

in this case.

15

Thank you so much @Dan! for sharing this tutorial, I created my account because I had some troubles implementing your tutorial so i could ask some questions. Luckily I figured it out now :D, Snips is integrated with openhab now and I can control my lights. Really happy with microphone of the PS eye, picks up my voice from the entire room.

16

Great tutorial, everything works great. But I have one problem, after saying command Snips ends session with error. Can someone help me to send endsession command to Snips via MQTT? I have no idea how to do it

17

If you scroll to the bottom of the Snips jython tutorial I wrote (Tutorial for using the Snips Voice Assistant with Jython and Mqtt for switching and dimming lights) you can see how to do it. But the basis is to extract the session ID of the original message object to a variable and than publish a session end with the extracted session ID using the mqtt publish action in the correct format to hermes/dialogueManager/endSession.
You can find details on the session ID here:
https://docs.snips.ai/reference/dialogue#intent
and on the session end here:
https://docs.snips.ai/reference/dialogue#end-session

Best regards Johannes

18

Hi,

I have big trouble to get the mqtt2 binding running with snips.
I´m very new in the topic of mqtt and it is my first work with this binding.

I tried this tutorial in this post but nothing happens.
I also tried this one https://community.openhab.org/t/tutorial-for-using-the-snips-voice-assistant-with-jython-and-mqtt-for-switching-and-dimming-lights/76689 but with no result.

With the Paper Ui configuration the first error is when I try to set up a channel for the generic mqtt thing is “ERROR 500 - Internal Server Error”. When I can create such a channel and I type
hermes/intent/#:state:default
for the state Topic Im recieving: java.lang.Exception: No MQTT client and again nothing is working. With the mqtt Binding everything works fine. But I want to use the newer one.

It is exasperating. I don´t no where is my mistake. Can someone Help me please.

19

Hmm, you will probably have to give us some more information on your setup. Unfortunately I don’t know how to set up the mqtt binding with thing files as I only use it on conjunction with paper Ui.
Did you create a broker thing connected to the mosquito broker that snips installs and uses? When you use the connection details that you use for the Snips broker thing in openhab to connect to it with something like mqtt-explorer do you get a successful connection and can you see snips publish to the topics you want to use?
Did you choose the right broker (the Snips mosquito thing you created) when you created your generic mqtt thing?
It looks to me like either something is wrong with your broker thing connecting to the Snips broker or something is wrong with your generic thing definition.
Have you tried following the steps of the tutorial I wrote concerning the mqtt connection step by step using paper Ui?
You will probably have to post more details to get to the bottom of this.
Best regards Johannes

20

It is working now.

@JGKK thank you for your answer.
The solution was in post from @David_Graeff

that will not work, because the topics are invalid. “#” must be the last character of a topic name