Cuevox - A Rule Based Voice Interpreter [4.0.0.0;4.9.9.9]

[JanMattner]

Description

Cuevox is a simple but extensible voice control tool for openHAB. The OH Android App can be used to send voice commands to OH and interpret them using voice command rules that can be easily extended in the rule script. Apart from the voice command itself, no OH internal data needs to be sent to any external API.

The voice command is split in tokens, which are tried to be matched for the pre-defined rules.

Features:

• Supported languages: English, German
• Send ON/OFF or UP/DOWN command to a single item (matched by label or synonym)

  • turn off the kitchen light

  • lower the living room blinds

  • Assuming in above examples that kitchen light and living room blinds are the labels of single items.
• Turn all lights on/off or raise/lower all roller shutters in a group, e.g. a location.
  • All descendant items of the respective group can be matched.

  • turn off the light in the first floor

    • This will match all Switch items with the Light tag.

  • raise all shutters in the living room

  • Assuming in above examples that first floor and living room are group items.
• Item matching via the item label or Synonym metadata.
• Create custom voice command rules, including custom script execution upon matching.

Examples (English)

Light Controls

Turning lights on/off in a specific location:

• “Turn on all the lights in the living room.”
• “Switch the light off in the kitchen.”
• “Turn all lights off in the bedroom.”
• “Switch the lights on.”

Turning lights on/off with labels:

• “Turn the kitchen light off.”
• “Switch on the bedroom lights.”
• “Turn off the living room lights.”

Rollershutter Controls

Raising or lowering rollershutters:

• “Put the rollershutter up in the office.”
• “Pull all the blinds down in the living room.”
• “Lower the shutters in the bedroom.”
• “Raise all the roller blinds in the kitchen.”

Specifying rollershutters by label:

• “Lower the kitchen shutters.”
• “Raise the office blinds.”
• “Pull the living room blinds down.”
• “Put the bathroom roller shutter up.”

Generic Item Commands

Turning items on/off:

• “Turn the fan on.”
• “Switch the heater off.”
• “Turn the lamp off.”

Adjusting items up/down:

• “Pull the screen down.”
• “Lower the projector.”
• “Raise the curtain.”

Examples (German)

Lichtsteuerung (Light Controls)

Schalten von Lichtern ein/aus in einem bestimmten Raum:

• “Schalte alle Lichter im Wohnzimmer ein.”
• “Mach das Licht in der Küche aus.”
• “Schalt alle Lampen im Schlafzimmer aus.”
• “Mach die Lichter im Flur an.”

Schalten von Lichtern mit Labels:

• “Schalte die Wohnzimmerlampe aus.”
• “Mach das Küchenlicht an.”
• “Schalt das Schlafzimmerlicht ein.”
• “Mach die Flurlampe aus.”

Rolladensteuerung (Rollershutter Controls)

Hoch- oder Herunterfahren von Rollos in einem Raum:

• “Fahr die Rollos im Wohnzimmer hoch.”
• “Mach das Rollo im Schlafzimmer runter.”
• “Fahre die Rolladen in der Küche hoch.”
• “Fahr die Rolläden im Büro herunter.”

Hoch- oder Herunterfahren von Rollos mit Labels:

• “Fahr das Küchenrollo hoch.”
• “Mach den Wohnzimmerrolladen runter.”
• “Fahre die Schlafzimmerrolläden herunter.”
• “Fahr die Bürorollos hoch.”

Generische Befehle (Generic Commands)

Ein- und Ausschalten von Geräten:

• “Schalte den Ventilator ein.”
• “Mach die Heizung aus.”
• “Schalt die Lampe an.”

Hoch- oder Herunterfahren von Geräten:

• “Fahre den Bildschirm runter.”
• “Mach die Leinwand hoch.”
• “Fahr die Jalousie herunter.”

Requirements

Language: JavaScript, Scripting with openhab-js

Dependencies:

• JavaScript Scripting add-on
• openhab-js 4.5.1 or later
• A voice command String item, which stores the spoken text
• The voice command item configured under Rule Voice Interpreter and the openHAB Android App, connected to your OH instance and configured to send voice commands to OH
  • Alternatively: any other system that captures your voice and stores the spoken text in the voice command item.

How to setup the voice command item to use with the openHAB Android App

1. Create the item, called e.g. “VoiceCommand”
   

   

2. Configure it in the settings as target for the rule voice interpreter: Settings->Rule Voice Interpreter->Voice Command Item
   

   
   
   →
   
   

Create Custom Voice Command Rules

As of now, custom voice command rules must be defined in the inline script.
NOTE: If the rule template has been updated and you’d like to update your rule, you must create a new rule from the latest template. Any changes to your previous rule/script, particularly your custom voice command rules, must be copied from the previous rule script to the newly created rule script.

At the very bottom of the script, the // RULES section defines the rules. Below, there is a section for your custom voice command rules:

  // ** CUSTOM RULES *********************
  // Add your rules here
  // *************************************

You can define variables for common expressions or reuse the ones from the pre-defined rules and add the rule to the rbi object (the instance of the RuleBasedInterpreter class), e.g.

let the = opt("the");
let lowerRaise = alt(cmd("lower", DOWN),cmd("raise", UP));
rbi.addRule(seq(lowerRaise, the, itemLabel()));

    /**
     * Adds a rule.
     * Either the expression must contain a function to execute (e.g. send a command) or a specific function must be given.
     * @param {Expression} expression 
     * @param {function} executeFunction    If a specific function is given, it will override any function from the expression.
     */
    addRule(expression, executeFunction)

The following expressions are available to define a rule:

/**
 * Creates an alternative expression. All given expressions can be used alternatively, i.e. using an OR logic.
 * @param  {...Expression} expressions Any expression types.
 */
function alt(...expressions)
/**
 * Creates a sequential expression. All given expressions are processed sequentially in that order.
 * @param  {...Expression} expressions Any expression types.
 */
function seq(...expressions)
/**
 * Creates an optional expression. The given expression is not mandatory for a match.
 * @param  {Expression} expression 
 */
function opt(expression)
/**
 * Creates a command expression.
 * If the given expression is matched, the given command is sent to all found items of that rule.
 * @param {Expression} expression 
 * @param {string} command 
 */
function cmd(expression, command)
/**
 * Creates an item label expression.
 * It will try to match an item's label or its synonyms to the tokens at this point.
 * Only a single item must be matched.
 * The found item can be included in the final execution parameter, e.g. to send a command to that item.
 * @param {boolean} includeInExecuteParameter   Default: true.
 * @param {boolean} isGroup                     Default: false. If true, only group items (type: "Group") are matched.
 * @returns 
 */
function itemLabel(includeInExecuteParameter, isGroup)
/**
 * Creates an item properties expression.
 * It tries to filter items according to their properties: tags, item type or parent group.
 * If none of filter properties are given, then all items in the registry will be matched.
 * All matched items will be included in the final execution parameter, e.g. to send a command to these items.
 * @param {Expression} expression   The expression to match.
 * @param {string[]} tags           Default: []. Only items that have all the given tags will be matched.
 * @param {boolean} groupRequired   Default: true. If true, the expression must contain a group Item Label Expression. Only descendants of that group will be matched.
 * @param {string} itemType         Default: null. If a type is given, then only items of that type will be matched.
 */
function itemProperties(expression, tags, groupRequired, itemType)

Future Improvements

The rule template source code is maintained at GitHub: JanMattner/voice-control-openhab: A simple but extensible rule based voice control tool for openHAB.
Feel free to add your ideas for improvements there or in this forum thread.

• Add more out of the box rules, including other languages
• Verify rules in various scenarios
• Include custom rules from a separate script, so the update of the rule template is easier without the risk of data loss

Changelog

Version 0.2

• rename and update documentation

Version 0.1

• initial release

Resources

https://raw.githubusercontent.com/JanMattner/voice-control-openhab/main/rule-template/cuevox.yaml

[JanMattner]

Just published the template, first testing was successful.

Dear community,
I’d highly appreciate feedback from you testing it in your system, as I have tested it only in my limited environment, which surely not covers diverse settings (I just remember this discussion with @JustinG - I do not use the full semantic model or various device types).
Much appreciating also any contribution (rules or translation in other languages) or ideas for other rules.
Thanks!

[dantalion]

I tried installing this through the add-on marketplace but it failed with an http 429m error. Just to be sure the source I found here is the same add-on right? GitHub - JanMattner/voice-control-openhab: A simple but extensible rule based voice control tool for openHAB.

I proceeded to use the steps as found on github and that works great! I have some ideas that could make it even better potentially.

First, Could we extend the interpreter to ignore none alphabetical characters that shouldn’t matter. For instance I found that `turn on the lights in the house’ works for me but ‘Turn on the lights in the house.’ doesn’t. I think filtering things like , and . would be low hanging fruit and really nice as Speech to Text engines tend to insert them frequently.

Secondly, Do you it would be possible to use fuzzy matching for the interpretation, this could be used to make things like `LivingRoom’ and ‘living room’ equivalent. This one is probably more work but libraries already exist in almost any programming language that can achieve this. Otherwise some of those libraries might serve as excellent examples to write one from scratch.

Thanks for putting this together, I have the full chain working, Rustpotter, Habspeaker, Whishper-CPP, Mimic3 and your interpreter. I am thinking I should describe this in a blog post to attract more users as its quite an involved procedure.

Cheers

[JanMattner]

Thanks, good to see that it’s useful for you too!

I have added two issues for your enhancement ideas in GitHub:
Ignore none-alphabetical characters · Issue #12 · JanMattner/voice-control-openhab
Add fuzzy matching · Issue #13 · JanMattner/voice-control-openhab

However, the fuzzy matching would need a some rework (which existing libraries cannot solve), as each word is currently a single token and matching is done token by token, but for fuzzy matching you’d need to include several tokens at once in a look-ahead. But let’s see.

However, I think I will create a new add-on as a workaround for the error with this existing add-on.

[dantalion]

However, I think I will create a new add-on as a workaround for the error with this existing add-on.

Do let me know when that is up so I can test installing it for you

[JanMattner]

After changing the rule template resource from inline script to a GitHub link, the problem seems to be resolved for me. I can install the add-on now again without error.
You might need to refresh your add-ons. Much appreciating if you could give it a try and provide feedback.

[dantalion]

Installation worked but it doesn’t function yet, I am thinking it might be because I simply disabled the rule and script rather then deleting them so now the names conflict?

Will try again after a backup when I have deleted both files.

[dantalion]

I tried again after deleting everything cleanly, but it still does not seem to work. Is there I way I could verify the files from the add-on have ended up in there correct location?

[JanMattner]

I could force reloading the add-ons by doing the following (after removing the add-ons):

1. Go to Settings
2. Go to “Community Marketplace”
3. toggle “Show Unpublished Entries” and click save
4. toggle it back in its original state and click save

Then the add-on should be the latest version, which you can install. Maybe this works for you too.

[dantalion]

Unfortunately, after days of trying, reinstalling, restarting openhab, removing and recreating items, changing the settings back and forth between internal / rule interpreter. I still coulnd’t get it to work.

Sadly, I am usually quite pressed for time, so I will go back to the manual npm install for now.

I really appreciate the add-on and look forward to some freetime to extend the current rules. For instance I’d really like to set color temperature or color.

73,
Dantali0n (PD3SU)