This is a wiki topic, regular members as well as staff may edit it freely and are encouraged to do so! The content will eventually be migrated somewhere in the main documentation when it’s stabilized and enough feedback has been received. When replying to posts on this series, please try to stay on-topic - there will be several of them, each covering a particular feature or area, so please look for the appropriate topic and reply there, or create a new topic. Thanks!
When using HABot, one of the main challenges is to figure out the right information from the original natural language query. Items are openHAB’s conceptual model of your home so this is where the information will be found. To do that, the interpreter and associated skills have to transform Entities extracted by Natural Language Processing into actual items to display or act on.
There has been several iterations on how to perform this task, but ultimately it led to the introduction of the semantics core bundle in the Eclipse SmartHome framework.
By applying semantic tags to your items, you will able to describe more accurately what they actually represent (an equipment, a location) as well as relations between them; it will then become possible for clients like HABot to perform advanced queries on them, like “items related to a certain property (Temperature) located in a certain place (Ground Floor)”; additionally, attributes to refer to them will be provided for you in several languages so clients won’t have to rely on the items’ labels alone. It is also an attempt at unifying how to describe and match items among different natural language “assistant” integrations like Alexa, Google etc. and avoid custom implementations of this particular problem in each of them. Please note that semantics are a very recent addition to ESH, so they are still subject to change and will be expanded over time!
The Eclipse SmartHome Ontology & Semantic Tags
The ESH ontology was built according to continuing discussions about a tag library, and the Brick Schema as a baseline. The schema below illustrates its initial version. The up to date version is available on github.
The ontology defines 4 main types: Location, Property, Point and Equipment, with associated tags for each of them. As you can see on the schema, the entire model is an actual Java class hierarchy (the 4 main types are also derived from the generic Tag interface). Therefore GarageDoor is a kind of Door, OpenState is a kind of Status, and Basement is a kind of Floor, which is also a location inherited from Indoor.
According to the schema, currently you can only tag an item with one of the Location, Equipment or Point tags, but you can combine a Point tag and a Property tag on the same item.
The following is also applicable and noteworthy:
- The hasLocation relations and the isPartOf relation for locations will be using group memberships to determine objects in a location, so it only makes sense to put Location tags on Group items which represent a location, like a floor or a room
Group gFF "First Floor" <firstfloor> ["FirstFloor"]
Group gGF "Ground Floor" <groundfloor> ["GroundFloor"]
Group gC "Cellar" <cellar> ["Basement"]
Group Garden "Garden" <garden> ["Garden"]
Group GF_Living "Living Room" <video> (gGF) ["LivingRoom"]
Group GF_Kitchen "Kitchen" <kitchen> (gGF) ["Kitchen"]
- Points and Properties should usually used together, so that the “relatesTo” relation is implied by having a single item tagged with a Point & Property. If a Property tag is found without the associated Point, the item will implicitely be considered to be a Status if the item is read only, or a Control if it can accept commands.
Number Temperature_GF_Corridor "Corridor [%.1f °C]" ["Temperature", "Measurement"]
Switch Light_FF_Bath_Ceiling "Ceiling" ["Light"]
Dimmer Temperature_Setpoint "Temperature [%.1f °C]" ["Setpoint", "Temperature"]
- In theory (!) Equipment tags are only valid on groups, similary to locations, so that the group memberships build the hasPoint and isPointOf relations; that being said, nothing currently prevents a non-Group item from representing an Equipment without specifying which kind of Point it measures or controls - this is useful when there is only one item for that equipment for the unspecified main functionality (or Point) of that equipment:
# Equipment with multiple Points
Group gKitchenSensor (gKitchen) ["MotionDetector"]
Contact KitchenSensor_MotionDetected (gKitchenSensor) ["Status"]
Switch KitchenSensor_LowBattery (gKitchenSensor) ["LowBattery"]
Switch KitchenSensor_Tampered (gKitchenSensor) ["Tampered"]
# Equipments with no explicit Points
Rollershutter Shutter_GF_Living "Livingroom" (GF_Living) ["Blinds"]
Contact Garage_Door "Garage Door [MAP(en.map):%s]" ["GarageDoor"]
Contact Window_GF_Kitchen "Kitchen [MAP(en.map):%s]" ["Window"]
Switch Heating_GF_Corridor "Corridor" (GF_Corridor) ["HVAC"]
(in the future it will likely be possible to specify a Point and Property along with an Equipment).
Tag-provided Terms and User-defined Synonyms
As mentioned above, tagged items will automatically be provided with default labels, plurals and synonyms predefined in OHC in several languages useable in queries. These Terms may also be expanded on a per-item basis, when:
- there is no tag in the ontology exactly matching the item’s nature;
- the item needs to be referred to with a specific name in queries (in addition to the label), like “Amy’s Room”.
For these scenarios, a comma-separated list of user-supplied synonyms can be provided for individual items using metadata in the synonyms
namespace. Just tag the item with the closest matching tag you can find in the ontology and add one or several synonyms, separating them with commas - note that the item’s label will also implicitely added as a term for the item, therefore no need to add it as a synonym!
Group GF_Toilet "Toilet" (gGF) ["Bathroom"] { synonyms="Toilets,WC,Restroom" }
Group FF_Office "Office" (gFF) ["Room"] { synonyms="Study" }
Group FF_Son "Oliver's Room" (gFF) ["Bedroom"] { synonyms="Oli's Room" }
Group FF_Daughter "Amelia's Room" (gFF) ["Bedroom"] { synonyms="Amy's Room" }
Group FF_Bed "Bedroom" (gFF) ["Bedroom"] { synonyms="Master Bedroom,Parents' Room" }
If you define item synonyms, plurals have to be supplied too (when it makes sense) so you can refer to a group of them using the plural forms.
In short, when looking for items related to a particular term, the framework will look in:
- The item tags’ label and synonyms (in the current language);
- The item’s label, without the format information between brackets;
- The user-supplied synonyms in the
synonyms
namespace.
How HABot Resolves Items
One of the components of the natural language processing in HABot’s case apart from intent classification is the entity extraction, that is, extracting the important parts of the sentence and determining their type.
A lot of training focuses on sentences containing “<what> in the <where>”, which will be extracted in entities with the object (“what”) and location (“where”) types.
Note that “object” is not in ESH’s ontology; for technical and historical reasons, HABot is unable to distinguish between equipments, points and properties, therefore an object in HABot’s terminology can refer indifferently to a Equipment, Point or Property as defined in ESH’s ontology.
Terms associated with individual items apart from its label are called “(named) attributes” in HABot’s terminology for the same reasons; in other words, attributes are what HABot considers how each item can be referred to. When chatting, the interpreter will resolve items from the entities extracted from the sentence (object=Lights, location=Kitchen) by looking at items and their attributes.
You can review which attributes are associated with your items in HABot’s UI: click on Settings in the sidebar, then the View item attributes option:
The chips in the last column indicate whether the attribute is an object or a location (with icons), and whether it comes from a tag (in blue) or a synonym defined with metadata (in black). Remember, the label is also automatically considered.
Next Steps
The next article will introduce a cool feature of the HABot Progressive Web App - how you can have it app-like full screen experience on your Android phone or desktop computer instead of running it in the browser.