Am on OH 5.3.0.M3 and installed the Gemini binding (model Gemini 3.1 Flash Lite which is the default) incl. the required API key. The chat is basically working fine.
But the chat string item “last response” is not updated (NULL). Need this to operate a rule based on this string item for a voice output of Gemini’s response. Can anybody point me into the right direction?
According to the docs there is no “last response” Channel. The response comes as the return value from the Rule Action. So if you have an Item where you want to store the last response, you need a update that Item yourself in the rule that calls one of the sendMessage Actions.
But I wonder if the docs need to be updated (I’m on the latest milestone) as the list of Channels doesn’t match what I’m seeing on the actual Thing. But what I do see there matches what the ChatGPT binding does with the single “Chat” Item.
You sendCommand the prompt and the binding will update the same Item with the result. The Item is not updated with the response when you call the sendMessage Action. If you use the action, you need to update that Item yourself.
Thanks for your input. I see a single channel “chat” and created a “response item” (string item) linked to this channel assuming that something should happen. My feeling was already that i misunderstood something.
Will give the rule approach a try.
Here is how I use an Item linked to that Channel in a rule.
// Clear and seed the OpenAI prompt proxy channel
console.debug('Sending prompt to LLM: ' + prompt);
items.OpenAI.postUpdate('');
items.OpenAI.sendCommand(prompt);
// Wait for LLM to respond to generate the message
timer.loop(() => {
console.debug('Waiting for LLM response...');
// sleep for a second if not yet ready TODO: add timeout
if(items.OpenAI.state == prompt || items.OpenAI.state == '') {
console.debug('No response yet');
return 'PT1S';
}
Note timer is a LoopingTimer from openHAB Rules Tools.
Though I wrote that when I wasn’t aware that there was a sendMessage Action or maybe there wasn’t one back then.
A more proper implementation would be something like:
const geminiActions = actions.get("gemini", "gemini:account:geminiAPI");
const response = geminiActions.sendMessage(prompt); // will block until there is a reply
// If you want to keep the response in an Item
items.GeminiLastResponse.postUpdate(response);
Thanks Rich. Am 2 days off now and will test it on Saturday and come back to you.
I must admit that i don’t get anything running. I think that i have still some basic understanding issues.
What i want to achieve is only that the openHAB chat response is spoken by the Piper TTS which i have configured. Have verified that Piper is working using the Karaf console.
My understanding is in order for the Chat on Main UI to work it needs to be configured with an LLM add-on that supported it and right now Gamini is the only one that supports it.
Gemini can be called directly from a rule or even the UI independently from the Chat field by commanding the Item linked to the Chat channel or using the rule actions.
Correct, this is working as designed. Interactions through the Chat feature is separate and independent from the Chat Channel.
The Chat channel can only take in text and output text. The Chat feature can cause things to happen in response to the input, like turn on a light.
The Chat feature in MainUI knows all about your OH instance and can read the states of Items and command Items.
The chat channel knows nothing about your (Zh instance. It’s like going to https://gemini.google.com and typing into the chat there. It only knows what you tell it.
In short, the Chat feature in MainUI and the chat channel are completely independent.
If the Chat feature supports voice output (I don’t use that so I don’t know if it does), it’s going to be configured through the whole generic OH voice and HLI pipeline and not through Items and Channels and Rules.
Also, be aware that the input and output to/from the LLM includes a while lot more than just the text you see in MainUI. Outputting that as voice may not be what you want.
Thanks for supporting me. Now i make progress in understanding how things work out.
Learned now, that webaudio as audio sink has to be enabled on each device/browser where it is expected to work.
My goal was to output the Chat response text as voice. TTS tests to webaudio as audio sink are working. The Chat response conversion to voice is still not working but according to the docs it should? Have not identified an extra setting for this.
Gemini & LLM-powered HLI author/“code owner” here:
There a three distinct concepts that seem to be mixed up:
You’d call Voice.interpret.
Many thanks for clarification. The docs link and its content is also excellent. Didn’t find this earlier.
My confusion mixing functionality up was a bit supported by the chat feature. It controls items but can also send a request to Gemini which is answered similar to the chat channel or thing actions.
Also, especially in this context, the AI search results are not really helpful because they mix up the concepts as well.
I may come back to you with “webaudio” issues. Initial trials led to problems with BasicUI and also MainUI.
Am going to test “webaudio” in order to get the desired voice output on my iPhone.
As soon as i send a voice output using a Blockly test script, voice output is working, but the webview sections are no longer reloaded or take an extremely long time. If i restart the BasicUI, everything seems to be normal again until the next voice output.
Any idea why this happens?
Edit 23.10.2026: i believe that every WebView item within my sitemap counts like another open browser tab. If webaudio is activated for this browser, a limitation for available http-channels occurs because webaudio broadcasts to all sessions in parallel.
Can somebody confirm this?