Native mic button in Android app doesn't work with Gemini (LLM-based) HLI as default interpreter

Apps & Services Android App
1

Hi all,

I’m running openHAB 5.2.0 with the new Gemini binding, configured as the
default Human Language Interpreter (Settings → Voice → Default Human
Language Interpreter = Gemini).

The Chat feature in Main UI works perfectly, both in the browser and
inside the Android app (loaded via WebView) — it correctly interprets
natural language commands and controls Items through Gemini’s tool/function
calling.

However, the native microphone button in the Android app itself (top
right icon, not the Chat inside Main UI) does not work:

  • Android’s speech recognizer correctly transcribes what I say
    (confirmed on screen)
  • Nothing happens afterwards — no Item state changes, no error shown
  • Nothing appears in openhab.log, not even with DEBUG logging enabled
    for org.openhab.binding.gemini
  • Confirmed via adb logcat that… (rellena aquí si llegas a hacer la
    prueba con adb, por ejemplo si ves o no una petición HTTP saliendo de
    la app)

According to the official docs ( Android App | openHAB ), the
“VoiceCommand” Item mechanism only applies when Default Human Language
Interpreter is set to “Rule-based Interpreter”. Since my default
interpreter is Gemini (not Rule-based), I created a VoiceCommand Item
anyway just to test, and it made no difference — as expected, since the
doc says it only applies to the Rule-based Interpreter.

Questions:

  1. Does the Android app’s native mic button currently support LLM-based
    HLIs (like Gemini) as the default interpreter, or is it still
    hardcoded to only work with the Rule-based Interpreter / VoiceCommand
    Item mechanism?
  2. If it’s not supported yet, is there a workaround, or is this a known
    limitation that will be addressed?

Setup details:

  • openHAB version: 5.2.0 (confirm exact build if different)
  • Android app version: 3.20.7
  • Gemini binding, model: gemini-flash-lite-latest
  • Server: LXC container on Proxmox VE
  • Tested with mic permission granted, battery optimization disabled for
    the app, connected via local WiFi (not remote URL)

Thanks in advance for any pointers!

2

What does it say in adb?

The app sends to command to the configured voice interpreter. If it’s unavailable it falls back to the legacy `VoiceCommand` item.

3

Thanks for pointing me to the source code — that was very helpful for
narrowing this down.

I confirmed via adb logcat that the app does call the rest/voice/interpreters
endpoint successfully (WorkManager reports SUCCESS, HTTP-wise the request
completes), so it’s not a connectivity or auth issue. The problem is deeper:
it’s a difference in how the Gemini binding builds the request depending on
which endpoint is used.

I enabled DEBUG logging on org.openhab.binding.gemini and compared the two
request paths side by side, sending the exact same command
(“enciende la luz del despacho” / “turn on the office light”) each time,
same Thing config, same model (gemini-flash-lite-latest).

Via Main UI Chat (/rest/voice/conversations) — works correctly:

The outgoing request to generateContent includes a full “tools” block:

"tools" : [ {
  "functionDeclarations" : [ {
    "name" : "get-date-time", ...
  }, {
    "name" : "item-get-state", ...
  }, {
    "name" : "item-send-command",
    "description" : "Control items via commands...",
    "parameters" : { ... }
  } ]
} ]

Gemini correctly returns a functionCall, the binding executes it, and the
Item is updated. This has worked reliably across many turns of conversation.

Via the Android app’s native mic button (/rest/voice/interpreters) — fails every time:

The outgoing request to generateContent has “contents”, “systemInstruction”
and “generationConfig” — but no “tools” block at all. The
systemInstruction still tells the model to “use the tools provided”, but no
tool/function declarations are actually attached to the request.

Gemini’s response in this case is:

"finishReason": "MALFORMED_FUNCTION_CALL",
"finishMessage": "Malformed function call: call:openHAB:sendCommand(itemName=\"Luz_Despacho\", command=\"ON\")"

My reading: since the model is instructed to use tools but none are declared
in the request, it still attempts to emit a function call based on the
system prompt description, and the result comes back malformed because the
model has no real function schema to conform to. This happens 100% of the
time when going through /rest/voice/interpreters, on every model I’ve tested
(2.0, 2.5, 3.1, 3.5, 3.6 — where quota allowed testing), while the same
models work fine through the Chat / conversations endpoint.

Summary / suspected root cause:
It looks like the Gemini HLI’s interpret() code path (used by
/rest/voice/interpreters, and therefore by the app’s native voice button)
isn’t attaching the function/tool declarations to the generateContent
request, while the conversation-based path used by Main UI Chat does attach
them correctly.

Setup: openHAB 5.2.0, Gemini binding, Android app 3.20.7, tested with both
a corporate-restricted phone and a Realme Pad (RMP2103, Android 11) to rule
out device-specific causes — same result on both.

Happy to provide the full raw request/response JSON for both paths (redacted
of course) if that helps track this down, or to test a fix/snapshot build if
one becomes available.

4

The /rest/voice/interprets endpoint expects a comma separated list of llmtools to use. Available tools can be retrieved with the /rest/voice/llmtools endpoint. Either we need to adjust the default endpoint to select all tools by default or make the Android app to select the tools.

5

Wouldn’t the former make more sense in terms of backwards compatibility?

6

Yes.
Though „backward“ compatibility is kind of the wrong term for such new functionality …

7

Why? The Android app uses the rest/voice/interpreters endpoint since a long time. It used to work fine, but now - depending on choice of HLI - doesn’t anymore, at least not without client changes. That’s a text book example for a backwards compat issue in my book.

8
9

The PR has been merged a few minutes ago and the issue will be fixed with openHAB 5.2.1.
If none LLM tools are explicitly selected, openHAB will default to providing all tools to the LLM.

10

The code alekanda linked shows it should log ‘Try to send update to voice interpreters endpoint’ before attempting the call. Did you check the Android app’s logs (via adb logcat or the app’s own debug output) to see if that message even appears when you tap the mic button? If it’s not there, the app isn’t reaching that code path at all. If it IS there but you’re getting no response from the server end, that’d point to something on the openHAB side not exposing the endpoint correctly for Gemini.

11

This problem has been fixed in version 5.2.1