How to configure a ESPHome Voice Assistent Device

Off-Topic
21

I guess we need to introduce a way for bindings like ESPHome to get updates from the DialogProcessor when reaching these pipeline stages. I can offer looking into this once I’ve finished my org.openhab.core.voice restructuring. Want to break up the DialogProcessor and VoiceManagerImpl into smaller classes anyway …

22

Maybe some tasks can already start before the previous is finished.
For the recording the stream can already be stream to the transcription before it is finsihed, and maybe transcription parts can already be provided to the LLM.

Maybe that optimisation is for later, i hope we are able to achieve less then 2 second delay after the last word is spoken until the noticable action/response.

I’ll watch this closely, very interesting.

23

That could be difficult, especially if the processing part is faster than the part that provides the data.

For example, recording 5 seconds of a voice takes 5 seconds. will never be faster :wink:

But processing 5 seconds of speech takes, for example, 2 seconds.

If I start processing after 2 seconds, I’ll run out of data 1 second later. I’m not sure if an LLM can handle blocking voice streams.

that could work is streaming the answer early, because creating voice is always faster than realtime. Means we run never out of data.

24

WAV support (similarly to FLAC) is only compiled in if the compiler preprocessor directive is enabled — we’d have to check that in the firmware of the voice satellite.
Anyway, right now I am working on FLAC support for openHAB and transcoding between our supported audio formats. PCM is pretty much standard for TTS and STT, so we need to transcode from TTS’ PCM to FLAC.

25

Just when I’m having fun with Hermes agent, you guys come up with this… fantastic :slight_smile: I might try one of the cheaper options from m5stack though.
I’ll be following this thread with great interest!

26

If you use the OpenAI TTS API, like speaches ai, you can request FLAC directly. In this case, it just pass through.

28

Hi all,

a quick update from my side:

I’ve got the Home Assistant Voice Preview Edition up-and-running with openHAB!
Many thanks for sponsoring me the hardware @holger_hees :folded_hands:

The main issue was ESPHome Voice satellites require FLAC, but most TTS services generate signed PCM. If you use an “external” service like Holger, it may support FLAC output (transcoding the output internally), but our Piper TTS add-on only supports WAV (= signed PCM + RIFF header).
I’ve added support for returning signed PCM directly from Piper TTS to avoid adding and then removing the RIFF header again, and added FLAC audio format declarations to core.
I’ll try to get my audio transcoding PR ([audio] Implement transcoding support by florian-h05 · Pull Request #5878 · openhab/openhab-core · GitHub) into the next milestone.

Using the Python snippet from How to configure a ESPHome Voice Assistent Device - #18 by holger_hees, you can now talk to openHAB. Remainint task is to implement that interpreter loop inside openHAB, which I’ll do after modularising our voice infrastructure a bit.

Since @lsiepel asked for responsiveness, I got a few numbers.
My setup:

Usually, I receive a spoken response less than 5 seconds after the last spoken word. It heavily depends on the ambient noise, if there is too much the voice activity detection keeps the stream open too long and delays the start of the speech-to-text step. In my setup, STT and TTS finish somewhere in the range of a few hundred milliseconds, most of the time is spent in the interpretation stage (~ 1-3 seconds).

STT word error rate however isn’t the best with Whisper tiny, even though I use a fine-tuned version for German. I am currently exploring other options, especially NVIDIA’s Parakeet model looks promising to me.