OrangeAssist - Google Assistant Integration

aweeeeeeesssssssommmmee ! Have fun with it :slight_smile:

2 Likes

Another question :slight_smile:

I am trying to get my Pi to output the audio response from google but I am not hearing any audio when I make the request.

Request:

{
    "request": "what is the weather",
    "uuid": 123,
    "output_audio_file": "output.wav",
    "language": "en-US",
    "is_play_audio": true,
    "is_return_html": true
}

If I quit orangeassist and navigate to the output directory I can see the output.wav file has been created.
When I execute aplay output.wav I hear the google response on the speakers connected to the Pi.

There are no errors on the console when the response is received by orangeassist.
@luckymallari any ideas?

Also, side note: I had to manually create the /output sub-folder as it seems the code is hard-coded to look for this and I was getting 500 errors when it did not exist. As you can see from my config file, I am just specifying the file name and orangeassist creates the file in the output sub-folder automatically.

There’s enable_speaker on the config.json Make sure that’s set to true

Arg! Sorry I didn’t spot that, it is not listed in the OP but I should have seen it anyway!
I set enable_speaker to true and now I am getting a 500:

"POST /assist/ask HTTP/1.1" 500 412 0.058130
ERROR:flask.app:Exception on /assist/ask [POST]
Traceback (most recent call last):
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/openhabian/orange-assist/tmp/or.py", line 98, in decorated
    return f(*args, **kwargs)
  File "/home/openhabian/orange-assist/tmp/or.py", line 126, in create_task
    t, h = answer(r)
  File "/home/openhabian/orange-assist/tmp/or.py", line 82, in answer
    text_response, html_response = orange.assist(r)
  File "/home/openhabian/orange-assist/tmp/orangeassist.py", line 173, in assist
    flush_size=DEFAULT_AUDIO_DEVICE_FLUSH_SIZE
  File "/home/openhabian/orange-assist/tmp/helpers/audio_helpers.py", line 190, in __init__
    blocksize=int(block_size/2),  # blocksize is in number of frames.
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/sounddevice.py", line 1264, in __init__
    **_remove_self(locals()))
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/sounddevice.py", line 681, in __init__
    samplerate)
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/sounddevice.py", line 2490, in _get_stream_parameters
    info = query_devices(device)
  File "/home/openhabian/orange-assist/env/lib/python3.5/site-packages/sounddevice.py", line 488, in query_devices
    raise PortAudioError('Error querying device {0}'.format(device))
sounddevice.PortAudioError: Error querying device -1

This is probably another Linux issue vs orangeassist, so once again I appologise for my lack of knowledge.

Follow steps for configuring audio devices

https://developers.google.com/assistant/sdk/guides/service/python/embed/audio

I went through the guide and tested the speakers as described. I can hear the voice playing.
I then launched orangeassist and ran my query again, same error.
:confused:

# speaker-test -t wav

speaker-test 1.1.3

Playback device is default
Stream parameters are 48000Hz, S16_LE, 1 channels
WAV file(s)
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 512 to 65536
Period size range from 512 to 65536
Using max buffer size 65536
Periods = 4
was set period_size = 16384
was set buffer_size = 65536
 0 - Front Left
Time per period = 0.374149
 0 - Front Left
Time per period = 1.372050
 0 - Front Left
Time per period = 1.707663
 0 - Front Left
^CTransfer failed: Bad address
(env) [11:51:45] root@openHABianPi:/home/openhabian/orange-assist/tmp# python -m or
Cred file: /root/.config/google-oauthlib-tool/credentials.json
 + Found!
# orange assist v1 #
Server OrangeAssist with config:
{
    "credentials_file": "/root/.config/google-oauthlib-tool/credentials.json",
    "delete_output_files_sec": 60,
    "device_id": "OpenHAB",
    "device_model_id": "celeste-openhab-orangeassist",
    "enable_speaker": true,
    "host": "192.168.1.80",
    "is_debug": false,
    "is_verbose": true,
    "on_success": null,
    "password": null,
    "port": "2828",
    "project_id": "celeste-openhab-orangeassist",
    "screen_mode": "SCREEN_MODE_UNSPECIFIED",
    "username": null
}
Cred file: /root/.config/google-oauthlib-tool/credentials.json
 + Found!
Ready!
ERROR:flask.app:Exception on /assist/ask [POST]
Traceback (most recent call last):
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/openhabian/orange-assist/tmp/or.py", line 98, in decorated
    return f(*args, **kwargs)
  File "/home/openhabian/orange-assist/tmp/or.py", line 126, in create_task
    t, h = answer(r)
  File "/home/openhabian/orange-assist/tmp/or.py", line 82, in answer
    text_response, html_response = orange.assist(r)
  File "/home/openhabian/orange-assist/tmp/orangeassist.py", line 173, in assist
    flush_size=DEFAULT_AUDIO_DEVICE_FLUSH_SIZE
  File "/home/openhabian/orange-assist/tmp/helpers/audio_helpers.py", line 190, in __init__
    blocksize=int(block_size/2),  # blocksize is in number of frames.
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/sounddevice.py", line 1264, in __init__
    **_remove_self(locals()))
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/sounddevice.py", line 681, in __init__
    samplerate)
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/sounddevice.py", line 2490, in _get_stream_parameters
    info = query_devices(device)
  File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/sounddevice.py", line 488, in query_devices
    raise PortAudioError('Error querying device {0}'.format(device))
sounddevice.PortAudioError: Error querying device -1
192.168.1.143 - - [2019-01-31 11:52:32] "POST /assist/ask HTTP/1.1" 500 412 0.025555

Can you paste what’s in your .asoundrc file

.asoundrc

pcm.!default {
  type asym
  capture.pcm "mic"
  playback.pcm "speaker"
}
pcm.mic {
  type plug
  slave {
    pcm "hw:0,0"
  }
}
pcm.speaker {
  type plug
  slave {
    pcm "hw:0,0"
  }
}

result of aplay -l

**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 7/7
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

arecord -l lists no devices so I have tried the .asoundrc file with the pcm.mic section and the reference removed but it didn’t make any difference.

speaker-test -t wav

speaker-test 1.1.3

Playback device is default
Stream parameters are 48000Hz, S16_LE, 1 channels
WAV file(s)
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 512 to 65536
Period size range from 512 to 65536
Using max buffer size 65536
Periods = 4
was set period_size = 16384
was set buffer_size = 65536
 0 - Front Left
Time per period = 0.381968
 0 - Front Left
Time per period = 1.362102

And I can hear the test voice

Did you install portaudio

sudo apt-get install portaudio19-dev libffi-dev libssl-dev

The only other thing I could think of is that your audio devices is busy. Do you know how to setup snooping for pi? If not, let me know and I will give you instructions

Edit: looks like I did here

That will allow your audio devices to be used concurrently by any process

Yes, but I did it again just to be sure:

 sudo apt-get install portaudio19-dev libffi-dev libssl-dev libmpg123-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libffi-dev is already the newest version (3.2.1-6).
libmpg123-dev is already the newest version (1.23.8-1).
libssl-dev is already the newest version (1.1.0j-1~deb9u1).
portaudio19-dev is already the newest version (19.6.0-1).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

Tested and same issue.


I updated my .asoundrc file to match yours in the thread you linked (my HW IDs are the same):

pcm.snoopedmic {
    type dsnoop
    ipc_key 2241234
    slave {
        pcm "hw:0,0"
        channels 1
    }
}
pcm.mixedspeaker {
    type dmix
    ipc_key 2241234
    slave {
        pcm "hw:0,0"
        channels 1
    }
}

pcm.!default {
        type asym
        playback.pcm {
                type plug
                slave.pcm "mixedspeaker"
        }
        capture.pcm {
                type plug
                slave.pcm "snoopedmic"
        }
}

Still the same issue:

...
File "/home/openhabian/orange-assist/tmp/env/lib/python3.5/site-packages/sounddevice.py", line 488, in query_devices
    raise PortAudioError('Error querying device {0}'.format(device))
sounddevice.PortAudioError: Error querying device -1
192.168.1.143 - - [2019-01-31 15:57:17] "POST /assist/ask HTTP/1.1" 500 412 0.024712

Yep. It looks like this project is probably going to replace that one, if I can just get some time. I’m also trying to get Shinobi set up and having some “fun” reading the H.264 stream from one of my RPi cams. MJPEG works just fine so I may just give up and move on. I only have three cameras so it’s not like I need that many resources anyway.

I’ll get back to this this weekend I hope.

@luckymallari, I never touched python before.
Now I followed your tutorial and it worked up to this point:
image
I’m able to talk to my assistant and it switches lights for example.

But I don’t understand how to connect to this python-machine via http.
Which port/url do I use?

And I had to choose a device-type - there I selected “speaker”.
Is this relevant?

EDIT: It works now :slight_smile: After setting debug to false I reached the host/port from the config.json

EDIT2:
WORKS: Switch on < light >
WORKS: Remind me to < something > at < time >
DOESN’T WORK: play < movie > on netflix on shield
DOESN’T WORK: play < song > on < device or group >

Go to config.json and set debug to false. You will more configuration stuff there. Use the html tester so you know what to send to the url when you configure OH

@luckymallari technically it works now - but many commands don’t get a result. Especially the “play music on device” commands. Neither in english nor in german.

Did you register your device using the device tool (part of the SDK) and you can confirm your instance shows under your Google Home app?

I am having problem to create an instance of the device.

googlesamples-assistant-devicetool register-device --model "rpi-assistant-7cf1b-rpi-assistant-o137ha"

i just receive Error: Missing option “–project-id”.
Have tried with

googlesamples-assistant-devicetool register-device --project-id "rpi-assistant-7cf1b"  --model "rpi-assistant-7cf1b-rpi-assistant-o137ha"

Same error.

How did you do it

EDIT I did not success with the command but now i see it in my Google Home :grinning:

/Mike

Is it possible to run tester from a remote computer?

I am running this on a pi without browser.

/Mike

Yes. It’s a rest api

What is the url to tester.html.

/Mike