sounddevice.PortAudioError: Error opening RawStream: Device unavailable [PaErrorCode -9985] in the grpc sample

[moham96]

Hi,
This issue is related to the stream fix work that was done by @proppy in #188.
Testing on the raspberry pi 3 (running under python3)
If I use the master branch (includes the fix) I get the following error:

(env3dev) ➜  ~ googlesamples-assistant-pushtotalk
INFO:root:Connecting to embeddedassistant.googleapis.com
Traceback (most recent call last):
  File "/home/pi/env3dev/bin/googlesamples-assistant-pushtotalk", line 11, in <module>
    sys.exit(main())
  File "/home/pi/env3dev/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/pi/env3dev/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/pi/env3dev/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pi/env3dev/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/pi/env3dev/lib/python3.5/site-packages/googlesamples/assistant/grpc/pushtotalk.py", line 347, in main
    flush_size=audio_flush_size
  File "/home/pi/env3dev/lib/python3.5/site-packages/googlesamples/assistant/grpc/audio_helpers.py", line 190, in __init__
    blocksize=int(block_size/2),  # blocksize is in number of frames.
  File "/home/pi/env3dev/lib/python3.5/site-packages/sounddevice.py", line 1256, in __init__
    **_remove_self(locals()))
  File "/home/pi/env3dev/lib/python3.5/site-packages/sounddevice.py", line 772, in __init__
    'Error opening {0}'.format(self.__class__.__name__))
  File "/home/pi/env3dev/lib/python3.5/site-packages/sounddevice.py", line 2563, in _check
    raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Error opening RawStream: Device unavailable [PaErrorCode -9985]

While if I don't use these fixes (using the 0.4.4 sable release ) I can run the sample with no errors and i can interact with the assistant but sometimes I get the usual audio errors:

(env) ➜  ~ googlesamples-assistant-pushtotalk
INFO:root:Connecting to embeddedassistant.googleapis.com
Press Enter to send a new request...
INFO:root:Recording audio request.
WARNING:root:SoundDeviceStream read overflow (3200, 6400)
INFO:root:Transcript of user request: "tell".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me a".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me a joke".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell  me a joke".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me  a joke".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me a  joke".
INFO:root:Playing assistant response.
INFO:root:Transcript of user request: "tell me a joke".
INFO:root:Playing assistant response.
INFO:root:End of audio request detected
INFO:root:Transcript of user request: "tell me a joke".
INFO:root:Playing assistant response.
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
WARNING:root:SoundDeviceStream write underflow (size: 1600)
INFO:root:Finished playing assistant response.
Press Enter to send a new request...
INFO:root:Recording audio request.
INFO:root:End of audio request detected
INFO:root:Finished playing assistant response.
Press Enter to send a new request...
INFO:root:Recording audio request.
INFO:root:End of audio request detected
INFO:root:Finished playing assistant response.
Press Enter to send a new request...
INFO:root:Recording audio request.
INFO:root:End of audio request detected
INFO:root:Finished playing assistant response.
Traceback (most recent call last):
  File "/home/pi/env/bin/googlesamples-assistant-pushtotalk", line 11, in <module>
    sys.exit(main())
  File "/home/pi/env/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/pi/env/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/pi/env/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pi/env/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/pi/env/lib/python3.5/site-packages/googlesamples/assistant/grpc/pushtotalk.py", line 424, in main
    continue_conversation = assistant.assist()
  File "/home/pi/env/lib/python3.5/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/home/pi/env/lib/python3.5/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/home/pi/env/lib/python3.5/site-packages/tenacity/__init__.py", line 252, in iter
    return fut.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/home/pi/env/lib/python3.5/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/home/pi/env/lib/python3.5/site-packages/googlesamples/assistant/grpc/pushtotalk.py", line 169, in assist
    self.conversation_stream.stop_playback()
  File "/home/pi/env/lib/python3.5/site-packages/googlesamples/assistant/grpc/audio_helpers.py", line 288, in stop_playback
    self._source.stop()
  File "/home/pi/env/lib/python3.5/site-packages/googlesamples/assistant/grpc/audio_helpers.py", line 221, in stop
    self.flush()
  File "/home/pi/env/lib/python3.5/site-packages/googlesamples/assistant/grpc/audio_helpers.py", line 211, in flush
    self._audio_stream.write(b'\x00' * self._flush_size)
  File "/home/pi/env/lib/python3.5/site-packages/sounddevice.py", line 1202, in write
    _check(err)
  File "/home/pi/env/lib/python3.5/site-packages/sounddevice.py", line 2563, in _check
    raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Can't write to an input only stream [PaErrorCode -9974]

This error I get only on the raspberry pi, if I run the sample on my laptop I have no problem using the master branch

Thanks.

[moham96]

I just discovered that on my laptop if I have a .asoundrc file in my home directory that uses the devices directly I get the same error
sounddevice.PortAudioError: Error opening RawStream: Device unavailable [PaErrorCode -9985]

.asoundrc content:

pcm.!default {
  type asym
  capture.pcm "mic"
  playback.pcm "speaker"
}
pcm.mic {
  type plug
  slave {
    pcm "hw:0,0"
  }
}
pcm.speaker {
  type plug
  slave {
    pcm "hw:0,0"
  }
}

while if I change the content of asoundrc file to use pulseaudio the Device unavailable goes a way and I can interact with the assistant with no problems

pcm.!default {
  type pulse
  fallback "sysdefault"
  hint {
    show on
    description "Default ALSA Output (currently PulseAudio Sound Server)"
  }
}

ctl.!default {
  type pulse
  fallback "sysdefault"
}

[PizlaTheDeveloper]

You will have to set PulseAudio default devices.
e.g. [(https://askubuntu.com/questions/14077/how-can-i-change-the-default-audio-device-from-command-line)]

or uninstall PulseAudio :)

[moham96]

@PizlaTheDeveloper I don't think you understood the problem, actually, PulseAudio is the solution(more precisely a workaround) for the Device Unavailable error.

On the raspberry pi I don't have PulseAudio --> I get Device Unavailable error.

On the laptop running manjaro I have PulseAudio installed --> no Device Unavailable error and the pushtotalk sample works just fine.

[moham96]

Ok I did more tests on the raspberry pi.
I installed PulseAudio and started it and voila the Device Unavailable error is gone and the grpc sample works fine(Although the sound is stuttering, might be related to raspberrypi/linux#994 ?).
I didn't have to change the .asoundrc configuration to get the sample working, only having pulseaudio running in the background solves the problem even if I delete ~/.asoundrc and /etc/asound.conf .
This is not Ideal since I don't like using pulse and it would be better to have the sample working on alsa only system.

[Whuzz]

I think this has something to do with the changes in pushtotalk.py.
In main, after : # Configure audio source and sink.
Both audio_source and audio_sink call upon the SoundDeviceStream, causing two in- or output streams to be opened. (The second one cause a device unavailable error)

I might be completely mistaken though, since I'm no expert in python (yet).
I'm trying to fix it by splitting the audio_helper's rawstream into a rawinputstream and a rawoutputstream. So far, I've got it work, with the "minor" flaw of the assistant now sounding like a chipmunk. I saw that problem somewhere, so it should be fixable.

At least no more "Device Unavailable" anymore.

Edit: Fixed the chipmunk. Used too many channels on the RawOutputStream. (should be 1)

I now get underflow errors again though...

[moham96]

@Whuzz can you post your changes So I can try?

[moham96]

Strangely enough, I ran the audio_helpers.py and was able to record and playback the audio with no "Device Unavailable" error. So what is the problem in the pushtotalk.py that is causing the issue! I can't see any difference in pushtotalk.py compared to audio_helpers.py that might cause the issue.
@Whuzz can you run the googlesamples-assistant-audiotest and see If You have the "Device Unavailable" error?

[Whuzz]

My main change is in audio_helpers.py:

def __init__(self, sample_rate, sample_width, block_size, flush_size, device_used, channels_used, direction):
        if sample_width == 2:
            audio_format = 'int16'
        else:
            raise Exception('unsupported sample width:', sample_width)
        if direction == 0:
            self._audio_stream = sd.RawInputStream(
                samplerate=sample_rate, dtype=audio_format, channels=1,
                blocksize=int(block_size/2), device=device_used # blocksize is in number of frames.
            )
            self._block_size = block_size
            self._flush_size = flush_size
            self._sample_rate = sample_rate
            self.device_used = device_used
            self.channels_used = channels_used
            
        if direction == 1:
            self._audio_stream = sd.RawOutputStream(
                samplerate=sample_rate, dtype=audio_format, channels=1,
                blocksize=int(block_size/2), device=device_used # blocksize is in number of frames.
            )
            self._block_size = block_size
            self._flush_size = flush_size
            self._sample_rate = sample_rate
            self.device_used = device_used
            self.channels_used = channels_used

I set the channels to hardcoded 1 at the moment, but of course you can give them as arguments.
In pushtotalk.py I added some arguments to the audio_source and audio_sink definitions, so that each has its own "direction".

In pushtotalk.py, look for this: #Configure audio source and sink.
Below that you will find the creation of "audio_source" and "audio_sink" (that was changed by the last patch). I added some extra arguments to it, so that it handles the arguments I added in audio_helpers.py.
Instead of using one "RawStream" object, I use seperate objects (RawOutputStream / RawInputStream), to prevent both the audio_source and the audio_sink from engaging the same devices.

It works, but it seems to crash (hang) at random intervals at the moment, so I'll need to make it more robust.

I have a rather "difficult" setup, as I use an 8-microphone array for the audio-in. So many of the standard tests don't work.

Please note, I'm very inexperienced in Python, so treat my stuff as "uneducated guesses". I might be totally wrong and doing stupid stuff ;)

[moham96]

@Whuzz I used your changes only I ditched device=device_used and self.channels_used = channels_used and It Fixed the problem.
When you said that you get underflow error is it fatal(does the script exits) or does the script runs with no problem? I think this error is not important If the assistant mic and speakers works fine.
@proppy Can You take a look at @Whuzz fix above ?

[Whuzz]

I don't think it's the underflows that are fatal, no.
The crashes that I suffer from don't even exit the script, they completely hang it. Terminal becomes unresponsive (ctrl+C doesn't do anything except show ^C). Sometimes the whole system hangs.

It might have something to do with the threads? I'm thinking that a "stop/start recording" causes trouble for an RawOutputStream (as it doesn't record) and a "stop/start playback" casues trouble for a RawInputStream in their respective threads. But I'm not sure... I'm a threading-noob, coming from an embedded-C background ;)

Sometimes it works for a while (including underflows) and I can do several conversations before it hangs. Sometimes it hangs right after the first keypress. Maybe some race condition?

This seems related to #185

Glad that it seems to work for you at least!

[moham96]

Maybe It's because of your "difficult" audio setup, try using a single mic, I think you can disable the 7 mics from Alsa and leave only one mic working.

[Whuzz]

I need (well... really want) to use the 8 mics. In ALSA they are available as a single (beamformed) device, so it should be fine. Works with the assistant SDK library just fine. I'm mainly using the GRPC right now since I believe that might help me solve the bluetooth problem (see other bug report).

Once it's working, I plan on injecting "silent frames" before each assistant response to prevent audio loss.

[Starlon87]

@moham96 I met the same error on Raspberry Pi 3B

"I used your changes only I ditched device=device_used and self.channels_used = channels_used and It Fixed the problem."
Could you told me more details to fix it?

[proppy]

@moham96 @Whuzz @Starlon87 can you share mode details about your audio setup? (Which sound card, alsa configuration, pulseaudio or not?)

[Whuzz]

Soundcard for output: Standard Raspberry Pi, Jack out. (or bluealsa / Bluez, but for now testing with basic settings)
Sound for input: Matrix Creator 8 mic array

Alsa:

pcm.!default {
  type asym
  capture.pcm "mic"
  playback.pcm "speaker"
}

pcm.mic {
   type softvol
   slave {
     pcm "array"
   }
   control {
     name "MicArray Master"
        card 0
   }
}

pcm.speaker {
  type plug
  slave {
    pcm "hw:0,0"
  }
}

pcm.array {
  type plug
  slave {
    pcm "hw:2,0"
  }
}

No pulseaudio. Stuff like portaudio, sounddevice, etc all installed as per instructions: https://developers.google.com/assistant/sdk/guides/service/python/embed/install-sample