Bug 220599

Summary: WebRTC live Opus audio stream stutters
Product: WebKit Reporter: Mikael Nousiainen <tmnousia>
Component: WebRTCAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Major CC: darin, eric.carlson, ews-watchlist, glenn, hta, jer.noble, philipj, sergio, tmnousia, tommyw, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 14   
Hardware: All   
OS: All   
See Also: https://bugs.webkit.org/show_bug.cgi?id=221504
Attachments:
Description Flags
Stuttering audio from live WebRTC Opus stream
none
Patch
none
Patch
none
WebRTC/Media logs from iPhone where the audio is not playing none

Description Mikael Nousiainen 2021-01-13 13:22:16 PST 
Created attachment 417555 [details]
Stuttering audio from live WebRTC Opus stream

The symptoms of this bug are very similar to bug https://bugs.webkit.org/show_bug.cgi?id=218762
but this is clearly a separate issue, as fixing that bug did not fix this one.

WebRTC live Opus audio stream stutters all the time.
The audio sounds like a "machine gun" (there's an audio clip as an attachment), where the audio is sliced into small pieces.

Here's a small JavaScript application, which reproduces the issue consistently every time.
The application should start playing back a 750 Hz sine wave tone after clicking the Play button.

https://webrtctest.incompleteopus.net:9001/webrtctest.html

There is Janus Gateway running in the background and the audio is generated by GStreamer in real time.
There is also a Coturn STUN/TURN server running too.

This sounds like some buffering issue in Safari or something related -- at least based on bug mentioned above, which was fixed.

The sine wave stream plays back cleanly in Firefox and Chrome (also on macOS) and the exact same WebRTC streaming setup was working
fine in Safari too until June/July 2020, somewhere around iOS 13.5 release. After that it has been broken and all Safari 14
release I've tested are broken too.

The issue can be reproduced on both iOS and macOS Safari.

I've tested this on Safari Technology Preview Release 118 (Safari 14.1, WebKit 16611.1.9.3.1)
and verified that the bug still exists on this latest TP release at the moment.
Comment 1 Mikael Nousiainen 2021-01-13 13:33:49 PST 
A note about the test application implementation:

The incoming WebRTC audio stream is played back through an AudioContext (to allow audio processing and visualization) and streaming is initiated by creating an Audio element where the stream is first assigned as srcObject and then wrapped via createMediaStreamSource() to be connected into the AudioContext.

The source code can be read from the example website.
Comment 2 Mikael Nousiainen 2021-01-14 01:03:42 PST 
This is a real blocker for my use case and I have not found any workarounds for it. I wonder that if anyone else has the same issue with live Opus streams? Seems they simple don't work -- not with AudioContext at least...

Anyway, I've updated the test application to work properly on desktop Safari 14 (there were some ES6 features in the code only supported by Safari Technology Preview). Now the test app works on desktop Safari 14 (14.0.2) and reproduces the issue on macOS.

However, while I can play back the audio on iOS 14 in the application the test case originates from, this example seems to produce only silence on iOS 14 (iPhone 7) currently. I have checked logs on Web Inspector and get no errors or other indications. I'm using a user gesture to trigger the audio playback and specifically resuming AudioContext in the click handler too.
Comment 3 Radar WebKit Bug Importer 2021-01-14 02:41:55 PST 
<rdar://problem/73190139>
Comment 4 youenn fablet 2021-01-14 02:58:20 PST 
Thanks Mikeal, I can reproduce the issue.
Comment 5 youenn fablet 2021-01-15 01:08:22 PST 
I wrote https://jsfiddle.net/b5sr8ey9/ which seems to be a reduced test case for this issue.
Creating two media stream source nodes on the same track is not working properly.
@Mikael, can you update your code to not do that and validate it fixes everything on your side?

A fix is needed on WebKit side.
Comment 6 youenn fablet 2021-01-15 01:24:11 PST 
Reduced test case with just local capture audio track: https://jsfiddle.net/7kf8r3de/
Comment 7 youenn fablet 2021-01-15 02:03:56 PST 
Created attachment 417689 [details]
Patch
Comment 8 Mikael Nousiainen 2021-01-15 03:39:54 PST 
Thanks for finding out the root cause, providing a patch and even pointing out a mistake in my code! (example updated)

I somehow never realized the stream was being created twice (because of some internal logic on Janus Gateway client code).

I can confirm that the stream plays without glitches in Safari 14 when there's only one source node attached.

However, what still bothers me is why iOS Safari produces only silence with this example. I'm using an iPhone 7 and just upgraded to latest iOS 14.4 Beta 2 yesterday.

Can you find out if there's another bug only on iOS -- or what is preventing  playback on mobile devices?
Comment 9 youenn fablet 2021-01-15 06:17:45 PST 
> Can you find out if there's another bug only on iOS -- or what is preventing
> playback on mobile devices?

It might be auto play policy restrictions.
You can try to do suspend/resume in inspector or enable Media/WebRTC logging to check that within the inspector.
Comment 10 youenn fablet 2021-01-15 06:18:30 PST 
(In reply to youenn fablet from comment #9)
> > Can you find out if there's another bug only on iOS -- or what is preventing
> > playback on mobile devices?
> 
> It might be auto play policy restrictions.
> You can try to do suspend/resume in inspector or enable Media/WebRTC logging
> to check that within the inspector.

If you execute JS from inspector, make sure to enable the "Emulate User Gesture" check box.
Comment 11 youenn fablet 2021-01-15 06:27:42 PST 
Created attachment 417695 [details]
Patch
Comment 12 Mikael Nousiainen 2021-01-15 08:44:24 PST 
I'm testing the example app directly on an iPhone with no Web Inspector or anything attached and it doesn't play anything.

There is code to make sure that AudioContext is resumed in the button click handler, but it doesn't seem to help.

Any other ideas? Are you able to test this yourself? Shouldn't the macOS and iOS Safari behave the same way here?
Comment 13 Mikael Nousiainen 2021-01-15 13:33:22 PST 
I enabled WebRTC and Media logging and got some log statements regarding audio playback not being permitted because it's not initiated by user gesture. However, I do get the exact same logs in desktop/macOS Safari and the playback works.

I've attached the full log file to this bug, but here are some interesting parts:

[Log] HTMLMediaElement::HTMLMediaElement(C5E20E3B8BE94F27) 
[Log] MediaSessionManageriOS::addSession(0) (B71CA082AB67FBF9)
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), Video(0), Audio(1), VideoAudio(0), WebAudio(5)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = AmbientSound, policy = Default
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureForFullscreen
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequirePageConsentToLoadMedia
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToAutoplayToExternalDevice
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequirePlaybackToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding InvisibleAutoplayNotPermitted
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureForAudioRateChange
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToShowPlaybackTargetPicker
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding AutoPreloadingNotPermitted
[Log] MediaElementSession::clientWillBeginAutoplaying(C5E20E3B8BE94F27) state = Idle
[Log] MediaElementSession::setState(C5E20E3B8BE94F27) Autoplaying
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), Video(0), Audio(1), VideoAudio(0), WebAudio(5)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = AmbientSound, policy = Default
[Info] HTMLMediaElement::setSrcObject(C5E20E3B8BE94F27) 
[Log] HTMLMediaElement::prepareForLoad(C5E20E3B8BE94F27) gesture = false
[Info] HTMLMediaElement::createMediaPlayer(C5E20E3B8BE94F27) 
[Info] HTMLMediaElement::cancelPendingEventsAndCallbacks(C5E20E3B8BE94F27) 
[Log] HTMLMediaElement::setPlaybackRate(C5E20E3B8BE94F27) 1
[Log] MediaElementSession::clientWillBeginAutoplaying(C5E20E3B8BE94F27) state = Autoplaying
[Log] HTMLMediaElement::setShouldDelayLoadEvent(C5E20E3B8BE94F27) true
[Info] MediaElementSession::removeBehaviorRestriction(C5E20E3B8BE94F27) removed RequirePageConsentToLoadMedia
[Info] RTCPeerConnection::dispatchEvent(912A778D) dispatching 'addstream'
[Log] RealtimeIncomingAudioSource::setMuted(BC77DDBD1D697ED7) false
[Log] RealtimeIncomingAudioSource::start(BC77DDBD1D697ED7) 
[Log] MediaStreamPrivate::trackStarted(6D8380859E058110) (BC77DDBD1D697ED7)
[Log] MediaStreamPrivate::trackMutedChanged(6D8380859E058110) (BC77DDBD1D697ED7) false


[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) loadstart
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(C5E20E3B8BE94F27) lambda(), task fired
[Log] HTMLMediaElement::mediaEngineWasUpdated(C5E20E3B8BE94F27)
[Info] MediaElementSession::mediaEngineUpdated(C5E20E3B8BE94F27)
[Log] VideoLayerManagerObjC::setVideoFullscreenFrame(C5E20E3B8BE94F27) 0, 0, 0, 0
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) progress
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) suspend
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) durationchange
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) loadedmetadata
[Log] BaseAudioContext::createMediaStreamSource(9589982D114A0B5C)
[Log] AudioNode::AudioNode(9589982D114A0002)
[Log] AudioNode::setNodeType(9589982D114A0002) NodeTypeMediaStreamAudioSource
[Info] AudioNode::addOutput(9589982D114A0002) NodeTypeMediaStreamAudioSource
[Log] BaseAudioContext::createGain(9589982D114A0B5C)
[Log] AudioNode::AudioNode(9589982D114A0003)
[Log] AudioParam::AudioParam(9589982D114A0001) name = gain, value = 1, default = 1, min = -3.4028234663852886e+38, max = 3.4028234663852886e+38, units = 0
[Log] AudioNode::setNodeType(9589982D114A0003) NodeTypeGain
[Info] AudioNode::addInput(9589982D114A0003) NodeTypeGain
[Info] AudioNode::addOutput(9589982D114A0003) NodeTypeGain
[Log] AudioNode::setChannelCount(9589982D114A0003) 2
[Log] AudioNode::setChannelCountMode(9589982D114A0003) 0
[Log] AudioNode::setChannelInterpretation(9589982D114A0003) 0
[Debug] AudioParam::setValue(9589982D114A0001) 1
[Log] AudioNode::connect(9589982D114A0002) NodeTypeGain, output = 0, input = 0
[Log] AudioNode::connect(9589982D114A0003) NodeTypeDestination, output = 0, input = 0
[Log] HTMLMediaElement::scheduleUpdateMediaState(C5E20E3B8BE94F27) lambda(), task fired
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
Comment 14 Mikael Nousiainen 2021-01-15 13:34:14 PST 
Created attachment 417729 [details]
WebRTC/Media logs from iPhone where the audio is not playing
Comment 15 Mikael Nousiainen 2021-01-15 13:37:42 PST 
The code initiates the WebRTC connection from user gesture (in button click handler).

However, the Audio element and the AudioContext graph are created in WebRTC remote stream callback when we have the remote MediaStream available. But I'm not sure how that could be different, as that's where we get the MediaStream...
Comment 16 EWS 2021-01-18 00:40:59 PST 
Committed r271575: <https://trac.webkit.org/changeset/271575>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 417695 [details].
Comment 17 youenn fablet 2021-01-18 02:29:40 PST 
(In reply to Mikael Nousiainen from comment #15)
> The code initiates the WebRTC connection from user gesture (in button click
> handler).
> 
> However, the Audio element and the AudioContext graph are created in WebRTC
> remote stream callback when we have the remote MediaStream available. But
> I'm not sure how that could be different, as that's where we get the
> MediaStream...

I tried on my iPhone and it seems to work now.