Bug 244203

Summary:	recording audio using audioworklet produces corrupted data
Product:	WebKit	Reporter:	jk
Component:	Web Audio	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	asi, cdumez, eric.carlson, jer.noble, mkahlau, rychouwei, webkit-bug-importer, youennf
Priority:	P2	Keywords:	InRadar
Version:	Safari Technology Preview
Hardware:	Mac (Intel)
OS:	macOS 12

Description jk 2022-08-22 08:07:06 PDT

When the issue is hit. Recording someone counting numbers 1,2,3,4,5,6,7,8,9,10 will yield samples containing something along the lines of 2,3, 5,6,7, 9,10.

We have created a minimal test case for the issue. It records audio from the first microphone returned by media query using audioworklet's port.postMessage() to send it to the js thread. Then after recording plays the received samples using same sample rate as reported by the audio context used for recording.

STEPS TO REPRODUCE:
I don't have a reproducible sequence of events to lead to the issue, but it does occur frequently and seems to be easy to reproduce again and again when the correct preconditions are hit.

To try to reproduce, load the page using audioworklet for recording: https://jk.numfum.com/mic-issue/index.html
and open https://jsfiddle.net/st___h/36w7vxn2/27/show to another tab.
See: https://jk.numfum.com/mic-issue/tabs.mov for reproduction where the fiddle is run in Version 15.6.1 (17613.3.9.1.16) and the audio worklets in Tech preview Release 151 (Safari 16.0, WebKit 17615.1.1.2)

In the jsfiddle, start playback of the audio, then in the audioworklet page, press "record", after 10 or so seconds, press "stop", close the fiddle tab, so it doesn't make any noise and on the audioworklet page press "playback".
Playing back the received samples, when the issue is reproduced, will skip 1-2 seconds every 1-2 seconds. Estimating the recorded sample rate based on the received samples and duration of recording yields roughly 25.5k, when all requested and reported sample rates are 48k. When the issue doesn't occur, the estimated sample rate is reported pretty accurately a 48k (+/-0.1k)

See: https://jk.numfum.com/mic-issue/spotify.mov for another reproduction using spotify as audio source.

We have reproduced this on at least:
Release 151 (Safari 16.0, WebKit 17615.1.1.2) (intel)
safari Version 15.6 (17613.3.9.1.5) (m1)

Another interesting symptom when reproducing this is that when the recording is started, playback of audio _in another Safari tab/safari process_ speeds up (and changes the pitch). This seems to be reproducible almost 100% on my machine, so I don't know if the issues are related. Worth noting anyway.
As seen in the videos, when bootsrapping the microphone, the cpu usage of coreaudiod goes briefly very high. The latter of the above videos shows over 500%.

files for the test as a zip: https://jk.numfum.com/mic-issue/mic-issue.zip

Comment 1 Radar WebKit Bug Importer 2022-08-23 06:29:10 PDT

<rdar://problem/99030355>

Comment 2 jk 2022-09-23 02:25:22 PDT

I retested this on:
Safari Beta Version 16.1 (17614.2.5, 17614)
and
Safari Tech Preview Release 154 (Safari 16.0, WebKit 17615.1.6.2)

On 16.1 I don't hear the pitch shifting anymore, but the issue with roughly half of the recorded samples being missing is still happening.
On Tech Preview both the pitch shift and missing samples are happening. Though I believe the Pitch shift was marked as fixed in TP 153 release notes (https://github.com/WebKit/WebKit/commit/65ae54bdedbb604855de7306576ed9572dd7bfe3)

FWIW, another change in these versions is that trying to use an aggregate device created in Audio MIDI Setup fails with "A MediaStreamTrack ended due to a capture failure" while it used to work in previous versions.

It also looks like playing something on Spotify desktop client while starting the recording makes it more likely to trigger missing samples issue.

Comment 3 mkahlau 2023-04-18 02:27:29 PDT

I have probably faced the same issue using Safari and Chrome on iOS 15.7.3 on an iPhone 7 Plus.

When recording with the built-in microphone using AudioWorklet (and latencyHint "playback") by pushing each sample of each audio block of channel 0 to an array as the audio blocks are coming into the AudioWorkletProcessor.process method, some samples are regularly missing. It's easy to notice if you count numbers, like 1 to 20 (about seconds apart). When playing back the recordings I noticed that every few numbers 1 to 3 numbers are missing or cut off, e.g.: 1, 2, 3, 4, 5, 8, 9, 10, 11, 12, (13), 15, 16, 17, 18, 19

Note: With Safari and Chrome on iOS, the sample rate mentioned in the track settings is 44.100 Hz. However, the sampleRate retrieved from the AudioContext is 48.000 Hz. I wonder why the settings are not the same.

The mentioned behavior unfortunately makes AudioWorklet-based recording unusable with Webkit-based browsers. On the other hand, if you record with ScriptProcessorNode, this problem does not occur. Unfortunately, ScriptProcessorNode has been deprecated for some time, and AudioWorklet would be state-of-the-art. 

On other operating systems like Ubuntu (Chrome, Firefox) and Android (Chrome) I could not reproduce the problem with AudioWorklet. I therefore assume that it is Webkit-specific. 

Any help highly appreciated. Let me know, if you need more info or another minimal test case.