We are seeing quite a few failures on Ventura EWS that are only occurring on EWS and not on post-commit testers. The tests I have found so far are: http/tests/media/fairplay/fps-hls-update-reject.html http/tests/media/fairplay/fps-hls.html http/tests/media/fairplay/fps-init-data-skd.html http/tests/media/fairplay/fps-mse-unmuxed-key-renewal.html http/tests/workers/service/getnotifications-stop.html http/tests/workers/service/getnotifications.html http/tests/workers/service/shownotification-allowed-document.html http/tests/workers/service/shownotification-allowed.html http/tests/workers/service/shownotification-invalid-data.html http/wpt/push-api/pushEvent.any.serviceworker.html imported/w3c/web-platform-tests/notifications/idlharness.https.any.html imported/w3c/web-platform-tests/notifications/idlharness.https.any.serviceworker.html imported/w3c/web-platform-tests/notifications/idlharness.https.any.sharedworker.html imported/w3c/web-platform-tests/notifications/idlharness.https.any.worker.html It's a combination of Failing and timing out as observed by the results page here: https://ews-build.s3-us-west-2.amazonaws.com/macOS-AppleSilicon-Ventura-Debug-WK2-Tests-EWS/44c05b08-225-clean-tree/results.html We can tell that they're not occurring on post-commit bots with the following history link: https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=http%2Ftests%2Fmedia%2Ffairplay%2Ffps-hls.html&test=http%2Ftests%2Fmedia%2Ffairplay%2Ffps-init-data-skd.html&test=http%2Ftests%2Fmedia%2Ffairplay%2Ffps-mse-unmuxed-key-renewal.html&test=http%2Ftests%2Fworkers%2Fservice%2Fgetnotifications-stop.html&test=http%2Ftests%2Fworkers%2Fservice%2Fgetnotifications.html&test=http%2Ftests%2Fworkers%2Fservice%2Fshownotification-allowed-document.html&test=http%2Ftests%2Fworkers%2Fservice%2Fshownotification-allowed.html&test=http%2Ftests%2Fworkers%2Fservice%2Fshownotification-invalid-data.html&test=http%2Fwpt%2Fpush-api%2FpushEvent.any.serviceworker.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fnotifications%2Fidlharness.https.any.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fnotifications%2Fidlharness.https.any.serviceworker.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fnotifications%2Fidlharness.https.any.sharedworker.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fnotifications%2Fidlharness.https.any.worker.html&test=media%2FaudioSession%2FgetUserMedia.html
<rdar://problem/103411154>
It appears that this may be a test-list ordering issue. For the time being I am going to mark expectations for the above tests only for Ventura Apple Silicon WK2 arm64 below:
Test gardening commit 257949@main (c9019ce76e4b): <https://commits.webkit.org/257949@main> Reviewed commits have been landed. Closing PR #7712 and removing active labels.
Test gardening commit 258751@main (7e9151364f67): <https://commits.webkit.org/258751@main> Reviewed commits have been landed. Closing PR #8491 and removing active labels.
From what I could tell this was no longer happening. It may have been related to us having the wrong Xcode/Mac SDK on the EWS bots that has since been corrected. I have removed the expectations above. If it starts happening again we will need to re-file.