imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub.html is frequently failing on macOS WK1 bots after the re-sync in https://commits.webkit.org/255124@main --- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-expected.txt +++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/reporting-api-sends-reports-on-violation.https.sub-actual.txt @@ -3,5 +3,5 @@ PASS Test that image does not load PASS Event is fired FAIL Report is observable to ReportingObserver assert_equals: expected 53 but got 0 -PASS Violation report status OK. +FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting-api%2Freporting-api-sends-reports-on-violation.https.sub.html
<rdar://problem/101045517>
Seeing a similar failure with another WPT test in https://bugs.webkit.org/show_bug.cgi?id=246351, probably related?
Pull request: https://github.com/WebKit/WebKit/pull/5240
Test gardening commit 255395@main (4900b6842429): <https://commits.webkit.org/255395@main> Reviewed commits have been landed. Closing PR #5240 and removing active labels.
Unfortunate this is flaky on WK1. The part that is failing is in an async defer <script> tag. Is that feature well supported on WK1?
Looks like this may also apply to imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which was updated in the same re-sync: --- /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-expected.txt +++ /Volumes/Data/worker/Apple-Monterey-Release-AppleSilicon-WK1-Tests/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub-actual.txt @@ -3,5 +3,5 @@ PASS Test that image does not load PASS Event is fired FAIL Report is observable to ReportingObserver assert_equals: expected 54 but got 0 -FAIL Violation report status OK. undefined is not an object (evaluating 'data[0]["body"]') +PASS Violation report status OK. https://results.webkit.org/?suite=layout-tests&test=imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html
Yeah, both the failures are happening in the 'checkReport.sub.js' script. I don't think the 'async defer' bits are relevant, because they would only impact the timing of the page being fully parsed/constructed (whether it waits for this script to load first). Since the script runs an XHR, and continues when that XHR receives data, it seems like it should be resilient to timing differences. However, this test failure makes it seem like the XHR completes (without error), but contains no data. It's like the WPT Stash thing is sometimes returning an empty value, but is not treating that as an error state.
(In reply to Brent Fulgham from comment #7) > Yeah, both the failures are happening in the 'checkReport.sub.js' script. I > don't think the 'async defer' bits are relevant, because they would only > impact the timing of the page being fully parsed/constructed (whether it > waits for this script to load first). > > Since the script runs an XHR, and continues when that XHR receives data, it > seems like it should be resilient to timing differences. > > However, this test failure makes it seem like the XHR completes (without > error), but contains no data. > > It's like the WPT Stash thing is sometimes returning an empty value, but is > not treating that as an error state. Details: https://web-platform-tests.org/tools/wptserve/docs/stash.html