Bug 248561

The test failures seen in the following WK1 run were consistently failing on ToT WK1 until they were rebaselined last night, but the failures were still triggering retries in spite of our new results database checks.

https://ews-build.webkit.org/#/builders/73/builds/20801

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcss%2Fcss-cascade%2Fall-prop-initial-xml.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2FPositionOptions.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2Fpermission.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faccumulation-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faddition-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Finterpolation-per-property-002.html


The issue might be that when we query the results database to analyze failures, we are only checking for platform (macOS/iOS) and configuration (release/debug), which will return results from both WK1 and WK2 and throw off the pass/fail percentage.

Checking Results database for failing tests. Identifier: 257097@main, configuration: {'platform': 'mac', 'style': 'release'}
imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html: pass_rate: 72, pre-existing-failure=False
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 72, 'text': 27, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/geolocation-API/PositionOptions.https.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/geolocation-API/permission.https.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/accumulation-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/addition-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/interpolation-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}


If we had been checking for WK1 results specifically, it seems likely that we would have ignored the failures and skipped the retry step.