Bug 245329

Summary:	EWS false positive when crashes didn't reproduce on run-layout-tests-without-change
Product:	WebKit	Reporter:	Alexey Proskuryakov <ap>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	aakash_jain, clopez, jbedard, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=219500 https://bugs.webkit.org/show_bug.cgi?id=174537 https://bugs.webkit.org/show_bug.cgi?id=231999

Description Alexey Proskuryakov 2022-09-17 19:58:08 PDT

https://github.com/WebKit/WebKit/pull/4441 got wrongly marked as failing. Buidbot link: https://ews-build.webkit.org/#/builders/73/builds/14733

These tests crashed on first try:

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-insensitive.html
inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

Retry:

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

But nothing crashed on run-layout-tests-without-change, which only ran these two tests.

Was this just an unfortunate coincidence due to flakiness? Or did crashing depend on having run preceding tests? I'm feeling uneasy about only running tests that failed in run-layout-tests-without-change, we seem to be getting a lot of false positives lately.

Comment 1 Carlos Alberto Lopez Perez 2022-09-19 04:42:36 PDT

> Was this just an unfortunate coincidence due to flakiness? Or did crashing depend on having run preceding tests?

In this case it looks to me it was an unfortunate coincidence due to flakiness.

See: https://results.webkit.org/?suite=layout-tests&test=inspector%2Fdebugger%2Fsymbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html  looks flaky (crash/pass) on "Big Sur E wk1 Release on Macmini8,1 with x86_64" and other Apple platforms.

I suggest to mark this test as flaky on the TestExpectations for the Apple ports.

> I'm feeling uneasy about only running tests that failed in run-layout-tests-without-change, we seem to be getting a lot of false positives lately.

The EWS GTK or WPE bots use a new class added in bug 231999 (LayoutTestClass = RunWebKitTestsRedTree) which repeats failures at least 10 times to discard flakies. We couldn't make reliable in our ports the default layout test runner, because of the unpredictable random flakies, so we took this big hammer approach to discard flakies on the EWS itself by repeating them 10 times.

So If you wish, I don't have any issue at this moment with removing the optimizations added on bug 219500 and bug 231265 (the GTK/WPE bots don't use that code as said above). But removing that is going to cause the EWS bots that use the default layout test runner to do more work, so they will take longer to finish.

Comment 2 Radar WebKit Bug Importer 2022-09-24 19:59:17 PDT

<rdar://problem/100370073>