NEW 137883
Many (most?) transitions tests are very flaky, frequently failing all at once 
https://bugs.webkit.org/show_bug.cgi?id=137883
Summary Many (most?) transitions tests are very flaky, frequently failing all at once
Alexey Proskuryakov
Reported 2014-10-20 09:37:44 PDT
These tests are very very flaky, and have been for quite a while. Time to start ignoring them. transitions/background-transitions.html [ Failure Pass ] transitions/border-radius-transition.html [ Failure Pass ] transitions/change-values-during-transition.html [ Failure Pass ] transitions/clip-transition.html [ Failure Pass ] transitions/color-transition-all.html [ Failure Pass ] transitions/color-transition-premultiplied.html [ Failure Pass ] transitions/color-transition-rounding.html [ Failure Pass ] transitions/created-while-suspended.html [ Failure Pass ] @@ -1,3 +1,5 @@ -PASS - "background-position" property for "box" element at 0.5s saw something close to: 10,10 -PASS - "-webkit-background-size" property for "box2" element at 0.5s saw something close to: 20,20 +CONSOLE MESSAGE: line 240: Failed to pause 'background-position' transition on element 'box' +CONSOLE MESSAGE: line 240: Failed to pause '-webkit-background-size' transition on element 'box2' +FAIL - "background-position" property for "box" element at 0.5s expected: 10,10 but saw: 20,20 +FAIL - "-webkit-background-size" property for "box2" element at 0.5s expected: 20,20 but saw: 30,30
Attachments
tests.txt (259.48 KB, text/plain)
2014-11-14 14:57 PST, Alexey Proskuryakov 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2014-10-20 09:40:58 PDT
Marked them as flaky in <http://trac.webkit.org/changeset/174886>.
Simon Fraser (smfr)
Comment 2 2014-10-20 09:46:39 PDT
Would be nice to understand why these 8 all flake at once. Maybe some earlier test leaves bad state around?
Alexey Proskuryakov
Comment 3 2014-10-20 10:06:10 PDT
> Would be nice to understand why these 8 all flake at once. It definitely would! > Maybe some earlier test leaves bad state around? I tried running tests in exactly the same order as a failing tester on a bot did, and couldn't reproduce the issue. So it must be something more involved.
Simon Fraser (smfr)
Comment 4 2014-10-20 10:17:54 PDT
It's also interesting that the same bot hits this only sporadically. Does EWS hit it more reliably?
Alexey Proskuryakov
Comment 5 2014-10-20 10:43:04 PDT
EWS hits this in release builds, while regular bots only do in debug.
Alexey Proskuryakov
Comment 6 2014-11-13 17:07:43 PST
Something changed, and now it's 30 at once that fail, not counting ones that were already marked as flaky. This happens on EWS (Mountain Lion WK2 release) and on regular bots (Yosemite WK2 release or debug). Many of these are already marked flaky, perhaps we should mark all the rest?
Alexey Proskuryakov
Comment 7 2014-11-13 17:08:11 PST
*** Bug 113227 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 8 2014-11-14 10:06:12 PST
Skipped a lot more in <http://trac.webkit.org/r176125>, because having 30+ flaky failures breaks EWS. We should fix EWS (filed bug 138743), and we should fix this bug, but right now, marking the tests as flaky is the only way out. https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r176079%20(479)/results.html is an example of this problem occurring on a regular regression test bot. Looks like it's not only transitions tests that are affected, and this is a general timer problem.
Alexey Proskuryakov
Comment 9 2014-11-14 14:44:56 PST
This happens very rarely and is thus difficult to pinpoint. But looks like this started to happen for more tests about two weeks ago. Could it be <http://trac.webkit.org/changeset/175441>?
Alexey Proskuryakov
Comment 10 2014-11-14 14:57:04 PST
Created attachment 241628 [details] tests.txt This is the sequence of tests that a single worker process on a bot ran when this happened recently. But I cannot reproduce when re-running in the same order locally.
Alexey Proskuryakov
Comment 11 2015-07-25 19:21:19 PDT
This still happens, so we still can't have any test coverage for transitions. How fun is that?
Radar WebKit Bug Importer
Comment 12 2017-09-08 16:43:06 PDT
<rdar://problem/34342774>
Alexey Proskuryakov
Comment 13 2017-09-11 09:06:10 PDT
The original issue with 8 tests failing all at once doesn't seem to be happening any more. But we have many more transition tests marked as flaky now, and a few of them still are flaky. And a few are marked as flaky, but fail 100% of time. This will take some effort to clean up.
Note You need to log in before you can comment on or make changes to this bug.