Bug 217414

Summary: REGRESSION (r267402): [ iOS Mojave+ Release ] imported/w3c/web-platform-tests/user-timing/measure-l3.any.html is a flaky failure
Product: WebKit Reporter: Karl Rackler <rackler>
Component: New BugsAssignee: Diego Pino <dpino>
Status: RESOLVED FIXED    
Severity: Normal CC: achristensen, clopez, dpino, ews-watchlist, ntim, sam, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://github.com/web-platform-tests/wpt/pull/26192
Attachments:
Description Flags
Patch
none
Patch none

Description Karl Rackler 2020-10-06 16:51:32 PDT 
Description:
imported/w3c/web-platform-tests/user-timing/measure-l3.any.html

The first failure that I see this on the dashboard was 07/29/20 at r265034 for iOS 13, then 08/25/20 began to be consistent passing for iOS and macOS, with a few flaky failures on iOS.  On 09/22/20 at r267403 macOS and then iOS began to flaky fail together, but this commit does not appear to be relevant to what would have caused this.

I can reproduce this on r267403, but unable to reproduce on r267401 or earlier.
This test was modified in the layout test import in r267402, so that probably introduced the flakiness.

REPRODUCTION STEPS
Command: 
run-webkit-tests --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --iterations 50 imported/w3c/web-platform-tests/user-timing/measure-l3.any.html

Result: 
Regressions: Unexpected text-only failures (1)
  imported/w3c/web-platform-tests/user-timing/measure-l3.any.html [ Failure ]

History:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fuser-timing%2Fmeasure-l3.any.html&platform=ios&platform=mac&limit=50000

Diff Log:
Catalina wk2
--- /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 9.000000000000002 but got 9
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively

Mojave wk2
--- /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 9.000000000000002 but got 9
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively
 
iOS wk2
--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 51.00000000000001 but got 51
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively
Comment 1 Radar WebKit Bug Importer 2020-10-06 16:53:24 PDT 
<rdar://problem/70024016>
Comment 2 Karl Rackler 2020-10-06 17:16:34 PDT 
I have marked this test as failing while this issue is investigated.
https://trac.webkit.org/changeset/268093/webkit
Comment 3 Diego Pino 2020-10-22 07:43:46 PDT 
Created attachment 412095 [details]
Patch
Comment 4 EWS Watchlist 2020-10-22 07:44:39 PDT 
This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess
Comment 5 Diego Pino 2020-10-22 07:46:05 PDT 
Link to same change in WPT repo: https://github.com/web-platform-tests/wpt/pull/26192
Comment 6 Diego Pino 2020-10-22 13:51:32 PDT 
Created attachment 412129 [details]
Patch
Comment 7 Alex Christensen 2020-10-23 11:00:42 PDT 
Comment on attachment 412129 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=412129&action=review

> LayoutTests/imported/w3c/ChangeLog:8
> +        Sometimes lhs of comparison is higher by an infinitisimal margin.

infinitisimal isn't quite correct but whatevs.
Comment 8 EWS 2020-10-23 11:03:16 PDT 
Committed r268928: <https://trac.webkit.org/changeset/268928>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 412129 [details].
Comment 9 Alex Christensen 2021-01-08 11:02:49 PST 
(In reply to Diego Pino from comment #5)
> Link to same change in WPT repo:
> https://github.com/web-platform-tests/wpt/pull/26192

Diego, would you mind including https://bugs.webkit.org/show_bug.cgi?id=217118 in your PR?
Comment 10 Diego Pino 2021-11-16 17:08:40 PST 
(In reply to Alex Christensen from comment #9)
> (In reply to Diego Pino from comment #5)
> > Link to same change in WPT repo:
> > https://github.com/web-platform-tests/wpt/pull/26192
> 
> Diego, would you mind including
> https://bugs.webkit.org/show_bug.cgi?id=217118 in your PR?

Thank you Alex, I added your changes to the PR https://github.com/web-platform-tests/wpt/pull/26192