Bug 186132

Summary: export-w3c-test-changes can not export reftests
Product: WebKit Reporter: Frédéric Wang (:fredw) <fred.wang>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW    
Severity: Normal CC: ews-watchlist, glenn, graouts, gsnedders, lforschler, ntim, philip, rwlbuis, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=187003
Bug Depends on:    
Bug Blocks: 167911    
Attachments:
Description Flags
Patch none

Frédéric Wang (:fredw)
Reported 2018-05-31 02:14:39 PDT
In WebKit, reftests are NAME.html NAME-expected.html Moreover, the <link rel="match"> points to the non-existing NAME-ref.html file. When export-w3c-test-changes runs the lint tool, it complains about 1) NAME-expected.html being a support file in the wrong directory. 2) NAME.html pointing to non-existent ref file. The support script should rename NAME-expected.html to NAME-ref.html before exporting in order to fix these errors.
Attachments
Patch (6.91 KB, patch)
2018-05-31 08:41 PDT, Frédéric Wang (:fredw) 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Frédéric Wang (:fredw)
Comment 1 2018-05-31 08:41:00 PDT
Created attachment 341663 [details] Patch
youenn fablet
Comment 2 2018-05-31 12:21:27 PDT
The issue with ref tests is that one ref file in WPT might be used for several test files, so I am not sure we can always update correctly all ref files. In the simple one test/one ref case, this should work so maybe we could add a warning or add a check for this multi case
Frédéric Wang (:fredw)
Comment 3 2018-05-31 12:26:56 PDT
So you mean that when we have a single ref for multiple reftests, the import script will generate multiple -expected.html? I guess it's a problem with the import / way WebKit handles reftest then. I saw a FIXME in the import script about that.
Frédéric Wang (:fredw)
Comment 4 2018-05-31 12:33:35 PDT
I suppose for new reftests, it's safe to just rename -expected => -ref. For existing reftests using a single ref for multiple files or when upstream names are not of the form *-ref.html or similar (e.g. ./css/css-fill-stroke/reference/paint-order-001-ref.tentative.html) I guess the patch will just fail to apply.
Frédéric Wang (:fredw)
Comment 5 2018-07-11 06:21:56 PDT
Cancelling review since it seems we don't have a clear agreement on how we will handle the mismatch between WebKit / WPT formats...
Philip Jägenstedt
Comment 6 2019-11-05 06:25:00 PST
Is it Tools/Scripts/import-w3c-tests that renames references to -expected.html?
Frédéric Wang (:fredw)
Comment 7 2019-11-05 06:48:30 PST
(In reply to Philip Jägenstedt from comment #6) > Is it Tools/Scripts/import-w3c-tests that renames references to > -expected.html? Yes it is.
Philip Jägenstedt
Comment 8 2019-11-07 15:34:50 PST
That'll definitely make it challenging to work effectively with reftests, and it looks like plenty of WebKit-specific failures in WPT are reftests. Is there any possible fix here short of getting rid of the renaming, with all that would entail?
Radar WebKit Bug Importer
Comment 9 2023-02-10 04:59:19 PST
<rdar://problem/105269974>
Antoine Quint
Comment 10 2023-02-10 05:00:07 PST
Not sure the name of this Radar is completely correct. You can export reftests so long as you have both -expected.html and -ref.html files in your WebKit repository. It's far from ideal though.
Tim Nguyen (:ntim)
Comment 11 2023-02-10 23:14:39 PST
(In reply to Antoine Quint from comment #10) > Not sure the name of this Radar is completely correct. You can export > reftests so long as you have both -expected.html and -ref.html files in your > WebKit repository. It's far from ideal though. It wasn't possible before because we renamed the -ref.html file and deleted the original -ref.html. Now we have duplicates, which allows exporting, but is far from ideal.
Note You need to log in before you can comment on or make changes to this bug.