Bug 249720 - import-w3c-tests appears to create a redundant *-expected.html file for each *-ref.html file
Summary: import-w3c-tests appears to create a redundant *-expected.html file for each ...
Status: RESOLVED DUPLICATE of bug 203784
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: David Kilzer (:ddkilzer)
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2022-12-21 09:13 PST by David Kilzer (:ddkilzer)
Modified: 2022-12-21 12:57 PST (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Kilzer (:ddkilzer) 2022-12-21 09:13:15 PST
import-w3c-tests appears to create a redundant *-expected.html file for each *-ref.html file.

In Bug 249691, I imported one new WPT (dom/xslt/sort.html), but three files were added:

* LayoutTests/imported/w3c/web-platform-tests/dom/xslt/sort-expected.html: Add.
* LayoutTests/imported/w3c/web-platform-tests/dom/xslt/sort-ref.html: Add.
* LayoutTests/imported/w3c/web-platform-tests/dom/xslt/sort.html: Add.

The `sort-expected.html` file doesn't exist upstream, and appears to have been created by `import-w3c-tests` in Tools/Scripts/webkitpy/w3c/test_importer.py (I haven't debugged this--it's just a guess that this code is responsible):

```
                if 'reference' in test_info.keys():
                    reftests += 1
                    total_tests += 1
                    test_basename = self.filesystem.basename(test_info['test'])

                    # Add the ref file, following WebKit style.
                    # FIXME: Ideally we'd support reading the metadata
                    # directly rather than relying  on a naming convention.
                    # Using a naming convention creates duplicate copies of the
                    # reference files.
                    ref_file = self.filesystem.splitext(test_basename)[0] + '-expected'
                    if 'type' in test_info and test_info['type'] == 'mismatch':
                        ref_file += '-mismatch'
                    ref_file += self.filesystem.splitext(test_info['reference'])[1]

                    copy_list.append({'src': test_info['reference'], 'dest': ref_file, 'reference_support_info': test_info['reference_support_info']})
                    copy_list.append({'src': test_info['test'], 'dest': filename})
```

There are 293 `*-ref.html` files without corresponding `*-expected.html` files:

$ for F in `find LayoutTests/imported/w3c/web-platform-tests -name \*-ref.html`; do G="`echo $F | sed -e 's/-ref.html/-expected.html/'`"; if [ ! -e $G ]; then ls -la $F; fi; done | wc -l
     293

While there are 1947 `*-ref.html` files with a `*-expected.html` file:

$ for F in `find LayoutTests/imported/w3c/web-platform-tests -name \*-ref.html`; do G="`echo $F | sed -e 's/-ref.html/-expected.html/'`"; if [ -e $G ]; then ls -la $G; fi; done | wc -l
    1947

In most cases the `*-ref.html` and `*-expected.html` files are the same, but not always:

$ cksum LayoutTests/imported/w3c/web-platform-tests/quirks/body-fills-html-quirk-{expected,ref}.html
707962029 242 LayoutTests/imported/w3c/web-platform-tests/quirks/body-fills-html-quirk-expected.html
3896252966 153 LayoutTests/imported/w3c/web-platform-tests/quirks/body-fills-html-quirk-ref.html
Comment 1 Radar WebKit Bug Importer 2022-12-21 09:14:32 PST
<rdar://problem/103600959>
Comment 2 David Kilzer (:ddkilzer) 2022-12-21 09:14:54 PST
Anyone know what the "expected" (*pun*) behavior is when a `*-ref.html` file exists?

I'm guessing the `*-expected.html` files can be deleted, and that the `import-w3c-tests` script shouldn't be creating them.  Is that correct?
Comment 3 David Kilzer (:ddkilzer) 2022-12-21 09:36:54 PST
And for "reference" (*pun*), there are no `*-expected.html` files for `*-ref.html` files in the W3C web-platform-tests repository:

```
$ for F in `find WebKitBuild/w3c-tests/web-platform-tests -name \*-ref.html`; do G="`echo $F | sed -e 's/-ref.html/-expected.html/'`"; if [ -e $G ]; then ls -la $G; fi; done | wc -l
       0
```
Comment 4 David Kilzer (:ddkilzer) 2022-12-21 11:54:19 PST
Pull request: https://github.com/WebKit/WebKit/pull/7971
Comment 5 David Kilzer (:ddkilzer) 2022-12-21 12:57:34 PST
Dupe of this:

Bug 203784: Add support for finding test references via <link rel="match"> and use that in WPT

*** This bug has been marked as a duplicate of bug 203784 ***