Bug 212623 - Change ANGLE's header postprocessing script to not rely on timestamps
Summary: Change ANGLE's header postprocessing script to not rely on timestamps
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: ANGLE (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Keith Rollin
URL:
Keywords: InRadar
Depends on: 199212
Blocks: 212444
  Show dependency treegraph
 
Reported: 2020-06-01 19:53 PDT by Kenneth Russell
Modified: 2020-06-02 20:59 PDT (History)
18 users (show)

See Also:


Attachments
Patch (4.52 KB, patch)
2020-06-02 01:20 PDT, Keith Rollin
no flags Details | Formatted Diff | Diff
Patch (4.59 KB, patch)
2020-06-02 18:56 PDT, Keith Rollin
no flags Details | Formatted Diff | Diff
Patch (4.59 KB, patch)
2020-06-02 19:03 PDT, Keith Rollin
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kenneth Russell 2020-06-01 19:53:14 PDT
In discussions related to Bug 212444, krollin@ points out that the script adjust-angle-include-paths.sh added in Bug 199212 must be changed to stop using timestamps to determine whether it needs to do work. Instead it needs to do something like compute a hash of the contents of each file, and skip touching the file if the hash is unchanged since the last time the script ran.
Comment 1 Radar WebKit Bug Importer 2020-06-01 21:02:31 PDT
<rdar://problem/63856997>
Comment 2 Keith Rollin 2020-06-02 01:20:48 PDT
Created attachment 400795 [details]
Patch
Comment 3 EWS Watchlist 2020-06-02 01:21:31 PDT
Note that there are important steps to take when updating ANGLE. See http://trac.webkit.org/wiki/UpdatingANGLE
Comment 4 Keith Rollin 2020-06-02 01:23:11 PDT
To the GTK developers and CMakeLists.txt maintainers: I tried updating the CMakeLists.txt file to take into account that we no longer use an angle.timestamp file. I have no idea if I got it right, so please look at that change closely.
Comment 5 David Kilzer (:ddkilzer) 2020-06-02 08:51:04 PDT
Comment on attachment 400795 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=400795&action=review

r=me if you adjust DERIVED_FILE_DIR for CMake and update the `diff` command line.

> Source/ThirdParty/ANGLE/ChangeLog:32
> +        * adjust-angle-include-paths.sh:

This script would be much more efficient as a Makefile (in the spirit of Source/JavaScriptCore/DerivedSources.make or Source/WebCore/DerivedSources.make or Source/WebKitLegacy/mac/MigrateHeaders.make).  It could compare the timestamps of the original source files with the derived output sources and only update the individual files that had newer timestamps, but that's out of scope of this change.

> Source/ThirdParty/ANGLE/CMakeLists.txt:155
> +    add_custom_target(ANGLE-webgl-headers
>          DEPENDS LibGLESv2EntryPointsHeaders ANGLEWebGLHeaders
> -        COMMAND BUILT_PRODUCTS_DIR=${ANGLE_FRAMEWORK_HEADERS_DIR}/
> +        COMMAND DERIVED_FILE_DIR=/tmp

DERIVED_FILE_DIR should probably not be at the top level of /tmp (since it's going to dump every individual file into that directory).  Maybe something like:

    DERIVED_FILE_DIR=/tmp/ANGLE-webgl-headers

Other projects like JavaScriptCore and WebCore have variables like ${JavaScriptCore_DERIVED_SOURCES_DIR} and ${WebCore_DERIVED_SOURCES_DIR} in their CMake files.  I can't figure out how/where they're set, but using an ${ANGLE_DERIVED_SOURCES_DIR} would be better than /tmp or /tmp/ANGLE-webgl-headers if it's not too hard to set up.

> Source/ThirdParty/ANGLE/adjust-angle-include-paths.sh:44
> +' < "$i" > "$temp_file"
> +    if ! diff "$temp_file" "$i" &> "$temp_file.diff"
> +    then

You should use `diff -q` here since you're just discarding the output.

Also, why not just send diff output to /dev/null instead of a unique file?  (If this script isn't run on Windows, using /dev/null should be fine.)
Comment 6 Kenneth Russell 2020-06-02 14:25:39 PDT
Comment on attachment 400795 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=400795&action=review

Looks good to me with David's comments addressed; one additional question.

>> Source/ThirdParty/ANGLE/CMakeLists.txt:155
>> +        COMMAND DERIVED_FILE_DIR=/tmp
> 
> DERIVED_FILE_DIR should probably not be at the top level of /tmp (since it's going to dump every individual file into that directory).  Maybe something like:
> 
>     DERIVED_FILE_DIR=/tmp/ANGLE-webgl-headers
> 
> Other projects like JavaScriptCore and WebCore have variables like ${JavaScriptCore_DERIVED_SOURCES_DIR} and ${WebCore_DERIVED_SOURCES_DIR} in their CMake files.  I can't figure out how/where they're set, but using an ${ANGLE_DERIVED_SOURCES_DIR} would be better than /tmp or /tmp/ANGLE-webgl-headers if it's not too hard to set up.

Will /tmp work portably? I suspect it won't on Windows. Can the temporary files be generated into the output directory, and deleted if they don't differ from the already-generated output files?
Comment 7 Kenneth Russell 2020-06-02 14:25:59 PDT
By the way thanks Keith for picking this up!
Comment 8 Keith Rollin 2020-06-02 16:20:21 PDT
(In reply to David Kilzer (:ddkilzer) from comment #5)
> Comment on attachment 400795 [details]
> > Source/ThirdParty/ANGLE/adjust-angle-include-paths.sh:44
> > +' < "$i" > "$temp_file"
> > +    if ! diff "$temp_file" "$i" &> "$temp_file.diff"
> > +    then
> 
> You should use `diff -q` here since you're just discarding the output.
> 
> Also, why not just send diff output to /dev/null instead of a unique file? 
> (If this script isn't run on Windows, using /dev/null should be fine.)

Actually, that's the way the patch should have been, and the way it was originally. I removed the -q and piped the output to a file for debugging purposes, and forgot to revert that scaffolding. Good catch.
Comment 9 Keith Rollin 2020-06-02 16:22:41 PDT
(In reply to Kenneth Russell from comment #6)
> Comment on attachment 400795 [details]
> Patch
> 
> >> Source/ThirdParty/ANGLE/CMakeLists.txt:155
> >> +        COMMAND DERIVED_FILE_DIR=/tmp
> > 
> > DERIVED_FILE_DIR should probably not be at the top level of /tmp (since it's going to dump every individual file into that directory).  Maybe something like:
> > 
> >     DERIVED_FILE_DIR=/tmp/ANGLE-webgl-headers
> > 
> > Other projects like JavaScriptCore and WebCore have variables like ${JavaScriptCore_DERIVED_SOURCES_DIR} and ${WebCore_DERIVED_SOURCES_DIR} in their CMake files.  I can't figure out how/where they're set, but using an ${ANGLE_DERIVED_SOURCES_DIR} would be better than /tmp or /tmp/ANGLE-webgl-headers if it's not too hard to set up.
> 
> Will /tmp work portably? I suspect it won't on Windows. Can the temporary
> files be generated into the output directory, and deleted if they don't
> differ from the already-generated output files?

OK, I'll take this approach. I didn't like using /tmp, but I wasn't aware of where the scratch build space was in our cmake facility (if any).
Comment 10 Keith Rollin 2020-06-02 18:56:43 PDT
Created attachment 400876 [details]
Patch
Comment 11 Keith Rollin 2020-06-02 19:03:03 PDT
Created attachment 400877 [details]
Patch
Comment 12 ChangSeok Oh 2020-06-02 20:54:09 PDT
(In reply to Keith Rollin from comment #4)
> To the GTK developers and CMakeLists.txt maintainers: I tried updating the
> CMakeLists.txt file to take into account that we no longer use an
> angle.timestamp file. I have no idea if I got it right, so please look at
> that change closely.

Nothing broken for the gtk port with USE_ANGLE_WEBGL enabled. =)
Comment 13 Keith Rollin 2020-06-02 20:55:55 PDT
Thanks! My local macOS-based tests look good, too. I'll go ahead and land this.
Comment 14 EWS 2020-06-02 20:59:50 PDT
Committed r262474: <https://trac.webkit.org/changeset/262474>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 400877 [details].