| Summary: | [git-webkit] Update fork via API instead of local push | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Simon Fraser (smfr) <simon.fraser> |
| Component: | Tools / Tests | Assignee: | Jonathan Bedard <jbedard> |
| Status: | RESOLVED FIXED | ||
| Severity: | Normal | CC: | aakash_jain, ap, jbedard, ryanhaddad, simon.fraser, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | Safari Technology Preview | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| See Also: | https://bugs.webkit.org/show_bug.cgi?id=255999 | ||
| Bug Depends on: | |||
| Bug Blocks: | 239082 | ||
|
Description
Simon Fraser (smfr)
2022-07-27 21:28:48 PDT
Interrupting gave:
^CTraceback (most recent call last):
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Volumes/Data/Development/system/webkit/Internal/Tools/Scripts/git-webkit", line 89, in <module>
sys.exit(program.main(
File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 148, in main
return parsed.main(
File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 507, in main
return cls.create_pull_request(repository, args, branch_point)
File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 382, in create_pull_request
if run([repository.executable(), 'push', '-f', target, repository.branch], cwd=repository.root_path).returncode:
File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/subprocess_utils.py", line 51, in run
return subprocess.run(*popenargs, timeout=timeout, **kwargs)
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 495, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1020, in communicate
self.wait()
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1083, in wait
return self._wait(timeout=timeout)
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1806, in _wait
(pid, sts) = self._try_wait(0)
File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
Reapting the same command gives: LayoutTests % git webkit pr --no-update --issue 237812 Creating a pull-request for '237812' but we're on 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out' With verbose logging: INFO:webkitscmpy:Pushing 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out' to 'fork'... Enumerating objects: 3350, done. Counting objects: 100% (3350/3350), done. Delta compression using up to 20 threads Compressing objects: 100% (1586/1586), done. Writing objects: 100% (2681/2681), 6.47 MiB | 960.00 KiB/s, done. Total 2681 (delta 1732), reused 1855 (delta 1025), pack-reused 0 LayoutTests % GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push fork -f -v [lots of logging) 21:55:46.877281 http.c:703 <= Recv header, 0000000023 bytes (0x00000017) 21:55:46.877284 http.c:715 <= Recv header: x-frame-options: DENY 21:55:46.877286 http.c:703 <= Recv header, 0000000055 bytes (0x00000037) 21:55:46.877288 http.c:715 <= Recv header: x-github-request-id: C0FF:9D5A:824128:997753:62E216D2 21:55:46.877291 http.c:703 <= Recv header, 0000000002 bytes (0x00000002) 21:55:46.877293 http.c:715 <= Recv header: Writing objects: 100% (2672/2672), 6.47 MiB | 972.00 KiB/s, done. Total 2672 (delta 1725), reused 1855 (delta 1025), pack-reused 0 and nothing more. I did a `git fetch fork` and it magically resolved this. Still seeing this; seems worse when my fork is more out of date. I'm going to use this bug to track removing of what I believe to be the offending push in most workflows. What I believe is going on here is that we're attempting to update `main` in your personal fork by push hundreds or thousands of commits from your local checkout to your personal fork. According to GitHub's documentation, this is the correct way to update a local fork with raw `git` tools. Unsurprisingly, though, this is a slow operation. To make things worse, this operation gets slower as we add more protections in our pre-push hook. Pushing commits costs both local compute and network traffic which scales with the number of commits behind a fork is. However, we don't actually need to push _any_ pre-existing commits in principle, because GitHub already has all these commits server-side because the pre-existing commits are originating from `WebKit/WebKit`. We should be able to invoke https://docs.github.com/en/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository before we do any pushes, then the only commits we're actually pushing are the ones a contributor has created locally. Pull request: https://github.com/WebKit/WebKit/pull/12978 Committed 263251@main (556bf5c84d3e): <https://commits.webkit.org/263251@main> Reviewed commits have been landed. Closing PR #12978 and removing active labels. |