Bug 251267

Summary:	[ews] Regression: git steps in ews builds have started taking long time
Product:	WebKit	Reporter:	Aakash Jain <aakash_jain>
Component:	Tools / Tests	Assignee:	Jonathan Bedard <jbedard>
Status:	ASSIGNED ---
Severity:	Normal	CC:	aakash_jain, ap, jbedard, ryanhaddad, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	Other
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=251211 https://bugs.webkit.org/show_bug.cgi?id=248895

Description Aakash Jain 2023-01-27 06:44:22 PST

Regression: git steps in ews builds have started taking long time. They used to take 20-30 seconds earlier. But now, they typically takes 2-3 minutes. This slows down each build and queue. The queues which used to be instanteneous have also started to get backlog of pending build at times. Slow queues have gotten slower because of this. We should look into it and see if we can speed it up.

e.g.: 
git operations took ~3 minutes (178 seconds) in recent build https://ews-build.webkit.org/#/builders/6/builds/104972  (clean-up-git-repo: 1 min 43s, checkout-source: 27s, update-working-directory: 11s, checkout-pull-request: 22s, show-identifier: 14s, fetch-branch-references: 1s).

Comment 1 Aakash Jain 2023-01-27 07:14:30 PST

Comparison of time spend in major git operation, over time:

On Jan 1, 2023: 178s in https://ews-build.webkit.org/#/builders/6/builds/101700 (clean-up-git-repo: 106s, clean-and-update-working-directory: 22s, update-working-directory: 7s, checkout-pull-request: 25s, show-identifier: 17s, fetch-branch-references: 1s)

On Dec 1, 2022 118s in https://ews-build.webkit.org/#/builders/6/builds/98400 (clean-up-git-repo: 39s, clean-and-update-working-directory: 26s, update-working-directory: 6s, checkout-pull-request: 27s, show-identifier: 15s, fetch-branch-references: 5s)

On Oct 1, 2022: 72s in https://ews-build.webkit.org/#/builders/6/builds/91200 (clean-up-git-repo: 18s, clean-and-update-working-directory: 29s, update-working-directory: 3s, checkout-pull-request: 17s, show-identifier: 3s, fetch-branch-references: 2s)

On July 1, 2022: 80s in https://ews-build.webkit.org/#/builders/6/builds/81000  (clean-up-git-repo: 28s, clean-and-update-working-directory: 22s, update-working-directory: 5s, checkout-pull-request: 10s, show-identifier: 13s, fetch-branch-references: 2s)

On January 1, 2022: 26s in https://ews-build.webkit.org/#/builders/6/builds/65050 (clean-and-update-working-directory: 20s, update-working-directory: 2s, show-identifier: 3s, fetch-branch-references: 1s)

On January 1, 2021: 18s in https://ews-build.webkit.org/#/builders/6/builds/36979 (clean-and-update-working-directory: 13s, update-working-directory: 1s)

On January 1, 2020: 14s in https://ews-build.webkit.org/#/builders/6/builds/10000 (clean-and-update-working-directory: 17s, update-working-directory: 1s)


(Note that time for hidden steps like show-identifier was fetched using buildbot api, e.g.: https://ews-build.webkit.org/api/v2/builders/6/builds/65050/steps).

Comment 2 Radar WebKit Bug Importer 2023-01-27 07:14:48 PST

<rdar://problem/104744000>

Comment 3 Aakash Jain 2023-01-27 08:46:28 PST

257496@main added "git prune" command in "clean-up-git-repo" step and it increased the run-time of that step by ~1 minute.

e.g.: https://ews-build.webkit.org/#/builders/6/builds/101700/steps/4/logs/stdio shows that git prune took 67 seconds
Similarly, in https://ews-build.webkit.org/#/builders/6/builds/104972/steps/4/logs/stdio it took 69 seconds.

Comment 4 Aakash Jain 2023-01-27 09:34:29 PST

One idea is to increase kern.maxvnodes value to make git operations faster (e.g.: sudo sysctl kern.maxvnodes=$((512*1024)))