Bug 216536

Summary:

Add further temporary debug logging for wptserve

Product:

WebKit

Reporter:

Sam Sneddon [:gsnedders] <gsnedders>

Component:

Tools / Tests

Assignee:

Sam Sneddon [:gsnedders] <gsnedders>

Status:

RESOLVED WONTFIX

Severity:

Normal

CC:

aakash_jain, ap, clopez, ews-watchlist, youennf

Priority:

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=215829

Attachments:

Description	Flags
Patch	none
Patch	none
Patch	none
Patch	aakash_jain: review-, ews-feeder: commit-queue-

Description Sam Sneddon [:gsnedders] 2020-09-15 06:06:11 PDT

This just checks what processes and open files we have, to try and see why we're running into "too many open files"

Comment 1 Sam Sneddon [:gsnedders] 2020-09-15 06:11:24 PDT

Created attachment 408814 [details]
Patch

Comment 2 EWS Watchlist 2020-09-15 06:12:29 PDT

This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess

Comment 3 Aakash Jain 2020-09-15 08:15:18 PDT

Comment on attachment 408814 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=408814&action=review

> LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:254
> +            self.logger.debug(subprocess.check_output(["ps", "aux"])

missing parenthesis

Comment 4 Sam Sneddon [:gsnedders] 2020-09-15 09:15:52 PDT

Created attachment 408824 [details]
Patch

Comment 5 Aakash Jain 2020-09-15 09:43:27 PDT

Where are we running into "too many open files"? Any bug/radar with the details?

Comment 6 Sam Sneddon [:gsnedders] 2020-09-15 09:50:24 PDT

See the see also bug: 215829 :)

Comment 7 Sam Sneddon [:gsnedders] 2020-09-15 11:06:43 PDT

Created attachment 408834 [details]
Patch

Comment 8 Sam Sneddon [:gsnedders] 2020-09-15 11:30:59 PDT

Created attachment 408838 [details]
Patch

Comment 9 Sam Sneddon [:gsnedders] 2020-09-15 14:25:54 PDT

This entire approach is doomed to failure, because lsof can block and make everything worse.

Comment 10 Aakash Jain 2020-09-15 14:27:07 PDT

Comment on attachment 408838 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=408838&action=review

> LayoutTests/imported/w3c/web-platform-tests/tools/wptserve/wptserve/response.py:257
> +                self.logger.debug(subprocess.check_output(["ps", "aux"]))

This ('ps aux' and 'lsof -P +c0') will produce huge amount of output (tens of thousands of lines). Are you sure you want to add that much output in the logs? Maybe you just need a count of number of lines of these.

Also, when the machine is already running out of file handles, these commands might just slow down/hang the machine further. This might explain why https://ews-build.webkit.org/#/builders/30/builds/17561/steps/11/logs/stdio hanged with this patch.

Comment 11 Alexey Proskuryakov 2020-09-16 16:40:18 PDT

We can manually run a job on one of the machines dumping lsof every few seconds, this way we could catch it right before the condition occurs. As long as it's not a super quick explosion, that could be good enough.