Created attachment 117899 [details] Patch

Part of the point of this policy was to avoid slowing down the bots too much when some low-level crash is introduced in WebKit. In many cases, all 20 tests will have crashed for the same reason, and any other tests that would have been run would also have crashed for the same reason, so running more tests doesn't give more diagnostic information. And since we have to wait for Crash Reporter for each crashed test, running more tests can significantly slow down the bots.

I don't think you want to change this. At least not unless you fix https://bugs.webkit.org/show_bug.cgi?id=37797. But I don't think we want that behavior on the bots anyway.

(In reply to comment #2) > Part of the point of this policy was to avoid slowing down the bots too much when some low-level crash is introduced in WebKit. In many cases, all 20 tests will have crashed for the same reason, and any other tests that would have been run would also have crashed for the same reason, so running more tests doesn't give more diagnostic information. And since we have to wait for Crash Reporter for each crashed test, running more tests can significantly slow down the bots. Right. However, in the world where we have multiple worker processes, 20 might be unrealistically low since many tests that get paged out temporarily will end up taking much longer time to run. This problem might go away once you add more RAM to Mac bots. For example, I haven't had this issue on my MacPro (has 12GB of RAM).

It is possible we should re-evaluate this for the multi-proesss world.

I don't think we need to re-evaluate this. I think we need to fix the bugs that are causing these timeouts. For comparison, we don't have these issues on the chromium bots, and no one has found the exit-early behavior to be a problem.

(In reply to comment #6) > I don't think we need to re-evaluate this. I think we need to fix the bugs that are causing these timeouts. If we can fix the underlying issues, that's clearly the best resolution. > For comparison, we don't have these issues on the chromium bots, and no one has found the exit-early behavior to be a problem. Is this just because we have a lot of tests listed as expected to timeout/crash flakily so they don't contribute to the count?

Created attachment 146669 [details] Patch

rniwa / ojan, please take a look?

Created attachment 146673 [details] change 0 to None

Comment on attachment 146673 [details] change 0 to None setting cq- because we should update the chromium bots to pass these flags first.

Comment on attachment 146673 [details] change 0 to None I'd prefer a test showing that None does what we think it does.

(In reply to comment #12) > (From update of attachment 146673 [details]) > I'd prefer a test showing that None does what we think it does. I'm not entirely sure what you're looking for? It's the default argument, so the code path is being executed. However, I'm not sure how we would demonstrate that we never exit early? Given that there are many tests that include failures, is that sufficient? E.g., run_webkit_tests_integrationtest.MainTest.test_all shows that we run all of the tests and that N of them fail ...

(In reply to comment #13) > (In reply to comment #12) > > (From update of attachment 146673 [details] [details]) > > I'd prefer a test showing that None does what we think it does. > > I'm not entirely sure what you're looking for? It's the default argument, so the code path is being executed. However, I'm not sure how we would demonstrate that we never exit early? Given that there are many tests that include failures, is that sufficient? E.g., run_webkit_tests_integrationtest.MainTest.test_all shows that we run all of the tests and that N of them fail ... You're right. I didn't think it through very well.

Comment on attachment 146673 [details] change 0 to None Clearing flags on attachment: 146673 Committed r120007: <http://trac.webkit.org/changeset/120007>

All reviewed patches have been landed. Closing bug.