| Summary: | [CSS Backgrounds] ASSERTION FAILED: !image->size().isEmpty() on css/css-backgrounds/background-size/vector/zero*ratio-auto-5px.html | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Carlos Alberto Lopez Perez <clopez> |
| Component: | CSS | Assignee: | Nobody <webkit-unassigned> |
| Status: | REOPENED --- | ||
| Severity: | Normal | CC: | ap, bfulgham, dpino, webkit-bot-watchers-bugzilla, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | WebKit Nightly Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=205872 https://bugs.webkit.org/show_bug.cgi?id=206897 https://bugs.webkit.org/show_bug.cgi?id=207574 https://bugs.webkit.org/show_bug.cgi?id=247005 |
||
|
Description
Carlos Alberto Lopez Perez
2020-01-22 04:39:19 PST
It's not really OK to leave 100% crashing tests enabled. Generating crash reports is somewhat costly, and can impact stability of other tests, or prevent us from getting crash reports from other tests. Please skip them. (In reply to Alexey Proskuryakov from comment #1) > It's not really OK to leave 100% crashing tests enabled. > Sorry, but that I should mark crashing tests as Skip its totally news to me. And after thinking about that, I don't think its a good practice. I always mark crashing tests as Crash, for the very same reasons I mark tests failing as Failure, and tests timing-out as Timeout, and not simply skip them: 1) If there its a change in WebKit that makes the test start passing I want to be notified of that (new pass). 2) If the test starts failing (or timing out) instead of crashing I also want to get notified. 3) But if the test its skipped we won't get ever notified of the new behavior until someone manually un-skips it. > Generating crash > reports is somewhat costly, and can impact stability of other tests, or > prevent us from getting crash reports from other tests. Well, its also costly to mark tests timing-out as Timeout, and not simply skip them.. but we don't do that, do we? Also I don't see why it should impact stability or prevent from getting crashing reports from other tests? Its that a bug on WTR or on the tooling for running tests? Its the first time I hear this. Can you elaborate on why its this a problem? Thanks > Well, its also costly to mark tests timing-out as Timeout, and not simply skip them.. but we don't do that, do we? We do. Please do, too. A lot of the work people are doing importing WPT tests is overall counter-productive, because it makes WebKit regression tests slower and less reliable. It is also a huge cost that someone else then has to find and triage flaky tests. > Also I don't see why it should impact stability or prevent from getting crashing reports from other tests? Generating a crash log is a CPU and disk access intensive operation. Since we already over-commit on the number of parallel processes, this is not trivial. Additionally, crash reporter on Apple platforms throttles reporting when there are too many reports to avoid impacting user experience. If you look at any recent Mac or iOS test run, you'll see that most crashes don't have crash logs. It is because of this. (In reply to Alexey Proskuryakov from comment #3) > > Well, its also costly to mark tests timing-out as Timeout, and not simply skip them.. but we don't do that, do we? > > We do. Please do, too. > Well, on the GTK and WPE ports (so far) we don't do that. We mark tests as Crash or Timeout if they end always-crashing or always-timing-out. Maybe we should start doing thinks like you do, it makes little sense that we don't follow the same rules. Its there a document or wiki page where its documented the policy you follow regarding this? > A lot of the work people are doing importing WPT tests is overall > counter-productive, because it makes WebKit regression tests slower and less > reliable. It may be counter-productive from the point of view of getting the layout test step to run as fast as possible and without flaky results since WPT tests are a lot of extra tests to run, but its not counter-productive from the point of view of getting more test coverage. I have just checked it, and currently of 53340 layout tests, 17314 are WPT tests.. that's around 1/3 of the layout tests. Maybe we should re-think how we run WPT tests and perhaps run them in a new different step from the layout-test one. > It is also a huge cost that someone else then has to find and > triage flaky tests. > I agree, flaky tests are a time sink problem. > > Also I don't see why it should impact stability or prevent from getting crashing reports from other tests? > > Generating a crash log is a CPU and disk access intensive operation. Since > we already over-commit on the number of parallel processes, this is not > trivial. Good point, I see how leaving tests expectations as Crash can contribute to cause flaky tests (specially unexpected timeouts). It still seems not ideal to me to skip crashing tests, but having unexpected flaky tests seems a bigger problem. So, I will change the expectation for this tests to Skip. Committed r257573: <https://trac.webkit.org/changeset/257573> I skipped them on r257573. Re-opening bug closed by mistake. This assert is still firing in current WebKit (262723@main) |