Bug 69118
| Summary: | NRWT should have an option to not exit early because of image failures | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Simon Fraser (smfr) <simon.fraser> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED FIXED | ||
| Severity: | Normal | CC: | dpranke, eric, simon.fraser |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Bug Depends on: | |||
| Bug Blocks: | 88680 | ||
Simon Fraser (smfr)
NRWT should never exit early because of image failures. When running on a new OS version, you can have lots of image failures without necessarily having layout failures.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Dirk Pranke
I'm not sure I agree with this; it depends on whether pixel tests are enabled or not. If they are enabled, it seems like we should exit early (and running on a new/unsupported OS version seems like the exception rather than the rule). You can change the --exit-after-n-failures threshold to some really high number if you want to defeat this, or just turn off pixel tests.
We can certainly change the defaults to do something different if we detect we're on the "future" version of the O/S, though. I'm not yet convinced that that's the right thing to do, either, but it would be easier to convince me of this than to change it for the general case.
Simon Fraser (smfr)
OK, I retitled :)
Dirk Pranke
I guess I'm not sure what you want ... if you know you're on a new release that doesn't have correct images yet, is it good enough to just turn off pixel tests?
Simon Fraser (smfr)
No, because I'd still like to see what the pixel diffs look like.
Maybe --exit-after-n-failure=10000000 is enough for me. I'd prefer a shorthand for 'never exit' though, and didn't see one in the help output.
Dirk Pranke
Ah, I see. Would you want to still want to exit early after N text diffs, and just ignore the pixel failures? Otherwise, as you say, --exit-after-n-failures would work.
From a quick glance at the code, I believe that --exit-after-n-failures=0 is equivalent to "never exit". That should probably be documented in the help.
Dirk Pranke
smfr ... is --exit-after-n-failures=0 good enough, or would you like image failures to be treated separately?
Dirk Pranke
in r120007 we changed the default for --exit-after-n-failures to None (i.e., never exit early) (see bug 73843).
Hopefully this is good enough and we don't need to add another flag.
Please reopen if there's not the case.