I'm not sure I agree with this; it depends on whether pixel tests are enabled or not. If they are enabled, it seems like we should exit early (and running on a new/unsupported OS version seems like the exception rather than the rule). You can change the --exit-after-n-failures threshold to some really high number if you want to defeat this, or just turn off pixel tests. We can certainly change the defaults to do something different if we detect we're on the "future" version of the O/S, though. I'm not yet convinced that that's the right thing to do, either, but it would be easier to convince me of this than to change it for the general case.

OK, I retitled :)

I guess I'm not sure what you want ... if you know you're on a new release that doesn't have correct images yet, is it good enough to just turn off pixel tests?

No, because I'd still like to see what the pixel diffs look like. Maybe --exit-after-n-failure=10000000 is enough for me. I'd prefer a shorthand for 'never exit' though, and didn't see one in the help output.

Ah, I see. Would you want to still want to exit early after N text diffs, and just ignore the pixel failures? Otherwise, as you say, --exit-after-n-failures would work. From a quick glance at the code, I believe that --exit-after-n-failures=0 is equivalent to "never exit". That should probably be documented in the help.

smfr ... is --exit-after-n-failures=0 good enough, or would you like image failures to be treated separately?

in r120007 we changed the default for --exit-after-n-failures to None (i.e., never exit early) (see bug 73843). Hopefully this is good enough and we don't need to add another flag. Please reopen if there's not the case.