Bug 207986

Summary: [GTK][Tests] Layout test result upload step fails with too many files exception (needs Zip64)
Product: WebKit Reporter: Lauro Moura <lmoura>
Component: Tools / TestsAssignee: Lauro Moura <lmoura>
Status: RESOLVED WORKSFORME    
Severity: Critical CC: bugs-noreply, clopez, ews-watchlist, glenn, jbedard, lmoura
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Tentative patch, replacing shutil with raw ZipFile none

Lauro Moura
Reported 2020-02-19 19:56:18 PST
Sample trace: 18:14:23.533 85898 Finished writing JSON file for the test results server. 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit Release (Tests) 18:14:27.317 85898 JSON uploaded. 18:14:27.361 85898 18:14:27.361 85898 Preparing to upload test archive ... LargeZipFile raised: Files count would require ZIP64 extensions Traceback (most recent call last): File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 91, in main run_details = run(port, options, args, stderr) File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 485, in run run_details = manager.run(args) File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 325, in run shutil.make_archive(archive, 'zip', self._results_directory) File "/usr/lib/python2.7/shutil.py", line 595, in make_archive filename = func(base_name, base_dir, **kwargs) File "/usr/lib/python2.7/shutil.py", line 497, in _make_zipfile zf.write(path, path) File "/usr/lib/python2.7/zipfile.py", line 1171, in write self._writecheck(zinfo) File "/usr/lib/python2.7/zipfile.py", line 1137, in _writecheck " would require ZIP64 extensions") LargeZipFile: Files count would require ZIP64 extensions program finished with exit code 254 elapsedTime=3965.470351 Python 3.4 enabled by allowZip64 by default (in 2014), but shutil.make_archive does not expose this option.
Attachments
Tentative patch, replacing shutil with raw ZipFile (2.78 KB, patch)
2020-02-19 20:05 PST, Lauro Moura 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Lauro Moura
Comment 1 2020-02-19 20:05:49 PST
Created attachment 391248 [details] Tentative patch, replacing shutil with raw ZipFile
Lauro Moura
Comment 2 2020-02-19 20:10:16 PST
Maybe instead of the patch we could run `run-webkit-tests` with python 3.4+ in the GTK bots?
Carlos Alberto Lopez Perez
Comment 3 2020-02-20 03:19:39 PST
(In reply to Lauro Moura from comment #0) > Sample trace: > > 18:14:23.533 85898 Finished writing JSON file for the test results server. > 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit > Release (Tests) > 18:14:27.317 85898 JSON uploaded. > 18:14:27.361 85898 > 18:14:27.361 85898 Preparing to upload test archive ... > > LargeZipFile raised: Files count would require ZIP64 extensions Which caused this? This was not happening before.
Carlos Alberto Lopez Perez
Comment 4 2020-02-20 03:27:46 PST
(In reply to Carlos Alberto Lopez Perez from comment #3) > (In reply to Lauro Moura from comment #0) > > Sample trace: > > > > 18:14:23.533 85898 Finished writing JSON file for the test results server. > > 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit > > Release (Tests) > > 18:14:27.317 85898 JSON uploaded. > > 18:14:27.361 85898 > > 18:14:27.361 85898 Preparing to upload test archive ... > > > > LargeZipFile raised: Files count would require ZIP64 extensions > > Which caused this? This was not happening before. Also.. why it only happens on GTK but not on WPE? I'm seeing a different kind of exception on the GTK bot now: 00:35:59.580 100943 Preparing to upload test archive ... 00:36:05.478 100943 Uploading archive to https://results.webkit.org ... 00:36:05.531 100943 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 00:36:13.093 100943 https://results.webkit.org:443 "POST /api/upload/archive HTTP/1.1" 403 146 00:36:13.094 100943 Upload to https://results.webkit.org failed: 00:36:13.095 100943 <html> 00:36:13.095 100943 <head><title>403 Forbidden</title></head> 00:36:13.095 100943 <body> 00:36:13.095 100943 <center><h1>403 Forbidden</h1></center> 00:36:13.095 100943 <hr><center>Apple</center> 00:36:13.095 100943 </body> 00:36:13.095 100943 </html> 00:36:13.095 100943 This error is not fatal, continuing 00:36:13.128 100943 Testing completed, Exit status: 50 Something smells weird here... this have to be investigated to know what its causing this
Lauro Moura
Comment 5 2020-02-20 04:09:24 PST
(In reply to Carlos Alberto Lopez Perez from comment #4) > Also.. why it only happens on GTK but not on WPE? WPE buid 17060: 41170 tests ran as expected, 120 didn't GTK failed build 12712: 53188 tests ran as expected, 118 didn't Maybe just a matter of time before we hit this with WPE too? > > I'm seeing a different kind of exception on the GTK bot now: > > 00:35:59.580 100943 Preparing to upload test archive ... > 00:36:05.478 100943 Uploading archive to https://results.webkit.org ... > 00:36:05.531 100943 Starting new HTTPS connection (1): results.webkit.org:443 > /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/ > webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > InsecureRequestWarning, > 00:36:13.093 100943 https://results.webkit.org:443 "POST /api/upload/archive > HTTP/1.1" 403 146 > 00:36:13.094 100943 Upload to https://results.webkit.org failed: > 00:36:13.095 100943 <html> > 00:36:13.095 100943 <head><title>403 Forbidden</title></head> > 00:36:13.095 100943 <body> > 00:36:13.095 100943 <center><h1>403 Forbidden</h1></center> > 00:36:13.095 100943 <hr><center>Apple</center> > 00:36:13.095 100943 </body> > 00:36:13.095 100943 </html> > 00:36:13.095 100943 This error is not fatal, continuing > 00:36:13.128 100943 Testing completed, Exit status: 50 > > > Something smells weird here... this have to be investigated to know what its > causing this Mojave WK2 with the same 403 issue: https://build.webkit.org/builders/Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10510/steps/layout-test/logs/stdio
Carlos Alberto Lopez Perez
Comment 6 2020-02-20 04:22:44 PST
(In reply to Lauro Moura from comment #5) > (In reply to Carlos Alberto Lopez Perez from comment #4) > > Also.. why it only happens on GTK but not on WPE? > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > Maybe just a matter of time before we hit this with WPE too? > > LargeZipFile: Files count would require ZIP64 extensions Which limit its being hit that caused this to be raised? on the GTK Release test bot I see layout-test-results.zip at /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb.
Lauro Moura
Comment 7 2020-02-20 04:38:18 PST
(In reply to Carlos Alberto Lopez Perez from comment #6) > (In reply to Lauro Moura from comment #5) > > (In reply to Carlos Alberto Lopez Perez from comment #4) > > > Also.. why it only happens on GTK but not on WPE? > > > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > > > Maybe just a matter of time before we hit this with WPE too? > > > > > > > LargeZipFile: Files count would require ZIP64 extensions > > Which limit its being hit that caused this to be raised? > > on the GTK Release test bot I see layout-test-results.zip at > /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb. Number of files in the archive. Regular zip files have a limit of 64k entries. https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64
Carlos Alberto Lopez Perez
Comment 8 2020-02-20 04:45:14 PST
(In reply to Lauro Moura from comment #7) > (In reply to Carlos Alberto Lopez Perez from comment #6) > > (In reply to Lauro Moura from comment #5) > > > (In reply to Carlos Alberto Lopez Perez from comment #4) > > > > Also.. why it only happens on GTK but not on WPE? > > > > > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > > > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > > > > > Maybe just a matter of time before we hit this with WPE too? > > > > > > > > > > > > LargeZipFile: Files count would require ZIP64 extensions > > > > Which limit its being hit that caused this to be raised? > > > > on the GTK Release test bot I see layout-test-results.zip at > > /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb. > > Number of files in the archive. Regular zip files have a limit of 64k > entries. > > https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64 I still don't see how we can be hitting that. For the last run on the GTK bot, 256917 I see the zip generated only contains 15.8K files inside. You can find the zips generated here: https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/ cd $(mktemp -d) wget 'https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/r256917%20%2812705%29.zip' unzip 'r256917 (12705).zip' $ find|wc -l 15800
Carlos Alberto Lopez Perez
Comment 9 2020-02-20 05:40:10 PST
I triggered a clean build on the GTK bot and it seems to be working fine noew: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/12718
Carlos Alberto Lopez Perez
Comment 10 2020-02-20 05:44:03 PST
(In reply to Carlos Alberto Lopez Perez from comment #9) > I triggered a clean build on the GTK bot and it seems to be working fine > noew: > > https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/12718 Well, at least the zip creation seems working... There its still the failure with 05:37:28.676 93202 Preparing to upload test archive ... 05:37:41.563 93202 Uploading archive to https://results.webkit.org ... 05:37:41.638 93202 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 05:37:47.138 93202 https://results.webkit.org:443 "POST /api/upload/archive HTTP/1.1" 403 146 05:37:47.140 93202 Upload to https://results.webkit.org failed: 05:37:47.141 93202 <html> 05:37:47.142 93202 <head><title>403 Forbidden</title></head> 05:37:47.142 93202 <body> 05:37:47.142 93202 <center><h1>403 Forbidden</h1></center> 05:37:47.142 93202 <hr><center>Apple</center> 05:37:47.142 93202 </body> 05:37:47.142 93202 </html> 05:37:47.142 93202 This error is not fatal, continuing 05:37:47.186 93202 Testing completed, Exit status: 36
Lauro Moura
Comment 11 2020-02-20 06:40:12 PST
(In reply to Carlos Alberto Lopez Perez from comment #10) > Well, at least the zip creation seems working... > > There its still the failure with > snip. > 05:37:47.142 93202 <head><title>403 Forbidden</title></head> This seems to be happening to Mojave since Feb 13th at least: https://build.webkit.org/builders/Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10311/steps/layout-test/logs/stdio And in GTK release, build 12520 from Jan 30th shows 403 error but no detailed html error message. Thee detailed message started with r255500 (gtk release build 12535 is the first one with it) (https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/12535). So, it seems there is indeed something odd with uploading results (but kinda expected, as 403 is not a fatal error). But not related to this ZIP64 issue. Should we close this with invalid or wontfix? In case the exception appears again we could reopen it.
Jonathan Bedard
Comment 12 2020-02-20 07:39:01 PST
(In reply to Lauro Moura from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > Well, at least the zip creation seems working... > > > > There its still the failure with > > > > snip. > > > 05:37:47.142 93202 <head><title>403 Forbidden</title></head> > > This seems to be happening to Mojave since Feb 13th at least: > > https://build.webkit.org/builders/ > Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10311/steps/layout-test/ > logs/stdio > > And in GTK release, build 12520 from Jan 30th shows 403 error but no > detailed html error message. > > Thee detailed message started with r255500 (gtk release build 12535 is the > first one with it) > (https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/12535). > > So, it seems there is indeed something odd with uploading results (but kinda > expected, as 403 is not a fatal error). But not related to this ZIP64 issue. > > Should we close this with invalid or wontfix? In case the exception appears > again we could reopen it. That's a failure server side. At the moment, we're deliberately ignoring them because there is still work to be done with the reporting service to make upload archives useful, and we're still not 100% sure we want to go with this approach. Basically, we have a load-balancer on https://results.webkit.org that is rejecting upload archives because they're too large. We will, in the relatively near future, be fixing this, so I think this change is worth our time.
Jonathan Bedard
Comment 13 2020-02-20 07:44:58 PST
Comment on attachment 391248 [details] Tentative patch, replacing shutil with raw ZipFile View in context: https://bugs.webkit.org/attachment.cgi?id=391248&action=review Heads up, EWS does not exercise this code-path. So a green EWS doesn't guarantee this its actually working. > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:180 > + def _make_archive(self, archive, results_directory): Since we aren't actually using self, can we make this an @staticmethod or an @classmethod?
Lauro Moura
Comment 14 2020-03-02 06:13:33 PST
(In reply to Jonathan Bedard from comment #13) > Comment on attachment 391248 [details] > Tentative patch, replacing shutil with raw ZipFile > > View in context: > https://bugs.webkit.org/attachment.cgi?id=391248&action=review > > Heads up, EWS does not exercise this code-path. So a green EWS doesn't > guarantee this its actually working. > > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:180 > > + def _make_archive(self, archive, results_directory): > > Since we aren't actually using self, can we make this an @staticmethod or an > @classmethod? Thanks for the review Jonathan. I'll update the patch if this happens again.
Note You need to log in before you can comment on or make changes to this bug.