Bug 216705

Summary: Make ews-app robust against config.json issues
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, jbedard, ryanhaddad, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Aakash Jain 2020-09-18 10:34:22 PDT 
If config.json is invalid temporarily (e.g.: merge conflict while updating checkout on the buildbot server), then the ews-app might encounter a crash and stop polling for new patches. I noticed this on UAT instance. Noticed this exception in logs:

2020-09-17 20:51:17,020 - 22 r? patches, 0 patches need to be sent to Buildbot: []
2020-09-17 20:51:47,050 - Fetching: https://ews-build.webkit-uat.org/config.json
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/ews/OpenSource/Tools/BuildSlaveSupport/ews-app/ews/fetcher.py", line 50, in run
    Buildbot.update_icons_for_queues_mapping()
  File "/var/ews/OpenSource/Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py", line 99, in update_icons_for_queues_mapping
    config = cls.fetch_config()
  File "/var/ews/OpenSource/Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py", line 95, in fetch_config
    return config_data.json()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 795, in json 
    return json.loads(self.content.decode(encoding), **kwargs)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting property name enclosed in double quotes: line 358 column 1 (char 6271)
Comment 1 Aakash Jain 2020-09-18 10:37:43 PDT 
Created attachment 409145 [details]
Patch
Comment 2 Jonathan Bedard 2020-09-18 11:14:55 PDT 
Comment on attachment 409145 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409145&action=review

> Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py:97
> +        except Exception as e:

Nit: Don't need the "as e"
Comment 3 Aakash Jain 2020-09-18 12:06:27 PDT 
Comment on attachment 409145 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409145&action=review

>> Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py:97
>> +        except Exception as e:
> 
> Nit: Don't need the "as e"

Updated the patch to add a log which includes the error as well. This log might help in debugging.
Comment 4 Aakash Jain 2020-09-18 12:06:39 PDT 
Created attachment 409159 [details]
Patch
Comment 5 EWS 2020-09-18 12:46:39 PDT 
Committed r267277: <https://trac.webkit.org/changeset/267277>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409159 [details].
Comment 6 Radar WebKit Bug Importer 2020-09-18 12:47:23 PDT 
<rdar://problem/69169472>
Comment 7 Alexey Proskuryakov 2020-09-19 11:22:04 PDT 
Comment on attachment 409159 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409159&action=review

> Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py:105
>              _log.warn('Unable to fetch buildbot config.json')

Looks like we now call both _log.error and _log.warn.
Comment 8 Aakash Jain 2020-09-21 12:19:39 PDT 
Comment on attachment 409159 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=409159&action=review

>> Tools/BuildSlaveSupport/ews-app/ews/common/buildbot.py:105
>>              _log.warn('Unable to fetch buildbot config.json')
> 
> Looks like we now call both _log.error and _log.warn.

Changed this one to _log.error as well in https://trac.webkit.org/changeset/267353/webkit