PatZilla croaks with “HTTPBadGateway: Could not connect to OPS servers”

PatZilla Troubleshooting / Fehleranalyse
#1

After installing PatZilla in development mode on a fresh Ubuntu 18.10 box, customers are reporting a problem to us. While PatZilla starts fine and is apparently configured properly, it bails out when actually accessing the OPS services with this exception:

2019-01-31 18:11:00,896 INFO     [patzilla.access.epo.ops.client          ][waitress] OpsOAuthClientFactory.create_session: identifier=system, client_id=<redacted>
2019-01-31 18:11:00,898 INFO     [patzilla.navigator.services.ops         ][waitress] query raw: pn=EP666666
2019-01-31 18:11:00,898 INFO     [patzilla.util.expression                ][waitress] Parsing search expression "pn=EP666666" with syntax "cql" and grammar "default"
2019-01-31 18:11:01,168 INFO     [patzilla.navigator.services.ops         ][waitress] query cql: pn=EP0666666
2019-01-31 18:11:01,177 DEBUG    [patzilla.access.epo.ops.api             ][waitress] OPS request with client-id <redacted>
2019-01-31 18:11:01,453 DEBUG    [patzilla.util.web.identity.store        ][waitress] Measure transfer: userid=389e0df4-a846-440d-a445-a2c873871fca, upstream=ops, volume=86
2019-01-31 18:11:01,462 ERROR    [patzilla.access.epo.ops.client          ][waitress] OpsOAuth2Session HTTPError: 401 Client Error: Unauthorized for url: https://ops.epo.org/3.2/auth/accesstoken. client_id=<redacted>
2019-01-31 18:11:01,463 WARNING  [patzilla.access.epo.ops.client          ][waitress] Invalidating token and closing connection for client_id=<redacted>
2019-01-31 18:11:01,478 CRITICAL [patzilla.navigator.services             ][waitress] ops-search error: query="pn=EP0666666", reason=pyramid.httpexceptions.HTTPBadGateway: Could not connect to OPS servers.
response:
None
exception:
Traceback (most recent call last):
  File "/home/develop/patselect-dev/ip-navigator/patzilla/navigator/services/ops.py", line 105, in ops_published_data_search_handler
    result = ops_published_data_search(constituents, search.expression, range)
  File "/home/develop/patselect-dev/ip-navigator/.venv27/local/lib/python2.7/site-packages/beaker/cache.py", line 599, in cached
    return cache[0].get_value(cache_key, createfunc=go)
  File "/home/develop/patselect-dev/ip-navigator/.venv27/local/lib/python2.7/site-packages/beaker/cache.py", line 322, in get
    return self._get_value(key, **kw).get_value()
  File "/home/develop/patselect-dev/ip-navigator/.venv27/local/lib/python2.7/site-packages/beaker/container.py", line 378, in get_value
    v = self.createfunc()
  File "/home/develop/patselect-dev/ip-navigator/.venv27/local/lib/python2.7/site-packages/beaker/cache.py", line 597, in go
    return func(*args, **kwargs)
  File "/home/develop/patselect-dev/ip-navigator/patzilla/access/epo/ops/api.py", line 208, in ops_published_data_search
    return ops_published_data_search_real(constituents, query, range)
  File "/home/develop/patselect-dev/ip-navigator/patzilla/access/epo/ops/api.py", line 223, in ops_published_data_search_real
    response = client.get(url, headers={'Accept': 'application/json'}, params={'q': query, 'Range': range})
  File "/home/develop/patselect-dev/ip-navigator/.venv27/local/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/home/develop/patselect-dev/ip-navigator/patzilla/access/epo/ops/client.py", line 192, in request
    raise error
HTTPBadGateway: Could not connect to OPS servers.

2019-01-31 18:11:01,481 INFO     [patzilla.navigator.services.ops         ][waitress] query finished

We have been able to confirm this problem.

#2

Investigation

We confirmed successful connectivity from the machine running PatZilla using HTTPie

http POST https://ops.epo.org/3.2/auth/accesstoken

and the python-epo-ops-client using the same OPS credentials as configured in patzilla-development.ini:

import epo_ops

client = epo_ops.Client(key='abc', secret='xyz')        # Instantiate client
response = client.published_data(                       # Retrieve bibliography data
  reference_type = 'publication',                       # publication, application, priority
  input = epo_ops.models.Docdb('1000000', 'EP', 'A1'),  # original, docdb, epodoc
  endpoint = 'biblio',                                  # optional, defaults to biblio in case of published_data
  constituents = []                                     # optional, list of constituents
)

Both tests worked flawlessly.


#3

Analysis

We haven’t been able to find the exact root cause for this problem. Obviously, it had something to do with the OAuth authentication against OPS. We decided to stop spending time on investigating the problem further as we headed for a more sustainable solution.

Solution

As George Song’s fine python-epo-ops-client library worked well, we finally started using it in favor of the custom OPS client implementation based on requests+requests-oauthlib we were having before.

Not all OPS services are ported yet, so this amendment is currently living in the epo-ops-client branch per https://github.com/ip-tools/ip-navigator/commit/9d9019f4. It should help you when aiming to run PatZilla on a recent Ubuntu 18.04.1 LTS (Bionic Beaver) or others.

Credits

Thanks George for conceiving and maintaining such a nice drop-in replacement we have been switch over to with minimal effort.

Appendix

https://github.com/ip-tools/ip-navigator/commit/9d9019f4

2019Q01
2019Q01
#4

After upgrading our development environment on macOS (now on Python 2.7.15 through Homebrew), we are experiencing the same error. We conclude it must have something do to with a recent change to Python itself, as all other package dependencies are nailed to their respective versions. Also, we haven’t been hit by this problem on our production servers yet, which are running vanilla Debian Linux.

While errors of this kind are rare, they probably have been caused by some glitch in the area of Python vs. OpenSSL, as the OAuth modules (requests-oauthlib · PyPI and oauthlib · PyPI) for authenticating with OPS most likely build upon that.

#5

After others also found this annoying problem [1,2], we finally have been able to finish the switchover from the - now broken - custom OPS client implementation to the python-epo-ops-client library. The new PatZilla release 0.162.0 released on Feb 13, 2019 resolves the OPS accessibility problem you might have experienced before.

Thanks to all who reported this problem to us and confirmed that everything works again. Kudos again to George Song for conceiving and maintaining this excellent OPS client wrapper library.

Enjoy your research.

[1] How to configure data source DEPATISnet · Issue #8 · ip-tools/ip-navigator · GitHub
[2] PatZilla croaks with “HTTPBadGateway: Could not connect to OPS servers” · Issue #9 · ip-tools/ip-navigator · GitHub

#6