CERTIFICATE_VERIFY_FAILED error

Python Django error:

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
unable to get local issuer certificate (_ssl.c:1131)>

Exception Location: /usr/lib/python3.8/urllib/request.py, line 1357, in do_open

Running: Python 3.8.10, Django 4.0.3, Ubuntu 20.04, Apache 2

I'm using Django for a simple contact form app, which currently works and throws no errors. The CERTIFICATE_VERIFY_FAILED error occurs when I use this Django library to add hCaptcha to the form.

The issue doesn't appear to be with django-hcaptcha-field; it appears to be Ubuntu and SSL certificates when the hCaptcha API is accessed.

I've looked at multiple questions on SO (esp. , even though it's for OS X) and Ask Ubuntu Certificate problems .

I've tried all these "fixes":

pip install pyOpenSSL --upgrade
apt-get install --reinstall python3-certifi
pip install --upgrade certifi --force
apt install --reinstall openssl
apt install ca-certificates
update-ca-certificates --fresh
export SSL_CERT_DIR=/etc/ssl/certs

I "force" updated my Let's Encrypt SSLs.

I tried updated certs:

wget --quiet
export SSL_CERT_FILE=$HOME/cacert.pem

Nothing in my Python code requires import ssl

What else can I try?

Diagnostic outputs:

dpkg -l | grep cert returns

ica-certificates 20210119~20.04.2 all Common CA certificates
certbot 0.40.0-1ubuntu0.1 all automatically configure HTTPS using Let's Encrypt
dirmngr 2.2.19-3ubuntu2.1 amd64 GNU privacy guard - network certificate management service
python-certbot-apache 0.36.0-1 all transitional dummy package
python3-certbot 0.40.0-1ubuntu0.1 all main library for certbot
python3-certbot-apache 0.39.0-1 all Apache plugin for Certbot
ipython3-certifi 2019.11.28-1 all root certificates for validating SSL certs and verifying TLS hosts (python3)
ssl-cert 1.0.39 all simple debconf wrapper for OpenSSL

dpkg -l | grep openssl returns

libxmlsec1-openssl:amd64 1.2.28-2 amd64 Openssl engine for the XML security library
openssl 1.1.1f-1ubuntu2.12 amd64 Secure Sockets Layer toolkit - cryptographic utility
perl-openssl-defaults:amd64 4 amd64 version compatibility baseline for Perl OpenSSL packages
python3-openssl 19.0.0-1build1 all Python 3 wrapper around the OpenSSL library

whereis openssl returns

openssl: /usr/bin/openssl /usr/local/bin/openssl /usr/include/openssl /usr/share/man/man1/openssl.1ssl.gz

which openssl /usr/bin/openssl returns

/usr/local/bin/openssl
/usr/bin/openssl

ldd $(which wget) returns

linux-vdso.so.1 (0x00007ffd9f10f000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007efdb3e3d000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007efdb3e34000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007efdb3e12000)
libssl.so.1.1 => /usr/local/lib/libssl.so.1.1 (0x00007efdb3d7a000)
libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x00007efdb3a8e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efdb3a72000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007efdb3a5d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efdb386b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efdb3848000)
/lib64/ld-linux-x86-64.so.2 (0x00007efdb3f6b000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007efdb36c6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efdb36c0000)

dpkg -l | grep python3-certifi returns

python3-certifi 2019.11.28-1 all root certificates for validating SSL certs and verifying TLS hosts (python3)

Traceback:

Request Method: POST
Request URL:
Django Version: 4.0.3
Python Version: 3.8.10
Installed Applications:
['django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'contactform.apps.ContactformConfig', 'encrypted_files', 'hcaptcha_field']
Installed Middleware:
['django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback (most recent call last): File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3.8/http/client.py", line 1425, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib/python3.8/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake()
During handling of the above exception ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)), another exception occurred: File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/var/www/html/", line 26, in contact if form.is_valid(): File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 205, in is_valid return self.is_bound and not self.errors File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 200, in errors self.full_clean() File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 433, in full_clean self._clean_fields() File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 445, in _clean_fields value = field.clean(value) File "/usr/local/lib/python3.8/dist-packages/django/forms/fields.py", line 199, in clean self.validate(value) File "/usr/local/lib/python3.8/dist-packages/hcaptcha_field/fields.py", line 129, in validate response = opener.open(request, timeout=hcaptcha_settings.TIMEOUT) File "/usr/lib/python3.8/urllib/request.py", line 525, in open response = self._open(req, data) File "/usr/lib/python3.8/urllib/request.py", line 542, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(*args) File "/usr/lib/python3.8/urllib/request.py", line 1397, in https_open return self.do_open(http.client.HTTPSConnection, req, File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open raise URLError(err)
Exception Type: URLError at /contact/contact/
Exception Value: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

fields.py of hcaptcha_field of

import json
import logging
import ssl # added #############
import certifi # added #############
from urllib.error import HTTPError
from urllib.parse import urlencode
from urllib.request import build_opener, Request, ProxyHandler
from django import forms
from django.core.exceptions import ValidationError
from django.utils.translation import gettext_lazy as _
from hcaptcha_field.settings import hcaptcha_settings
from hcaptcha_field.widgets import hCaptchaWidget
LOGGER = logging.getLogger('hcaptcha_field')
DATA_ATTRIBUTE_CONFIG = frozenset([ 'theme', 'size', 'tabindex', 'callback', 'expired-callback', 'chalexpired-callback', 'open-callback', 'close-callback', 'error-callback',
])
QUERY_PARAMETER_CONFIG = frozenset([ 'onload', 'render', 'hl', 'recaptchacompat'
])
class hCaptchaField(forms.Field): widget = hCaptchaWidget default_error_messages = { 'error_hcaptcha': _( # Translators: Error shown when an internal server error occurred. 'Something went wrong while verifying the hCaptcha. ' 'Please try again.' ), 'invalid_hcaptcha': _( # Translators: Error shown when visitor did not pass the hCaptcha check. 'hCaptcha could not be verified.' ), 'required': _( # Translators: Error shown when visitor forgot to fill in the hCaptcha. 'Please prove you are human.' ), } def __init__(self, sitekey=None, **kwargs): """ Initializer for `hCaptchaField` class. It determines data attributes for the widget class and constructs a widget if none is given. This constructed widget receives the URL of the JavaScript resource for the hCaptcha integration and the `sitekey` of the site to protect. """ # Retrieve settings DEFAULT_CONFIG = hcaptcha_settings.DEFAULT_CONFIG JS_API_URL = hcaptcha_settings.JS_API_URL SITEKEY = hcaptcha_settings.SITEKEY # Determine widget data attributes self.widget_data_attrs = {} for setting in DATA_ATTRIBUTE_CONFIG: if setting in kwargs: self.widget_data_attrs[setting] = kwargs.pop(setting) elif setting in DEFAULT_CONFIG: self.widget_data_attrs[setting] = DEFAULT_CONFIG[setting] # If the `widget` argument is not given, instantiate `self.widget` with # the hCaptcha API url and the sitekey if 'widget' not in kwargs: # Determine hCaptcha API url query_params = {} for setting in QUERY_PARAMETER_CONFIG: if setting in kwargs: query_params[setting] = kwargs.pop(setting) elif setting in DEFAULT_CONFIG: query_params[setting] = DEFAULT_CONFIG[setting] if query_params: js_api_url = '%s?%s' % (JS_API_URL, urlencode(query_params)) else: js_api_url = JS_API_URL # Determine hCaptcha sitekey self.sitekey = sitekey or SITEKEY # Instantiate widget kwargs['widget'] = self.widget( js_api_url=js_api_url, sitekey=self.sitekey) super().__init__(**kwargs) def widget_attrs(self, widget): """ Returns the widget attributes, including all the data attributes determined in the initializer. """ attrs = super().widget_attrs(widget) for key, value in self.widget_data_attrs.items(): attrs['data-%s' % key] = value return attrs def validate(self, value): """ Validates the field by verifying the value of the hidden field `h-captcha-response` with their API endpoint. """ super().validate(value) # Build request opener = build_opener(ProxyHandler(hcaptcha_settings.PROXIES)) post_data = urlencode({ 'secret': hcaptcha_settings.SECRET, 'response': value, 'sitekey': self.sitekey, }).encode('utf-8') request = Request(hcaptcha_settings.VERIFY_URL, post_data) # Perform request try: context=ssl.create_default_context(cafile=certifi.where()) # added ############ response = opener.open(request, timeout=hcaptcha_settings.TIMEOUT) except HTTPError: LOGGER.exception("Failed to verify response with hCaptcha API.") raise ValidationError( self.error_messages['error_hcaptcha'], code='error_hcaptcha' ) # Check response response_data = json.loads(response.read().decode('utf-8')) if not response_data.get('success'): LOGGER.error("Failed to pass hCaptcha check: %s", response_data) raise ValidationError( self.error_messages['invalid_hcaptcha'], code='invalid_hcaptcha' )

1 Answer

It looks like the context needs to be passed into an HTTPSHandler in order to make use of the certifi certificates.

Try patching your local copy of /usr/local/lib/python3.8/dist-packages/hcaptcha_field/fields.py with the following.

To the imports, as you've already done, add:

import certifi
import ssl

In addition, expand the import from urllib.request to include HTTPSHandler:

from urllib.request import build_opener, Request, ProxyHandler, HTTPSHandler

Also as you've already done, create an SSL context that uses the certifi certificates. I would add this right under the import block, though, rather than deep in the module code:

context=ssl.create_default_context(cafile=certifi.where())

Then, immediately below this new context=... line, create a new HTTPSHandler instance using context:

https_handler = HTTPSHandler(context=context)

Then, revise the line where opener is defined using the build_opener(...) call to include the new HTTPSHandler instance:

opener = build_opener(https_handler, ProxyHandler(hcaptcha_settings.PROXIES))

If I'm reading the build_opener docs correctly, the build_opener machinery is already using a default instance of HTTPSHandler as it tries to open the URL. Hopefully, replacing that default instance with this new https_handler instance, which will operate with the SSL context that includes the certifi certificates, will allow the urlopen call to work.

If this does work, then an issue/PR to the upstream project might be a good idea, to integrate this logic and either to provide a default behavior that pulls in the certifi certificates automatically if they're available, or to include a configuration option to let the user wire in the certifi certificates.

UPDATE 2022-04-14: Since this did fix the problem, I'll add some more detail as to why this fix was necessary and how it worked.

HTTPS requires the website you're accessing to provide a certificate that validates its identity. Otherwise, how do you know your traffic hasn't been messed with, man-in-the-middle attacked, etc.? The certificate that a website provides to you when you browse to it is issued to the operators of the website by a certificate issuing authority. This authority serves as a trusted third party, to "vouch for" the website operators and that, "yes, you can trust this certificate from them".

There are several of these issuing authorities out there. Those authorities issue their own certificates ("certificate authority certificates", or "CA certificates"), which are completely separate from the ones that individual websites provide. Your browser (or URL opener) needs to have access to these CA certificates in order to complete the 'trusted third party' verification of the certificate that the website presents. The key error the OP is facing here is the inability of the URL opener to find a good set of CA certificates to use for this third-party validation:

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

The message unable to get local issuer certificate means that urlopen cannot find the needed issuer certificate (CA certificate) on the local system.

So, the solution is to point urlopen to the CA certificates provisioned by the certifi package:

• In order to tell the opener where to find them, you have to provide their location when you create opener from build_opener

• build_opener works based upon a collection of handler objects, so the CA certificate information must be included within a suitable handler

• The only handler (that I could find in the docs) that includes information about the SSL context is the HTTPSHandler

• The way that HTTPSHandler takes in information about SSL-related configuration is through the context keyword argument when creating a new instance, which takes an 'SSL context' object

• ssl.create_default_context is (as best I understand) a factory function for creating SSL context objects, customizeable in various ways

• One of the ways it's customizeable is with the location of a CA certificate store (via the cafile argument)

• certifi provides the location on disk of its curated CA certificate store through the certifi.where() function

If you scan these bullets from bottom to top, they map pretty well to each of the steps in the answer.