The Day Our Python gRPC Connections Died
On the 30th of September 2021, a heavily-used root certificate — DST Root CA X3 — expired. You can read all about it here.
According to a handful of forum posts and github issues I’ve come across, the change has caused a fair amount of pain to those unfortunates who failed to heed the warnings, but for most of us this really wasn’t a surprise. For our team, the expiration date came and went and we didn’t even notice! Until our primary in-house testing tool began failing its connection tests with the following:
Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Our gRPC connection tests are written in Python (using the grpcio and grpcio-tools packages), and run on a variety of linux machines and Docker images. Hunting through the forums, it looked like upgrading to the latest versions of the grpcio dependencies should do the trick, but it didn’t.
At least not by itself.
We eventually determined that the problem was that DST Root CA X3 was still registered as a certificate authority, and it took so long to figure out how to remove it on Debian that I realized that I had to post about it:
- To see if the DST Root CA X3 certificate is configured as a root authority, list the contents of…
