The Usual - Renewing Certificates
At work, we supply custom APIs to our customers for integrating their systems with our SaaS apps. Each tenant gets its own API Gateway and domain per integrated service, being served under a custom domain name for the tenant. As some of the tenant systems rely on certificate pinning, we issue our own CA-signed certificates to better control when we need to change them in the tenant systems. (looking at you, managed ACM certificates, with your sudden certificate renewals two times a week)
One of our legacy services - at this point running on Node.js 14 in a Fargate container - is used to transform and route requests to our internal services and tenant integration API Gateways, e.g. receiving a request with a captured event, transforming the event payload into a specific request body and then sending the request to the tenant API. Those tenant APIs are hosted under the custom domains mentioned above.
Our wildcard certificate renewal had been due - so weeks before we created and signed the new certificate via an authority and told our tenants to update the certificate in their systems. Last week we uploaded the new one to ACM after every tenant had finished the update. All fine and good, tenant systems still had access to their APIs. Certificate renewed, done and done.
The Unexpected - The Service Suddenly Fails
A few minutes later, we were getting alarms from our legacy service - a lot of requests had failed and were now slumbering in DLQs, waiting for a redrive. Looking into the logs, we quickly found the culprit:
{
"statusText": "SELF_SIGNED_CERT_IN_CHAIN",
"status": -1,
"body": {
"message": "self signed certificate in certificate chain"
}
}Huh, Node TLS aborted the handshake as it encountered a self signed certificate in the chain - but that looked improbable, because we used a public authority to sign it. We had been rightfully puzzled by this one.
Why the Error Message is Misleading
That error can also show up when the runtime does not yet trust a root in the certificate chain. In our case, the problem had been Node 14 and its older CA bundle. The chain included the "SSL.com TLS RSA Root CA 2022" certificate, which had only been added to Node in August of 2023[1] - four months after Node 14 had reached its EOL. The Node runtime did not yet trust the root certificate, and thus rejected the request.
Why not the much clearer UNABLE_TO_GET_ISSUER_CERT_LOCALLY error, you might
ask? Turns out that the certificate we uploaded contained the whole chain,
including the root. The server returned the whole chain, which ended in a
certificate that OpenSSL could not verify because no matching anchor certificate
was found in the CA bundle. So OpenSSL correctly identified it as self-signed
and returned the not-so-clear error message. Had we stripped the root
certificate from the chain
(as the TLS RFC allows[2]),
the error would have been the more obvious UNABLE_TO_GET_ISSUER_CERT_LOCALLY.
The Hotfix - Supplying the Missing Root Certificate
As a hotfix, we supplied the necessary certificates to the Node process via
NODE_EXTRA_CA_CERTS in the Dockerfile. One quick redeploy and everything
worked again.
FROM node:14-alpine
WORKDIR /usr/src/app
# One PEM file that may contain multiple concatenated certificates.
COPY certs/ ./certs/
# Extend Node's bundled CA store with your additional certificates.
ENV NODE_EXTRA_CA_CERTS=/usr/src/app/certs/extra-ca-certs.pem
# Do your thingKeep an eye out for the path to your PEM file - if Node cannot find it at startup, the service will start nevertheless, only logging a small warning. Check this, if your TLS handshake still fails after adding the certificates.
Now, letting the service chug along on a Node version that has been EOL for over three years? Not optimal. We are upgrading to Node 24 in the next few weeks to mitigate further "surprises" like this 😉.