1

I recently moved my TLS termination back to my reverse proxy from the backend server, and I have run into this very specific problem.

When connecting to my nextcloud site firefox says SEC_ERROR_UNKNOWN_ISSUER.

More info on the error (cert info at bottom):

https://nextcloud.domain.com/

Peer’s Certificate issuer is not recognized.


HTTP Strict Transport Security: false

HTTP Public Key Pinning: false

The weird part is that this problem does not occur on any other browser, just firefox (Firefox 95, Ubuntu 20.04, Windows 10, and Android 12; no extensions). It only occurs after browsing initially to the site, then closing firefox, then reopening it and browsing to any url on nextcloud and the error occurs. Unfortunately its not exact that the error occurs on the second browsing, but it will happen whether its the second, third, or fourth time browsing after a firefox restart. I can resolve the error by restarting nginx on my reverse proxy.

To be clear, it appears the firefox restart triggers it; closing the tab and reopening does not produce the error. I have also tried clearing Firefox of the site data and restarting, error still happens.

The stranger part is that I have a couple other sites on the same reverse proxy using lets encrypt certs (different sub domain) and I cannot replicate the error there. It seems to be confined to firefox+nextcloud+nginx. Although restarting the backend nginx/server does not affect it. So I suppose there should be a specific configuration that I am missing on my reverse proxy.

I am at a loss here, the only thing that has changed is where the TLS is being terminated and a new Lets Encrypt cert. Hopefully I am missing something obvious and this isn't a bigger problem.

Full nginx configuration (minus other subdomains and mime types):

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}
http {
        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/sites-enabled/*;

        ##
        # Hardening
        ##
 
        add_header Allow "GET, POST, HEAD" always;
        add_header X-XSS-Protection "1; mode=block";
}


# configuration file /etc/nginx/snippets/ssl-params.conf:
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA HIGH !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 valid=300s;
resolver_timeout 5s;
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer" always;
add_header X-Download-Options "noopen" always;
add_header X-Permitted-Cross-Domain-Policies "none" always;
add_header X-Robots-Tag "none" always;
add_header X-XSS-Protection "1; mode=block" always;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

# configuration file /etc/nginx/sites-enabled/nextcloud.domain.com:
server {
        listen 443 ssl http2;
        ssl_certificate /etc/letsencrypt/live/nextcloud.domain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/nextcloud.domain.com/privkey.pem;
        include snippets/ssl-params.conf;
        server_name nextcloud.domain.com;
        location / {
                proxy_pass https://BACKENDIP;
                proxy_set_header X-Real-IP $remote_addr;
        }
}

# configuration file /etc/nginx/sites-enabled/reverseproxy.conf:
server {
        listen 80;
        server_name _;
        return 301 https://$host$request_uri;
}

Some info from the server cert via firefox:

Subject Name
Common Name nextcloud.domain.com

Issuer Name
Country US
Organization Let's Encrypt
Common Name R3

Validity
Not Before Fri, 10 Dec 2021 19:29:21 GMT
Not After Thu, 10 Mar 2022 19:29:20 GMT

Intermediate cert:

Subject Name
Country US
Organization Let's Encrypt
Common Name R3

Issuer Name
Country US
Organization Internet Security Research Group
Common Name ISRG Root X1

Validity
Not Before Fri, 04 Sep 2020 00:00:00 GMT
Not After Mon, 15 Sep 2025 16:00:00 GMT

Root Cert:

Subject Name
Country US
Organization Internet Security Research Group
Common Name ISRG Root X1

Issuer Name
Organization Digital Signature Trust Co.
Common Name DST Root CA X3

Validity
Not Before Wed, 20 Jan 2021 19:14:03 GMT
Not After Mon, 30 Sep 2024 18:14:03 GMT

No errors on the proxy or server associated with anything. The HTTP request shows up as code 200.

Here is a comparison between two certs on two of my servers that were both issued through Lets Encrypt Certbot minutes apart:

This is the cert firefox has a problem with: Bad Cert

This is the cert firefox accepts: Good Cert

A slightly concerning problem is that when browsing to the site firefox doesn't trust, in chrome, I get a root certificate that is NOT issued by the same organization.

  • The certificate listing CA X3, has one of its root CAs, is obviously the problem since it’s expired back on September 30 – Ramhound Dec 11 '21 at 03:00
  • What you label a root is not; it is a bridge (to the expired DST root) that [LE uses to support ancient, unupdatable Android devices](https://letsencrypt.org/2020/12/21/extending-android-compatibility.html) (@Ramhound). Firefox's own truststore has had the ISRG X1 root for years now, and should be using it: have you configured some option(s) like [security.enterprise_roots.enabled](https://serverfault.com/questions/722563/how-to-make-firefox-trust-system-ca-certificates)? If you look in burger Options Priv&Sec ViewCerts Authorities is Internet Security Research Group / ISRG X1 there? – dave_thompson_085 Dec 11 '21 at 05:45
  • Yes, Firefox does contain ISRG Root X1 and R3. –  Dec 11 '21 at 12:23
  • @Ramhound Where do you see the expired cert? Every cert in the chain that I see expires years from now (2024,2025) –  Dec 11 '21 at 12:25
  • @ehammer - `DST Root CA X3` expired on September 30, 2021. The issue is the Nginx configuration is not necessarily Firefox. I mean the error message, specifically, call's out, the issuer which I assume is ``DST Root CA X3` – Ramhound Dec 11 '21 at 14:42
  • I'm not entirely sure how I can solve this if the cert that is the problem is one that I don't manage? –  Dec 11 '21 at 16:45
  • Is the server sending all the certificates required to verify the end-entity certificate? That is, is it sending the intermediate CA certificate(s) as part of the handshake? Remember that Firefox does not use the AIA extension's caIssuer field to fetch missing certificates and expects servers to be configured correctly. – garethTheRed Dec 12 '21 at 19:49
  • The certificate (albeit limited) information I posted originally was all the server sent to the client. I did a pcap on the connection but then I realized TLS1.3 encrypts the server hello. –  Dec 13 '21 at 22:33
  • Taking a look at certs on my other servers, there is a difference between those that are working and those that arent. The bridge issuer CN for the server that doesnt work is DST Root CA X3, whereas the server that works is issued by ISRG Root X1. This makes me think firefox doesn't like certificate chains where the "root" certificate is only a bridge and not the actual root certificate. –  Dec 13 '21 at 23:05
  • Updated my post with pictures of what I am talking about –  Dec 13 '21 at 23:14
  • Yes; I will go back to my statement about the certificate being signed by the CA X3 certificate, which has expired, and is no longer technically valid. The differences between the two certificates highlight this fact. Only the ISRG X1 certificate should be an issuer on the certificate FIrefox does NOT accept. – Ramhound Dec 13 '21 at 23:18
  • Yeah I understand that now. But why is firefox browser getting the wrong issuer for the root certificate. Chrome pulls a different root cert issuer for the same site. –  Dec 13 '21 at 23:21
  • I attempted to regenerate the problem certs with certbots --preferred-chain "ISRG Root X1" and now firefox is only seeing the server cert and the R3 intermediate cert (signed by ISRG Root X1). I may be wrong, but it seems like firefox won't connect to certs where the last one in the chain isn't "self signed" i.e. the final root cert. –  Dec 14 '21 at 21:32
  • Deleting the expired DST Root CA X3 in Firefox and reloading the website suddenly makes it work. This seems less a problem with nginx if I can fix the issue purely on the client side. Why does firefox even still have the DST Root CA X3 if its expired? I'd think the newest firefox would be the first to remove it. –  Dec 15 '21 at 18:09

0 Answers0