You are here: Home > Participate > Join a Discussion > RIPE Forum
This mailing list interface will be retired once we upgrade our mailing list system. Click here to use the new RIPE NCC Forum.

RIPE Atlas

Threaded
Collapse

[atlas] TLS Certificate probes fail ("handshake failure") against Vercel servers

User Image

Daniel Gillmor

2023-01-25 18:16:07 CET

Hi RIPE Atlas folks--

I noticed today that using the "SSL" probe (gathering X.509 certificates
from a TLS endpoint) appears to fail against servers operated by
Vercel.  the failure is a uniform "handshake failure" according to the
web interface.

I think this might be a bug in the probe's implementation of the client
side of TLS.  Where could i find the specific implementation of that
probe type so that i can debug it further?

Or, if this is a known issue, where can i read up on it in more detail?

I've demonstrated this by probing Vercel's own servers, since they
appear to be eating their own dogfood.

You can see this with measurement 49131334:
https://atlas.ripe.net/measurements/49131334/#probes

here is the definition of the query i made:

{
 "definitions": [
  {
   "target": "vercel.com",
   "af": 4,
   "port": 443,
   "hostname": "vercel.com",
   "description": "SSL measurement to vercel.com",
   "resolve_on_probe": true,
   "skip_dns_check": false,
   "type": "sslcert"
  }
 ],
 "probes": [
  {
   "type": "area",
   "value": "WW",
   "requested": 10
  }
 ],
 "is_oneoff": true,
}


This is pretty odd, since normal TLS clients appear to be able to
connect to them fine.  Here's me running a manual handshake debugger
against vercel.com's servers:

$ gnutls-cli-debug vercel.com
GnuTLS debug client 3.7.8
Checking vercel.com:443
whether the server accepts default record size (512 bytes)... yes
                  whether %ALLOW_SMALL_RECORDS is required... no
                        whether we need to disable TLS 1.2... no
                        whether we need to disable TLS 1.1... no
                        whether we need to disable TLS 1.0... no
                        whether %NO_EXTENSIONS is required... no
                             for TLS 1.0 (RFC2246) support... no
 for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no
                             for TLS 1.1 (RFC4346) support... no
                                  fallback from TLS 1.1 to... failed
                             for TLS 1.2 (RFC5246) support... yes
                             for TLS 1.3 (RFC8446) support... yes
                    for known TLS or SSL protocols support... yes
                       TLS1.2 neg fallback from TLS 1.6 to... failed (server requires fallback dance)
                                     for HTTPS server name... Vercel
                               for certificate chain order... sorted
                  for safe renegotiation (RFC5746) support... yes
                    for encrypt-then-MAC (RFC7366) support... no
                   for ext master secret (RFC7627) support... no
                           for heartbeat (RFC6520) support... no
                       for version rollback bug in RSA PMS... yes
            whether the server ignores the RSA PMS version... yes
whether small records (512 bytes) are tolerated on handshake... yes
    whether cipher suites not in SSL 3.0 spec are accepted... yes
whether a bogus TLS record version in the client hello is accepted... yes
         whether the server understands TLS closure alerts... yes
            whether the server supports session resumption... yes
                      for anonymous authentication support... no
                              for RSA key exchange support... no
|<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice
                      for ephemeral Diffie-Hellman support... yes
|<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice
                        for RFC7919 Diffie-Hellman support... no
                   for ephemeral EC Diffie-Hellman support... yes
for VKO GOST-2012 (draft-smyshlyaev-tls12-gost-suites) support... no
                             for curve SECP256r1 (RFC4492)... yes
                             for curve SECP384r1 (RFC4492)... yes
                             for curve SECP521r1 (RFC4492)... yes
                                for curve X25519 (RFC8422)... yes
                      for AES-GCM cipher (RFC5288) support... yes
                      for AES-CCM cipher (RFC6655) support... no
                    for AES-CCM-8 cipher (RFC6655) support... no
                      for AES-CBC cipher (RFC3268) support... no
                 for CAMELLIA-GCM cipher (RFC6367) support... no
                 for CAMELLIA-CBC cipher (RFC5932) support... no
                     for 3DES-CBC cipher (RFC2246) support... no
                  for ARCFOUR 128 cipher (RFC2246) support... no
            for CHACHA20-POLY1305 cipher (RFC7905) support... yes
for GOST28147-CNT cipher (draft-smyshlyaev-tls12-gost-suites) support... no
                                       for MD5 MAC support... no
                                      for SHA1 MAC support... no
                                    for SHA256 MAC support... no
for GOST28147-IMIT MAC (draft-smyshlyaev-tls12-gost-suites) support... no
                     for max record size (RFC6066) support... no
                for OCSP status response (RFC6066) support... yes
$

Thanks for any pointers,

       --dkg

User Image

Michel Stam

2023-01-25 18:30:59 CET

RIPE NCC staff member

Hi Daniel,

I think this may be because the measurement code doesn’t support TLS 1.3 yet, and vercel.com does. It’s a known issue, we’d like to add TLS 1.3 at some point.

You can find the relevant code here:
https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/blob/7c03fba082e93b7a1f0f14cc3769bb31e83909e3/eperd/sslgetcert.c#L927
ripe-atlas-probe-measurements/sslgetcert.c at 7c03fba082e93b7a1f0f14cc3769bb31e83909e3 · RIPE-NCC/ripe-atlas-probe-measurements
github.com

Cheers,

Michel

> On 25 Jan 2023, at 18:16, Daniel Kahn Gillmor <dkg _at_ fifthhorseman _dot_ net> wrote:
> 
> Hi RIPE Atlas folks--
> 
> I noticed today that using the "SSL" probe (gathering X.509 certificates
> from a TLS endpoint) appears to fail against servers operated by
> Vercel.  the failure is a uniform "handshake failure" according to the
> web interface.
> 
> I think this might be a bug in the probe's implementation of the client
> side of TLS.  Where could i find the specific implementation of that
> probe type so that i can debug it further?
> 
> Or, if this is a known issue, where can i read up on it in more detail?
> 
> I've demonstrated this by probing Vercel's own servers, since they
> appear to be eating their own dogfood.
> 
> You can see this with measurement 49131334:
> https://atlas.ripe.net/measurements/49131334/#probes
> 
> here is the definition of the query i made:
> 
> {
> "definitions": [
>  {
>   "target": "vercel.com",
>   "af": 4,
>   "port": 443,
>   "hostname": "vercel.com",
>   "description": "SSL measurement to vercel.com",
>   "resolve_on_probe": true,
>   "skip_dns_check": false,
>   "type": "sslcert"
>  }
> ],
> "probes": [
>  {
>   "type": "area",
>   "value": "WW",
>   "requested": 10
>  }
> ],
> "is_oneoff": true,
> }
> 
> 
> This is pretty odd, since normal TLS clients appear to be able to
> connect to them fine.  Here's me running a manual handshake debugger
> against vercel.com's servers:
> 
> $ gnutls-cli-debug vercel.com
> GnuTLS debug client 3.7.8
> Checking vercel.com:443
> whether the server accepts default record size (512 bytes)... yes
>                  whether %ALLOW_SMALL_RECORDS is required... no
>                        whether we need to disable TLS 1.2... no
>                        whether we need to disable TLS 1.1... no
>                        whether we need to disable TLS 1.0... no
>                        whether %NO_EXTENSIONS is required... no
>                             for TLS 1.0 (RFC2246) support... no
> for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no
>                             for TLS 1.1 (RFC4346) support... no
>                                  fallback from TLS 1.1 to... failed
>                             for TLS 1.2 (RFC5246) support... yes
>                             for TLS 1.3 (RFC8446) support... yes
>                    for known TLS or SSL protocols support... yes
>                       TLS1.2 neg fallback from TLS 1.6 to... failed (server requires fallback dance)
>                                     for HTTPS server name... Vercel
>                               for certificate chain order... sorted
>                  for safe renegotiation (RFC5746) support... yes
>                    for encrypt-then-MAC (RFC7366) support... no
>                   for ext master secret (RFC7627) support... no
>                           for heartbeat (RFC6520) support... no
>                       for version rollback bug in RSA PMS... yes
>            whether the server ignores the RSA PMS version... yes
> whether small records (512 bytes) are tolerated on handshake... yes
>    whether cipher suites not in SSL 3.0 spec are accepted... yes
> whether a bogus TLS record version in the client hello is accepted... yes
>         whether the server understands TLS closure alerts... yes
>            whether the server supports session resumption... yes
>                      for anonymous authentication support... no
>                              for RSA key exchange support... no
> |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice
>                      for ephemeral Diffie-Hellman support... yes
> |<1>| FFDHE groups advertised, but server didn't support it; falling back to server's choice
>                        for RFC7919 Diffie-Hellman support... no
>                   for ephemeral EC Diffie-Hellman support... yes
> for VKO GOST-2012 (draft-smyshlyaev-tls12-gost-suites) support... no
>                             for curve SECP256r1 (RFC4492)... yes
>                             for curve SECP384r1 (RFC4492)... yes
>                             for curve SECP521r1 (RFC4492)... yes
>                                for curve X25519 (RFC8422)... yes
>                      for AES-GCM cipher (RFC5288) support... yes
>                      for AES-CCM cipher (RFC6655) support... no
>                    for AES-CCM-8 cipher (RFC6655) support... no
>                      for AES-CBC cipher (RFC3268) support... no
>                 for CAMELLIA-GCM cipher (RFC6367) support... no
>                 for CAMELLIA-CBC cipher (RFC5932) support... no
>                     for 3DES-CBC cipher (RFC2246) support... no
>                  for ARCFOUR 128 cipher (RFC2246) support... no
>            for CHACHA20-POLY1305 cipher (RFC7905) support... yes
> for GOST28147-CNT cipher (draft-smyshlyaev-tls12-gost-suites) support... no
>                                       for MD5 MAC support... no
>                                      for SHA1 MAC support... no
>                                    for SHA256 MAC support... no
> for GOST28147-IMIT MAC (draft-smyshlyaev-tls12-gost-suites) support... no
>                     for max record size (RFC6066) support... no
>                for OCSP status response (RFC6066) support... yes
> $
> 
> Thanks for any pointers,
> 
>       --dkg
> 
> -- 
> ripe-atlas mailing list
> ripe-atlas _at_ ripe _dot_ net
> https://lists.ripe.net/mailman/listinfo/ripe-atlas

User Image

Daniel Gillmor

2023-01-25 22:27:01 CET

Hi Michael--

Thanks for your followup!  More below…

On Wed 2023-01-25 18:30:59 +0100, Michel Stam wrote:
> I think this may be because the measurement code doesn’t support TLS
> 1.3 yet, and vercel.com does. It’s a known issue, we’d like to add TLS
> 1.3 at some point.

Hm, i don't think that's the full story, because the same probe actually
succeeds for sites that also support TLS 1.3
(e.g. https://www.aclu.org/).  And, when i try to connect to it from a
client that has TLS 1.3 deliberately disabled (e.g. "gnutls-cli
-priority NORMAL:-VERS-TLS1.3 vercel.com") i still have no problem connecting.

Digging into it a bit further, it looks to me like Vercel servers send
an alert if we do not emit the ec_point_format TLS extension.

This is probably a bug on Vercel's side, but it shouldn't block the Atlas'
ability to harvest certificates from it.

> You can find the relevant code here:
> https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/blob/7c03fba082e93b7a1f0f14cc3769bb31e83909e3/eperd/sslgetcert.c#L927

Thanks for this pointer!

I've provided a (mainly untested) pull request with a pretty simple
patch that should hopefully fix the issue:

   https://github.com/RIPE-NCC/ripe-atlas-probe-measurements/pull/15

If anyone on this list has the ability to test this patch and follow up
on that issue, i'd appreciate any review.

Regards,

        --dkg