Couldn't connect to Amazon Keyspaces from EC2-hosted C# app using CassandraCSharpDriver but CAN connect from my dev machine

I'm trying to connect to Amazon Keyspaces from EC2 Windows instance, and couldn't, BUT I can successfully connect to it from my local PC with absolutely same trivial console application:

using System;
using Cassandra;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;

namespace cassandraconnectiotest;
internal class Program
{
    static void Main(string[] args)
    {
        try
        {
            var amazoncert = new X509Certificate2("AmazonRootCA1.der");
            // I obtained AmazonRootCA1.der by executing:
            // curl https://www.amazontrust.com/repository/AmazonRootCA1.pem -o AmazonRootCA1.pem
            // openssl x509 -outform der -in AmazonRootCA1.pem -out AmazonRootCA1.der


            var sslOptions = new SSLOptions(System.Security.Authentication.SslProtocols.Tls13, false, null) //it will NOT work with simple new SSLOptions() for some reason
            // and will return Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.190:9142: IOException 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..')
                .SetCertificateCollection(new X509Certificate2Collection { amazoncert })
                .SetRemoteCertValidationCallback((sender, cert, chain, sslPolicyErrors) =>
                {
                    Console.WriteLine(cert.Subject);
                    if (sslPolicyErrors == SslPolicyErrors.None)
                    {
                        Console.WriteLine($"SSL Certificate is valid!");
                        return true; // Certificate is valid
                    }
                    else
                    {
                        // Log the SSL policy errors
                        Console.WriteLine($"SSL Certificate Error: {sslPolicyErrors}");

                        // Optionally, log details about the certificate and chain
                        if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateChainErrors) != 0 && chain != null)
                        {
                            foreach (var chainStatus in chain.ChainStatus)
                            {
                                Console.WriteLine($"Chain Status: {chainStatus.Status} - {chainStatus.StatusInformation}");
                            }
                        }

                        // Return false to reject the certificate
                        return false;
                    }
                });

            var cluster = Cluster.Builder()
                 .AddContactPoints("cassandra.us-east-2.amazonaws.com")
                 .WithPort(9142)
                 .WithAuthProvider(new PlainTextAuthProvider("cassandra_keyspaces+1-at-NNNNhere", @"xLk35password/2ndpartofit="))
                 .WithSSL(sslOptions)
                 .Build();
            var session = cluster.Connect();
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }
    }
}

When I'm connecting from my local machine, I'm getting number of messages like this:

CN=cassandra.us-east-2.amazonaws.com
SSL Certificate is valid!
CN=cassandra.us-east-2.amazonaws.com
SSL Certificate is valid!

so I'm assuming it's negotiation phase.

But when I'm running same app from EC2 instance, I'm getting:

Cassandra.NoHostAvailableException: All hosts tried for query failed (tried 3.12.23.188:9142: AuthenticationException 'Authentication failed, see inner exception.')
   at Cassandra.Connections.Control.ControlConnection.Connect(Boolean isInitializing)
   at Cassandra.Connections.Control.ControlConnection.InitAsync()
   at Cassandra.Tasks.TaskHelper.WaitToCompleteAsync(Task task, Int32 timeout)
   at Cassandra.Cluster.Init()
   at Cassandra.Cluster.ConnectAsync(String keyspace)
   at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout)
   at Cassandra.Tasks.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout)
   at Cassandra.Cluster.Connect(String keyspace)
   at Cassandra.Cluster.Connect()
   at cassandraconnectiotest.Program.Main(String[] args) in D:\git\beholdtech\trivialcassandraconnectiontest\cassandraconnectiotest\Program.cs:line 56

At the same time on EC2 instance:

tracert cassandra.us-east-2.amazonaws.com

gives:

Tracing route to cassandra.us-east-2.amazonaws.com [3.12.23.165]... 

(seems to be working)

C:\>nslookup cassandra.us-east-2.amazonaws.com
Server:  ip-172-31-0-2.us-east-2.compute.internal
Address:  172.31.0.2

Non-authoritative answer:
Name:    cassandra.us-east-2.amazonaws.com
Address:  3.12.23.190



telnet cassandra.us-east-2.amazonaws.com 9142

gives black screen with some effect when I type.

c:\> openssl s_client -connect cassandra.us-east-2.amazonaws.com:9142 -msg -debug

gives

Connecting to 3.12.23.184
CONNECTED(00000140)
... dumps... 
subject=CN=cassandra.us-east-2.amazonaws.com
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)

openssl s_client -connect cassandra.us-east-2.amazonaws.com:9142 -CAfile "C:\AmazonRootCA1.der" returns:

E4130000:error:05800088:x509 certificate routines:X509_load_cert_crl_file_ex:no certificate or crl found:crypto\x509\by_file.c:264:

If I use another certificate, sf-class2-root.crt, or sf-class2-root.cer obtained via:

$client = new-object System.Net.WebClient
$client.DownloadFile("https://certs.secureserver.net/repository/sf-class2-root.crt","d:\sf-class2-root.crt")                                                                                                            
Export-Certificate -Cert $cert -FilePath "c:\sf-class2-root.cer"

c:>openssl s_client -connect cassandra.us-east-2.amazonaws.com:9142 -CAfile "C:\NEWTEST\sf-class2-root.crt"

will give me:

Connecting to 3.12.23.176
CONNECTED(00000140)
depth=4 C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
verify return:1
depth=3 C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M01
verify return:1
depth=0 CN=cassandra.us-east-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:CN=cassandra.us-east-2.amazonaws.com
   i:C=US, O=Amazon, CN=Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov  8 00:00:00 2024 GMT; NotAfter: Oct 26 23:59:59 2025 GMT
 1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M01
   i:C=US, O=Amazon, CN=Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
 2 s:C=US, O=Amazon, CN=Amazon Root CA 1
   i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
 3 s:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   i:C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT

...
-----END CERTIFICATE-----
subject=CN=cassandra.us-east-2.amazonaws.com
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5565 bytes and written 408 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

DONE

but starting application with this sf-class2-root.crt will lead to the same result: everything works from localhost and error on EC2 instance.

Can you please advice, what to do? How to add any logging? what could be the reason? My EC2 instance has static IP and all outbound connections allowed.

It was supposed to be trivial, very quick, easy task. It took me already DAYS trying to figure out... Is there another way to connect using CassndraCSharpDriver? TRIVIAL, which will just work from any computer, either local or EC2 (I can modify code during deployment as well, adopting behavior to environment).

Become part of our
growing community!
Welcome to Planet Cassandra, a community for Apache Cassandra®! We're a passionate and dedicated group of users, developers, and enthusiasts who are working together to make Cassandra the best it can be. Whether you're just getting started with Cassandra or you're an experienced user, there's a place for you in our community.
A dinosaur
Planet Cassandra is a service for the Apache Cassandra® user community to share with each other. From tutorials and guides, to discussions and updates, we're here to help you get the most out of Cassandra. Connect with us and become part of our growing community today.
© 2009-2023 The Apache Software Foundation under the terms of the Apache License 2.0. Apache, the Apache feather logo, Apache Cassandra, Cassandra, and the Cassandra logo, are either registered trademarks or trademarks of The Apache Software Foundation. Sponsored by Anant Corporation and Datastax, and Developed by Anant Corporation.

Get Involved with Planet Cassandra!

We believe that the power of the Planet Cassandra community lies in the contributions of its members. Do you have content, articles, videos, or use cases you want to share with the world?