Encrypting communication between an app that uses an ODBC/DSN (with ADODB) and SQL Server 2008 R2

Question

I've been doing a lot of reading the last couple of days on how we can encrypt db communication between our product app and a customer's SQL Server db, but cannot make it work as expected. The app uses a ODBC/DSN to connect to the SQL Server db. I use this
ODBC app to setup the DSN (on a Windows 7 PC):
C:\Windows\SysWOW64\odbcad32.exe
The  DSNconnection uses the SQL Server driver 6.01.7601.17514 and has these properties:
- WinNT authentication.
- Client Config button: TCP/IP to <server-name>\<instance.-name>
- Change to default db: <name-of-app-db>
- Everything else is default setting.
SQL Server is on the same Windowns 7 PC and has a self-signed cert installed (used IIS to generate it) and has the Force Enryption set to "yes".
I have a test C# program that uses the ADODB 2.7.0.0 COM-wrapper, made by Visual Studio after adding a reference to the ADO 2.7 library version 6.1.7601.17857. The program creates an ADODB.Connection object that has a simple connection string: "DSN=<san-name>;UID=<user>;PWD=<password>". The
program then creates an ADODB.Recordset object and reads and displays a field from a table.
Works fine.
If I go into SQL Server and set Force Encryption to "no," clear the cert, restart the SQL service, and then re-run the program, it works fine.
Here's the kick. If I go into the DSN and select "Use strong encryption for data" the Test button on the DSN works—why does it work? The SQL Server is no longer encrypting the connection so an error should occur. If I run the test program, it works
as well—why? I can look at the connection properties in the test program and see that ADODB has added the ";Encrypt-yes" stuff to the end of the connection string. Yet that option seems to have no effect.
If I set the SQL instance back to Force Encryption:yes, enable the cert, restart the SQL service, and clear the DSN's "Use strong encryption for data" option, I can still connect to the db with the
test program—why?
What am I doing wrong? I need to be able to ensure that the communication between our app product and the SQL instance is encrypted, and that we get an error if the SQL instance does not support encrypted communications. We really don't want the customer
to have to enable Force Encryption because they have other db's on their SQL Server that do not use encrypted communication, but they want to know that our product's communication channel with the db is encrypted.
No, I can't change the app product's code at this point in time. I'm stuck with working with what a DSN called from ADODB has to offer.
Also, how can I be sure that communications are encrypted? I mean, I've tried things like "SELECT * FROM sys.dm_exec_connections" but that doesn't help because I have no idea how to tie the list of sessions shown back to my test program, although
I guess it's a good thing that some of the sessions listed show encrypt_option as TRUE.
-glenn-

Answer 1

Ah yes, very good point. It's easy to miss because you have to delete then recreate the DSN in order to change drivers. So I switched the DSN over to the SQL Native 11 driver.
Now when I try to connect to the SQL instance as <computer-name>\<instance-name>, and without a cert on the server, I get "the target principal name is incorrect". Perfect; now we're getting somewhere!
Change my DSN to use <fqdn>\<instance-name> and it works. This tells me that SQL Server has auto-generated a cert and named it <fqdn>. I would have expected the "cert fail" error, not a cert
name mismatch, because I'm not using Trust Server Cert.
So I load up my self-signed cert, and that works too.
I am still confused as to why I'm not seeing the "cert fail" error when I have no cert loaded on the SQL Server. I am not using Force Encryption on the server at all, so wouldn't expect SQL Server to auto-create a cert when an Encrypt=yes request comes in,
but apparently it does?
I also ran into a problem with this:
select c.session_id, c.encrypt_option, s.client_interface_name
from sys.dm_exec_connections c
join sys.sysprocesses s
  on c.session_id = s.session_id
where s.dbid = db_id('MyDatabase')
There is no s.client_interface_name, probably should be s.hostname. There's also no s.session_id. I thought maybe this should be s.sid, but then no rows ever come back. The c.session_id looks like 51 and 52, but the s.sid looks like a very long binary number,
so these two fields cannot be joined. I don't know how to convert the sid's properly so that the join would work. Ah wait, I just found the s.spid column; the join works when that column is used (I assume that's correct anyhow).
If I add a Thread.Sleep(30 seconds) to my C# program just before the connection is closed, this query shows me the session for the correct hostname has encrypt_option=TRUE.
And I have to keep my fingers crossed that all the app I/O will still work properly after
changing the driver. Probably a safe bet though.
Think I'm ready to throw in the towel on getting the "cert fail/no SSL" error to appear.
It does look like I am able to sufficiently show that the connection is encrypted when Encrypt=yes is used with the newer driver.
Thank for all the help!
-glenn-