L2TP based VPN with OpenS/WAN server, OpenSSL machine certificates

Question

I cannot seem to get OSX to accept the machine certificates for a VPN connection using Internet Connect.
I have generated OpenSSL x509 certificates for the server and client side, the same process has generated certificates that work just dandy with WindowsXP. The certificates have "subjectAltName=" key/value pairs assigned to the IP address of the VPN server.
Once generated I import the certificates into OS X (you have to run KeyChain Access with "sudo" from the console to get this to work). The certificate authority seems to be ok, the CA has been added to the x509Roots, and when I examine the machine certificate for my OS X install using KeyChain Access the certificate is marked valid.
I generated the hash link for the certificate:
ln -s /etc/racoon/certs/certname.pem /etc/racoon/certs/'openssl x509 -noout -in certname.pem'.0
From the console I run '
openssl verify certname.pem
It fails unless I specify '-CAPath /etc/racoon/certs', then it passes.
When Internet Connect is setup to use the certificates I can see in the OpenS/WAN logs that the OS X box connects and negotiates IPSEC to MAIN_3. At this point pluto logs the following:
ignoring informational payload, type INVALIDCERTAUTHORITY
This repeats for several re-tries before the OS X side gives up. No useful logging is generated on the OS X side for me to debug, and everything from the OpenS/WAN side seems to be kosher, it appears to be an oakley/racoon issue with validating the machine certificate provided by OpenS/WAN to the OS X side, with the OS X side unable to verify the certificate.
Has anyone solved this? Any ideas on how to improve the logging output from OS X so I can see what racoon/oakley is carping about in the certificate files it is using?

Answer 1

I'm having the same problem. I've got a machine cert on my Mac OS 10.4.6 client that was issued by my Win2003 CA. When I try and connect, it just hangs and then dies. In the Security Logs on the 2003 L2TP server, I even see a successful IKE negotiation (MS Event ID 541 and 543 below).
EventID 541:
IKE security association established.
Mode:
Key Exchange Mode (Main Mode)
Peer Identity:
Certificate based Identity.
Peer Subject C=US, S=City, L=State, O=Company, OU=group, CN=machine.subdomain.company.com, E=[email protected]
Peer SHA Thumbprint peerthumbrint
Peer Issuing Certificate Authority O=company.com, CN=Certificate Authority
Root Certificate Authority O=company.com, CN=Certificate Authority
My Subject CN=server.subdomain.company.com
My SHA Thumbprint mythumbrint
Peer IP Address: x.x.x.x
Filter:
Source IP Address x.x.x.x
Source IP Address Mask 255.255.255.255
Destination IP Address x.x.x.x
Destination IP Address Mask 255.255.255.255
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr x.x.x.x
IKE Peer Addr x.x.x.x
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr
Parameters:
ESP Algorithm Triple DES CBC
HMAC Algorithm SHA
Lifetime (sec) 3600
MM delta time (sec) 1
EventID 543:
IKE security association ended.
Mode: Key Exchange (Main mode)
Filter:
Source IP Address X.X.X.X
Source IP Address Mask 255.255.255.255
Destination IP Address X.X.X.X
Destination IP Address Mask 255.255.255.255
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr X.X.X.X
IKE Peer Addr X.X.X.X
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr
At least give me a some methods to debug with.