Diffie Hellman Public Key from openSSL is throwing InvalidKeySpecException

Question

Ladies and Gents,
I am trying to write a client application in Java to replace an existing client app that is written in C++. The current client and server use openSSL for the crypto. The client and server perform a Diffie Hellman Key exchange and then encrypt the data streams. Communication is via net sockets and is pure TCP.
I can get a Java app to Java app DH key exchange to work as well as a C++/openSSL app to C++/openSSL app. A problem arises when I attempt to have a Java client perform a DH Key exchange with the C++/openSSL server.
I have narrowed down the error to this codeblock (I added the line numbers for clarity):
164       KeyFactory keyFac = KeyFactory.getInstance("DH");
165       X509EncodedKeySpec dhKeySpec = new X509EncodedKeySpec(peerPublicKeyBytes);
166       DHPublicKey peerPublicKey = (DHPublicKey) keyFac.generatePublic(dhKeySpec);Here is the error thrown:
java.security.spec.InvalidKeySpecException: Inappropriate key specification
     at com.sun.crypto.provider.DHKeyFactory.engineGeneratePublic(DashoA13*..)
     at java.security.KeyFactory.generatePublic(Unknown Source)
     at jClient.crypto.DHCipher.bytesToPublicKey(DHCipher.java:166)Now based on the error given, I can guess that the format of the Public Key I received is somehow incorrect. The server sends its generated public key in the following manner:
keyLen(4 bytes), keyBytes (keyLen bytes)
Which is simple enough, so I assumed that simply stripping off the first 4 bytes off the byte[] would yield a viable PublicKey, but I get the above error.
Additionally, I have tried to modify the code to use a DHPublicKeySpec instead of a X509EncodedKeySpec:
163       KeyFactory keyFac = KeyFactory.getInstance("DH");
164       BigInteger peerPubKeyBI = new BigInteger(peerPublicKeyBytes);
165       DHPublicKeySpec dhKeySpec = new DHPublicKeySpec(peerPubKeyBI, this.p, this.g);
166       DHPublicKey peerPublicKey = (DHPublicKey) keyFac.generatePublic(dhKeySpec);This seems to accept the openSSL generated public key, but ultimately results in non-matching client and server SecretKeys.
I normally don't post for help, choosing instead to just keep hammering away at a problem till I win, but this is going on 3 weeks and I have a deadline coming up. Any help would be appreciated. If anymore information is needed, just say so.
Thanks in advance,
Dave
Edited by: claymore1977 on May 26, 2009 4:47 PM

Answer 1

claymore1977 wrote:
Which is simple enough, so I assumed that simply stripping off the first 4 bytes off the byte[] would yield a viable PublicKey, but I get the above error.Too simple. If you look at the Javadocs for X509EncodedKeySpec, you can see that it is much more complicated beast that contains object IDs in addition to the DH public bytes. You could try to get openssl to build the more complicated X509EncodedKeySpec, but I see below that you have found a simpler solution.
>
>
Additionally, I have tried to modify the code to use a DHPublicKeySpec instead of a X509EncodedKeySpec:
163       KeyFactory keyFac = KeyFactory.getInstance("DH");
164       BigInteger peerPubKeyBI = new BigInteger(peerPublicKeyBytes);
165       DHPublicKeySpec dhKeySpec = new DHPublicKeySpec(peerPubKeyBI, this.p, this.g);
166       DHPublicKey peerPublicKey = (DHPublicKey) keyFac.generatePublic(dhKeySpec);
This looks reasonable, the only thing I can see wrong is the BigInteger constructor you are using. If the data sent is "negative", the resulting BigInteger will be negative and you'll get wrong answers. See if using the sign=magnitude constructor works better for you, i.e. BigInteger peerPubKeyBI = new BigInteger(1, peerPublicKeyBytes);