Invalid Key Exception: Unsupported key type: Sun RSA public key, 1024 bits

Question

I am trying to retrieve certificates from Microsoft Keystore and extract its keys using SunMSCAPI in jdk 1.6. It gives me an invalid key exception, when I am trying to wrap the Symmetric key (which was previously used to perform AES encryption on data), using RSA algorithm.
Code snippet:
           // RSA 1024 bits Asymmetric encryption of Symmetric AES key             
            // List the certificates from Microsoft KeyStore using SunMSCAPI.
                  System.out.println("List of certificates found in Microsoft Personal Keystore:");
                   KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI");
                   ks.load(null, null) ;
                   Enumeration en = ks.aliases() ;
                   PublicKey RSAPubKey = null;
                   Key RSAPrivKey = null;
                   int i = 0;
                   while (en.hasMoreElements()) {
                        String aliasKey = (String)en.nextElement() ;              
                        X509Certificate c = (X509Certificate) ks.getCertificate(aliasKey) ;     
                        String sss = ks.getCertificateAlias(c);
                        if(sss.equals("C5151997"))
                        System.out.println("---> alias : " + sss) ;
                        i= i + 1;
                        String str = c.toString();
                        System.out.println(" Certificate details : " + str ) ;
                      RSAPubKey = c.getPublicKey();
                        RSAPrivKey = ks.getKey(aliasKey, null);  //"mypassword".toCharArray()
                        Certificate[] chain = ks.getCertificateChain(aliasKey);     
                   System.out.println("No of certificates found from Personal MS Keystore: " + i);
            // Encrypt the generated Symmetric AES Key using RSA cipher      
                    Cipher rsaCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", ks.getProvider().getName());            
                   rsaCipher.init(Cipher.WRAP_MODE, RSAPubKey);
                   byte[] encryptedSymmKey = rsaCipher.wrap(aeskey);   
                   System.out.println("Encrypted Symmetric Key :" + new String(encryptedSymmKey));
                   System.out.println("Encrypted Symmetric Key Length in Bytes: " + encryptedSymmKey.length);
                   // RSA Decryption of Encrypted Symmetric AES key
                   rsaCipher.init(Cipher.UNWRAP_MODE, RSAPrivKey);
                   Key decryptedKey = rsaCipher.unwrap(encryptedSymmKey, "AES", Cipher.SECRET_KEY);Output:
List of certificates found in Microsoft Personal Keystore:
---> alias : C5151997
Certificate details : [
Version: V3
Subject: CN=C5151997, O=SAP-AG, C=DE
Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5
Key: Sun RSA public key, 1024 bits
modulus: 171871587533146191561538456391418351861663300588728159334223437391061141885590024223283480319626015611710315581642512941578588886825766256507714725820048129123720143461110410353346492039350478625370269565346566901446816729164309038944197418238814947654954590754593726047828813400082450341775203029183105860831
public exponent: 65537
Validity: [From: Mon Jan 24 18:17:49 IST 2011,
               To: Wed Jan 23 18:17:49 IST 2013]
Issuer: CN=SSO_CA, O=SAP-AG, C=DE
SerialNumber: [    4d12c509 00000005 eb85]
Certificate Extensions: 6
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 07 E5 83 A1 B2 B7 DF 6B 4B 67 9C 1D 42 C9 0D F4 .......kKg..B...
0010: 35 76 D3 F7 5v..
[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: E4 C4 2C 93 20 AF DA 4C F2 53 68 4A C0 E7 EC 30 ..,. ..L.ShJ...0
0010: 8C 0C 3B 9A ..;.
[3]: ObjectId: 1.3.6.1.4.1.311.21.7 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 30 30 2E 06 26 2B 06 01 04 01 82 37 15 08 82 .00..&+.....7...
0010: D1 E1 73 84 E4 FE 0B 84 FD 8B 15 83 E5 90 1B 83 ..s.............
0020: E6 A1 43 81 62 84 B1 DA 50 9E D3 14 02 01 64 02 ..C.b...P.....d.
0030: 01 1B ..
[4]: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
RFC822Name: [email protected]
[5]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
[6]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:false
PathLen: undefined
Algorithm: [SHA1withRSA]
Signature:
0000: B3 C5 92 66 8D D7 ED 6D 51 12 63 CC F4 52 18 B9 ...f...mQ.c..R..
0010: B8 A6 78 F7 ED 7D 78 18 DA 71 09 C9 AE C8 49 23 ..x...x..q....I#
0020: F5 32 2F 0F D1 C0 4C 08 2B 6D 3C 11 B9 5F 5B B5 .2/...L.+m<.._[.
0030: 05 D9 CA E6 F9 0A 94 14 E7 C6 7A DB 63 FE E5 EC ..........z.c...
0040: 48 94 8C 0D 77 92 59 DE 34 6E 77 1A 24 FE E3 C1 H...w.Y.4nw.$...
0050: D8 0B 52 6A 7E 22 13 71 D7 F8 AF D1 17 C8 64 4F ..Rj.".q......dO
0060: 83 EA 2D 6A CA 7F C3 84 37 15 FE 99 73 1D 7C D1 ..-j....7...s...
0070: 6D B4 99 09 62 B9 0F 18 33 4C C6 66 7A 9F C0 DB m...b...3L.fz...
No of certificates found from Personal MS Keystore: 1
Exception in thread "main" java.security.InvalidKeyException: Unsupported key type: Sun RSA public key, 1024 bits
modulus: 171871587533146191561538456391418351861663300588728159334223437391061141885590024223283480319626015611710315581642512941578588886825766256507714725820048129123720143461110410353346492039350478625370269565346566901446816729164309038944197418238814947654954590754593726047828813400082450341775203029183105860831
public exponent: 65537
     at sun.security.mscapi.RSACipher.init(RSACipher.java:176)
     at sun.security.mscapi.RSACipher.engineInit(RSACipher.java:129)
     at javax.crypto.Cipher.init(DashoA13*..)
     at javax.crypto.Cipher.init(DashoA13*..)
     at com.sap.srm.crpto.client.applet.CryptoClass.main(CryptoClass.java:102)
Edited by: sabre150 on 18-Jul-2011 03:47
Added [ code] tags to make code readable.

Answer 1

A bit of research indicates that the classes of the keys obtained by
                      RSAPubKey = c.getPublicKey();
                           RSAPrivKey = ks.getKey(aliasKey, null);  //"mypassword".toCharArray()are sun.security.rsa.RSAPublicKeyImpl and sun.security.*mscapi*.RSAPrivateKey . It seems that for Cipher objects from the SunMSCAPI provider cannot accept RSA public keys of class sun.security.rsa.RSAPublicKeyImpl and that the SunMSCAPI will only accept RSA private keys of class sun.security.mscapi.RSAPrivateKey.
This came up under different guise a couple of years ago. It makes sense since encrypting/wrapping with a public key does not represent a security problem (there is nothing secret in any of the encryption operations) when done outside of MSCAPI so one can use any provider that has the capability BUT the decryption/unwrapping must be done with the SunMSCAPI provider which delegates it to the MSCAPI.
My working test code based on your code implementing this approach is :
        // RSA 1024 bits Asymmetric encryption of Symmetric AES key             
        // List the certificates from Microsoft KeyStore using SunMSCAPI.
        System.out.println("List of certificates found in Microsoft Personal Keystore:");
        KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI");
        ks.load(null, null);
        Enumeration en = ks.aliases();
        PublicKey RSAPubKey = null;
        Key RSAPrivKey = null;
        int i = 0;
        while (en.hasMoreElements())
            String aliasKey = (String) en.nextElement();
            X509Certificate c = (X509Certificate) ks.getCertificate(aliasKey);
            String sss = ks.getCertificateAlias(c);
            if (sss.equals("rsa_key")) // The alias for my key - make sure you change it back to your alias
                System.out.println("---> alias : " + sss);
                i = i + 1;
                String str = c.toString();
                System.out.println(" Certificate details : " + str);
                RSAPubKey = c.getPublicKey();
         System.out.println(RSAPubKey.getClass().getName());
               RSAPrivKey = ks.getKey(aliasKey, null);  //"mypassword".toCharArray()
        System.out.println(RSAPrivKey.getClass().getName());
                Certificate[] chain = ks.getCertificateChain(aliasKey);
        System.out.println(ks.getProvider().getName());
        System.out.println("No of certificates found from Personal MS Keystore: " + i);
        Cipher rsaCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");//, ks.getProvider().getName());       !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
            rsaCipher.init(Cipher.WRAP_MODE, RSAPubKey);
        byte[] keyBytes =
            1, 2, 3, 4, 5, 6, 7, 8, 2, 3, 4, 5, 6, 7, 8, 9
        SecretKey aeskey = new SecretKeySpec(keyBytes, "AES");
        byte[] encryptedSymmKey = rsaCipher.wrap(aeskey);
        System.out.println("Encrypted Symmetric Key :" + Arrays.toString(encryptedSymmKey));
        System.out.println("Encrypted Symmetric Key Length in Bytes: " + encryptedSymmKey.length);
        // RSA Decryption of Encrypted Symmetric AES key
        Cipher unwrapRsaCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", ks.getProvider().getName());       //!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
        unwrapRsaCipher.init(Cipher.UNWRAP_MODE, RSAPrivKey);
        Key decryptedKey = unwrapRsaCipher.unwrap(encryptedSymmKey, "AES", Cipher.SECRET_KEY);
        System.out.println("Decrypted Symmetric Key :" + Arrays.toString(decryptedKey.getEncoded())); // Matches the 'keyBytes' above