Signing with openssl verify with java fails

Question

Hi!
I've been itching my head for the last 6 hours on why I can't sign something with openssl and verify it in java. Anyone out there that can help me out?
I'm using OpenSSL 0.9.8k 25 Mar 2009 on Ubuntu 10.04 and using java version:
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)
1. Generation of the private key
openssl genrsa -out private_key.pem 2048
echo "Testing signing" >data.txt
openssl rsautl -sign -in data.txt -inkey private_key.pem -out data.signedOk, now I have the signature for the content of data.txt in the file data.signed. Let's try and verify this signature.
2. I'm using Bouncy Castle version 1.45 with the following code (just check verify for this case)
package test.signature;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.KeyPair;
import java.security.NoSuchAlgorithmException;
import java.security.Security;
import java.security.Signature;
import java.security.SignatureException;
import java.security.spec.InvalidKeySpecException;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openssl.PEMReader;
public class BouncySignatureHandler  {
     static String TYPE = "SHA1withRSA";
     public static void main(String[] args) throws Exception {
          if (args.length < 2)
               printInfoAndExit();
          String cmd = args[0];
          Security.addProvider(new BouncyCastleProvider());
          Signature signature = Signature.getInstance(TYPE, "BC");
          if ("verify".equalsIgnoreCase(cmd)) {
               String pubKeyFileName = args[1];
               String dataFileName = args[2];
               String signatureFileName = args[3];
               System.out.println("Using key " + pubKeyFileName
                         + " to verify signature " + signatureFileName
                         + " with the content of " + dataFileName);
               byte[] dataBytes = readFile(dataFileName);
               byte[] signatureBytes = readFile(signatureFileName);
               FileReader fr = new FileReader(new File(pubKeyFileName));
               PEMReader pemReader = new PEMReader(fr);
               KeyPair keyPair = (KeyPair) pemReader.readObject();
               fr.close();
               Signature verifier = Signature.getInstance(TYPE, "BC");
               verifier.initVerify(keyPair.getPublic());
               verifier.update(dataBytes);
               if (verifier.verify(signatureBytes)) {
                    System.out.println("Signature is valid");
               } else {
                    System.out.println("Signature is NOT valid");
          } else if ("sign".equalsIgnoreCase(cmd)) {
               String pemFileName = args[1];
               String dataFileName = args[2];
               System.out.println("Creating a signature with " + pemFileName
                         + " over the content of " + dataFileName);
               byte[] data = readFile(dataFileName);
               FileReader fr = new FileReader(new File(pemFileName));
               PEMReader pemReader = new PEMReader(fr);
               KeyPair keyPair = (KeyPair) pemReader.readObject();
               fr.close();
               signature.initSign(keyPair.getPrivate());
               signature.update(data);
               byte[] signatureBytes = signature.sign();
               writeFile(signatureBytes, dataFileName + ".signed");
               String encoded = Base64.encode(signatureBytes);
               writeFile(encoded.getBytes(), dataFileName + ".signed.base64");
               System.out.println("Signature for " + dataFileName + " saved in "
                         + dataFileName + ".signed");
          } else {
               printInfoAndExit();
     private static void writeFile(byte[] signatureBytes, String fName)
               throws IOException {
          FileOutputStream fout = new FileOutputStream(new File(fName));
          fout.write(signatureBytes);
          fout.close();
     private static void printInfoAndExit() {
          System.out
                    .println("BouncySignatureHandler <sign privKeyFile dataFile |verify pubkeyFile dataFile signedDataFile>");
          System.exit(1);
     public static byte[] readFile(String fileName) throws IOException {
          File f = new File(fileName);
          FileInputStream fis = new FileInputStream(f);
          DataInputStream dis = new DataInputStream(fis);
          byte[] bytes = new byte[(int) f.length()];
          dis.readFully(bytes);
          dis.close();
          return bytes;
}3. Running this code (no classpath here)
java -cp .... BouncySignatureHandler verify private_key.pem data.txt data.txt.signedThis gives the output, ..... NOT valid
Any ideas what I'm doing wrong????
Edited by: trouthunter on Sep 7, 2010 3:50 PM
Edited by: trouthunter on Sep 7, 2010 3:53 PM

Answer 1

The "openssl rsautl -sign" is not doing what you think it is. If one applies                   Cipher cipher = Cipher.getInstance("RSA", "BC");
                    cipher.init(Cipher.DECRYPT_MODE, keyPair.getPublic());
                    byte[] decryptedSignature= cipher.doFinal(signatureBytes);
                    HexEncoder hexEncoder =new HexEncoder();
                    System.out.println("Decrypted signature (hex) = " + hexEncoder.encodeAsString(decryptedSignature));
                    System.out.println("Decrypted as String = " + new String(decryptedSignature, decryptedSignature.length-16, 16));
one gets as a result
Decrypted signature (hex) = 01ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0054657374696e67207369676e696e670a
Decrypted as String = Testing signingwhich indicates that the openssl signature using the 'rsautl' command does not actually encrypt the digest of the message. It just pads the message with something approaching PKCS1 and encrypts that with the private key.
If you need the background to signing then [RFC3447|http://www.ietf.org/rfc/rfc3447.txt] describes how a signature for a given digest and encryption scheme is formed.
Since this is a Java forum, I will leave it to you to correct your use of openssl command.