RSA decryption Error: Data must start with zero

Question

Because of some reasons, I tried to use RSA as a block cipher to encrypt/decrypt a large file. When I debug my program, there some errors are shown as below:
javax.crypto.BadPaddingException: Data must start with zero
     at sun.security.rsa.RSAPadding.unpadV15(Unknown Source)
     at sun.security.rsa.RSAPadding.unpad(Unknown Source)
     at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356)
     at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:394)
     at javax.crypto.Cipher.doFinal(Cipher.java:2299)
     at RSA.RRSSA.main(RRSSA.java:114)
From breakpoint, I think the problem is the decrypt operation, and Cipher.doFinal() can not be operated correctly.
I searched this problem from google, many people met the same problem with me, but most of them didn't got an answer.
The source code is :
Key generation:
package RSA;
import java.io.FileOutputStream;
import java.io.ObjectOutputStream;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.util.logging.Level;
import java.util.logging.Logger;
public class GenKey {
      * @param args
                 * @author tang
     public static void main(String[] args) {
          // TODO Auto-generated method stub
             try {
                  KeyPairGenerator KPG = KeyPairGenerator.getInstance("RSA");
                  KPG.initialize(1024);
                  KeyPair KP=KPG.genKeyPair();
                  PublicKey pbKey=KP.getPublic();
                  PrivateKey prKey=KP.getPrivate();
                  //byte[] publickey = decryptBASE64(pbKey);
                  //save public key
                  FileOutputStream out=new FileOutputStream("RSAPublic.dat");
                  ObjectOutputStream fileOut=new ObjectOutputStream(out);
                  fileOut.writeObject(pbKey);
                  //save private key
                      FileOutputStream outPrivate=new FileOutputStream("RSAPrivate.dat");
                  ObjectOutputStream privateOut=new ObjectOutputStream(outPrivate);
                             privateOut.writeObject(prKey)
     }Encrypte / Decrypt
package RSA;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.security.Key;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.crypto.Cipher;
//import sun.misc.BASE64Decoder;
//import sun.misc.BASE64Encoder;
public class RRSSA {
      * @param args
     public static void main(String[] argv) {
          // TODO Auto-generated method stub
            //File used to encrypt/decrypt
             String dataFileName = argv[0];
             //encrypt/decrypt: operation mode
             String opMode = argv[1];
             String keyFileName = null;
             //Key file
             if (opMode.equalsIgnoreCase("encrypt")) {
             keyFileName = "RSAPublic.dat";
             } else {
             keyFileName = "RSAPrivate.dat";
             try {
             FileInputStream keyFIS = new FileInputStream(keyFileName);
             ObjectInputStream OIS = new ObjectInputStream(keyFIS);
             Key key = (Key) OIS.readObject();
             Cipher cp = Cipher.getInstance("RSA/ECB/PKCS1Padding");//
             if (opMode.equalsIgnoreCase("encrypt")) {
             cp.init(Cipher.ENCRYPT_MODE, key);
             } else if (opMode.equalsIgnoreCase("decrypt")) {
             cp.init(Cipher.DECRYPT_MODE, key);
             } else {
             return;
             FileInputStream dataFIS = new FileInputStream(dataFileName);
             int size = dataFIS.available();
             byte[] encryptByte = new byte[size];
             dataFIS.read(encryptByte);
             if (opMode.equalsIgnoreCase("encrypt")) {
             FileOutputStream FOS = new FileOutputStream("cipher.txt");
             //RSA Block size
             //int blockSize = cp.getBlockSize();
             int blockSize = 64 ;
             int outputBlockSize = cp.getOutputSize(encryptByte.length);
             /*if (blockSize == 0)
                  System.out.println("BLOCK SIZE ERROR!");       
             }else
             int leavedSize = encryptByte.length % blockSize;
             int blocksNum = leavedSize == 0 ? encryptByte.length / blockSize
             : encryptByte.length / blockSize + 1;
             byte[] cipherData = new byte[outputBlockSize*blocksNum];
             //encrypt each block
             for (int i = 0; i < blocksNum; i++) {
             if ((encryptByte.length - i * blockSize) > blockSize) {
             cp.doFinal(encryptByte, i * blockSize, blockSize, cipherData, i * outputBlockSize);
             } else {
             cp.doFinal(encryptByte, i * blockSize, encryptByte.length - i * blockSize, cipherData, i * outputBlockSize);
             //byte[] cipherData = cp.doFinal(encryptByte);
             //BASE64Encoder encoder = new BASE64Encoder();
             //String encryptedData = encoder.encode(cipherData);
             //cipherData = encryptedData.getBytes();
             FOS.write(cipherData);
             FOS.close();
             } else {
            FileOutputStream FOS = new FileOutputStream("plaintext.txt");
             //int blockSize = cp.getBlockSize();
             int blockSize = 64;
             //int j = 0;
             //BASE64Decoder decoder = new BASE64Decoder();
             //String encryptedData = convert(encryptByte);
             //encryptByte = decoder.decodeBuffer(encryptedData);
             int outputBlockSize = cp.getOutputSize(encryptByte.length);
             int leavedSize = encryptByte.length % blockSize;
             int blocksNum = leavedSize == 0 ? encryptByte.length / blockSize
                       : encryptByte.length / blockSize + 1;
             byte[] plaintextData = new byte[outputBlockSize*blocksNum];
             for (int j = 0; j < blocksNum; j++) {
             if ((encryptByte.length - j * blockSize) > blockSize) {
                  cp.doFinal(encryptByte, j * blockSize, blockSize, plaintextData, j * outputBlockSize);
                  } else {
                  cp.doFinal(encryptByte, j * blockSize, encryptByte.length - j * blockSize, plaintextData, j * outputBlockSize);
             FOS.write(plaintextData);
             //FOS.write(cp.doFinal(encryptByte));
             FOS.close();
}Edited by: sabre150 on Aug 3, 2012 6:43 AM
Moderator action : added [ code] tags so as to make the code readable. Please do this yourself in the future.
Edited by: 949003 on 2012-8-3 上午5:31

Answer 1

1) Why are you not closing the streams when writing the keys to the file?
2) Each block of RSA encrypted data has size equal to the key modulus (in bytes). This means that for a key size of 1024 bits you need to read 128 bytes and not 64 bytes at a time when decrypting ( this is probably the cause of your 'Data must start with zero exception'). Since the input block size depends on the key modulus you cannot hard code this. Note - PKCS1 padding has at least 11 bytes of padding so on encrypting one can process a maximum of the key modulus in bytes less 11. Currently you have hard coded the encryption block at 64 bytes which is OK for your 1024 bits keys but will fail for keys of modulus less than about 936 bits.
3) int size = dataFIS.available(); is not a reliable way to get the size of an input stream. If you check the Javadoc for InputStream.available() you will see that it returns the number of bytes that can be read without blocking and not the stream size.
4) InputStream.read(byte[]) does not guarantee to read all the bytes and returns the number of bytes actually read. This means that your code to read the content of the file into an array may fail. Again check the Javadoc. To be safe you should used DataInputStream.readFully() to read a block of bytes.
5) Reading the whole of the cleartext or ciphertext file into memory does not scale and with very large files you will run out of memory. There is no need to do this since you can use a "read a block, write the transformed block" approach.
RSA is a very very very slow algorithm and it is not normal to encrypt the whole of a file using it. The standard approach is to perform the encryption of the file content using a symmetric algorithm such as AES using a random session key and use RSA to encrypt the session key. One then writes to the ciphertext file the RSA encrypted session key followed by the symmetric encrypted data. To make it more secure one should actually follow the extended procedure outlined in section 13.6 of Practical Cryptography by Ferguson and Schneier.