IOS Mobile Device Management - The SCEP server returned an invalid response

Question

I am in the process of writing an open source iOS mobile device management module in Java. For this I am referring the Apple provided Ruby code at [1]. I have set this up and it works fine for me. Now I need to convert this code to Java. So far I have accomplished to do that up to PKIOperation. In the PKI operation I get "The SCEP server returned an invalid response" which I believe is due to wrong response I sent to device upon PKIOperation.
However when I do search on the internet I get this is something to do with the "maxHttpHeaderSize" as I am using the server as Apache Tomcat. Although I increase that since still it does not get resolved.
Here is the code I need to convert - taken from Apple provided Ruby script
if query['operation'] == "PKIOperation"
    p7sign = OpenSSL::PKCS7::PKCS7.new(req.body)
    store = OpenSSL::X509::Store.new
    p7sign.verify(nil, store, nil, OpenSSL::PKCS7::NOVERIFY)
    signers = p7sign.signers
    p7enc = OpenSSL::PKCS7::PKCS7.new(p7sign.data)
    csr = p7enc.decrypt(@@ra_key, @@ra_cert)
    cert = issueCert(csr, 1)
    degenerate_pkcs7 = OpenSSL::PKCS7::PKCS7.new()
    degenerate_pkcs7.type="signed"
    degenerate_pkcs7.certificates=[cert]
    enc_cert = OpenSSL::PKCS7.encrypt(p7sign.certificates, degenerate_pkcs7.to_der,
        OpenSSL::Cipher::Cipher::new("des-ede3-cbc"), OpenSSL::PKCS7::BINARY)
    reply = OpenSSL::PKCS7.sign(@@ra_cert, @@ra_key, enc_cert.to_der, [], OpenSSL::PKCS7::BINARY)
    res['Content-Type'] = "application/x-pki-message"
    res.body = reply.to_der
end
So this is how I written this in Java using Bouncycastle library.
X509Certificate generatedCertificate = generateCertificateFromCSR(
                privateKeyCA, certRequest, certCA.getIssuerX500Principal()
                        .getName());
        CMSTypedData msg = new CMSProcessableByteArray(
                generatedCertificate.getEncoded());
        CMSEnvelopedDataGenerator edGen = new CMSEnvelopedDataGenerator();
        edGen.addRecipientInfoGenerator(new JceKeyTransRecipientInfoGenerator(
                receivedCert).setProvider(AppConfigurations.PROVIDER));
        CMSEnvelopedData envelopedData = edGen
                .generate(
                        msg,
                        new JceCMSContentEncryptorBuilder(
                                CMSAlgorithm.DES_EDE3_CBC).setProvider(
                                AppConfigurations.PROVIDER).build());
        CMSSignedDataGenerator gen = new CMSSignedDataGenerator();
        ContentSigner sha1Signer = new JcaContentSignerBuilder(
                AppConfigurations.SIGNATUREALGO).setProvider(
                AppConfigurations.PROVIDER).build(privateKeyRA);
        List<X509Certificate> certList = new ArrayList<X509Certificate>();
        CMSTypedData cmsByteArray = new CMSProcessableByteArray(
                envelopedData.getEncoded());
        certList.add(certRA);
        Store certs = new JcaCertStore(certList);
        gen.addSignerInfoGenerator(new JcaSignerInfoGeneratorBuilder(
                new JcaDigestCalculatorProviderBuilder().setProvider(
                        AppConfigurations.PROVIDER).build()).build(
                sha1Signer, certRA));
        gen.addCertificates(certs);
        CMSSignedData sigData = gen.generate(cmsByteArray, true);
        return sigData.getEncoded();
The returned result here will be output in to the servlet output stream with the content type "application/x-pki-message".
It seems I get the CSR properly and I generate the X509Certificate using following code.
public static X509Certificate generateCertificateFromCSR(
        PrivateKey privateKey, PKCS10CertificationRequest request,
        String issueSubject) throws Exception {
    Calendar targetDate1 = Calendar.getInstance();
    targetDate1.setTime(new Date());
    targetDate1.add(Calendar.DAY_OF_MONTH, -1);
    Calendar targetDate2 = Calendar.getInstance();
    targetDate2.setTime(new Date());
    targetDate2.add(Calendar.YEAR, 2);
    // yesterday
    Date validityBeginDate = targetDate1.getTime();
    // in 2 years
    Date validityEndDate = targetDate2.getTime();
    X509v3CertificateBuilder certGen = new X509v3CertificateBuilder(
            new X500Name(issueSubject), BigInteger.valueOf(System
                    .currentTimeMillis()), validityBeginDate,
            validityEndDate, request.getSubject(),
            request.getSubjectPublicKeyInfo());
    certGen.addExtension(X509Extension.keyUsage, true, new KeyUsage(
            KeyUsage.digitalSignature | KeyUsage.keyEncipherment));
    ContentSigner sigGen = new JcaContentSignerBuilder(
            AppConfigurations.SHA256_RSA).setProvider(
            AppConfigurations.PROVIDER).build(privateKey);
    X509Certificate issuedCert = new JcaX509CertificateConverter()
            .setProvider(AppConfigurations.PROVIDER).getCertificate(
                    certGen.build(sigGen));
    return issuedCert;
The generated certificate commonn name is,
Common Name: mdm(88094024-2372-4c9f-9c87-fa814011c525)
Issuer: mycompany Root CA (93a7d1a0-130b-42b8-bbd6-728f7c1837cf), None
[1] - https://developer.apple.com/library/ios/documentation/NetworkingInternet/Concept ual/iPhoneOTAConfiguration/Introduction/Introduction.html

Answer 1

I am in the process of writing an open source iOS mobile device management module in Java. For this I am referring the Apple provided Ruby code at [1]. I have set this up and it works fine for me. Now I need to convert this code to Java. So far I have accomplished to do that up to PKIOperation. In the PKI operation I get "The SCEP server returned an invalid response" which I believe is due to wrong response I sent to device upon PKIOperation.
However when I do search on the internet I get this is something to do with the "maxHttpHeaderSize" as I am using the server as Apache Tomcat. Although I increase that since still it does not get resolved.
Here is the code I need to convert - taken from Apple provided Ruby script
if query['operation'] == "PKIOperation"
    p7sign = OpenSSL::PKCS7::PKCS7.new(req.body)
    store = OpenSSL::X509::Store.new
    p7sign.verify(nil, store, nil, OpenSSL::PKCS7::NOVERIFY)
    signers = p7sign.signers
    p7enc = OpenSSL::PKCS7::PKCS7.new(p7sign.data)
    csr = p7enc.decrypt(@@ra_key, @@ra_cert)
    cert = issueCert(csr, 1)
    degenerate_pkcs7 = OpenSSL::PKCS7::PKCS7.new()
    degenerate_pkcs7.type="signed"
    degenerate_pkcs7.certificates=[cert]
    enc_cert = OpenSSL::PKCS7.encrypt(p7sign.certificates, degenerate_pkcs7.to_der,
        OpenSSL::Cipher::Cipher::new("des-ede3-cbc"), OpenSSL::PKCS7::BINARY)
    reply = OpenSSL::PKCS7.sign(@@ra_cert, @@ra_key, enc_cert.to_der, [], OpenSSL::PKCS7::BINARY)
    res['Content-Type'] = "application/x-pki-message"
    res.body = reply.to_der
end
So this is how I written this in Java using Bouncycastle library.
X509Certificate generatedCertificate = generateCertificateFromCSR(
                privateKeyCA, certRequest, certCA.getIssuerX500Principal()
                        .getName());
        CMSTypedData msg = new CMSProcessableByteArray(
                generatedCertificate.getEncoded());
        CMSEnvelopedDataGenerator edGen = new CMSEnvelopedDataGenerator();
        edGen.addRecipientInfoGenerator(new JceKeyTransRecipientInfoGenerator(
                receivedCert).setProvider(AppConfigurations.PROVIDER));
        CMSEnvelopedData envelopedData = edGen
                .generate(
                        msg,
                        new JceCMSContentEncryptorBuilder(
                                CMSAlgorithm.DES_EDE3_CBC).setProvider(
                                AppConfigurations.PROVIDER).build());
        CMSSignedDataGenerator gen = new CMSSignedDataGenerator();
        ContentSigner sha1Signer = new JcaContentSignerBuilder(
                AppConfigurations.SIGNATUREALGO).setProvider(
                AppConfigurations.PROVIDER).build(privateKeyRA);
        List<X509Certificate> certList = new ArrayList<X509Certificate>();
        CMSTypedData cmsByteArray = new CMSProcessableByteArray(
                envelopedData.getEncoded());
        certList.add(certRA);
        Store certs = new JcaCertStore(certList);
        gen.addSignerInfoGenerator(new JcaSignerInfoGeneratorBuilder(
                new JcaDigestCalculatorProviderBuilder().setProvider(
                        AppConfigurations.PROVIDER).build()).build(
                sha1Signer, certRA));
        gen.addCertificates(certs);
        CMSSignedData sigData = gen.generate(cmsByteArray, true);
        return sigData.getEncoded();
The returned result here will be output in to the servlet output stream with the content type "application/x-pki-message".
It seems I get the CSR properly and I generate the X509Certificate using following code.
public static X509Certificate generateCertificateFromCSR(
        PrivateKey privateKey, PKCS10CertificationRequest request,
        String issueSubject) throws Exception {
    Calendar targetDate1 = Calendar.getInstance();
    targetDate1.setTime(new Date());
    targetDate1.add(Calendar.DAY_OF_MONTH, -1);
    Calendar targetDate2 = Calendar.getInstance();
    targetDate2.setTime(new Date());
    targetDate2.add(Calendar.YEAR, 2);
    // yesterday
    Date validityBeginDate = targetDate1.getTime();
    // in 2 years
    Date validityEndDate = targetDate2.getTime();
    X509v3CertificateBuilder certGen = new X509v3CertificateBuilder(
            new X500Name(issueSubject), BigInteger.valueOf(System
                    .currentTimeMillis()), validityBeginDate,
            validityEndDate, request.getSubject(),
            request.getSubjectPublicKeyInfo());
    certGen.addExtension(X509Extension.keyUsage, true, new KeyUsage(
            KeyUsage.digitalSignature | KeyUsage.keyEncipherment));
    ContentSigner sigGen = new JcaContentSignerBuilder(
            AppConfigurations.SHA256_RSA).setProvider(
            AppConfigurations.PROVIDER).build(privateKey);
    X509Certificate issuedCert = new JcaX509CertificateConverter()
            .setProvider(AppConfigurations.PROVIDER).getCertificate(
                    certGen.build(sigGen));
    return issuedCert;
The generated certificate commonn name is,
Common Name: mdm(88094024-2372-4c9f-9c87-fa814011c525)
Issuer: mycompany Root CA (93a7d1a0-130b-42b8-bbd6-728f7c1837cf), None
[1] - https://developer.apple.com/library/ios/documentation/NetworkingInternet/Concept ual/iPhoneOTAConfiguration/Introduction/Introduction.html