AES 256 Revision 6 (PDF 2.0) Encryption

Question

I am trying to implement decryption of AES 256 Revision 6 (PDF 2.0) as described in the ISO 32000-2 spec and having some success but getting some peculiar results that I cannot resolve and would appreciate some suggestions.
Using Acrobat X on a PC and encrypting using password encryption compatible with Acrobat X, I created a set of about 20 Acrobat X encrypted PDF tests. When I ran these through our implementation to decrypt as following ISO 32000-2 particularly Algorithms 2.A and 2.B most decrypted successfully and produced correct output. However a few failed either in the authentication phase or in the intermediate key computation stage, with the latter showing an error by an invalid decryption of the first stream of PDF encountered. Next I tried another set of different tests and also got a similar pass rate. Finally I took one unencrypted PDF test and chose the same security settings of compatible with Acrobat X, restrict editing, and printing, and allowed print and used the same password for 15 generated versions of this PDF test. 13 of these 15 resultant encrypted tests ran successfully with our implementation of the Revision 6 decryption algorithm. Two failed, with one failing a match on both user and owner key and the other failing computing the intermediate owner key.
In the past when we have implemented earlier Revision 5 256 AES, or even older compatibility versions it always was the case that you either had the software working or you didn’t. And the new PDF 2.0 2B algorithm with 64+ hashes and 64+ aes encryptions of data along with 16 byte mod 3 math computations leaves little room for error.
  I believe that Acrobat when encrypting is choosing a random AES IV and all data including input /U /O /UE, and /OE strings differ. Even for the case described above of the same input test, same password, and same Acrobat encryption options. Thus the input into Algorithm 2-B will differ but the output should for authentication match the first 32 bytes of the O or U key or should result in a correct final result for intermediate owner or user key if the corresponding match occurred above.
However for the few exceptions that fail the above decryption it is not easy to determine what went wrong. Just about any change to the implementation of Algorithm 2.B breaks all working test cases instead of giving a clue as to what the issue is. The possible suspects are the new SHA-384 and SHA-512 and the encrypt code. We have used SHA-256 and the AES decrypt portion in earlier implementation of revision 5 and had no problems. The AES and hash code we are using is from Gladman1. I was wondering what others are using? It looks like Acrobat X is using RSA BSAFE Crypto – C2 at least for FIPS. Could Leonard or somebody else at Adobe tell me if this RSA software is also used in general with Acrobat X?
And I think that it would be very beneficial to have and publish a set of test vectors given input into algorithm 2.B along with correct intermediate results for each step. For each hash – including which method used per step show hash results, and also encryption step results, number of steps beyond 64 minimum, as well as final result. For the 80% of tests I have working I could produce this info. For those tests I cannot get working I would need help. Perhaps someone at Adobe or elsewhere who has had greater success than I have can help? I can provide input for the problematic tests either through this forum or privately at [email protected].
1) http://www.gladman.me.uk/
2) http://blogs.adobe.com/security/2011/05/update-fips-validation-certificates-for-acrobat-an d-reader-x.html

Answer 1

I create a simple file called 256encrypt.pdf and encrypted with aes256
I am using "Algorithm 2.B: Computing a hash" from ISO32000-2 to verify the user password
user password: password
User string from the PDF test file : f4 65 f1 69 9a e2 ea 71 ba e7 6b 48 bb 12 8f 1f 18 74 e3 d3 e2 97 7e b8 d6 fe 9f 7f 86 b0 6d 89 c9 38 40 c5 64 dc 5a 32 04 4d 9c 6f 28 d2 98 d0
User string hash value:  f4 65 f1 69 9a e2 ea 71 ba e7 6b 48 bb 12 8f 1f 18 74 e3 d3 e2 97 7e b8 d6 fe 9f 7f 86 b0 6d 89
User Validatiaon salt: c9 38 40 c5 64 dc 5a 32
User Key salt:04 4d 9c 6f 28 d2 98 d0
The input for the "Algorithm 2.B: Computing a hash" is as follows:
user Validation Salt: c9 38 40 c5 64 dc 5a 32
password: 70 61 73 73 77 6f 72 64 (password)
step 1: SHA256(password+user Validation Salt)
the result is  K = 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
step 2: Make a new string K1 with 64 repetitions of the input password and K
K1= 70 61 73 73 77 6f 72 64 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
       70 61 73 73 77 6f 72 64 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
       70 61 73 73 77 6f 72 64 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
        70 61 73 73 77 6f 72 64 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
Total 64 times , total length = 0xa00
step3: Encrypt K1 with AES_128(CBC)
AES_CBC_128_NOPADDING:
Key = 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee
IV =   be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
K1 64 repeat of the 70 61 73 73 77 6f 72 64 9d 47 2d 4e f0 96 cd dd 7a 8c 04 8d b4 d2 b8 ee be fe b9 9f 7f cc e1 29 ea 63 ad f2 a3 d5 11 5b
Result : Total length is 0xa00. The beginning part of the E is
E =
47 df 2a 7f 90 8a c4 d9 f2 8b a0 f1 49 f0 8e 09 51 c4 a3 ce fd 28 48 f3 d7 c1 04 76 1b 6b 5b f2 6d 3d 2c 3f 03 26 76 06 d5 67 44 c8 2a b6
10 02 a5 8d a7 93 4f 94 02 b9 bf 93 b5 2d 17 82 02 3b f7 8e 8a 07 0f 18 ed 19 b3 ba 55 8b 14 b7 45 16 80 47 4f 6e c3 b6 20 d2 72 cd d1 46
2c d3 88 f7 c4 f7 e3 3a 04 3d 72 4f e0 d2 66 63 c4 9c 77 7c c5 53 fd 69 81 f6 3b 3d f5 8e b2 bd 66 4e 0f c6 1e 96 5e 91 e2 3d 60 5c 60 75
a3 13 49 58 85 e8 bb 37 93 91 4c 4f 79 a5 80 f2 13 be 44 22 aa e5 ee 6c 29 2c 76 50 a3 15 85 69 5e e9 c5 29 13 2a f6 67 51 8e 1e 7f 23 8a
90 a7 fe 93 c7 ff 45 ee 2a f0 c0 70 f1 78 2e 80 bd be 06 4f ad 69 4d 47 e6 3f ae e2 6a 76 ef 3e 56 8f 2d f5 c9 49 26 f3 7e 6e 61 8b 5e e6
e6 2d dd 76 cd 30 33 1d fe bf 11 60 ce 33 35 43 da b7 33 9b b9 6a 86 cd 35 a0 ca 84 99 0c ca 71 28 b3 01 b9 23 b4 a0 87 4e fb ff af b6 bd
step4:
The result of the first 16 bytes of E mod 3 is 1
step5:
Using SHA384 to get 48 bytes K
K = 29 de 28 c1 f0 17 c9 37 bd 93 97 e3 b5 51 b0 86 b9 0c 96 e0 77 28 87 1c 11 7b 41 ce 64 bf a8 7f f2 8b a2 7b 52 58 79 a9 63 c0 b2 31 f8 4e e4 6e
This is the end of round 1 and go back to step 2 using this new K
When round is equal or bigger than 64, check E[last byte], if E[last byte] > round -32, go back to step 2
The final round is 69. and the final result is
K = ab 7c c6 03 bc da 85 51 3f 3d 22 fb 58 8c 42 1d 45 67 55 92 9f 4f d2 41 b3 93 07 04 7d b1 30 6d
But this K does not match with the first 32 byte of the user string.