Someone please review...Is this a bug in SSLEngine?

Question

I have been dealing with this for days and finally wrote a JUnit test just against the SSLEngine itself. I am about to file a bug report, but can someone please verify that they are having the same problem.
Basically, on a rehanshake, just after a Runnable is retrieved and before it is run, the SSLEngine will not unwrap data. Here is an excerpt from the unit test....
          result = server.unwrap(encPacket, unencrPacket);
          assertEquals(HandshakeStatus.NEED_TASK, result.getHandshakeStatus());
          assertEquals(Status.OK, result.getStatus());     
          r = serverEngine.getDelegatedTask(); //get task but don't run it yet...wait until after decrypt of real data
          ByteBuffer dataOut = ByteBuffer.allocate(server.getSession().getApplicationBufferSize());
          dataOut.clear();
          helper.doneFillingBuffer(encData);
          log.fine("datain1="+encData+" out="+dataOut);
          result = serverEngine.unwrap(encData, dataOut);
          log.fine("datain2="+encData+" out="+dataOut);
          assertEquals(HandshakeStatus.NEED_TASK, result.getHandshakeStatus());
          assertEquals(Status.OK, result.getStatus());
FINE: datain1=java.nio.HeapByteBuffer[pos=0 lim=37 cap=16665] out=java.nio.HeapByteBuffer[pos=0 lim=16384 cap=16384]
Jul 4, 2005 3:45:50 PM biz.xsoftware.impl.nio.secure.test.TestNewAsynchSSLEngine testRawSSLEngine
FINE: datain2=java.nio.HeapByteBuffer[pos=0 lim=37 cap=16665] out=java.nio.HeapByteBuffer[pos=0 lim=16384 cap=16384]Notice the status of the engine is still status=OK and handshake=NEED_TASK after trying to read the data that came in after the Runnable was retrieved. The log statements log that the encData was not even read from and the dataOut was not read to. Here are the logs proving that encData and dataOut wasn't touched at all.......
FINE: datain1=java.nio.HeapByteBuffer[pos=0 lim=37 cap=16665] out=java.nio.HeapByteBuffer[pos=0 lim=16384 cap=16384]
Jul 4, 2005 3:45:50 PM biz.xsoftware.impl.nio.secure.test.TestNewAsynchSSLEngine testRawSSLEngine
FINE: datain2=java.nio.HeapByteBuffer[pos=0 lim=37 cap=16665] out=java.nio.HeapByteBuffer[pos=0 lim=16384 cap=16384]This may be "by design" such that data backs up until the Runnable is run which in myopinion is slightly dangerous.
Full code of JUnit test(not real implementation code, just reproduction of problem with SSLEngine unit test) can be found at
http://forum.java.sun.com/thread.jspa?threadID=641529&tstart=0
thanks,
dean

Answer 1

actually, I was thinking the status after trying to
unwrap my application data would have been...
handShakeState = NEED_TASK (since Runnable hadn't
been run yet).
status = BUFFER_UNDERFLOW or OK(depending on if the
application data had been decrypted or the engine
needed more data to decrypt the packet)
It seemed they had enough status to handle this
situation to me but I could be missing something else.You are right about this: I was thinking more about the internal SSLEngine state-machine states, if it really is a state machine. So, I have now had a look and there is an internal RENEGOTIATE state which is supposed to allow concurrent data and re-handshake, and from a very quick look at Sun's code this should work for both input and output of app data during renegotiation, so I am going to have another look at your code ...
One thing, the engine won't receive data in between the 2nd-last and the last WRAP of a re-handshake: this would be a TLS protocol violation (the Finished message must immediately follow the change_cipher_spec message).
I also found the following comment in the output code which I will risk Sun's wrath by quoting:
     * If we have a task outstanding, this MUST be done before
     * doing any more wrapping, because we could be in the middle
     * of receiving a handshake message, for example, a finished
     * message which would change the ciphers.
     */