Very slow retrieval of messages with IMAP

Question

I have a problem with javamail versions greater than 1.4.4:
I use IMAP and STARTTLS to connect to Exchange Server 2010
The mail I try to read in this example has RFC822.SIZE = 2317220 but the problem occurs with all mails.
This is the code where it happens ( line 3 )
msg = (MimeMessage)currentFolder.getMessage(1);
log.debug(mailBoxID +" start reading mail");
fullMessage = new MimeMessage(msg);
log.debug(mailBoxID +"   end reading mail");
With 1.4.5 and higher it takes approx. 8 minutes ( in debug mode ) to get the full mail.
With 1.4.4 it takes only 4 seconds ( also in debug mode ).
I tried 3 scenario's with 1.5.1: ( but I have the same problem with 1.4.5 and 1.4.6 )
mail.imap.fetchsize: 16384 ==> 8 minutes
mail.imap.fetchsize: 102400 ==> 7 minutes
mail.imap.partialfetch: false ==> 9 minutes
I tried 2 scenario's with 1.4.4:
mail.imap.fetchsize: 16384 ==> 4 seconds
mail.imap.partialfetch: false ==> 4 seconds
I suppose fetchsize and partialfetch are not part of the problem.
There must be some configuration issue but I don't find it.
And I have the same problem with the demo programs.
Here are some snippets from the protocol debug and the program debug for all scenario's:
DEBUG: setDebug: JavaMail version 1.5.1
DEBUG: getProvider() returning javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle]
DEBUG IMAP: mail.imap.fetchsize: 16384
DEBUG IMAP: mail.imap.ignorebodystructuresize: false
DEBUG IMAP: mail.imap.statuscachetimeout: 1000
DEBUG IMAP: mail.imap.appendbuffersize: -1
DEBUG IMAP: mail.imap.minidletime: 10
DEBUG IMAP: disable AUTH=PLAIN
DEBUG IMAP: disable AUTH=NTLM
DEBUG IMAP: enable STARTTLS
==> fetches in blocks of 16384
A10 FETCH 1 (BODY[]<0.16384>)
* 1 FETCH (BODY[]<0> {16384}
FLAGS (\Seen))
A11 OK FETCH completed.
A12 FETCH 1 (BODY[]<32768.16384>)
* 1 FETCH (BODY[]<32768> {16384}
FLAGS (\Seen))
A202 OK FETCH completed.
A203 FETCH 1 (BODY[]<3162112.16384>)
* 1 FETCH (BODY[]<3162112> {11800}
FLAGS (\Seen))
A203 OK FETCH completed.
2013-11-19 13:10:31 [Thread-4] DEBUG MailHandler :67 -  start reading mail
2013-11-19 13:18:31 [Thread-4] DEBUG MailHandler :69 -    end reading mail
==> 8 minutes
DEBUG: setDebug: JavaMail version 1.5.1
DEBUG: getProvider() returning javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle]
DEBUG IMAP: mail.imap.fetchsize: 102400
DEBUG IMAP: mail.imap.ignorebodystructuresize: false
DEBUG IMAP: mail.imap.statuscachetimeout: 1000
DEBUG IMAP: mail.imap.appendbuffersize: -1
DEBUG IMAP: mail.imap.minidletime: 10
DEBUG IMAP: disable AUTH=PLAIN
DEBUG IMAP: disable AUTH=NTLM
DEBUG IMAP: enable STARTTLS
==> fetches in blocks of 102400
FLAGS (\Seen))
A10 OK FETCH completed.
A11 FETCH 1 (BODY[]<102400.102400>)
* 1 FETCH (BODY[]<102400> {102400}
FLAGS (\Seen))
A39 OK FETCH completed.
A40 FETCH 1 (BODY[]<3072000.102400>)
* 1 FETCH (BODY[]<3072000> {101912}
FLAGS (\Seen))
A40 OK FETCH completed.
2013-11-19 14:23:42 [Thread-4] DEBUG MailHandler :67 -  start reading mail
2013-11-19 14:30:30 [Thread-4] DEBUG MailHandler :69 -    end reading mail
==> 7 minutes
DEBUG: setDebug: JavaMail version 1.5.1
DEBUG: getProvider() returning javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle]
DEBUG IMAP: mail.imap.partialfetch: false
DEBUG IMAP: mail.imap.ignorebodystructuresize: false
DEBUG IMAP: mail.imap.statuscachetimeout: 1000
DEBUG IMAP: mail.imap.appendbuffersize: -1
DEBUG IMAP: mail.imap.minidletime: 10
DEBUG IMAP: disable AUTH=PLAIN
DEBUG IMAP: disable AUTH=NTLM
DEBUG IMAP: enable STARTTLS
==> 1 big fetch
2013-11-19 13:21:35 [Thread-4] DEBUG MailHandler :67 -  start reading mail
2013-11-19 13:30:47 [Thread-4] DEBUG MailHandler :69 -    end reading mail
==> 9 minutes
DEBUG: setDebug: JavaMail version 1.4.4
DEBUG: getProvider() returning javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Sun Microsystems, Inc]
DEBUG: mail.imap.fetchsize: 16384
DEBUG: mail.imap.statuscachetimeout: 1000
DEBUG: mail.imap.appendbuffersize: -1
DEBUG: mail.imap.minidletime: 10
DEBUG: disable AUTH=PLAIN
DEBUG: disable AUTH=NTLM
DEBUG: enable STARTTLS
==> 1 big fetch
2013-11-19 13:55:47 [Thread-4] DEBUG MailHandler :67 -  start reading mail
2013-11-19 13:55:51 [Thread-4] DEBUG MailHandler :69 -    end reading mail
==> 4 seconds
DEBUG: setDebug: JavaMail version 1.4.4
DEBUG: getProvider() returning javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Sun Microsystems, Inc]
DEBUG: mail.imap.partialfetch: false
DEBUG: mail.imap.statuscachetimeout: 1000
DEBUG: mail.imap.appendbuffersize: -1
DEBUG: mail.imap.minidletime: 10
DEBUG: disable AUTH=PLAIN
DEBUG: disable AUTH=NTLM
DEBUG: enable STARTTLS
==> 1 big fetch
2013-11-19 14:02:52 [Thread-4] DEBUG MailHandler :67 -  start reading mail
2013-11-19 14:02:58 [Thread-4] DEBUG MailHandler :69 -    end reading mail
==> 4 seconds
Here is a listing of all properties:
java.runtime.name=Java(TM) SE Runtime Environment
sun.boot.library.path=H:\java\jre6\bin
java.vm.version=20.5-b03
java.vm.vendor=Sun Microsystems Inc.
java.vendor.url=http://java.sun.com/
path.separator=;
mail.mime.decodefilename=true
java.vm.name=Java HotSpot(TM) Client VM
file.encoding.pkg=sun.io
user.country=BE
sun.java.launcher=SUN_STANDARD
sun.os.patch.level=Service Pack 2
mail.imap.auth.ntlm.disable=true
java.vm.specification.name=Java Virtual Machine Specification
user.dir=H:\EclipseSpace\DigisMailBatchNewDev
java.runtime.version=1.6.0_30-b12
java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
mail.imap.fetchsize=102400
java.endorsed.dirs=H:\java\jre6\lib\endorsed
os.arch=x86
java.io.tmpdir=D:\Temp\
line.separator=
java.vm.specification.vendor=Sun Microsystems Inc.
user.variant=
os.name=Windows XP
sun.jnu.encoding=Cp1252
java.library.path=H:\java\jre6\bin;C:\WINDOWS\Sun\Java\...
mail.imap.auth.plain.disable=true
java.specification.name=Java Platform API Specification
java.class.version=50.0
mail.mime.address.strict=false
sun.management.compiler=HotSpot Client Compiler
os.version=5.1
user.home=C:\Documents and Settings\******
user.timezone=Europe/Paris
java.awt.printerjob=sun.awt.windows.WPrinterJob
java.specification.version=1.6
file.encoding=Cp1252
user.name=******
java.class.path=H:\EclipseSpace\DigisMailBatchNewDev;...
mail.mime.decodetext.strict=false
java.vm.specification.version=1.0
sun.arch.data.model=32
java.home=H:\java\jre6
sun.java.command=com.dexia.digis.mail.Launcher
mail.imap.partialfetch=true
java.specification.vendor=Sun Microsystems Inc.
user.language=nl
awt.toolkit=sun.awt.windows.WToolkit
java.vm.info=mixed mode
java.version=1.6.0_30
java.ext.dirs=H:\java\jre6\lib\ext;C:\WINDOWS\Sun\J...
sun.boot.class.path=H:\java\jre6\lib\resources.jar;H:\jav...
java.vendor=Sun Microsystems Inc.
file.separator=\
java.vendor.url.bug=http://java.sun.com/cgi-bin/bugreport...
mail.imap.starttls.enable=true
sun.cpu.endian=little
sun.io.unicode.encoding=UnicodeLittle
mail.mime.parameters.strict=false
sun.desktop=windows
sun.cpu.isalist=
And here is the complete protocol stacktrace up to the point where the mail is retrieved.
It's the same for all cases except the snippets above:
DEBUG IMAP: trying to connect to host "our-email-server.be", port 143, isSSL false
* OK The Microsoft Exchange IMAP4 service is ready.
A0 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI AUTH=PLAIN STARTTLS UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
A0 OK CAPABILITY completed.
DEBUG IMAP: AUTH: NTLM
DEBUG IMAP: AUTH: GSSAPI
DEBUG IMAP: AUTH: PLAIN
DEBUG IMAP: protocolConnect login, host=our-email-server.be, user=mydomain\userid\aliasname, password=<non-null>
A1 STARTTLS
A1 OK Begin TLS negotiation now.
A2 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI AUTH=PLAIN UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
A2 OK CAPABILITY completed.
DEBUG IMAP: AUTH: NTLM
DEBUG IMAP: AUTH: GSSAPI
DEBUG IMAP: AUTH: PLAIN
DEBUG IMAP: LOGIN command trace suppressed
DEBUG IMAP: LOGIN command result: A3 OK LOGIN completed.
A4 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 AUTH=NTLM AUTH=GSSAPI AUTH=PLAIN UIDPLUS CHILDREN IDLE NAMESPACE LITERAL+
A4 OK CAPABILITY completed.
DEBUG IMAP: AUTH: NTLM
DEBUG IMAP: AUTH: GSSAPI
DEBUG IMAP: AUTH: PLAIN
A5 LIST "" myfolder
* LIST (\HasNoChildren) "/" myfolder
A5 OK LIST completed.
DEBUG IMAP: connection available -- size: 1
A6 SELECT myfolder
* 1 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
* OK [UIDVALIDITY 39147] UIDVALIDITY value
* OK [UIDNEXT 141] The next unique identifier value
A6 OK [READ-WRITE] SELECT completed.
A7 EXPUNGE
* 1 EXISTS
A7 OK EXPUNGE completed.
A8 FETCH 1 (FLAGS)
* 1 FETCH (FLAGS (\Seen))
A8 OK FETCH completed.
A9 FETCH 1 (ENVELOPE INTERNALDATE RFC822.SIZE)
* 1 FETCH (ENVELOPE ("Fri, 9 Aug 2013 14:17:14 +0200" "subject" ........ INTERNALDATE "09-Aug-2013 14:17:14 +0200" RFC822.SIZE 2317220)
A9 OK FETCH completed.
A10 FETCH 1 (BODY[]<0.102400>)
* 1 FETCH (BODY[]<0> {102400}

Answer 1

The MimeMessage constructor you're using needs to copy the entire message from the server.  It uses the Message.writeTo method, writing the message content into a pipe that is read by the constructor.  It was a bug in previous releases that the writeTo method wasn't fetching the data in chunks, as it does in other cases.  Setting partialfetch=false will revert to the old behavior for writeTo, and for all other accesses of the message content.
It looks like both 1.4.4 and 1.5.1 are fetching the message content using a single FETCH command when you set partialfetch=false, according to your message.  If they're both sending the same commands to the server, I can't explain why it would be fast in one case and slow in the other.
I did similar testing using my Exchange 2010 server.  Fetching a similar size message took less than 20 seconds.  Setting partialfetch=false brought it down to less than 5 seconds.  Something's clearly wrong in your environment, but I don't know what.
Finally, I'm sure you understand that you only need to use that MimeMessage copy constructor in special circumstances, and you should probably avoid using it unless it's absolutely necessary.  Even in the best case it's wasting time and memory.