Swing app keyboard stops working, mystery ESCAPE keystrokes appear in EDT

Question

Java 6 Swing app. In our development environment, works great. In QA, they use it for a bit, type in a text field, click out to a Windows XP/7 app, click back in the text field, and the keyboard stops accepting keystrokes. The mouse continues to work, and the Swing app continues to paint to the screen.
I hooked up a KeyEventDispatcher to listen to what is going on. I'll post a more verbose log at the end of this post, but the short version is this. When the keyboard hangs, the log shows that 'escape' keys are being sent, though we do not do any keystroke injection in our app, ESCAPE or otherwise. Nothing on the Swing app can be determined visually to have focus.
Just before the app starts hanging, it has a side effect of not being able to be brought into the foreground by clicking on it, if, for example, one was working with Excel or Notepad, then try to click on the JFrame title of the app, or anywhere else on the app frame/internals. Once this condition happens, moving away to another Windows app, then going back to the Swing app, causes the keyboard to stop working and the KeyEventDispatcher to see 'escape' keystrokes being sent out.
Connecting remotely to the app via JVisualVM/JConsole does not show any of the threads hanging/blocked.
Sometimes you can work for hours before seeing this problem, and other times, you can start the app up and it happens right away. Once it happens, sometimes you can't recover, and sometimes you can click on a button on a navigator panel on the left side that displays an info panel on the right side, and you can start typing again in text fields.
Once this problem happens, you can start (or have already running) a completely different Swing app (ex.: StackTrace), and the keyboard will stop working for that app too, even though its running in its own separate VM.
This problem (ALMOST!) always happen when typing in a Swing text field (JTextField, JTextArea, etc.), clicking on and then typing in a Windows text area (Excel cell, Notepad), then clicking back into the Swing app's text field. A few times, we've gotten this to happen by typing in a text field, tabbing to a button, pressing a button, then tabbing/clicking back into the text field, all without leaving the Swing app. But this latter scenario is rare, usually going to/from Swing/Windows XP/7 apps cause the problem to occur more readily.
The QA computers normally use Citrix to connect and run the app, but this also happens if we run the app completely locally. But again, this only happens to some computers, all of the ones in the QA department; the development computers (the app has not been released into production yet) does not see this problem.
I had thought that our problem was this problem (Wrong characters in KeyEvents generated from input of barcode scanner but purposely slowing down the acceptance of KEY_PRESSED and KEY_RELEASED events before allowing them to go on to the Swing app (via a KeyDispatcher) did not solve the problem.
Also, we had thought it might be a Citrix problem and how it (or does it?) hook into the Windows keyboard. The fact that once one Swing app gets into this keyboard doesn't work and escape keys are being sent out by the EDT can affect another Swing app makes me wonder. We're not seeing any VM exceptions either like this (EXCEPTION_ACCESS_VIOLATION - JRE 6.0_23 - Citrix, windows 2003
Been trying to get this one solved for over a week, but with no luck. Any help/advice would be appreciated. Thank you in advance for your time.
P.S. Here's the detailed log info I generated via my KeyEventDispatch listener...
2011-04-01 11:58:17,493 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:36 - KEY1-keystroke [java.awt.DefaultKeyboardFocusManager@377369]: java.awt.event.KeyEvent[KEY_PRESSED,keyCode=27,keyText=Escape,keyChar=Escape,keyLocation=KEY_LOCATION_STANDARD,rawCode=27,primaryLevelUnicode=27,scancode=1] on javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,494 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:42 - KEY2-ActiveWindow: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,496 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:48 - KEY3-CurrentFocusCycleRoot: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,497 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:54 - KEY4-FocusedWindow: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,498 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:60 - KEY5-FocusOwner: javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,499 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:66 - KEY6-PermanentFocusOwner: javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,501 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:74 - KEY7-stacktrace...
com..client.util.MyKeyEventDispatcher$StackTraceGenerationException: This exception was created to generate a stack trace, and can be safely ignored.
     at com..client.util.MyKeyEventDispatcher.dispatchKeyEvent(MyKeyEventDispatcher.java:73)
     at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(Unknown Source)
     at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(Unknown Source)
     at java.awt.DefaultKeyboardFocusManager.dispatchEvent(Unknown Source)
     at java.awt.Component.dispatchEventImpl(Unknown Source)
     at java.awt.Container.dispatchEventImpl(Unknown Source)
     at java.awt.Window.dispatchEventImpl(Unknown Source)
     at java.awt.Component.dispatchEvent(Unknown Source)
     at java.awt.EventQueue.dispatchEvent(Unknown Source)
     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
     at java.awt.EventDispatchThread.run(Unknown Source)
2011-04-01 11:58:17,504 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:36 - KEY1-keystroke [java.awt.DefaultKeyboardFocusManager@377369]: java.awt.event.KeyEvent[KEY_RELEASED,keyCode=27,keyText=Escape,keyChar=Escape,keyLocation=KEY_LOCATION_STANDARD,rawCode=27,primaryLevelUnicode=27,scancode=1] on javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,506 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:42 - KEY2-ActiveWindow: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,507 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:48 - KEY3-CurrentFocusCycleRoot: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,508 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:54 - KEY4-FocusedWindow: javax.swing.JFrame[mainFrame,128,128,1024x768,invalid,layout=java.awt.BorderLayout,title=My - HQ - 17601,resizable,normal,defaultCloseOperation=DO_NOTHING_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,1024x768,invalid,layout=com.ep.skin.lnf.framework.internal.RootPaneUI$MetalRootLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.BorderUIResource@d0e678,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
2011-04-01 11:58:17,509 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:60 - KEY5-FocusOwner: javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,510 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:66 - KEY6-PermanentFocusOwner: javax.swing.JTextField[,320,28,175x18,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=com.ep.skin.lnf.framework.utils.internal.RoundedBorder@c3a7c0,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],disabledTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=0,left=0,bottom=0,right=0],selectedTextColor=sun.swing.PrintColorUIResource[r=51,g=51,b=51],selectionColor=javax.swing.plaf.ColorUIResource[r=184,g=207,b=229],columns=15,columnWidth=11,command=,horizontalAlignment=LEADING]
2011-04-01 11:58:17,512 [AWT-EventQueue-1] DEBUG MyKeyEventDispatcher.dispatchKeyEvent:74 - KEY7-stacktrace...
com..client.util.MyKeyEventDispatcher$StackTraceGenerationException: This exception was created to generate a stack trace, and can be safely ignored.
     at com..client.util.MyKeyEventDispatcher.dispatchKeyEvent(MyKeyEventDispatcher.java:73)
     at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(Unknown Source)
     at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(Unknown Source)
     at java.awt.DefaultKeyboardFocusManager.dispatchEvent(Unknown Source)
     at java.awt.Component.dispatchEventImpl(Unknown Source)
     at java.awt.Container.dispatchEventImpl(Unknown Source)
     at java.awt.Window.dispatchEventImpl(Unknown Source)
     at java.awt.Component.dispatchEvent(Unknown Source)
     at java.awt.EventQueue.dispatchEvent(Unknown Source)
     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
     at java.awt.EventDispatchThread.run(Unknown Source)
The above log info repeats multiple times, KEY_PRESSED and KEY_RELEASE, over and over again, for 'escape' keys.
Edited by: 850693 on Apr 7, 2011 10:16 AM (typo fix)
Edited by: 850693 on Apr 7, 2011 10:19 AM (Fixed links)

Answer 1

<discaimer>Don't put too much hope in my reply</disclaimer>
The only real difference is QA has the Citrix client installed on them, and development does not. You don't need to run our Swing app through a Citrix client though to cause the bug/freezing, just running it on a PC with the Citrix client seems to be enough (in theory). We've been working down a checklist of possible problems/solutions, and we've gotten to the "Install Citrix to the dev PC" item now, so I'll post back if that makes a difference or not in reproducing the problem.
I've also had QA people actually come over to a dev PC, without the Citrix client, and try to reproduce the problem, and they have not been able to. There's 'something' about their environment vs. ours, but not sure how that would manifest itself as a AWT/Swing keyboard event of mysterious escape keystrokes followed by the locking up on the keyboard, but not the whole Swing app. My personal guess is the Citirix client installing funky Windows-level keyboard driver(s), but I may be totally off on that. /shrugI read your initial post twice and couldn't find out whether you reproduce that on several different machines, so one "environmental" difference comes to mind: have you tried using another keyboard on the defective QA configuration?
Of course that doesn't explain in itself how the problem would manifest only after switching back and forth to a native Windows app, but then, with the hint that Citrix may make a difference, maybe one driver filters out repeated "Esc" keystrokes, while another doesn't, and that manifests only after a few app switches, and the system event queue, or whatever it's called in Windows, redirects the events to the target app/window?
Other than that, I wish you had investigated Jeanette's pacemaker hypothesis more... ;)
Otherwise, yeah. I have a hook in to see all AWTEvent's, but I still need to see/find what's posting the mysterious escape keys to the event queue.I assume it's what you mean, but I'll try to make that clear: you have to investigate, at the OS level, what is posting escape key events to the OS queue.
I am not a Windows developper, but I seem to understand that the following programs claims to capture keystrokes on the whole screen (not limited to a single window), with some limitations: http://www.codeproject.com/KB/winsdk/WIN_RECORDER.aspx
More generally, you should search for a Windows way to peek at the Windows' (keyboard?) event queue.
I'm not sure this could identify which DLL (if it's not originated by the material, see the defective keyboard hypothesis) is posting the events, but that's worth a try (Java for example, would enables to stuff a custom event queue that would trace info about a Java-poster).
Even if you can't identify the posting DLL, running the "key captuyre" on both the Windows host that has the Citrix window, and the Windows host that you are accessing via Citrix, may give you hints as to where the heck the key strokes originate...
Good luck, and keep us informed,
J.