Problem Hosting an ActiveX which starts VB6 Forms in WPF

Question

We need to “host” an older application written in VB6 into a WPF Shell. We need to make sure that this app is part of the new WPF shell.
The older app consists of:
1) ActiveX component that shows a grid with data and some buttons;
2) By selecting a record and double clicking a record a Modeless VB6 form is opened. This form is implemented in an in process vb6 app;
From these modeless form, other modeless forms and modal forms can be started by the user.
Showing these modeless forms that are implemented in the “in process” windowless ActiveX components gives us problems. We have tried out two options to get around the problems. But so far no success.
We have explored two options: 
1) Host ActiveX component directly in WPF, which gives us problems with modeless forms and ‘tabbing’ though the controls of the form;
2) Start vb6 exe, host the ActiveX component there and then host the output of that VB6 exe in WPF: we now face some challenges with the communication between form and WPF shell;   
In both cases we face some challenges with regards to the keyboard navigation. We want an have expert opinion on this matter. Does anyone have some tips on this topic? 

Answer 1

Thanks. 
Let me try to describe the problem in some more detail. I have build a solution, that is a very simple representation of the actual problem we are facing. But without our own system layer and controls
(etc).
The example solution consists of:
VB6 ActiveX dll, representing one of our maintance apps: containing a class and a form. The class creates instances of the form and shows the form;
VB6 control (ocx), representing our custom control, which instantiates the class;
VB6 exe, representing our 'old Shell', which hosts the VB 6 control;
WPF exe, representing our 'new Shell', which hosts the VB 6 control.
In the 'new shell' we have hosted an vb6 control (ocx), by using the WindowsFormsHost object. This is done in a way described in the "Walk through" (http://msdn.microsoft.com/en-us/library/ms742735.aspx
The VB6 control consist of two buttons: 'show modeless form' and 'show modal form'. The buttons create an instance of the class and instructs the class to either start the form modal or modeless,
by calling public methods of the class.
When starting the VB 6 app, the user can start new modeless forms and modal forms without any problems. This is how our current (VB6) Shell works.
Now we want to move our solution to a newer platform, but gradual, no big bang conversion: the real system is large and complex. In the first phase we don’t want the touch the legacy apps.
We created a WPF exe that hosts the VB6 control. The user can start the modal form without any problems. However clicking the 'show modeless button' raises an error:
Run-time error: '406' Non-Modal Forms cannot be displayed in this host application from an ActiveX DLL, ActiveX control or property page
We figured out that we 
have a fundamental problem here.
(some related docs we have consulted
http://www.kbalertz.com/247791/Modeless-Forms-ActiveX-Display-Clients.aspx ,
http://support.microsoft.com/default.aspx?scid=kb;EN-US;171978 ,
http://www.desaware.com/support/faq/spyworks/swfaqthreading.aspx )
Instead of doing the frm.Show vbModeless, we now call the ShowWindow() api. This shows the user the modeless form, with some side effects like a 'tab key' that doesn't work as expected (and probably
some other effects we didn't find out yet).
From the user perspective, only showing modal forms is not an option (as is suggested in 
http://support.microsoft.com/default.aspx?scid=kb;EN-US;171978)
We are now looking 
into hosting the VB 6 control in a VB6 exe and then host the VB6 exe in WPF. 
We then face some challenges in the communication (and navigation) between the two applications.
I have looked also into this document 
http://blogs.msdn.com/b/changov/archive/2009/10/26/hosting-wpf-ui-cross-thread-and-cross-process.aspx (and related docs) as well, but
the same issues persist here as described.
I hope this narrows down the problem a bit more. Do you have any other leads or paths we can follow?
I can send you the simple example I have created that reproduces our problem.
Sincerely,
Bart