General Protection Fault in module WIN87EM.DLL while Running Windows XP Mode

Question

I am attempting to run a legacy, 16-bit program, P3, on a Windows XP Mode Virtual Machine, hosted by a Windows 7 x64 box. When performing various activities in the program, I receive "P3 caused a General Protection Fault in module WIN87EM.DLL
at 0001:02C6." This occurs on multiple platforms (Core i5, Core 2 Duo).
Based on my research, I have found "The Windows 80x87 emulator library,
WIN87EM.DLL, works at the 16-bit-Windows level
to virtualize the coprocessor among multiple Windows-based applications that run inside the system VM."
It appears to me that, with the Virtual Machine using a Virtualized Processor (not related to the above reference to "virtualized coprocessor") as opposed to an emulated processor, the Virtual Math Coprocessor Device (VMCPD) and/or WIN87EM.DLL
generate an error from the virtualized processor (perhaps they do no recognize it?)
In addition "When the kernel loads an application, it checks to see if floating-point hardware is present. If a coprocessor is not present, the kernel uses the
fixup records to replace the actual floating-point instructions with
Interrupt 3x calls to emulation code in WIN87EM.DLL."
I have found some information that points to "hiding" the math coprocessor from the Virtual Machine, so it doesn't go to the processor. This apparently can be achieved by using a program called WinFloat which includes a tool called HIDE87, which is suppose
to hide the math coprocessor from the kernal.
The issue I'm running into is, to properly achieve this, HIDE87 must be loaded before Windows. I have tried a couple techniques, but I think I'm doing it wrong. I have tried adding the HIDE87 program to the System32 directory and then adding
HIDE87 (just before autocheck autochk *) to BootExecute under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
key.
For a little more info, when you run HIDE87 from the command prompt it reads "HIDE87: Int 11h hook installed (must reboot to remove)." The documentation says this is what needs to occur before Windows loads. You can find WinFloat here:
http://support.microsoft.com/kb/97265.
Any help or insight would be greatly appreciated.

Answer 1

This link helped point me to winfloat to solve the very same issue with WIN87EM.dll crashing under Windows XP on VMware 5 for an old legacy Windows 3.1.  I thought I would mention that the original approach taken by tstock_sjc should work fine.   First
off, winfloat is a self extracting executable and we really don't want to run winfloat, but rather hide87.com which is a tiny TSR that is inside the winfloat archive.  When you run any DOS or Windows 3.1 executable, Windows XP first boots up (if one isn't
already running) a mini-MSDOS 5 environment called NTVDM.exe.   Just as with the original MSDOS, when the shell loads, it loads files similar to config.sys and autoexec.bat.   Of course, there are more restrictions on drivers and not everything
under MSDOS is emulated but nevertheless, it is a fairly good approximation of an MSDOS 5 environment.  The issue that tstock_sjc had was NTVDM.exe looks for %WINDIR%\system32\autoexec.nt and %WINDIR%\system32\config.nt instead of autoexec.bat
and config.sys.  If you extract hide87.com out of winfloat.exe and place it into the %WINDIR%\system32 directory and add a line like %SystemRoot%\system32\hide87.com to the end of autoexec.nt , NTVDM will indeed load this TSR before launching its emulated
copy of of Windows 3.1 (wowexec.exe).  When the 16 bit application is finally launched after wowexec.exe, the coprocessor is then masked.  It worked fine for me anyway (after applying changes and rebooting).  Under process monitor I could clearly
see NTVDM.exe reading autoexec.nt and hide87.com and the WIN87EM.dll GP fault went away.