Windows Server 2003 (sp2 - 32bit) - OEM problem

Question

I inherited this in-house (closed) system a few years ago and have not had a problem until a couple of months ago.  It has been running for about 1 1/2 years.
Desc:   Oracle 11.1.0.6 (64bit) on Windows Server 2003 (sp2 - 32 bit)   [[ yes, I know, what the heck is a 64 bit Oracle running on a 32 bit windows?]]
     2 - instances running -  one production with associated ASM,  one RMAN cat with no ASM -  all folders are backed up to a large SANS.
          -  each instance has its own listener
          -  each instance has its own dbconsole service
Problem:
1.  a couple of months ago, when I attempted to access the production OEM, I started seeing
service-specific error 1 (0x1), NET HELPMSG 3547 
     -- the first time this happened, I immediately opened the OEM for rman and it was successful
          It was also fortunate in that we were doing some power upgrades and all of our systems were shutdown - and after this system was powered on, the production OEM worked fine.
     -- a few weeks later, the prod OEM was not accessible - giving the same 3547 error.  We were not able to shut down, so I stopped the Windows services for this instance, and stopped
               the windows listener services for both instances.  I then started only the production windows services, and was able to use the prod OEM.  Then I started the listener service for
               the rman and all was well.
     -- this Monday (today is Thurs), the prod OEM was inaccessible again and I tried the stopping and restarting of the services and started seeing 7024 errors.  I tested the rman OEM and
          it was fine.  I stopped both instances via sqlplus commands, used emctl to stop both dbconsoles (of course, the prod one was already not running), shut down the ASM instance and
          then stopped all the Windows services.  then I only brought up the prod services and confirmed with sqlplus commands that everything was started - but the final service, dbconsole
          would not start.  This morning I was able to reboot the server (after shutting down all the Oracle services) and now I can access the prod and the rman OEMs.
2.  What the heck?   Because I can get the prod OEM working with just a reboot, I don't know if I should have to rebuild dbconsole.   And since I'm the only one disuncombullated because
          of the prod OEM, it is not a fall-down-crying situation - I'm actually a sqlplus kinda guy, but when I'm just doing visual maintenance, it is nice to view OEM remotely for a quick check.
3.  I've been working with Linux systems and Oracle for many years and I've never experienced this kind of problem.   Oh, and since it is a closed system, it will be when pigs sprout wings
               and start flyng that I'll be allowed to install patches - but again, if a reboot 'fixes' this issue, what the heck?  Oh, finally, I don't believe in the common windows fix of rebooting - I'd
               like to find out WHY this is happening and literally fix it.  
ANYBODY have a clue?   Not me...
Thanks for your time.

Answer 1

user12183171 wrote:
I inherited this in-house (closed) system a few years ago and have not had a problem until a couple of months ago.  It has been running for about 1 1/2 years.
Desc:   Oracle 11.1.0.6 (64bit) on Windows Server 2003 (sp2 - 32 bit)   [[ yes, I know, what the heck is a 64 bit Oracle running on a 32 bit windows?]]
You'd better double check that.  It is fundamentally impossible for 64-bit software to run on a 32-bit OS.
     2 - instances running -  one production with associated ASM,  one RMAN cat with no ASM -  all folders are backed up to a large SANS.
          -  each instance has its own listener
One single listener, with the default name of LISTENER, using the default port of 1521, is quite capable of -- indeed, was designed to -- service multiple instances of multiple versions running from multiple homes.
          -  each instance has its own dbconsole service
Problem:
1.  a couple of months ago, when I attempted to access the production OEM, I started seeing
service-specific error 1 (0x1), NET HELPMSG 3547 
     -- the first time this happened, I immediately opened the OEM for rman and it was successful
          It was also fortunate in that we were doing some power upgrades and all of our systems were shutdown - and after this system was powered on, the production OEM worked fine.
     -- a few weeks later, the prod OEM was not accessible - giving the same 3547 error.  We were not able to shut down, so I stopped the Windows services for this instance, and stopped
               the windows listener services for both instances.  I then started only the production windows services, and was able to use the prod OEM.  Then I started the listener service for
               the rman and all was well.
     -- this Monday (today is Thurs), the prod OEM was inaccessible again and I tried the stopping and restarting of the services and started seeing 7024 errors.  I tested the rman OEM and
          it was fine.  I stopped both instances via sqlplus commands, used emctl to stop both dbconsoles (of course, the prod one was already not running), shut down the ASM instance and
          then stopped all the Windows services.  then I only brought up the prod services and confirmed with sqlplus commands that everything was started - but the final service, dbconsole
          would not start.  This morning I was able to reboot the server (after shutting down all the Oracle services) and now I can access the prod and the rman OEMs.
2.  What the heck?   Because I can get the prod OEM working with just a reboot, I don't know if I should have to rebuild dbconsole.   And since I'm the only one disuncombullated because
          of the prod OEM, it is not a fall-down-crying situation - I'm actually a sqlplus kinda guy, but when I'm just doing visual maintenance, it is nice to view OEM remotely for a quick check.
3.  I've been working with Linux systems and Oracle for many years and I've never experienced this kind of problem.   Oh, and since it is a closed system, it will be when pigs sprout wings
               and start flyng that I'll be allowed to install patches - but again, if a reboot 'fixes' this issue, what the heck?  Oh, finally, I don't believe in the common windows fix of rebooting - I'd
               like to find out WHY this is happening and literally fix it.  
ANYBODY have a clue?   Not me...
Thanks for your time.