Managed server independence mode startup : VERY slow (2h~)

Hello there,
I'm posting this thread to ask for some help.
In my production environnement, where I am running a cluster of several ALSB nodes, a problem showed up when a network card was unplugged on our admin server hardware.
You have to know that for some reason, we are not using a node manager for our admin operations on the managed servers, but rather scripts that are scheduled by a tool we use for several other things (backup,etc..). This tool runs WLST scripts to do some basic stuff on our servers.
So, during the shutdown/startup process, when we tried to bring back all the managed servers up, we faced this problem : When the managed servers were not able to contact the admin server, the startup process takes approx 2 hours, but only 10min when admin server is up and running. This is a real problem in our high availibility environnement, if we have to face this "no admin server" situation again.
My environnement is :
Weblogic 10.0
Solaris 10.6.0
Java HotSpot(TM) Server VM Version 1.5.0_16-b02
<Jul 22, 2010 10:07:03 AM MEST> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/xx/JMSReportingProviderStartupClass.jar:/xx/SID_AMM_ALSB_SECURITY.jar:/xx/content_system.jar:/xx/p13n-schemas.jar:/xx/p13n_common.jar:/xx/p13n_system.jar:/xx/wlp_services.jar:/xx/analytics_sys.jar:/xx/commerce_system.jar:/xx/groupspace_system.jar:/xx/netuix_common.jar:/xx/netuix_schemas.jar:/xx/netuix_system-full.jar:/xx/netuix_system.jar:/xx/nf-jspcmods.jar:/xx/nf-system.jar:/xx/wlp-schemas.jar:/xx/wlp_content_system.jar:/xx/wps_system.jar:/xx/wsrp-client.jar:/xx/wsrp-common.jar>
WebLogic Server Temporary Patch for CR382552 Fri Dec 12 13:12:27 IST 2008
WebLogic Server Temporary Patch for 9112835 Thu Nov 12 10:53:52 EST 2009
WebLogic Server Temporary Patch for CR368155 Mon Jul 07 15:51:15 PDT 2008
WebLogic Server Temporary Patch for CR370311 Tue Jun 03 14:40:28 IST 2008
WebLogic Server Temporary Patch for CR360676 Fri Apr 18 19:41:49 IST 2008
WebLogic Server Temporary Patch for CR360117 Wed Mar 12 14:47:42 PDT 2008
WebLogic Server Temporary Patch for CR354395 Mon Dec 10 08:10:43 PST 2007
WebLogic Server Temporary Patch for CR345092 Wed Oct 24 11:10:56 IST 2007
WebLogic Server Temporary Patch for CR338239 Fri Jan 11 17:14:11 EST 2008
WebLogic Server Temporary Patch for CR328282 Thu Nov 08 13:44:54 EST 2007
WebLogic Server Temporary Patch for CR315317 Mon Feb 25 09:45:40 PST 2008
WebLogic Server Temporary Patch for CR315317 Mon Feb 25 09:45:40 PST 2008
WebLogic Server Temporary Patch for CR344429 Mon Nov 26 16:11:55 EST 2007
WebLogic Server Temporary Patch for CR351304 Thu Feb 21 10:23:22 EST 2008
WebLogic Server Temporary Patch for CR350614 Sun Nov 11 18:02:20 PST 2007
WebLogic Server Temporary Patch for CR344375 Fri Oct 19 11:08:33 PDT 2007
WebLogic Server Temporary Patch for CR353586 Mon Feb 25 18:59:49 EST 2008
WebLogic Server Temporary Patch for CR353586 Mon Feb 25 18:59:49 EST 2008
WebLogic Server Temporary Patch for CR339736, CR346639 Mon Nov 26 18:13:36 PST 2007
WebLogic Server Temporary Patch for CR343580 Mon Oct 29 17:41:35 EDT 2007
WebLogic Server Temporary Patch for CR315317 Fri Feb 15 11:13:21 PST 2008
WebLogic Server Temporary Patch for CR351685, CR320630 Thu Dec 06 14:04:31 CST 2007
WebLogic Server Temporary Patch for CR345472 Thu Jan 17 16:13:48 EST 2008
WebLogic Server Temporary Patch for CR355338 Mon Jan 21 13:08:48 EST 2008
WebLogic Server Temporary Patch for CR356295 Wed Dec 19 15:37:45 EST 2007
WebLogic Server Temporary Patch for CR348244 Fri Oct 19 11:57:37 PDT 2007
WebLogic Server Temporary Patch for CR346064 Fri Oct 19 13:27:18 PDT 2007
WebLogic Server Temporary Patch for CR346063 Fri Oct 05 14:48:13 PDT 2007
WebLogic Server Temporary Patch for Cr346061 Thu Oct 18 15:38:04 PDT 2007
WebLogic Server Temporary Patch for Cr346061 Thu Oct 18 15:38:04 PDT 2007
WebLogic Server Temporary Patch for CR346060 Thu Oct 18 15:28:01 PDT 2007
WebLogic Server Temporary Patch for CR350528 Fri Nov 02 16:19:36 PDT 2007
WebLogic Server Temporary Patch for CR354048 Thu Dec 06 13:52:09 IST 2007
WebLogic Server Temporary Patch for CR348229 Wed Oct 24 13:49:07 PDT 2007
WebLogic Server Temporary Patch for CR362972, CR357095 Fri Mar 07 15:38:53 IST 2008
WebLogic Server Temporary Patch for CR358677 Wed Jan 23 16:28:40 PST 2008
WebLogic Server Temporary Patch for CR354390 Wed Feb 27 09:57:23 PST 2008
WebLogic Server Temporary Patch for CR357287 Thu Jan 31 14:51:56 PST 2008
WebLogic Server Temporary Patch for CR359323 Fri Mar 07 02:24:20 EST 2008
WebLogic Server Temporary Patch for CR322355 Tue Oct 23 10:53:45 PDT 2007
WebLogic Server 10.0 MP1 Thu Oct 18 20:17:44 EDT 2007 1005184 >
<Jul 22, 2010 10:10:36 AM MEST> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at http://xx.xx.xx.xx:xx.>
<Jul 22, 2010 10:10:36 AM MEST> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>
<Jul 22, 2010 10:10:36 AM MEST> <Info> <WebLogicServer> <BEA-000215> <Loaded License : xxxxxx>
<Jul 22, 2010 10:10:36 AM MEST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jul 22, 2010 10:10:36 AM MEST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<22/07/2010#10:10:36.794#+0200> <Notice> <Log Management> <BEA-170019> <The server log file xxxxxxx.log is opened. All server side log events will be written to this file.>
<22/07/2010#10:10:41.578#+0200> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<22/07/2010#10:10:42.75#+0200> <Alert> <Security> <BEA-090167> <Cannot retrieve identity certificate and private key on server xxmanaged1xx, because the keystore entry alias is not specified.>
<22/07/2010#10:10:42.77#+0200> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the JKS keystore file /xxxkeystore.>
<22/07/2010#10:16:02.665#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<22/07/2010#10:16:02.666#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<22/07/2010#10:45:48.374#+0200> <Alert> <ALSB Security> <BEA-387042> <There is no user password credential mapper provider configured in your security realm. AquaLogic Service Bus service account management will be disabled. Configure a user password credential mapper provider if you need ALSB service account support.>
<22/07/2010#10:45:48.564#+0200> <Error> <ClusterTimer> <BEA-000000> <Cannot contact Admin server. Therefore constructing the Cluster Authority Current time with the time skew 0
java.rmi.UnknownHostException: Could not discover URL for server 'xxadminxx'
     at weblogic.protocol.URLManager.findURL(URLManager.java:143)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getInitialContext(RemoteClusterTimerAuthority.java:72)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getTimerAuthority(RemoteClusterTimerAuthority.java:84)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getClusterTimerAuthorityCurrentTime(RemoteClusterTimerAuthority.java:44)
     at com.bea.wli.timer.ClusterTimerService.clusterTimeAuthorityCurrentTimeMillis(ClusterTimerService.java:173)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#10:45:48.567#+0200> <Error> <ClusterTimer> <BEA-000000> <Cannot contact the admin server. Therefore constructing the Cluster Authority Urzeit from the persisted value
java.rmi.UnknownHostException: Could not discover URL for server 'xxadminxx'
     at weblogic.protocol.URLManager.findURL(URLManager.java:143)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getInitialContext(RemoteClusterTimerAuthority.java:72)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getTimerAuthority(RemoteClusterTimerAuthority.java:84)
     at com.bea.wli.sb.init.RemoteClusterTimerAuthority.getUrzeit(RemoteClusterTimerAuthority.java:57)
     at com.bea.wli.timer.ClusterTimerService.initialize(ClusterTimerService.java:87)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#10:45:52.154#+0200> <Warning> <WliSbTransports> <BEA-381917> <MQ Transport could not be registered due to : Missing MQ Library>
<22/07/2010#11:54:55.765#+0200> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'ALSB Framework Starter Application'.
RegistrationException : java.rmi.RemoteException: [Deployer:149147]Could not reach the admin server via any of its URLS: 'http://xx.xx.xx.xx:xx'.
     at weblogic.deploy.service.internal.targetserver.TargetDeploymentService.registerHandler(TargetDeploymentService.java:166)
     at weblogic.deploy.service.internal.DeploymentService.registerHandler(DeploymentService.java:177)
     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.register(ServerDeploymentReceiver.java:82)
     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.<init>(ServerDeploymentReceiver.java:75)
     at com.bea.wli.config.weblogic.WebLogicConfigService$DeploymentReceiverLauncher.endOfStart(WebLogicConfigService.java:190)
     Truncated. see log file for complete stacktrace
RegistrationException : java.rmi.RemoteException: [Deployer:149147]Could not reach the admin server via any of its URLS: 'http://xx.xx.xx.xx:xx'.
     at weblogic.deploy.service.internal.targetserver.TargetDeploymentService.registerHandler(TargetDeploymentService.java:166)
     at weblogic.deploy.service.internal.DeploymentService.registerHandler(DeploymentService.java:177)
     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.register(ServerDeploymentReceiver.java:82)
     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.<init>(ServerDeploymentReceiver.java:75)
     at com.bea.wli.config.weblogic.WebLogicConfigService$DeploymentReceiverLauncher.endOfStart(WebLogicConfigService.java:190)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#11:54:56.271#+0200> <Error> <ConfigFwk> <BEA-000000> <Failed in endLoadNotification. Ignoring.
java.lang.NullPointerException
     at com.bea.wli.sb.resources.globaloperations.GlobalOperationsResourceLifecycleListener.endLoadNotification(GlobalOperationsResourceLifecycleListener.java:112)
     at com.bea.wli.config.impl.ResourceListenerNotifier.sendLoadNotificationsInContext(ResourceListenerNotifier.java:236)
     at com.bea.wli.config.impl.ResourceListenerNotifier.access$400(ResourceListenerNotifier.java:41)
     at com.bea.wli.config.impl.ResourceListenerNotifier$1.execute(ResourceListenerNotifier.java:151)
     at com.bea.wli.config.transaction.TransactionalTask._doExecute(TransactionalTask.java:217)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#11:54:57.153#+0200> <Error> <ConfigFwk> <BEA-000000> <Failed in endLoadNotification. Ignoring.
java.lang.NullPointerException
     at com.bea.wli.sb.service.ServiceChangeListener.updateAlerts(ServiceChangeListener.java:243)
     at com.bea.wli.sb.service.ServiceChangeListener.endLoadNotification(ServiceChangeListener.java:175)
     at com.bea.wli.config.impl.ResourceListenerNotifier.sendLoadNotificationsInContext(ResourceListenerNotifier.java:236)
     at com.bea.wli.config.impl.ResourceListenerNotifier.access$400(ResourceListenerNotifier.java:41)
     at com.bea.wli.config.impl.ResourceListenerNotifier$1.execute(ResourceListenerNotifier.java:151)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#11:54:59.828#+0200> <Error> <ALSB Statistics Manager> <BEA-473004> <Cannot recover from persistent Store : java.lang.NullPointerException
java.lang.NullPointerException
     at com.bea.wli.monitoring.statistics.collection.Collector.readPersistedRawSnapshot(Collector.java:357)
     at com.bea.wli.monitoring.statistics.StatisticsManager.onStatConfigLoad(StatisticsManager.java:347)
     at com.bea.wli.monitoring.statistics.ALSBStatisticsManager.onStatConfigLoad(ALSBStatisticsManager.java:379)
     at com.bea.wli.sb.service.ServiceStatisticsListener.updateRuntime(ServiceStatisticsListener.java:200)
     at com.bea.wli.sb.service.ServiceStatisticsListener.endLoadNotification(ServiceStatisticsListener.java:168)
     Truncated. see log file for complete stacktrace
>
<22/07/2010#11:55:02.434#+0200> <Warning> <Log Management> <BEA-170011> <The LogBroadcaster on this server failed to broadcast log messages to the admin server. The Admin server may not be running. Message broadcasts to the admin server will be disabled.>
<22/07/2010#11:55:02.557#+0200> <Notice> <Cluster> <BEA-000197> <Listening for announcements from cluster using unicast cluster messaging>
<22/07/2010#11:55:02.557#+0200> <Notice> <Cluster> <BEA-000133> <Waiting to synchronize with other running members of xxCLUSTERxx.>
<22/07/2010#11:55:32.595#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>
<22/07/2010#11:55:32.595#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>
<22/07/2010#11:57:17.835#+0200> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on xx.xx.xx.xx:14101 for protocols iiop, t3, CLUSTER-BROADCAST, ldap, snmp, http.>
<22/07/2010#11:57:17.835#+0200> <Notice> <WebLogicServer> <BEA-000358> <Started WebLogic Independent Managed Server "xxmanaged1xx" for domain "xxDOMAINxx" running in Production Mode>
<22/07/2010#11:57:17.840#+0200> <Warning> <JMX> <BEA-149510> <Unable to establish JMX Connectivity with the Adminstration Server xxadminxx at <JMXServiceURL:null>.>
<22/07/2010#11:57:20.13#+0200> <Notice> <Cluster> <BEA-000102> <Joining cluster xxCLUSTERxx on xx.xx.xx.xx:7001>
<22/07/2010#11:57:20.28#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
I have a hard time trying to see what's happening here.
If you have any input for me to use, I would be very thankful.
And by the way, please excuse me for my poor english (french here ^^)
Best regards,
Serge

As a test, I shutdown my admin server and then killed the managed server from the command line so that node manager would start it ( and not be able to find the admin server ).
I'm using WL 10.3.3 ( 11g + OSB + SOA ) on oracle linux (64bit) and the managed server started in 3 minutes.
So I tried the same in my 10g environment WL 10.3 running OSB 10g ( oracle linux 32-bit ) and the server started in 2 minutes.
I see a couple of times when your trace shows a pause:
1. when loading certificates22/07/2010#10:10:42.75#+0200> <Alert> <Security> <BEA-090167> <Cannot retrieve identity certificate and private key on server xxmanaged1xx, because the keystore entry alias is not specified.>
<22/07/2010#10:10:42.77#+0200> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the JKS keystore file /xxxkeystore.>
<22/07/2010#10:16:02.665#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
For this server, is the keystore alias entered in the server's SSL tab?
2. Just after entering STARTING<22/07/2010#10:16:02.666#+0200> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<22/07/2010#10:45:48.374#+0200> <Alert> <ALSB Security> <BEA-387042> <There is no user password credential mapper provider configured in your security realm. AquaLogic Service Bus service account management will be disabled. Configure a user password credential mapper provider if you need ALSB service account support.>
In my test, the "user password credential" message was preceded by messages about the Singleton Marker applications. Are the singleton apps deployed to this server, or to some other managed server? If they are deployed to some other managed server, are they deployed to a managed server on the same host as the (unreachable) admin server?
<Jul 19, 2010 11:11:54 AM EDT> <Info> <WorkManager> <BEA-002903> <Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "ALSB Cluster Singleton Marker Application">
<Jul 19, 2010 11:11:54 AM EDT> <Info> <WorkManager> <BEA-002903> <Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "ALSB Domain Singleton Marker Application">
<Jul 19, 2010 11:11:54 AM EDT> <Info> <WorkManager> <BEA-002903> <Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "ALSB Framework Starter Application">
<Jul 19, 2010 11:11:56 AM EDT> <Alert> <OSB Security> <BEA-387042> <There is no user password credential mapper provider configured in your security realm. Oracle Service Bus service account management will be disabled. Configure a user password credential mapper provider if you need OSB service account support.>
3. activating ALSB Framework 22/07/2010#10:45:52.154#+0200> <Warning> <WliSbTransports> <BEA-381917> <MQ Transport could not be registered due to : Missing MQ Library>
<22/07/2010#11:54:55.765#+0200> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'ALSB Framework Starter Application'.
In my test, this was logged immediately after the "user password credential mapper" message, with the following between:
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssModuleImpl.getInstance>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssModuleImpl.initialize>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <initializing WssHandlerImpl>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssHandlerImpl constructor>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssAuditor initilizing>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <auditingEnbled set to false>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <auditingEnbled set to false>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssAuditor successfully started>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityWss> <BEA-000000> <WssModuleImpl was successfully initialized>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityModule> <BEA-000000> <jndiLookup: [jndi-name: com.bea.alsb.internal.clusterTimerAuthority]>
<Jul 19, 2010 11:11:56 AM EDT> <Debug> <AlsbSecurityModule> <BEA-000000> <jndiLookup: [jndi-name: com.bea.alsb.internal.clusterTimerAuthority]>
<Jul 19, 2010 11:11:56 AM EDT> <Info> <Monitoring> <BEA-000000> <New Metadata version is 0>
<Jul 19, 2010 11:11:56 AM EDT> <Info> <Monitoring> <BEA-000000> <New Metadata version is 0>
<Jul 19, 2010 11:11:56 AM EDT> <Info> <Monitoring> <BEA-000000> <New Metadata version is 0>
<Jul 19, 2010 11:11:56 AM EDT> <Info> <Deployer> <BEA-149059> <Module Dummy of application ALSB Framework Starter Application is transitioning from STATE_NEW to STATE_PREPARED on server
The clusterTimerAuthority involves contacting the admin server in 11g, but not quite sure about 10g. ( I have a separate bug in 11g when the ClusterTime can't be reached if the admin server is using 2way SSL )
I can't run the same test in my multi-node environments, but is it possible for you to change the targets of the 2 Singleton applications to a managed server which runs on some machine other than the Admin server's machine? Then try starting that managed server while the Admin server is down, and also try restarting another managed server ( which is not on the admin server's host and is not targetted by the Singleton applications. )
I suspect the managed server with the singleton apps will start more quickly than the one without.

Similar Messages

OAM Identity Server user search is very slow after upgrading to 10.1.4.2

We recently upgraded Identity-Server from 7.0.4 to 10.1.4.2 + BP10. The new webpass (version 10.1.4.2) is on iPlanet webserver, which does not have any bundled patch available. After this upgrade, we found the user search is very slow. It is taking double the time compare to version 7.0.4. The search performance for NetPoint admin users is fine.
The new version is connecting to the same LDAP (Sun 5.2) as the old one. The 7.0.4 version was well tuned (like Ldap connections, caching, etc) for the performance. The migration suppose to carryover those performance configuration to the new version. Is there any new parameter (related to performance) I should look for in version 10 ? Anybody have faced these issues after migration and found a fix for it ?
Thanks!
Kabi

More in this thread - Re: OAM- "You do not have sufficient access rights" message with Master Adm
-Vinod

ITunes startup very slow

Hi
My iTunes is very slow, when it starts up, i have the newest version, i have just updated.
It has allways been a problem, do any of you have the same problem, and is there a solution on the problem??

How long is "forever"? I have been experimenting with startup times with my music collection (on a FW400 external drive) - on a G4 MDD it takes 60 secs to load, but on a MacBook it takes 30 secs. This even though the MacBook has much less RAM. So I am wondering if processor speed is what is being overloaded as the iTunes library file loads........ Can any of you provide actual numbers?
What Mac? How big a music collection? How long to load? Is there a pattern here?

Microsoft SQL Server 2012 Management Studio Express silent installation very slow

Iam doing silent installations often on several Systems.
I install the Microsoft SQL Server 2012 Management Studio Express with this command:
"SQLManagementStudio_x64_DEU.exe /ACTION=INSTALL /QUIET /IAcceptSQLServerLicenseTerms="True" /FEATURES=CONN,BC,SSMS /indicateprogress"
The installation will take almost an hour to finish.
I never get error messages or so. The installation afterwards is fine.
Is there a way to speed up the installation process?
Thanks!

Hi Dennis,
When installing SQL Server Management Studio, it is normal for us taking an hour to finish the installation.
Personally, if you want to speed up the installation process, I recommend improve your hardware device on your computer, such as CPU, RAM, hard disk and so on.
Or you can submit an submit the requirement at https://connect.microsoft.com/SQLServer/.
If the requirement mentioned by customers for many times, the product team may consider to add or improve this requirement in the future SQL Server version. Thanks for your understanding.
Thanks,
Sofiya Li
Sofiya Li
TechNet Community Support

11.2.0.1 rac startup very slow

Stack information:
RDBMS: 11.2.0.1
OS: AIX 5.3
Clusterware: 11.2.0.1 Oracle CRS
Issue:
We have recently upgraded from 10.2.0.4 to 11.2.0.1. Prior to the upgrade, our rac clusters would start in about 5 minutes maximum time. After upgrading to 11.2.0.1, startups take approximately 45 minutes. The way that the CRS daemons start is also very strange. We start CRS on node 1, and it takes about 15 minutes for the daemons to start. After they are started, we start the daemons on node 2. At this time, the daemons on node 1 become unavailable for another 10 minutes. After the daemons on node 1 are back, the same thing happens on node 2 and the daemons are unavailable for 10 minutes. Finally, the cluster is available and runs with no issues. There are no error messages that occur during this slow startup.
I have opened a SR ticket on this with Oracle, but have not made any progress on the issue. I have also found similar issues logged on metalink, but didn't see any workarounds to the issue. I wanted to create a post on this forum to see if anyone has seen issues such as this, and if you are aware of a workaround. Please let me know if you have any information.
Thank you

I would not think the CRS daemons as the primary culprits - simply "taking time" to start courtesy of some bug or something.
The CRS/OHA daemon has prerequisites. Such as opening the voting and OCR disks. Such as initialising its network end-points on the Interconnect interface. Such as IPC communication between daemons. Etc.
Take the OCR/voting disks prerequisite. The system level devices (LUNs/scsi targets/whatever) need to be created by the relevant kernel driver and these need to accept I/O.
This can only be done after the kernel driver "discovered" the actual devices via the storage fabric layer (HBAs/HCAs/etc).
If ASM is used as storage management layer, the ASM instance need to start and successfully mount these devices first.
The discovery process itself can be quite slow, depending on the fabric layer used. If for example the server crashed, the fabric layer may still have stale device entries for the LUNs for that server - and these need to reset when that server now wants "duplicate entries". Older versions of the SRP protocol (for Infiniband fabric) can be a bit cumbersome and even problematic in this regard.
Now, I'm not saying that the storage or Interconnect layers are the problem - simply that these need to be fully functional in order for CRS to start. And this is where I would start looking in your case to determine whether the prerequisites are met early/fast enough to meet CRS startup expectations.

SQL Server 2008 R2 Setup Very Slow

Hi,
We have a multi-node SQL Server 2008 R2 Failover Cluster.
I've noticed that the SQL Server 2008 R2 Setup is extremely slow and that it appears to only be using 1 of the 128 "CPUs" (i.e. in Task Manager).
We use this for creating new cluster groups and alike.
Is this tool only single threaded and is there a way to increase the performance? Our CPUs are 2.something GHz, we have 512GB of RAM and fast local disks.
Thanks

Hello,
What happen if you slipstream the SQL Server 2008 R2 with the latest SQL Server 2008 R2 service pack? Maybe SQL Server setup does better after the slipstream process.
http://blogs.msdn.com/b/petersad/archive/2011/07/13/how-to-slipstream-sql-server-2008-r2-and-a-sql-server-2008-r2-service-pack-1-sp1.aspx
http://msdn.microsoft.com/en-us/library/dd638062(v=sql.105).aspx (Slipstream a SQL Server 2008 R2
Update)
Hope this helps.
Regards,
Alberto Morillo
SQLCoffee.com

Installing Windows 2003 Server Standard/XP is very slow- how long does it take?

Hello, I'm looking for some ideas here... I tried installing Windows 2003 on my newly build computer, but after 3 hours it still wasn't installed, and seemed to just be really really slow, and not frozen. I ended up giving up and tried putting windows xp on it, but after an hour it still didn't finish installing. I figure there really has to be something wrong here... The only thing out of the ordinary is that I'm using 4GB ram DDR2 667. How long does it take to install windows xp on your system?!?
Things that I've tried have been:
disabled hypterthreading
loaded optimized settings
changed sata to mode II
MSI 945G Neo-F Socket T (LGA 775) Intel 945G ATX Intel Motherboard
Seagate Barracuda 7200.9 ST3160812AS 160GB 7200 RPM SATA 3.0Gb/s
Intel Pentium 4 650 Prescott 800MHz FSB LGA 775 EM64T
pqi TURBO $GB (4 x 1GB) 240-Pin DDR2 SDRAM DDR2 667 (PC2 5400) Unbuffered Dual
POwer: Ultra Xfinity 500Watt ATX +5v +3.3v +12v1 +12v2 -12v +5vsb
BIOS 1.2
Optial Drive: Generic Toshiba DVD

1&3 and 2&4 seem to be working okay, I've tested all these combinations (this is just plugging in the memory and looking at the cpu in task manager)
mem1 in slot 1 okay (tested slot 1 okay)
mem1 in slot 1
mem2 in slot 3 okay (test 2gigs in duel channel first set of memory okay)
mem3 in slot 1
mem4 in slot 3 okay (tested 2 gig in duel channel second set of memory)
mem1 in slot 2 okay (tested slot 2 okay)
mem1 in slot 2
mem2 in slot 4 okay (tested the second duel channel okay)
mem1 in slot 1
mem2 in slot 3
mem3 in slot 2
mem4 in slot 4 high cpu
mem3 in slot 1
mem1 in slot 2
mem4 in slot 3
mem2 in slot 4 high cpu
I upgraded to the latest bios, no luck... I haven't run any extensive memory tests, but through the tests i did, it seems that the memory is okay. Given these tests do you think it is more likely a problem with the motherboard running 4 slots? Or maybe some power issue, where the 4 sticks of memory are not getting enough power to work right.
The power supply I am using is Xfinity 500W
+5V/30A +3.3/30A +12V1/16A +12V2/18A -12V/0.6A +5VSB/2.5A
Thanks for your feedback..

SQL server 2000 Procedure executing very slower in SQL 2012

Hi,
I've migrated my database from SQL 2000 to SQL 2012 using SQL2008 as intermediate. After migration, I found that one of my procedure is timing out from web site. When I run that procedure in Management Studio, it takes 42 seconds to display result. The same
query is executed in 3 seconds in SQL 2000 version (in first run it takes 10 seconds in SQL 2000 and second run it took 3 seconds only). But in SQL 2012 second and third time also it took 36 seconds. Can anyone explain why the query takes longer time instead
of less time after the upgrade?
Our prod deployment date is approaching so any quick help is appreciated.
Thanks in advance.

You need to compare the execution plans of queries running in 2000 and 2012 to pin point why there is a difference.
However, have you made sure that you have reindexed the indexes and updated the stats and you still see the difference? If not, please do that and test again.
Could you post the execution plans?
Also read this -
http://blogs.msdn.com/b/psssql/archive/2015/01/30/frequently-used-knobs-to-tune-a-busy-sql-server.aspx
Regards, Ashwin Menon My Blog - http:\\sqllearnings.com

After intalling windows server 2008 R2 on database server, dstabase connections are very slow

Hi,
I upgraded my server from windows server 2003 to windows server 2008 R2. I have two application servers. Now applications on application servers are trying to execute queries but those are taking too long. It is very strange why it is happening.
Firewall rule is open for 1433 TCP and 1434 UDP.
Application servers are Windows Server 2003
Thanks in advance!
Thanks,
Sanjeev

Another possible cause is a problem with the application server. You might try a reboot of the problem app server to see of that changes performance.
If performance doesn't change after the reboot, try a coping a 100MB file from the database server to the application servers for performance comparison. Assuming the performance difference for the copy is roughly the same as the queries, that
will eliminate SQL Server for troubleshooting and allow you to focus on networking as the likely culprit. The fact that 10Mbps outperforms 100Mbps is concerning. Perhaps the OS upgrade resulted in different drivers or interface configuration, or maybe
there is something else going on external to the servers, such as incorrect routing.
I'm not a network guy but I would start by reviewing network interfaces on all servers to ensure they are configured properly and for full capabilities of the hardware and correct subnet and gateway. Also review all components if your network
topology for proper configuration.
For additional help with Windows networking, I suggest you start a new thread in the Windows Server Platform Networking TechNet forum (http://social.technet.microsoft.com/Forums/windowsserver/en-US/home?forum=winserverPN).
Include the results of the file copy test and a link back to this thread.
Dan Guzman, SQL Server MVP, http://www.dbdelta.com

Startup very slow

Hello,
I´ve create an instance with oradim and pfile=initTEST.ora
as follows:
After creating the service in Windows, I can connect as sys.
startup nomount takes up to 8 minutes to give any reaction.
Has somone an idea, what´s going wrong there?
Thank you for your help,
Shrek25
# include database configuration parameters
db_name = TEST
control_files = (D:\u02\TEST\controlTEST01.ctl,
D:\u02\TEST\controlTEST02.ctl,
D:\u02\TEST\controlTEST03.ctl)
# tuning parameters
db_files = 100
db_block_size = 8192
# db_file_multiblock_read_count = 8 # SMALL
db_file_multiblock_read_count = 16 # MEDIUM
# db_file_multiblock_read_count = 32 # LARGE
# db_cache_size = 10M
# shared_pool_size = 75000000
# PGA_AGGREGATE_TARGET = 89M
# db_cache_size = 160M
# shared_pool_size = 95000000
# PGA_AGGREGATE_TARGET = 180M
db_cache_size = 400M
shared_pool_size = 350M
PGA_AGGREGATE_TARGET = 430M
# db_cache_size = 70M
# shared_pool_size = 150M
# PGA_AGGREGATE_TARGET = 100M
# RAM unabhaengig
log_buffer = 1048576
log_checkpoint_interval = 10000
log_checkpoint_timeout = 0
java_pool_size = 0K
processes = 500
# dml_locks = 100 # SMALL
dml_locks = 300 # MEDIUM Standard 200
# dml_locks = 500 # LARGE
audit_trail = true # if you want auditing
# timed_statistics = true # if you want timed statistics
max_dump_file_size = 10240 # limit trace file size to 5 Meg each
#log_archive_start = true # if you want automatic archiving
log_archive_dest = D:\u03\archive\TEST
log_archive_format = logarcTEST%s.%t
# global_names = TRUE
compatible = 9.2.0
open_cursors = 400
nls_language = AMERICAN
nls_territory = GERMANY
max_enabled_roles = 140
sortelimination_cost_ratio = 10
job_queue_processes = 1
remote_os_authent = true
# Windows
remote_login_passwordfile = shared
undo_management = AUTO
undo_tablespace=UNDO
undo_retention=1200
undo_suppress_errors=TRUE
O7_DICTIONARY_ACCESSIBILITY = TRUE

RAM=4 GB
OS = Windows 2000
server = IBM Netfinity
I dont think its a hardware issue,since it was working fine with the same hardware configuration.

HT1338 Why my MacBook Pro startup very slow?

I hope some one may heip me resolve my problem.

Try rebooting the system into Safe Mode (hold the Shift key at startup) and then open the Disk Utility program and use it to run a verification of the boot drive, and also use it to run a permissions fix on the boot drive (these are available in the "First Aid" tab).

Managed server hangs after startup

For some reason we're unable to start one of our managed servers in one of our clusters.
After starting up, it takes about ten secons before it hangs.
The last message in the output logs is:
IIOP subsystem enabled
What could be going on? And how can we solve this?
We've already tried stuff like restarting nodemanager and adminserver, and we even rebooted the box.

Landro,
I had the same issue. I followed what was done here:
http://forums.bea.com/bea/message.jspa?messageID=300001317&tstart=0
and it worked. For me I believe it was an issue connecting to an Active Directory system.
-John

Startup very slow with 10.4.9 update

Hey all, I have the 24inch iMac core 2 2.33ghz, 2gb ram and was starting up in 23 secs before the update. Now I am getting over a minute for some reason and some freezes at times...
Generally slower performance than before.
I have already repaired permissions etc to no avail, any ideas please?
eX
iMac 24inch, 2.33ghz, 2GB RAM, PowerBook G4, Powermac G4 Mac OS X (10.4.9)

After an update of the operating system, some of the libraries (frameworks) and applications have changed. There's a process called "prelinking" that takes place afterwards -- namely it searches all the applications and libraries and essentailly makes some notes about where various functions are in the libraries so that an app doesn't waste its time doing that every time it loads.
The more apps and frameworks you've got, the longer this process takes. It should more or less happen in the background, but it could take an hour for a major update. Also, Spotlight will also start reindexing all the changes as well (also takes time).
Wait until you've logged a few hours of usage and see if the problem just doesn't go away.

Can't good sync to mozilla server. It's very slow or sometimes failed. i believe it's because of chinese [g][f][w]. what should i do?

can some computer network genius destroy the [g][f][w] ? a lot Chinese people angry at it.

i feel like it's going more and more difficult to success sync.
the chance of fail sync is getting big.
I'm worried that not long time later it will become completely impossible to success sync.

Libreoffice spell checker startup VERY SLOW

Hello everyone,
I installed the Libreoffice together with the latest openjdk. When I start Libreoffice Writer, everything goes fine, until I write something and the spell checker starts. I have to wait about one minute and a half in order to continue writing (after that everything goes fine). If I deactivate JRE this does not happen, but I need it for the Zotero Integration plugin. Is there anything I can do?
Thank you in advance,

OpenJDK 7 does not work well with Libreoffice. Use 6 and everything should work fine.

Managed server independence mode startup : VERY slow (2h~)

Similar Messages

Maybe you are looking for