Helps for WLP9.2 in a cluster: deploy errors, propagation errors

Question

I wrote a document for our portal project's production support that details how to fix several recuring problems with WLP9.2 (GA) in cluster. I'm sharing this with the community as a help.
Email me if you'd like updates or have additional scenarios / fixes or you'd like the word doc version.
Curt Smith, Atlanta WLP consultant, [email protected], put HELP WLP as a part of the subject please.
1     WLP 9.2 Environment
The cluster environment where these symptoms are frequently seen is described by the following:
1.     Solaris 9 on 4 cpu sparc boxes. 16Gb Ram. The VM is given ?Xmx1034m.
2.     Bea private patch: RSGT, fixed cluster deployments.
3.     Bea private patch: BG74, fixed session affinity, changed the session tracking cookie name back to the standard name: JSESSIONID.
4.     IBM UDB (DB2) using the Bea DB2 driver. FYI: the problems described below I don?t feel have any relationship to the DB used for a portal.
2     Common problems and their fixes
2.1     Failed deploy (Install) of a new portal application via the cluster console.
Problem symptoms:
After clicking Finish (or) Commit the console displays that there where errors. All errors require this procedure.
Fix steps:
1.     Shut down the whole cluster. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Admin server:
a.     Remove file: <domain>/config/config.lok
b.     Edit file: <domain>/config/config.xml
c.     Remove all elements of: <app-deployment> ? </app-deployment>
d.     Start the Admin server
e.     Make sure it comes up in the running state. Use the console: servers ? admin ? control - resume if needed.
3.     Both Managed servers:
a.     cd <domain>/config
b.     rm ?rf *
This forces the re-down load of the cleaned up config.xml.
c.     cd <domain>/servers/<managed_name>
d.     rm ?rf tmp cache stage data
This cleans up stale or jammed sideways deploys. Be sure to delete all three directorys: tmp, cache, stage and data.
e.     Start each managed server.
f.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
You can now use the console to Install your applications.
2.2     The portal throws framework / container exceptions on one managed instance.
Problem symptoms:
If you see exceptions from the classloader re a framework class not found, serialization failure or error etc. In general the symptom is that the container is not stable, running correctly, not making sense or your application works on one managed instance but not on the other.
Fix steps:
1.     Shut down the problem managed instance. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Perform these clean up steps on one or both managed instances:
a.     cd <domain>/config
b.     rm ?rf *
This forces the re-down load of the cleaned up config.xml.
c.     cd <domain>/servers/<managed_name>
d.     rm ?rf tmp cache stage data
This cleans up stale or jammed sideways deploys. Be sure to delete all three directorys: tmp, cache, stage and data.
e.     Start each managed server.
f.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
3.     The libraries and applications should auto deploy as the managed instance comes up. Once the managed instance goes into the running state, or you Resume into the running state. Your application should be accessible. Sometimes it takes a few seconds after going into the running state for all applications to be instantiated.
2.3     Content propagation fails on the commit step.
Problem symptoms:
In the log of the managed instance you specified in the propagation ant script you?ll see exceptions regarding not being able to create or instantiate a dynamic delegated role.
There is an underlying bug / robustness issue with WLP9.2 (GA) where periodically you can?t create delegated roles either with the PortalAdmin or via the propagation utility.
Important issue:
This procedure was supplied by Bea which will remove from the internal LDAP and the portal DB your custom / created roles. This will leave your cluster in the new installation state with just the default users and roles: weblogic and portaladmin. The implications are that you?ll have to boot your cluster with console user: weblogic / weblogic. You can then add back your secure console user/password but you?ll have to do this over and over as propagations fail. The observed failure rate is once every 2-3 weeks if you do propagations daily.
Note:
The following assumes that you left the default console user weblogic password to be the default password of: weblogic. The following procedure deletes the local LDAP but leaves the rows in the DB.users table including the SHA-1 hashed passwords. The following procedure should still work if you changed the password for weblogic, but it probably won?t work if you try to substitute your secure console user/pw because there will be no delegated authorization roles mapping to your custom console user. You might experiment with this scenario.
Fix steps:
1.     Shut down the whole cluster. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Admin server:
a.     Remove directory: <domain>/servers/AdminServer/data/ldap
b.     Run this SQL script after you edit it for your schema:
delete from yourschema.P13N_DELEGATED_HIERARCHY;
delete from yourschema.P13N_ENTITLEMENT_POLICY;
delete from yourschema.P13N_ENTITLEMENT_RESOURCE;
delete from yourschema.P13N_ENTITLEMENT_ROLE;
delete from yourschema.P13N_ENTITLEMENT_APPLICATION;
commit;
c.     Start the Admin server
d.     Make sure it comes up in the running state. Use the console: servers ? admin ? control - resume if needed.
3.     Both Managed servers:
a.     cd <domain>/servers/<managed_name>
b.     rm ?rf data
This forces the re-down load of the LDAP directory.
c.     Start each managed server.
d.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
2.4     The enterprise portal DB fails and needs to be restored OR switch DB instances
Restoring a portal DB or switching existing DBs are similar scenarios. The issues that you?ll face with WLP9.2 since it now uses a JDBCauthenticator to authenticate and authorize the console / boot user you need to first be able to connect to the DB before the admin and managed instances can boot. If you haven?t properly encrypted the DB user?s password in the <domain>/config/jdbc/*.xml files, then you?ll not be able to boot the admin server since you won?t be able to create a JDBC connection to the DB. The boot messages are not clear as to what the failure is.
You?ll need to know an Admin role user and password that?s in the DB you?re wanting to connect to, to put into boot.properties and on the managed instances in their boot.properties or startManaged scripts. Don?t forget that the managed instances have local credentials which is new for 9.2. They are in the startManaged script in clear text or a local boot.properties.
Note:
The passwords in the DB are SHA-1 hashed and there is no SHA-1 hash generator tool so you can?t change a password via SQL, but you can move the password from one DB to another. This is possible because the domain encryption salt is not used to generate the SHA-1 string. As it turns out, the SHA-1 string is compatible with all 9.2 cluster domains. IE the domain DES3 salt has nothing to do with password verification. The same password SHA-1 string taken from different domains or even same domain but different users will be different, this is just a randomization put into the algorithm yet every domain will be able to validate the given password against the DB?s SHA-1 string. Because of this, I?ve not had any problem moving DB instances between clusters, especially if I?ve given up on security and use weblogic/weblogic as the console user and this user / pw is in every DB.
Steps:
The assumption for restoring a DB is that the DB has been restored but it?s an older version and doesn?t have the console user/pw that is in boot.properties. At this point swaping a DB is the same as restoring an old version of the portal DB.
1.     Edit the console user and password as clear text into: <domain>/servers/AdminServers/security/boot.properties
This is where you may give in and use weblogic/weblogic.
2.     Set the correct DB access password encryption in the jdbc/*.xml files.
a.     cd <domain>/bin
b.     . ./setDomainEnv.sh or if you?re on windows just run: setDomainEnv.cmd
c.     java weblogic.security.Encrypt <the_db_password>
d.     Edit the returned string into every <domain>/config/jdbc/*.xml
e.     Make sure the *.xml files point to the correct DB host, port, schema, DB name, DB user.
3.     Start the Admin server. I should come up. If it doesn?t it has to be not being able to create a connection to the DB, which depends on the *.xml having the correct user and DES-3 encrypted password.
4.     Edit the new console user/password on the managed instance bin/startManaged script or the local boot.properties.
5.     Start the managed instances.
2.5     Install patches on a host that does not have internet access
The short description is to run smart update on a host that does have internet access. Fetch out of the <domain>/utils/bsu/cache_dir     directory the downloaded patch jar and xml. Manually apply the patch to your non-internet accessible hosts.

Answer 1

I wrote a document for our portal project's production support that details how to fix several recuring problems with WLP9.2 (GA) in cluster. I'm sharing this with the community as a help.
Email me if you'd like updates or have additional scenarios / fixes or you'd like the word doc version.
Curt Smith, Atlanta WLP consultant, [email protected], put HELP WLP as a part of the subject please.
1     WLP 9.2 Environment
The cluster environment where these symptoms are frequently seen is described by the following:
1.     Solaris 9 on 4 cpu sparc boxes. 16Gb Ram. The VM is given ?Xmx1034m.
2.     Bea private patch: RSGT, fixed cluster deployments.
3.     Bea private patch: BG74, fixed session affinity, changed the session tracking cookie name back to the standard name: JSESSIONID.
4.     IBM UDB (DB2) using the Bea DB2 driver. FYI: the problems described below I don?t feel have any relationship to the DB used for a portal.
2     Common problems and their fixes
2.1     Failed deploy (Install) of a new portal application via the cluster console.
Problem symptoms:
After clicking Finish (or) Commit the console displays that there where errors. All errors require this procedure.
Fix steps:
1.     Shut down the whole cluster. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Admin server:
a.     Remove file: <domain>/config/config.lok
b.     Edit file: <domain>/config/config.xml
c.     Remove all elements of: <app-deployment> ? </app-deployment>
d.     Start the Admin server
e.     Make sure it comes up in the running state. Use the console: servers ? admin ? control - resume if needed.
3.     Both Managed servers:
a.     cd <domain>/config
b.     rm ?rf *
This forces the re-down load of the cleaned up config.xml.
c.     cd <domain>/servers/<managed_name>
d.     rm ?rf tmp cache stage data
This cleans up stale or jammed sideways deploys. Be sure to delete all three directorys: tmp, cache, stage and data.
e.     Start each managed server.
f.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
You can now use the console to Install your applications.
2.2     The portal throws framework / container exceptions on one managed instance.
Problem symptoms:
If you see exceptions from the classloader re a framework class not found, serialization failure or error etc. In general the symptom is that the container is not stable, running correctly, not making sense or your application works on one managed instance but not on the other.
Fix steps:
1.     Shut down the problem managed instance. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Perform these clean up steps on one or both managed instances:
a.     cd <domain>/config
b.     rm ?rf *
This forces the re-down load of the cleaned up config.xml.
c.     cd <domain>/servers/<managed_name>
d.     rm ?rf tmp cache stage data
This cleans up stale or jammed sideways deploys. Be sure to delete all three directorys: tmp, cache, stage and data.
e.     Start each managed server.
f.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
3.     The libraries and applications should auto deploy as the managed instance comes up. Once the managed instance goes into the running state, or you Resume into the running state. Your application should be accessible. Sometimes it takes a few seconds after going into the running state for all applications to be instantiated.
2.3     Content propagation fails on the commit step.
Problem symptoms:
In the log of the managed instance you specified in the propagation ant script you?ll see exceptions regarding not being able to create or instantiate a dynamic delegated role.
There is an underlying bug / robustness issue with WLP9.2 (GA) where periodically you can?t create delegated roles either with the PortalAdmin or via the propagation utility.
Important issue:
This procedure was supplied by Bea which will remove from the internal LDAP and the portal DB your custom / created roles. This will leave your cluster in the new installation state with just the default users and roles: weblogic and portaladmin. The implications are that you?ll have to boot your cluster with console user: weblogic / weblogic. You can then add back your secure console user/password but you?ll have to do this over and over as propagations fail. The observed failure rate is once every 2-3 weeks if you do propagations daily.
Note:
The following assumes that you left the default console user weblogic password to be the default password of: weblogic. The following procedure deletes the local LDAP but leaves the rows in the DB.users table including the SHA-1 hashed passwords. The following procedure should still work if you changed the password for weblogic, but it probably won?t work if you try to substitute your secure console user/pw because there will be no delegated authorization roles mapping to your custom console user. You might experiment with this scenario.
Fix steps:
1.     Shut down the whole cluster. Using Bea?s shutdown script takes too long, or doesn?t work if the Admin is down or hung.
a.     Find the PID to kill: lsof ?C | grep <listen port number>
b.     kill <pid>
c.     Run kill again and if the pid is still running then do: kill -9 <pid>
2.     Admin server:
a.     Remove directory: <domain>/servers/AdminServer/data/ldap
b.     Run this SQL script after you edit it for your schema:
delete from yourschema.P13N_DELEGATED_HIERARCHY;
delete from yourschema.P13N_ENTITLEMENT_POLICY;
delete from yourschema.P13N_ENTITLEMENT_RESOURCE;
delete from yourschema.P13N_ENTITLEMENT_ROLE;
delete from yourschema.P13N_ENTITLEMENT_APPLICATION;
commit;
c.     Start the Admin server
d.     Make sure it comes up in the running state. Use the console: servers ? admin ? control - resume if needed.
3.     Both Managed servers:
a.     cd <domain>/servers/<managed_name>
b.     rm ?rf data
This forces the re-down load of the LDAP directory.
c.     Start each managed server.
d.     Make sure the managed instance comes up in the running state and not admin. Go to server-<instance>-control-Resume to set the run state to running.
2.4     The enterprise portal DB fails and needs to be restored OR switch DB instances
Restoring a portal DB or switching existing DBs are similar scenarios. The issues that you?ll face with WLP9.2 since it now uses a JDBCauthenticator to authenticate and authorize the console / boot user you need to first be able to connect to the DB before the admin and managed instances can boot. If you haven?t properly encrypted the DB user?s password in the <domain>/config/jdbc/*.xml files, then you?ll not be able to boot the admin server since you won?t be able to create a JDBC connection to the DB. The boot messages are not clear as to what the failure is.
You?ll need to know an Admin role user and password that?s in the DB you?re wanting to connect to, to put into boot.properties and on the managed instances in their boot.properties or startManaged scripts. Don?t forget that the managed instances have local credentials which is new for 9.2. They are in the startManaged script in clear text or a local boot.properties.
Note:
The passwords in the DB are SHA-1 hashed and there is no SHA-1 hash generator tool so you can?t change a password via SQL, but you can move the password from one DB to another. This is possible because the domain encryption salt is not used to generate the SHA-1 string. As it turns out, the SHA-1 string is compatible with all 9.2 cluster domains. IE the domain DES3 salt has nothing to do with password verification. The same password SHA-1 string taken from different domains or even same domain but different users will be different, this is just a randomization put into the algorithm yet every domain will be able to validate the given password against the DB?s SHA-1 string. Because of this, I?ve not had any problem moving DB instances between clusters, especially if I?ve given up on security and use weblogic/weblogic as the console user and this user / pw is in every DB.
Steps:
The assumption for restoring a DB is that the DB has been restored but it?s an older version and doesn?t have the console user/pw that is in boot.properties. At this point swaping a DB is the same as restoring an old version of the portal DB.
1.     Edit the console user and password as clear text into: <domain>/servers/AdminServers/security/boot.properties
This is where you may give in and use weblogic/weblogic.
2.     Set the correct DB access password encryption in the jdbc/*.xml files.
a.     cd <domain>/bin
b.     . ./setDomainEnv.sh or if you?re on windows just run: setDomainEnv.cmd
c.     java weblogic.security.Encrypt <the_db_password>
d.     Edit the returned string into every <domain>/config/jdbc/*.xml
e.     Make sure the *.xml files point to the correct DB host, port, schema, DB name, DB user.
3.     Start the Admin server. I should come up. If it doesn?t it has to be not being able to create a connection to the DB, which depends on the *.xml having the correct user and DES-3 encrypted password.
4.     Edit the new console user/password on the managed instance bin/startManaged script or the local boot.properties.
5.     Start the managed instances.
2.5     Install patches on a host that does not have internet access
The short description is to run smart update on a host that does have internet access. Fetch out of the <domain>/utils/bsu/cache_dir     directory the downloaded patch jar and xml. Manually apply the patch to your non-internet accessible hosts.