Production Portal hangs with timeouts

Question

Hi all,
Several times during the day the Portal 10.1.4.0.0 hangs with timeouts.
In order to recover, Portal needs to be re-started.
(The timeout also occurred during the time they had a db corruption, but now that corruption has been resolved, the hangs still occur.
After running the SVU in CLEANUP mode, the issue still occurs.
Timeout has been set to well over the default. Stall has been increased also. Neither have helped to resolve the issue.
The java portlets are from a specific OC4J Provider, and all of them are on pages under a Page Group.
These portlets reads (also trough views) data on portal schema in order to create the "style" and css of the portal pages. All of the database statements are select, exception of rare insert operation on a table of their custom schema.
User load:
The access statistics indicates that about 800 users access this site in 1 hour.
The more high user access in one hour was 1100 users.
The concurrent user access are about 500.
Thanks for any help.
Edited by: hmannila on May 11, 2010 11:50 PM

Answer 1

Hello Helenna,
In this particular case, the PPE timeout settings are set to very high values causing exhaustion of fetcher threads in the PPE fetcher thread pool :
<servlet>
   <servlet-name>page</servlet-name>
   <servlet-class>oracle.webdb.page.ParallelServlet</servlet-class>
   <init-param>
<param-name>requesttime</param-name>
<param-value>120</param-value>
   </init-param>
   <init-param>
       <param-name>minTimeout</param-name>
       <param-value>180</param-value>
   </init-param>
   <init-param>
       <param-name>stall</param-name>
       <param-value>200</param-value>
   </init-param> This should be the action plan for you :
>
As you already pointed out, the portal seems to become unresponsive because the Edison provider is not responding fast enough to requests from the Portal framework. Oracle Portal should be able to recover from thi in normal cases. Unfortunately the configuration settings in the Portal were changed from the default values making it almost impossible for the framework to recover from the error. To understand this, we will need to go through the theory :
The Oracle Portal framework collects information from various sources and assembles these to web pages which are then sent to the end users internet browser. Providers are one source of information. To retrieve the information from providers, Oracle Portal allocates a so-called fetcher thread in OC4J_Portal. This fetcher thread will create a session with the (remote) provider and fetch the content, similar to how a web browser would contact a web server to retrieve an HTML page. It is important to note that Oracle Portal allocates a fetcher thread for each portlet on a page. For a page with five portlets, five fetcher threads are allocated as the content is fetched in parallel.
In a default configuration with only one OC4J_Portal process, the number of fetcher threads is limited to 25. With five portlets on a page, the Portal will be able to serve up to five requests simultaneously before the pool of fetcher threads gets exhausted. This is normally not a problem because the fetcher threads are allocated for just a few seconds normally. Once they have received the response from the provider, they are returned to the pool and will be available to the Portal again.
The problems start when the provider's response becomes sluggish. The fetcher thread is not returned to the pool immediately. The requests start to queue up until the system doesn't have any fetcher threads available anymore. Once there are no fetcher threads available, the Portal will wait for a new fetcher thread for a certain period of time (as specified by the queueTimeout). After expiration of this queue timeout, the Portal will return an error to the Oracle HTTP Server. This is when end users will start to experience errors in the pages.
To keep the Portal alive and healthy, administrators should take care that the fetcher threads pool doesn't get exhausted. Obviously the best way to do this is to ensure that all providers respond in a reasonable amount of time. If there is no control on the response times of the providers, administrators can use the timeouts in the framework to assure that long running requests are killed off by the framework to assure that the fetcher threads are returned to the pool again. These parameters are configured in the web.xml from the application portal in the OC4J container called OC4J_Portal. Obviously users will see timeout errors for particular portlets on certain pages but the overall framework will be available. In addition, administrators can decide to increase the number of fetcher threads by starting more than one OC4J_Portal process. Each additional process will increase the amount of fetcher threads with 25. There is a memory penalty for this however as each process can consume between 256 - 512 Mb of memory. Administrators should only increase the amount of processes when they have checked that the Portal midtier system has enough free memory available to accommodate for these additional processes.
Action plan :
1. Change the timeout values for the Portal framework to the default values :
a. Navigate to $ORACLE_HOME/j2ee/OC4J_Portal/applications/portal/portal/WEB-INF
b. Copy the web.xml :
% cp web.xml web.xml_23MAR2010
c. Edit the web.xml :
For both the RepositoryServlet and the PortalServlet, remove the STALL timeout
For the page servlet, remove the requesttime, minTimeout, stall and queueTimeout parameters
2. Increase the number of OC4J_Portal processes from 1 to 4 :
a. Create a copy of the file $ORACLE_HOME/opmn/conf/opmn.xml :
% cp opmn.xml opmn.xml_23MAR2010
b. Open the file opmn.xml
c. Locate the following section :
<process-type id="OC4J_Portal" module-id="OC4J">
d. Within the section for the process OC4J_Portal, change the following line :
<process-set id="default_island" numprocs="1"/>
to
<process-set id="default_island" numprocs="4"/>
3. Synchronize the DCM repository :
% $ORACLE_HOME/dcm/bin/dcmctl updateconfig
4. Restart the midtier processes
% $ORACLE_HOME/opmn/bin/opmnctl stopall
% $ORACLE_HOME/opmn/bin/opmnctl startall
4. Check whether the changes have are in effect :
a. Number of OC4J_Portal processes :
Run opmnctl status and count the number of OC4J_Portal processes. You should see four of them.
b. Timeout parameters in OC4J_Portal
Open the file $ORACLE_HOME/j2ee/OC4J_Portal/application-deployments/portal/OC4J_Portal_default_island_1/application.log and check for the following line at the end of the file :
10/03/08 06:32:32 portal: PPE version : 10.1.2.0.2 (29022008)
10/03/08 06:32:32 portal: Tuning parameter values : poolSize(Fetchers)=25[default=25] : minTimeout(MinTimeout)=5s[default=5s] : requesttime(DefaultTimeout)=15s[default=15s] : stall(MaxTimeout)=65s[default=65s] :
queueTimeout(QueueTimeout)=10s[default=10s] maxParallelPortlets=20[default=20] maxParallelPagePortlets=10[default=10]