Performance issue for Report Regions

Question

We have an app which has as it's "Home Page" a portal-style page. The "portlets" are separate regions. Some of these regions are HTML forms, but most are report regions, invariably with PL/SQL returning the SQL query.
On our dev box this portal renders fine, in no more than 5 seconds (acceptable). On the production box a couple of the portlets go crazy, usually taking between 1-2 minutes each. I know this because I added the #TIMING# string to each region footer. The portal page has a META tag to refresh every 120 seconds.
The first thing I did to debug was to take the PL/SQL function body returning the SQL query, replace all the variables (which are just local vars, app items and page items) with hardcoded values, strip all quotes and run the "cleaned up" SQL query in SQLPlus. It executes very fast.
So my conclusion is that the problem lies with the application server (which is Oracle iAS). Is this a logical conclusion? What should my next move be to get to the bottom of this?
Even without the problem portlets the portal refresh speed varies widely, between <1 sec and >10 sec.
The application also has other pages displaying the portal regions individually. These pages (including those with the "problem portlet" queries) render ok (ie, order of 5 seconds or less). This tells me that there is an issue with having a bunch of query-based regions on the same page.
My guess is that I might need to examine the logs and/or httpd/modplsql settings of the app server. Number of simultaneous connections?
What would happen if hundreds of users used this app concurrently, navigated to the portal page and left it like that to auto-refresh every 2 minutes? Are such pages a no-no in HTMLDB?

Answer 1

Some more information:
When I run the page in debug mode, I see the following timings for one of the problem portlets:
11.53: show report
11.54: determine column headings
11.54: parse query
60.95: binding: ":F118_BUNDLE"="F118_BUNDLE" value="1"
60.95: binding: ":P52_STMETRICS_PRODBRANCH"="P52_STMETRICS_PRODBRANCH" value="RDBMS_MAIN"
60.95: binding: ":P52_STMETRICS_PLATFORM"="P52_STMETRICS_PLATFORM" value="LINUX"
61.35: print column headings
61.35: rows loop: 15 row(s)
[report is printed here]
61.49: FORMITEM: P52_STMETRICS_PLATFORM HIDDEN
61.54: FORMITEM: P52_STMETRICS_PRODBRANCH HIDDEN
This tells me that the bulk of the time (50 seconds in this case) is eaten up in parsing the query. What is this symptomatic of?
Another problem I've just noticed. It takes about 5 seconds to open up a region for editing (Region Definition) in HTMLDB. When I click "Apply Changes" (even without editing anything) it is taking on the order of minutes for the changes to be applied to the page. Our app server was bounced just before I noticed this.
These are the params in our DAD:
<Location /pls/toolsdb>
SetHandler pls_handler
Order deny,allow
PlsqlDatabaseConnectString ...
PlsqlDatabasePassword ...
PlsqlDatabaseUsername ...
PlsqlDefaultPage ...
PlsqlDocumentPath docs
</Location>