ODSI 10Gr3 audit logs common/time question

Question

Hi
With ODSI10GR3, We are investigating delays in processing of some DB2 Inserts
Inserst occur daily but th eproblem happens maybe once in a week
A review of the audit log during teh problem occurance shows the following
Comon/time is taking 33 seconds
common/time {
timestamp: Mon Mar 01 10:21:36 PST 2010
duration: 33323 }
with compile time being ~ 14 sec and insert time being ~ 4 seconds
Is it possible that things such as full gc ooccuring can impact this time?
We increased tx timeout to 120 seconds to avoid the timeout but would like to investigate further on this.
Thanks Much for any info
Best
####<Mar 1, 2010 10:22:09 AM PST> <Info> <ODSI> <qa-sc-eibapp02.corp.test.com> <ds_ms2> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <service.uateibsads> <> <> <1267467729490> <BEA-000000> <ClientDataspace> <DSPAuditEvent timestamp: Mon Mar 01 10:21:36 PST 2010 severity: FAILURE id: ClientDataspace:48:Mon Mar 01 10:21:36 PST 2010 {
common/application {
user: service.uateibsads
name: ClientDataspace
eventkind: update
server: ds_ms2
query/performance {
compiletime: 14869
update/relational {
source: CBS_DB2_DS
sql: INSERT INTO "S109935E"."C@CR538"."C@CUNEW01" ("C@STATUS", "C@SEQNBR", "CUBK", "CUNBR", "CUSTAT",
"CUALT", "CUNA1", "CUNA2", "CUNA3", "CUNA4", "CUNA5", "CUNA6", "CUZIP", "CUZIP2", "CUZIP3",
"CUZIP4", "CUSHRT", "CUSSNR@", "CUSSTY", "CUCLPH", "CUHMPH", "CUBUPH", "CUPOFF", "CUSOFF",
"CUPOF1", "CUPOF2", "CUOPDT", "CUTYPE", "CUTYP", "CUSIC", "CUSEX", "CURACE", "CUOWN", "CUYREM",
"CUINC", "CUSRIN", "CUBDTE", "CUDEP", "CUCTC", "CUCTCT", "CUCIRA", "CUMNBR", "CUNTID", "CUUSR1",
"CUCLNK", "CUUSR3", "CUCDCH", "CUCDCN", "CUCDCD", "CUCMCH", "CUCMNR", "CUCMCD", "CUCVSH",
"CUCVCN", "CUCVCD", "CUCATH", "CUCATN", "CUCATD", "CUCLNG", "CUCCCD", "CULGLR", "CUCWHP",
"CUCPSP", "CUCTXN", "CUCPRF", "CUSHKY", "CUITLD", "CUPSTL", "CUACOM", "CUBRCH", "CUMIDT@",
"CUMRTS", "CUMAIL", "CUSOLI", "CUSOCI", "CUCPNA", "CUBPNA", "CUPERS", "CUSALU", "CUFAX",
"CUTELX", "CUTXAN", "CUDOCF", "CUDCDT", "CUTINU", "CUTADT@", "CUWPRT", "CUCECD", "CUCELM",
"CUEXTF", "CUMTND", "CUCNCD", "CUMARK", "CUEMPL", "CUINQ", "CUMNT", "CUCENS", "CUCODT", "CUDEDT",
"CUACCD", "CUBYR1", "CUBYR2", "CUPREF", "CUJBDT", "CUJDDT", "CUEMA1", "CUEMA2", "CUOPT",
"CUOPTD", "CUSPFG", "CUOENTTYP", "CUCENTTYP", "CUAMDT", "CUESDT", "CUENA1", "CUENA2", "CUENA3",
"CUENA4", "CUENA5", "CUENA6", "CUEPST", "CUEZIP", "CUEZIS", "CUEZP3", "CUEZP4", "CUAAPL",
"CUAAKY", "CUAREC", "CUASTA", "CUANA1", "CUANA2", "CUANA3", "CUANA4", "CUANA5", "CUANA6",
"CUAZP1", "CUAZP2", "CUAZP3", "CUAZP4", "CUAPSD", "CUASTR", "CUASTP", "CUASTS", "CUAFLG",
"CUARFG", "CUCLS", "CURISK", "CURDT1", "CURSK2", "CURDT2", "CUCRLN", "CUCRDT", "CUCRFR",
"CUCRND", "CUCRPR", "CUFSDT", "CUFSFR", "CUFSND", "CUSALE", "CUCSTS", "CUNETI", "CUPRJI",
"CUASST", "CUCURA", "CUCASH", "CUACCR", "CUMKTS", "CUREAL", "CULIFE", "CUINVN", "CUFIXA",
"CULIAB", "CUCURL", "CULTRM", "CUNETW", "CUDIRL", "CUINDL", "CUDIRT", "CUINDT", "CUREDB",
"CULCRO", "CUOTHA", "CUOTHL", "CU5WHP", "CUIWHY", "CUWHEX", "CUFILL", "CUREC1", "CUSTAD",
"CUFRN1", "CUCHIB", "CUCHID", "CUCLOB", "CUCLOD", "CUCCDD", "CUCDD1", "CUCDD2", "CUCHD1",
"CUCHD2", "CUCHP1", "CUCHP2", "CUCHP3", "CUCHP4", "CUCOL1", "CUCOL2", "CUCOL3", "CUCOL4",
"CUCCDT", "CUCCYD", "CUCTYD", "CUCIDB", "CUCDIR", "CUCIND", "CUCSEC", "CUCUNS", "CUCILD",
"CUCOPN", "CUCTOD", "CUCNON", "CUCHGO", "CUCRNB", "CUCQAG", "CUCQHU", "CUCQHI", "CUCQLO",
"CUCQDD", "CUCQDT", "CUDPD1", "CUDPD2", "CUDPD3", "CUDPDP", "CUDPD", "CUIPD", "CUUPD", "CUCAGY",
"CUCIAM", "CUCIAY", "CUCPAM", "CUCPAY", "CUCLTC", "CUCLSO", "CUUCMO", "CULCCA", "CUDRSD",
"CUFACS", "CUINDS", "CUIPDS", "CUSECS", "CUUNSS", "CUDPDS", "CUUCMS", "CUBDIR", "CUBCMO",
"CUTHR1", "CUTHR2", "CUTHR3", "CUTHR4", "CUTHR5", "CUFIV1", "CUFIV2", "CUFIV3", "CUFIV4",
"CUFIV5", "CUTEN1", "CUTEN2", "CUTEN3", "CUTEN4", "CUTEN5", "CUTWN1", "CUTWN2", "CUTWN3",
"CUTWN4", "CUTWN5", "CUTHI1", "CUTHI2", "CUTHI3", "CUTHI4", "CUTHI5", "CUSEV1@", "CUSEV2",
"CUSEV3", "CUSEV4", "CUSEV5")
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
rowsmodified: 1
time: 3655
common/application {
exception: Transaction timed out after 29 seconds
BEA1-000021E917F59C34B15A
update/service {
procedure: InsertNewCIF
arity: 1
dataservice: ld:CoreBankingSystem/LogicalServices/CreateNewCIF.ds
script: declare namespace ns0="ld:CoreBankingSystem/LogicalServices/CreateNewCIF";
declare namespace ns1="http://www.test.com/schemas/client/cbs/logical";
declare variable $__fparam0 as element(ns1:NewCIF)* external;
{ return value ns0:InsertNewCIF($__fparam0); }
common/time {
timestamp: Mon Mar 01 10:21:36 PST 2010
duration: 33323 }

Answer 1

1) Is it possible that there is a database lock preventing the insert from being committed?
2) What does the audit look like for a successful update?
3) Notice that the "compile time" is non-zero. This indicates that the plan was not cached. Likely because it is the first time this was executed after the server was started. So not only do you have the extra query compilation time, there would also be time for loading classes and other initialization. (but 12 seconds of loading an initialization seems like a lot). Given that increasing the tx time to 120 seconds solves the problem (it does solve the problem, doesn't it?) I would say that this is the issue.
4) Given that you just started the server (right? see (3)), it's not likely this is due to GC. But you could enable gc verbose to see.