Monitoring long ops in OEM10g

Folks
We were able to monitor long SQL operations using OEM 9i. You could easily identify long op SQL and drill down to obtain the TAB which estimated the time for the query operation to complete. I have not been able to discover the same capability in OEM10g. Is this functionality still available? Has the approach changed for oem10g to monitor long ops to obtain a estimated time to complete?
Peter Johnson
Oracle ACS
613-720-2011

Yes in 10g this is part of ADDM
Refer the note 250655.1

Similar Messages

  • Ccms monitoring - Long running job

    I checked in CMMS monitoring (RZ20) version ECC 6.0, there is no option to monitor long running background job.
    How to monitor long running background job using transaction RZ20?

    Hi,
    Check this [link|Re: How to monitor long running background job.;
    Thanks
    Sunny

  • Monitoring Long Running SQL

    Hi all,
    Some users run reports for very large period of time (diffrence of start & end dates 2 years) and they end up scanning the whole database and crashing the application services.
    Is there any way to monitor this kind of queries and kill it before the application hangs??
    thanks in advance.

    If you set up PROFILES in your database (you need to set RESOURCE_LIMIT to true for this), you don't need to monitor any query, the session will be killed automatically when a threshold is exceeded.
    Consult the PROFILE concept and the CREATE PROFILE command in the documentation for further details.
    Sybrand Bakker
    Senior Oracle DBA

  • SCOM rule to monitor connector (OPS manager connetor) certificate expirary

    Hi guys,
    I need  a rule to monitor connector certificate expiray. This only information i have the sql query.
    Select * from Connectors
    Select * from ConnectorKey.
    Is there any way i can create rule to monitor this cert expiray date. Please advise
    Siva

    You could implement a script-based rule or monitor with powershell, and user something
    like this.
    Jonathan Almquist | SCOMskills, LLC (http://scomskills.com)

  • How to monitor DML statements

    Hi pals,
    I want to know how i can monitor DML statements, i can able to monitor select statments but don't DML statements, is there any way or script in following format:
    Executer,Statement,%Completed,%left,Exec_Time,Time_left
    Where
    Executer is a user who runs the statement
    Statement which statement runs
    %completed how much DML operation completed
    %left how much DML operation left
    Exec_Time elapsed time
    time_left how much time remaining in complete operation
    I really apperciate you pals if you can give me some idea how this can be possible??
    Thanks in advance
    Hassan Khan

    Hi,
    These info can be found in V$SESSION_LOGOPS. But the view isn't populated for every query and sometimes it's a bit weird (I actually had queries that ouput 290% of "work" - parallel query side effect). Also notice that this is for long ops, and not everything will be reported there.
    HTH,
    Yoann.

  • Custom Monitor Health State Troubleshooting Techniques

    Hi,
    I have created a Custom Timed Script 2-state Monitor through Ops Console in SCOM 2012 SP1 and the health state is misbehaving and remains Healthy always, even while the unhealthy expression should be evaluated as True.
    I have confirmed the monitor is targeted to class IIS FTP Server, as the monitor appears in Health Explorer for FTP Server and is displayed as enabled.
    I have used LogScriptEvent in the script to write an event of the results of the script on the agent and the value that will be passed back to the SCOM server, so I also know the script is executing and generating the correct results.
    The script returns a numeric value, either 0 or a positive value (ie. no negative value number).
    The healthy expression is = 0.  The unhealthy expression is > 0.
    The returned value represents the number of missing files and when it is for example 12, the health state is not changing to unhealthy and no alert is generated.
    I have updated the MP version and confirmed the agent receives the latest version.
    I have flushed the agent health state.
    I have reset and recalculated the monitor.
    I have reviewed agent and server event logs and do not see any related errors.
    I have used a simple alert description in case it itself had an error and was preventing a healthy state change.
    I have review the xml and see no issue. I updated the monitor xml manually to receive and compare the script value as an integer (instead of the default string type that the gui creates).
    I wish I knew how to see the results of the SCOM workflow that determines if a health state change is required.
    My next steps in the absence of a better action plan, is to reverse the healthy/unhealthy expressions to see what happens or recreate the MP from scratch.
    Any help would be greatly appreciated as I have been banging my head on this one!!

    Thanks for the response Andy.
    Actually, you are only partially correct. The returned value IS a string by default, however you CAN cast it as an Integer if you export the MP to xml, substitute value String with Integer, and reimport the MP.
    Root cause of the issue turned out to be a typo in the parameter syntax used in the health expression.
    It was an easy catch once I compared side-by-side a working monitoring against the unworking monitor.  Good example of how going too fast can actually slow you down :)
    Thanks again for taking the time to respond!

  • LONG running qurries in oracle database

    HI,
    How to check LONG running qurries in oracle database ? and also can you please tell me how to check current runnning quries in DB.
    Thanks,
    Shyamu.A.

    you mean long ops...
    SELECT sid, to_char(start_time,'hh24:mi:ss') stime,
    message,( sofar/totalwork)* 100 percent
    FROM v$session_longops
    WHERE sofar/totalwork < 1if you want locked objects
    select
      object_name,
      object_type,
      session_id,
      type,           -- Type or system/user lock
      lmode,         -- lock mode in which session holds lock
      request,
      block,
      ctime           -- Time since current mode was granted
    from
      v$locked_object, all_objects, v$lock
    where
      v$locked_object.object_id = all_objects.object_id AND
      v$lock.id1 = all_objects.object_id AND
      v$lock.sid = v$locked_object.session_id
    order by
      session_id, ctime desc, object_name
    /Query for currently running queries more than 60 secs...
    select s.username,s.sid,s.serial#,s.last_call_et/60 mins_running,q.sql_text from v$session s
    join v$sqltext_with_newlines q
    on s.sql_address = q.address
    where status='ACTIVE'
    and type <>'BACKGROUND'
    and last_call_et> 60
    order by sid,serial#,q.piece

  • UPDATE statement monitoring

    Does anybody know how to monitor update statement? I am updating 3 Million rows and it taking long time to complete. I just want to make sure this statement not hanged.

    With a minor bit of effort on your part, you can do a search on, for example, longops, and then copy/use one of the gazillion sample scripts used to monitor long operations.

  • Monitoring and Alerting tools for Business Object Data Integrator

    Hi,
    We have Business Objects DI installed in AIX 5.3 environment. Please advise any type of monitoring and alerting tools for DI.
    Monitor tools to monitor Job servers(like job service and web services running properly) and also to monitor long running extraction jobs and alerting the same. Thanks.
    Thanks.
    Raj.

    Hi Thomas
    Though both the tools objectives are for Data integration, the purpose for both tools are different.
    SAP BODS/ BOBJ Data Integrator is an ETL tool which involves combining data residing in different sources and providing users with a unified view of the data
    SAP PI is a Middleware/EAI tool which have a mechanism/technology that connects different parts of an application or a series of applications
    SAP BOBJ Data Integrator
    Bulk Data Integration
    Extraction Transformation and Loading
    Deals of huge volume of data
    SAP PI
    It is process based integration of application data
    It is a middle ware which integrates various project applications into a common programming package that can be operate/access as a single application
    SAP NetWeaver PI is SAP’s implementation of Service-oriented Architecture (SOA) middleware and facilitates the integration of business processes that span different departments, organizations, or companies.
    SAP PI uses SAP NetWeaver Application Server and message servers where in fail over cases the messages can be resent again.
    The below differences will gives you more clarity
    Verdict is we can use both the tools for data transfer/migration and for choosing the tool depends upon the business requirement
    Please reply back if you need more information.
    Regards
    Murthy

  • FAST enabled MView takes ages to refresh

    Hi All,
    I have a materlialized view which is based on a single fact table ( OAS_BALANCE_UNIONED 11,445,156 rows) and a bunch of dimension tables each containing limited number of rows (a typical start schema).
    The conditioins to make the mview fast refreshable have been met so the mview created successfully with the FAST option.
    The mview then worked just fine after than and had been refreshing FAST (6 minutes) for the whole day till other mviews started to refresh in the same time. Some of those mviews read from some of the base tables of the mview in question. Since then, the mivew has started to take ages (3 hours) to refresh. There is surely something wrong because the COMPLETE refresh was faster than that.
    Later, I stopped all the refresh jobs of the other mviews. Yet, the FAST mview still has the same problem of taking ages to refresh.
    When I monitored the session that refreshes the mview, I noticed in the "Long Ops" that
    most of the time is spent on a step named as "Hash Join" (it appears in the Message column). Hash join occurrs because of the outer join.
    Why my FAST mview suddenly started to take so long time to refresh?
    How can I make my FAST mivew really refresh fast?
    Technical Details are below:
    OS: Windows 2003 32-bit
    DB version: Oracle 10g R1 ( 10.1.0.2.0 )
    CREATE MATERIALIZED VIEW FACT_BAL_MV
    BUILD DEFERRED
    REFRESH FAST ON DEMAND
    WITH PRIMARY KEY
    AS
    SELECT B.cmpcode  BAL_KEY,
           B.CMPCODE,
           B.YR,
           B.PERIOD,
           B.BALCODE,
           B.CURCODE,
           B.REPBASIS,
           TO_NUMBER (B.YR || LPAD (B.PERIOD, 2, 0)) DIM_PERIOD_MV,
           D_CMP.DIM_KEY DIM_COMPANY,
           D_CUR.DIM_KEY DIM_CURRENCY,
           D_EL_1.DIM_KEY DIM_EL_1,
           D_EL_2.DIM_KEY DIM_EL_2,
           D_EL_3.DIM_KEY DIM_EL_3,
           D_EL_4.DIM_KEY DIM_EL_4,
           GRP_1_P_EL.DIM_KEY DIM_GRP_1_P_EL,
           D_GRP_1_B_EL.DIM_KEY DIM_GRP_1_B_EL,
           D_GRP_1_K_EL.DIM_KEY DIM_GRP_1_K_EL,
           D_GRP_2_D_EL.DIM_KEY DIM_GRP_2_D_EL,
           D_GRP_3_E_EL.DIM_KEY D_GRP_3_E_EL,
           B.ROWID X_OB_ROWID,
           D_CMP.ROWID X_CMP_ROWID,
           D_CUR.ROWID X_CUR_ROWID,
           D_EL_1.ROWID X_EL_1_ROWID,
           D_EL_2.ROWID X_EL_2_ROWID,
           D_EL_3.ROWID X_EL_3_ROWID,
           D_EL_4.ROWID X_EL_4_ROWID,
           GRP_1_P_EL.ROWID X_GRP_1_P_EL_ROWID,
           GRP_1_P.ROWID X_GRP_1_P_ROWID,
           D_GRP_1_B_EL.ROWID X_GRP_1_B_EL_ROWID,
           D_GRP_1_B.ROWID X_GRP_1_B_ROWID,
           D_GRP_1_K_EL.ROWID X_GRP_1_K_EL_ROWID,
           D_GRP_1_K.ROWID X_GRP_1_K_ROWID,
           D_GRP_2_D_EL.ROWID X_GRP_2_D_EL_ROWID,
           D_GRP_2_D.ROWID X_GRP_2_D_ROWID,
           D_GRP_3_E_EL.ROWID X_GRP_3_E_EL_ROWID,
           D_GRP_3_E.ROWID X_GRP_3_E_ROWID
      FROM CLVE_STAGING.OAS_BALANCE_UNIONED B,
           FINANCE.DW_DIM_COMPANY D_CMP,
           FINANCE.DW_DIM_CURRENCY D_CUR,
           FINANCE.DW_DIM_EL_1 D_EL_1,
           FINANCE.DW_DIM_EL_2 D_EL_2,
           FINANCE.DW_DIM_EL_3 D_EL_3,
           FINANCE.DW_DIM_EL_4 D_EL_4,
           FINANCE.DW_DIM_GRP_1_P_EL GRP_1_P_EL,
           FINANCE.DW_DIM_GRP_1_P GRP_1_P,
           FINANCE.DW_DIM_GRP_1_B_EL D_GRP_1_B_EL,
           FINANCE.DW_DIM_GRP_1_B D_GRP_1_B,
           FINANCE.DW_DIM_GRP_1_K_EL D_GRP_1_K_EL,
           FINANCE.DW_DIM_GRP_1_K D_GRP_1_K,
           FINANCE.DW_DIM_GRP_2_D_EL D_GRP_2_D_EL,
           FINANCE.DW_DIM_GRP_2_D D_GRP_2_D,
           FINANCE.DW_DIM_GRP_3_E_EL D_GRP_3_E_EL,
           FINANCE.DW_DIM_GRP_3_E D_GRP_3_E
    WHERE     B.CMPCODE = D_CMP.CODE
           AND (B.CMPCODE = D_CUR.CMPCODE(+) AND B.CURCODE = D_CUR.CODE(+))
           AND (B.CMPCODE = D_EL_1.CMPCODE(+) AND B.EL1 = D_EL_1.EL_CODE(+))
           AND (B.CMPCODE = D_EL_2.CMPCODE(+) AND B.EL2 = D_EL_2.EL_CODE(+))
           AND (B.CMPCODE = D_EL_3.CMPCODE(+) AND B.EL3 = D_EL_3.EL_CODE(+))
           AND (B.CMPCODE = D_EL_4.CMPCODE(+) AND B.EL3 = D_EL_4.EL_CODE(+))
           AND (GRP_1_P_EL.CMPCODE = GRP_1_P.CMPCODE(+)
                AND GRP_1_P_EL.GRP_CODE = GRP_1_P.GRP_CODE(+))
           AND (B.CMPCODE = GRP_1_P_EL.CMPCODE(+)
                AND B.EL1 = GRP_1_P_EL.EL_CODE(+))
           AND (D_GRP_1_B_EL.CMPCODE = D_GRP_1_B.CMPCODE(+)
                AND D_GRP_1_B_EL.GRP_CODE = D_GRP_1_B.GRP_CODE(+))
           AND (B.CMPCODE = D_GRP_1_B_EL.CMPCODE(+)
                AND B.EL1 = D_GRP_1_B_EL.EL_CODE(+))
           AND (D_GRP_1_K_EL.CMPCODE = D_GRP_1_K.CMPCODE(+)
                AND D_GRP_1_K_EL.GRP_CODE = D_GRP_1_K.GRP_CODE(+))
           AND (B.CMPCODE = D_GRP_1_K_EL.CMPCODE(+)
                AND B.EL1 = D_GRP_1_K_EL.EL_CODE(+))
           AND (D_GRP_2_D_EL.CMPCODE = D_GRP_2_D.CMPCODE(+)
                AND D_GRP_2_D_EL.GRP_CODE = D_GRP_2_D.GRP_CODE(+))
           AND (B.CMPCODE = D_GRP_2_D_EL.CMPCODE(+)
                AND B.EL2 = D_GRP_2_D_EL.EL_CODE(+))
           AND (D_GRP_3_E_EL.CMPCODE = D_GRP_3_E.CMPCODE(+)
                AND D_GRP_3_E_EL.GRP_CODE = D_GRP_3_E.GRP_CODE(+))
           AND (B.CMPCODE = D_GRP_3_E_EL.CMPCODE(+)
                AND B.EL3 = D_GRP_3_E_EL.EL_CODE(+));
    CREATE INDEX FINANCE.DW_FACT_BAL_MV_INX
    ON FACT_BAL_MV (X_OB_ROWID);
    SQL> select count(X_OB_ROWID) from FACT_BAL_MV ;
    COUNT(X_OB_ROWID)
             11444816
    Elapsed: 00:00:27.85
    SQL> select bytes/1024/1024/1024 GB from
    user_segments where segment_name='FACT_BAL_MV';
            GB
    4.4296875search id: redcorolla

    Hard to say why the refresh is now slow. Possibilties include
    * system resources implemented by the fast refresh
    * something changed on the query making the background query run slowly
    Look at the query first. 18 tables, with almost every join an outer join. Some of the tables appear more than once in the query.
    Get an executioion plan to see how the query is being executed.

  • Slow performance in Mapping

    hi ,
    i have a mapping which joins about 10 tables and inserts into the target table
    When i take the Select Query of final join and execute in sQL it is taking only 3 mins. But when i execute teh mapping through OWB it is taking 3 days.
    Please let me know the setting i can do for making the performance high

    Hi,
    Following are the ways you can find the perfromance bottleneck.
    1. It is always a good idea to monitor the performance of the mapping by a tool like TOAD. If you have toad, log into the session browser and find your session and see the LONG OPS section. This will give you an idea as to which operation is taking time.
    2. Create a temporary table where you put the output of the final join. Use this temporary table in the mapping just to verify if the rest of operators in the mapping flow is taking time.
    3. Get the intermediate results from the target table and run the SELECT query directly in any other SQL tool to see how long it takes. This SELECT query is the total query that appears as INSERT INTO ...SELECT *.
    4. Also you can see examine the execution plan and see which performance is costly.
    4. Select a loading hint like APPEND PARALLEL on the target table to see if it improves performance.
    5. Check if all the source tables are properly analysed and indexed.
    I guess I have listed out all methods I used to follow to get rid of such kind of performance problems. Hope it helps you a bit.
    Regards
    -AP

  • Top SQL's vs. Longops

    I'm trying to understand the difference vs TOP SQL statement vs. what is showing up in the longops table. I pull a report showing my top 25 SQL statements but yet when I look at the long ops table the statements taking more this 1 minute are different then what shows up as my top SQL.
    What should I be spending my time tuning whats in the long ops or my top SQL statements?

    Top sql shows in the statspack or AWR report shows the top sql during the time period for which you run the report.
    e.g if you run report for 4-5 pm,it will show the sql which consumes most resources(CPU,gets,reads etc)
    V$session_longops This view displays the status of various operations that run for longer than 6 seconds (in absolute time). These operations currently include many backup and recovery functions, statistics gathering, and query execution, and more
    operations are added for every Oracle release."You can also set rows in this view using the procedure dbms_application_info.set_session_longops"
    By Jonatan Lewis
    sqlexecprogression_cost with the description: "sql execution progression monitoring cost threshold" and the default value of 1,000
    cost = 1000 equates to elapsed time = 6 seconds

  • Update Statement Simply hanged but doing db file sequential read

    Hi,
    Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 18
    Unix process pid: 10048, image: oracle@prdfa001
    *** 2010-09-09 23:37:07.484
    *** ACTION NAME:() 2010-09-09 23:37:07.473
    *** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
    *** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
    *** SESSION ID:(289.54) 2010-09-09 23:37:07.473
    Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
    *** 2010-09-09 23:37:20.315
    Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
    WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
    WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
    WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
    WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
    WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
    WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
    WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
    WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
    WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
    WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
    WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
    WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
    WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
    WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
    WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
    WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
    WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
    WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
    WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
    WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
    WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
    WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
    WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
    WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
    WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
    WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
    Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 22
    Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
    *** ACTION NAME:() 2010-09-09 23:20:13.485
    *** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
    *** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
    *** SESSION ID:(290.697) 2010-09-09 23:20:13.485
    ===================================================
    SYSTEM STATE
    System global information:
         processes: base 47819b480, size 300, cleanup 4781a5638
         allocation: free sessions 47f1d6148, free calls 0
         control alloc errors: 0 (process), 0 (session), 0 (call)
         PMON latch cleanup depth: 0
         seconds since PMON's last scan for dead processes: 20
         system statistics:
    1171 logons cumulative
    19 logons current
    89219 opened cursors cumulative
    86 opened cursors current
    15095069 user commits
    5 user rollbacks
    58632904 user calls
    44023255 recursive calls
    224311 recursive cpu usage
    201424173 session logical reads
    0 session stored procedure space
    901812 CPU used when call started
    995437 CPU used by this session
    6814196 DB time
    0 cluster wait time
    22542300822 concurrency wait time
    3095 application wait time
    16479074661 user I/O wait time
    1284052668 session connect time
    1284067190 process last non-idle time
    189018343568 session uga memory
    1249667216 session uga memory max
    26059216 messages sent
    26059220 messages received
    239739 background timeouts
    162399896 session pga memory
    189662872 session pga memory max
    4 enqueue timeouts
    901146 enqueue waits
    0 enqueue deadlocks
    32122711 enqueue requests
    17819 enqueue conversions
    32122676 enqueue releases
    0 global enqueue gets sync
    0 global enqueue gets async
    0 global enqueue get time
    0 global enqueue releases
    2865667 physical read total IO requests
    262620 physical read total multi block requests
    270093476864 physical read total bytes
    select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
          hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
          namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
          lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
          pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
          ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
            LIBRARY OBJECT: object=471ee1d38
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 471ee1800 471ee1470 47df7dce0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47df7de48 471ee1e50 I/P/A/-/-    0 NONE   00
          SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
          LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
          namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
          lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
          pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
          ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
            LIBRARY OBJECT: object=472f8d9d8
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DEPENDENCIES: count=1 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb22ee0 472f8daf0 I/P/A/-/-    0 NONE   00
                6 472f8e508 46be86250 I/-/A/-/E    0 NONE   00
          SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
          name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
          hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
          lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
          pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
          ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
            LIBRARY OBJECT: object=472f8e6e0
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 472f8e1a8 472f8de18 47bb22fa0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb23108 472f8e7f8 I/P/A/-/-    0 NONE   00
          SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
          user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
          name=ALTER SESSION SET TIME_ZONE='+02:00'
          hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
          lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
          pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
          ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
            LIBRARY OBJECT: object=472fffc08
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bbde320 472fffd20 I/P/A/-/-    0 NONE   00
          SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
          (dummy) nxc=0, nlb=0  
        SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
                           event: 1132, last message event: 1132,
                           last message waited event: 1132,                        next message: 0(0), messages read: 0
                           channel: (47a2df4f8) system events broadcast channel
                                    scope: 2, event: 1132, last mesage event: 18,
                                    publishers/subscribers: 0/17,
                                    messages published: 1
        SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
        SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (osp req holder)
    PSEUDO PROCESS for group DEFAULT:
      SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
      (process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
                int error: 0, call error: 0, sess error: 0, txn error 0
      (post info) last post received: 0 0 0
                  last post received-location: No post
                  last process to post me: none
                  last post sent: 0 0 0
                  last post sent-location: No post
                  last process posted by me: none
        (latch info) wait_event=0 bits=0
        Process Group: DEFAULT, pseudo proc: 47a1eb7d0
        O/S info: user: , term: , ospid:  (DEAD)
        OSD pid info: Unix process pid: 0, image: PSEUDO
    Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
    4791BF530                   00000000 00000000          [........]
    4791BF540 00000000 00000000 00000000 00000000  [................]
      Repeat 31 times
    NO DETACHED BRANCHES.
    NO DETACHED NETWORK CONNECTIONS.
    CLEANUP STATE OBJECTS:
    SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: instance enqueue anchor state
    latch: 0x380009890
      SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
      (enqueue) TA-00000006-00000001     DID: 0001-000F-0000000B
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x47a28d020, mode: X, lock_flag: 0x0
      own: 0x0, sess: 0x0, prv: 0x47a28d030
    SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: switchable channel handle anch
    latch: 0x38000ac98
      SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
      (broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
                         event: 1, last message event: 1,
                         last message waited event: 1,                      next message: 0(0), messages read: 0
                         channel: (47a2e4190) KPON channel
                                  scope: 2, event: 1, last mesage event: 0,
                                  publishers/subscribers: 0/1,
                                  messages published: 0
    SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: TT shared object cleanup SO
    latch: 0x38001c6b8
    SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: SS shared object cleanup SO
    latch: 0x38001cd48
    END OF SYSTEM STATE
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    db file sequential read           2,347,652       9,215      4   64.5   User I/O
    db file scattered read              245,687       4,199     17   29.4   User I/O
    CPU time                                            974           6.8
    db file parallel write               50,082         408      8    2.9 System I/O
    log file parallel write               6,963          52      7    0.4 System I/O
    Time Model Statistics                DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Total time in database user-calls (DB Time): 14286.4s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                             14,280.3        100.0
    DB CPU                                                  974.5          6.8
    PL/SQL execution elapsed time                           531.8          3.7
    parse time elapsed                                       30.5           .2
    hard parse elapsed time                                  27.1           .2
    connection management call elapsed time                  14.9           .1
    hard parse (sharing criteria) elapsed time                3.4           .0
    hard parse (bind mismatch) elapsed time                   3.1           .0
    PL/SQL compilation elapsed time                           2.4           .0
    failed parse elapsed time                                 0.0           .0
    repeated bind elapsed time                                0.0           .0
    sequence load elapsed time                                0.0           .0
    DB time                                              14,286.4          N/A
    background elapsed time                                 670.2          N/A
    background cpu time                                     186.1          N/A
    Wait Class                            DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    User I/O                    2,593,484     .0           13,415       5     150.0
    System I/O                     87,506     .0              515       6       5.1
    Other                             839   11.4                6       7       0.0
    Commit                          3,225     .1                6       2       0.2
    Concurrency                     1,033     .0                5       5       0.1
    Configuration                   2,514   99.4                0       0       0.1
    Network                        47,559     .0                0       0       2.8
    Application                         7     .0                0       0       0.0
    Wait Events                          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    db file sequential read           2,347,652     .0       9,215       4     135.8
    db file scattered read              245,687     .0       4,199      17      14.2
    db file parallel write               50,082     .0         408       8       2.9
    log file parallel write               6,963     .0          52       7       0.4
    control file parallel write           6,203     .0          44       7       0.4
    control file sequential read         24,242     .0          11       0       1.4
    log file sync                         3,225     .1           6       2       0.2
    latch free                               84     .0           4      47       0.0
    os thread startup                        25     .0           3     120       0.0
    latch: session allocation                39     .0           1      33       0.0
    db file parallel read                    12     .0           1      92       0.0
    enq: TX - index contention              186     .0           1       3       0.0
    latch: shared pool                       47     .0           1      11       0.0
    LGWR wait for redo copy                 319    3.1           0       1       0.0
    library cache load lock                   2     .0           0     172       0.0
    buffer busy waits                       590     .0           0       0       0.0
    log file switch completion                6     .0           0      29       0.0
    SGA: allocation forcing comp             11   54.5           0      14       0.0
    latch: library cache lock                50     .0           0       3       0.0
    read by other session                    38     .0           0       4       0.0
    direct path read                         42     .0           0       3       0.0
    SQL*Net message to client            44,807     .0           0       0       2.6
    rdbms ipc reply                         207     .0           0       0       0.0
    SQL*Net more data from clien          1,014     .0           0       0       0.1
    latch: cache buffers chains              24     .0           0       1       0.0
    latch: library cache                     29     .0           0       1       0.0
    log file sequential read                  8     .0           0       3       0.0
    direct path write                        50     .0           0       0       0.0
    SQL*Net more data to client             398     .0           0       0       0.0
    latch: object queue header o             12     .0           0       1       0.0
    latch: In memory undo latch              78     .0           0       0       0.0
    undo segment extension                2,507   99.7           0       0       0.1
    latch: cache buffers lru cha              4     .0           0       1       0.0
    log file single write                     8     .0           0       0       0.0
    local write wait                          3     .0           0       1       0.0
    enq: RO - fast object reuse               3     .0           0       1       0.0
    buffer deadlock                          87   92.0           0       0       0.0
    enq: JS - queue lock                      1     .0           0       1       0.0
    cursor: pin S                            70     .0           0       0       0.0
    latch: row cache objects                  2     .0           0       1       0.0
    SQL*Net message to dblink             1,338     .0           0       0       0.1
    latch: checkpoint queue latc              2     .0           0       0       0.0
    reliable message                          3     .0           0       0       0.0
    log buffer space                          1     .0           0       1       0.0
    SQL*Net break/reset to clien              4     .0           0       0       0.0
    SQL*Net more data from dblin              2     .0           0       0       0.0
    SQL*Net message from client          44,949     .0     155,701    3464       2.6
    virtual circuit status                  621  100.0      18,156   29237       0.0
    Streams AQ: qmn slave idle w            664     .0      18,127   27299       0.0
    Streams AQ: qmn coordinator           1,339   50.4      18,099   13517       0.1
    Streams AQ: waiting for time             12  100.0       8,741  728394       0.0
    jobq slave wait                         130  100.0         380    2927       0.0
    PL/SQL lock timer                         1  100.0           1     978       0.0
    SQL*Net message from dblink           1,338     .0           0       0       0.1
    single-task message                       1     .0           0      38       0.0
    class slave wait                         11     .0           0       1       0.0
    SQL ordered by Elapsed Time          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
      Elapsed      CPU                  Elap per  % Total
      Time (s)   Time (s)  Executions   Exec (s)  DB Time    SQL Id
        13,664        906            0        N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
         8,792        195            0        N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
         2,524        368            1     2524.1    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
         1,414        177            1     1414.4     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           742        137            1      742.2     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
           274         11            1      274.2     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
           264          8           27        9.8     1.8 8szmwam7fysa3
    Module: DBMS_SCHEDULER
    insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
    lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
    ULL, NULL, 'FALSE', :5, 'FALSE'))
            99          1            1       99.4     0.7 1z0x41f66nvjr
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
            21         10            1       21.5     0.2 bbc1ck8594kvj
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
    IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
    ,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
    GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
            12          0            1       12.5     0.1 6xm9p9uy5kaap
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
    SQL ordered by CPU Time              DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
        CPU      Elapsed                  CPU per  % Total
      Time (s)   Time (s)  Executions     Exec (s) DB Time    SQL Id
           906     13,664            0         N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
           368      2,524            1      367.51    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
           195      8,792            0         N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
           177      1,414            1      176.93     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           137        742            1      137.38     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
            11        274            1       10.82     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
            10         21            1        9.65     0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39

    Hi,
    Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 18
    Unix process pid: 10048, image: oracle@prdfa001
    *** 2010-09-09 23:37:07.484
    *** ACTION NAME:() 2010-09-09 23:37:07.473
    *** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
    *** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
    *** SESSION ID:(289.54) 2010-09-09 23:37:07.473
    Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
    *** 2010-09-09 23:37:20.315
    Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
    WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
    WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
    WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
    WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
    WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
    WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
    WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
    WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
    WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
    WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
    WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
    WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
    WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
    WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
    WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
    WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
    WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
    WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
    WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
    WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
    WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
    WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
    WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
    WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
    WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
    WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
    Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 22
    Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
    *** ACTION NAME:() 2010-09-09 23:20:13.485
    *** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
    *** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
    *** SESSION ID:(290.697) 2010-09-09 23:20:13.485
    ===================================================
    SYSTEM STATE
    System global information:
         processes: base 47819b480, size 300, cleanup 4781a5638
         allocation: free sessions 47f1d6148, free calls 0
         control alloc errors: 0 (process), 0 (session), 0 (call)
         PMON latch cleanup depth: 0
         seconds since PMON's last scan for dead processes: 20
         system statistics:
    1171 logons cumulative
    19 logons current
    89219 opened cursors cumulative
    86 opened cursors current
    15095069 user commits
    5 user rollbacks
    58632904 user calls
    44023255 recursive calls
    224311 recursive cpu usage
    201424173 session logical reads
    0 session stored procedure space
    901812 CPU used when call started
    995437 CPU used by this session
    6814196 DB time
    0 cluster wait time
    22542300822 concurrency wait time
    3095 application wait time
    16479074661 user I/O wait time
    1284052668 session connect time
    1284067190 process last non-idle time
    189018343568 session uga memory
    1249667216 session uga memory max
    26059216 messages sent
    26059220 messages received
    239739 background timeouts
    162399896 session pga memory
    189662872 session pga memory max
    4 enqueue timeouts
    901146 enqueue waits
    0 enqueue deadlocks
    32122711 enqueue requests
    17819 enqueue conversions
    32122676 enqueue releases
    0 global enqueue gets sync
    0 global enqueue gets async
    0 global enqueue get time
    0 global enqueue releases
    2865667 physical read total IO requests
    262620 physical read total multi block requests
    270093476864 physical read total bytes
    select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
          hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
          namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
          lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
          pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
          ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
            LIBRARY OBJECT: object=471ee1d38
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 471ee1800 471ee1470 47df7dce0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47df7de48 471ee1e50 I/P/A/-/-    0 NONE   00
          SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
          LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
          namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
          lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
          pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
          ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
            LIBRARY OBJECT: object=472f8d9d8
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DEPENDENCIES: count=1 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb22ee0 472f8daf0 I/P/A/-/-    0 NONE   00
                6 472f8e508 46be86250 I/-/A/-/E    0 NONE   00
          SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
          name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
          hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
          lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
          pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
          ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
            LIBRARY OBJECT: object=472f8e6e0
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 472f8e1a8 472f8de18 47bb22fa0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb23108 472f8e7f8 I/P/A/-/-    0 NONE   00
          SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
          user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
          name=ALTER SESSION SET TIME_ZONE='+02:00'
          hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
          lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
          pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
          ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
            LIBRARY OBJECT: object=472fffc08
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bbde320 472fffd20 I/P/A/-/-    0 NONE   00
          SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
          (dummy) nxc=0, nlb=0  
        SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
                           event: 1132, last message event: 1132,
                           last message waited event: 1132,                        next message: 0(0), messages read: 0
                           channel: (47a2df4f8) system events broadcast channel
                                    scope: 2, event: 1132, last mesage event: 18,
                                    publishers/subscribers: 0/17,
                                    messages published: 1
        SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
        SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (osp req holder)
    PSEUDO PROCESS for group DEFAULT:
      SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
      (process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
                int error: 0, call error: 0, sess error: 0, txn error 0
      (post info) last post received: 0 0 0
                  last post received-location: No post
                  last process to post me: none
                  last post sent: 0 0 0
                  last post sent-location: No post
                  last process posted by me: none
        (latch info) wait_event=0 bits=0
        Process Group: DEFAULT, pseudo proc: 47a1eb7d0
        O/S info: user: , term: , ospid:  (DEAD)
        OSD pid info: Unix process pid: 0, image: PSEUDO
    Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
    4791BF530                   00000000 00000000          [........]
    4791BF540 00000000 00000000 00000000 00000000  [................]
      Repeat 31 times
    NO DETACHED BRANCHES.
    NO DETACHED NETWORK CONNECTIONS.
    CLEANUP STATE OBJECTS:
    SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: instance enqueue anchor state
    latch: 0x380009890
      SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
      (enqueue) TA-00000006-00000001     DID: 0001-000F-0000000B
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x47a28d020, mode: X, lock_flag: 0x0
      own: 0x0, sess: 0x0, prv: 0x47a28d030
    SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: switchable channel handle anch
    latch: 0x38000ac98
      SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
      (broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
                         event: 1, last message event: 1,
                         last message waited event: 1,                      next message: 0(0), messages read: 0
                         channel: (47a2e4190) KPON channel
                                  scope: 2, event: 1, last mesage event: 0,
                                  publishers/subscribers: 0/1,
                                  messages published: 0
    SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: TT shared object cleanup SO
    latch: 0x38001c6b8
    SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: SS shared object cleanup SO
    latch: 0x38001cd48
    END OF SYSTEM STATE
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    db file sequential read           2,347,652       9,215      4   64.5   User I/O
    db file scattered read              245,687       4,199     17   29.4   User I/O
    CPU time                                            974           6.8
    db file parallel write               50,082         408      8    2.9 System I/O
    log file parallel write               6,963          52      7    0.4 System I/O
    Time Model Statistics                DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Total time in database user-calls (DB Time): 14286.4s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                             14,280.3        100.0
    DB CPU                                                  974.5          6.8
    PL/SQL execution elapsed time                           531.8          3.7
    parse time elapsed                                       30.5           .2
    hard parse elapsed time                                  27.1           .2
    connection management call elapsed time                  14.9           .1
    hard parse (sharing criteria) elapsed time                3.4           .0
    hard parse (bind mismatch) elapsed time                   3.1           .0
    PL/SQL compilation elapsed time                           2.4           .0
    failed parse elapsed time                                 0.0           .0
    repeated bind elapsed time                                0.0           .0
    sequence load elapsed time                                0.0           .0
    DB time                                              14,286.4          N/A
    background elapsed time                                 670.2          N/A
    background cpu time                                     186.1          N/A
    Wait Class                            DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    User I/O                    2,593,484     .0           13,415       5     150.0
    System I/O                     87,506     .0              515       6       5.1
    Other                             839   11.4                6       7       0.0
    Commit                          3,225     .1                6       2       0.2
    Concurrency                     1,033     .0                5       5       0.1
    Configuration                   2,514   99.4                0       0       0.1
    Network                        47,559     .0                0       0       2.8
    Application                         7     .0                0       0       0.0
    Wait Events                          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    db file sequential read           2,347,652     .0       9,215       4     135.8
    db file scattered read              245,687     .0       4,199      17      14.2
    db file parallel write               50,082     .0         408       8       2.9
    log file parallel write               6,963     .0          52       7       0.4
    control file parallel write           6,203     .0          44       7       0.4
    control file sequential read         24,242     .0          11       0       1.4
    log file sync                         3,225     .1           6       2       0.2
    latch free                               84     .0           4      47       0.0
    os thread startup                        25     .0           3     120       0.0
    latch: session allocation                39     .0           1      33       0.0
    db file parallel read                    12     .0           1      92       0.0
    enq: TX - index contention              186     .0           1       3       0.0
    latch: shared pool                       47     .0           1      11       0.0
    LGWR wait for redo copy                 319    3.1           0       1       0.0
    library cache load lock                   2     .0           0     172       0.0
    buffer busy waits                       590     .0           0       0       0.0
    log file switch completion                6     .0           0      29       0.0
    SGA: allocation forcing comp             11   54.5           0      14       0.0
    latch: library cache lock                50     .0           0       3       0.0
    read by other session                    38     .0           0       4       0.0
    direct path read                         42     .0           0       3       0.0
    SQL*Net message to client            44,807     .0           0       0       2.6
    rdbms ipc reply                         207     .0           0       0       0.0
    SQL*Net more data from clien          1,014     .0           0       0       0.1
    latch: cache buffers chains              24     .0           0       1       0.0
    latch: library cache                     29     .0           0       1       0.0
    log file sequential read                  8     .0           0       3       0.0
    direct path write                        50     .0           0       0       0.0
    SQL*Net more data to client             398     .0           0       0       0.0
    latch: object queue header o             12     .0           0       1       0.0
    latch: In memory undo latch              78     .0           0       0       0.0
    undo segment extension                2,507   99.7           0       0       0.1
    latch: cache buffers lru cha              4     .0           0       1       0.0
    log file single write                     8     .0           0       0       0.0
    local write wait                          3     .0           0       1       0.0
    enq: RO - fast object reuse               3     .0           0       1       0.0
    buffer deadlock                          87   92.0           0       0       0.0
    enq: JS - queue lock                      1     .0           0       1       0.0
    cursor: pin S                            70     .0           0       0       0.0
    latch: row cache objects                  2     .0           0       1       0.0
    SQL*Net message to dblink             1,338     .0           0       0       0.1
    latch: checkpoint queue latc              2     .0           0       0       0.0
    reliable message                          3     .0           0       0       0.0
    log buffer space                          1     .0           0       1       0.0
    SQL*Net break/reset to clien              4     .0           0       0       0.0
    SQL*Net more data from dblin              2     .0           0       0       0.0
    SQL*Net message from client          44,949     .0     155,701    3464       2.6
    virtual circuit status                  621  100.0      18,156   29237       0.0
    Streams AQ: qmn slave idle w            664     .0      18,127   27299       0.0
    Streams AQ: qmn coordinator           1,339   50.4      18,099   13517       0.1
    Streams AQ: waiting for time             12  100.0       8,741  728394       0.0
    jobq slave wait                         130  100.0         380    2927       0.0
    PL/SQL lock timer                         1  100.0           1     978       0.0
    SQL*Net message from dblink           1,338     .0           0       0       0.1
    single-task message                       1     .0           0      38       0.0
    class slave wait                         11     .0           0       1       0.0
    SQL ordered by Elapsed Time          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
      Elapsed      CPU                  Elap per  % Total
      Time (s)   Time (s)  Executions   Exec (s)  DB Time    SQL Id
        13,664        906            0        N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
         8,792        195            0        N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
         2,524        368            1     2524.1    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
         1,414        177            1     1414.4     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           742        137            1      742.2     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
           274         11            1      274.2     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
           264          8           27        9.8     1.8 8szmwam7fysa3
    Module: DBMS_SCHEDULER
    insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
    lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
    ULL, NULL, 'FALSE', :5, 'FALSE'))
            99          1            1       99.4     0.7 1z0x41f66nvjr
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
            21         10            1       21.5     0.2 bbc1ck8594kvj
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
    IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
    ,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
    GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
            12          0            1       12.5     0.1 6xm9p9uy5kaap
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
    SQL ordered by CPU Time              DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
        CPU      Elapsed                  CPU per  % Total
      Time (s)   Time (s)  Executions     Exec (s) DB Time    SQL Id
           906     13,664            0         N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
           368      2,524            1      367.51    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
           195      8,792            0         N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
           177      1,414            1      176.93     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           137        742            1      137.38     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
            11        274            1       10.82     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
            10         21            1        9.65     0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39

  • Query works as two queries, but not as one

    I am beating my head against the wall trying to figure out why a query will not return, yet when I break it up into two queries, they both return successfully.
    There is a text index on the text_a field.
    This is my query:
    select distinct /*+ */ table_c.rec_id
    from table_d, table_n, table_c
    where ( ((create_date >='2006-03-16T00:00:00Z')
    and (misc_field='abcd')
    and (contains( text_a,'ABC or DEF or GHI or JKL or MNP') > 0 ))
    /* and (rownum <=2000) */)
    and table_c.rec_id = table_d.rec_id
    and table_c.create_date = table_d.create_date
    and table_c.rec_id = table_n.rec_id
    and table_c.create_date = table_n.create_date
    This query will not return data. I get an ora-1555 every time I let it run. The longest I let it run before getting the 1555 error was 1 hr and 50 min.
    Tests suspecting a text data problem:
    Now, here's what's puzzling. The query will run if 'JKL' is taken out. So, I thought this value may be the problem, so I ran the query with just the 'JKL' value in the "contains" clause and it did not return a value.
    Tests suspecting a date problem:
    Now, even more strange: When I ran the query (with all 5 conditions for misc_field) and looked from 2006-03-17T00:00:00Z, it returned records in less than 5 minutes. Then I queries just the 16th (create_date>='2006-03-16T00:00:00Z' and create_date < '2006-03-17T00:00:00Z' and it returned rows in less than 1 minute. So, I looked at the tokens for those dates, and there are values for all of them for each of the dates from the 16th to the present (the 21st). However, when I put them together in the query above, the query just seems to hang.
    The total number of rows that should be returned is around 650 (adding the two results that worked).
    Any suggestions? The indexes are valid, syncing is up-to-date, optimizing has no errors,a nd there are token_counts for the tokens in the $I table for the text_a field.
    This problem has crossed into a second day and the same value of the 16th (as in the query above) is still used and still does not return rows. What else should I be looking at?
    In advance, thanks for any suggestions/assistance.
    - Jenny

    I believe the 1555 error is an obscure response to (i.e. a result of) another problem.
    The create_date column is a date datatype. The NLS format matches at the session, instance and database level (nls_session_parameters, and instance, and database). The NLS_DATE_FORMAT is set to YYYY-MM-DD"T"HH24:MI:SS"Z" in each of those views. Would this still cause a potential problem?
    I neglected to say that we are on version 9.2.0.7 (both the data dictionary and the binaries).
    The response from raford suggests that the optimizer is causing the text index to be called in "functional lookup" mode. I tried testing it, forcing it to use an index. The query ran in 3.5 minutes. Then I tested the original query and received the same results. After checking with the other DBAs, one of them was updating statistics on partitions for previous days' data. The statistics were only being run on current data partitions for the current day, but apparently the method that the application uses to "update" data (which could be for previous days) is to delete it and then insert it. I am suspecting that this is the root cause of all our evil! We are currently working with the developers to get this design changed. In the meantime, I REALLY appreciate all the help/suggestions. If we see the problem occur again, we will add the hint to test and verify whether or not that is the cause. I suspect it will be. Thank you raford.
    Darn that optimizer!!!! ;-)
    Oh, just a hint: Using set autotrace traceonly explain will NOT show that each of the text index partitions is being scanned. This was what we were using for our explain plans. But, when another query, with a similar problem, was finally run, we used EM and the "Long Ops" tab to finally see that each partition was being scanned for the token, whether it needed it or not. I think I'll go back to Tom Kyte's website and look for that article on the set autotrace traceonly explain not giving a totally accurate explain plan.
    Thanks again.
    - Jenny

  • Slow extraction from Source System

    Experts,
    I'm on the basis team and we're trying to figure out what we can do to increase the performance of a BI extraction from an ECC source system.
    Our BI team is doing a Full extraction of Data Source 0UC_SALES_STATS_02 which has about 24 million records.  This has been running for about 2.5 days and has only extracted 12.5 million records according to RSMO
    I'm no expert on BI performance, but I'm trying to get there.  One thing we noticed is that we have over 1000 Data Packages and each has about 11K records per package.
    RSBIW "control params for data transfer" in the BI system is set as this:
    LOGSYS     --  Max KB.  --  MAX Lines  --  Freq  --  MAx PROC  -- Target Sys  -- Max DPs
    OURBWSYS --  20000 --          0             --  10    --        3            --     blank        --   blank
    OURECCSYS --  20000 --          0             --  10    --        3            --     blank        --   blank
    Also we only see one background process running on the ECC system which is the Zprogram that was written to perform the extract.
    We also checked IMG - sap netweaver - BI - links to other source systems - maintain contrl params for data xfer
    Freq/IDOC = 10
    pkg size = 50,000
    partition size = 1,000,000
    We are on NWEHP1 for our BI system
    We are on ECC6.0 EHP3sp5 for our ECC system
    We use Oracle 10.2.0.4.0 with it fully tuned
    We also think we have all the correct tuning parameters in our ECC/BI systems.  There are a few tweaks we could make but I don't see any glaring problems.  All our memory parameters seem to be in line with what is recommended.
    We do not see any kind of Memory/CPU bottlenecks.
    Do you think the many data packages and small records per package is a problem? 
    Any tips to increase performance would be helpful, I'm still looking.
    I saw this wiki thread and it mentions to look at tcode RSA3 in ECC system to monitor long extracts.   But I have looked at this and I have not figured out what all needs to go into the fields to correctly run this.
    Any help you can provide would be great.
    Thanks,
    NICK

    HI Nick,
    This problem is due to the huge volumes of data and the job is ruinng from long time onwards, normally we will kill the job if it exceeds more than 8 hours time while extracting the data.
    Please suggest your BI Team to pull data in small volumes by providing selections in info package. This will drastically redcues the time for extraction of huge data and will also not consume all the resources in system.
    Please pass below links to your BI team as this is completely releated to design issue.
    http://help.sap.com/saphelp_nw70/helpdata/en/39/b520d2ba024cfcb542ef455c76b250/frameset.htm
    Type of DSO should be write optimized to increase load performance.
    http://help.sap.com/saphelp_nw70/helpdata/en/34/1c74415cb1137de10000000a155106/frameset.htm
    2. And at your end look for the table spaces of the PSA tables, may be it is execeeding its capacity and make it as automatic. instead of manullay assigning the space at the regular intervals.
    Hope this helps.
    Regards,
    Reddy
    Edited by: Reddybl on Apr 2, 2010 12:23 AM

Maybe you are looking for

  • Error while calling a web service from BCM to CRM

    Dear Experts, I'm calling a CRM web service from BCM via Custom IVR. The web service works fine when run using SOAPUI. But when called from BCM, there is an error in the CEM logs as mentioned below: 11:16:51.515 (11224              ) ERR> Connection/

  • Urgent help needed... PL/SQL Insert statement

    Hi... I need some urgent help on this project. I have a 2 column table with values that need to be inserted into another existing table which has a sequence. This is the 2 column table: FUND YEAR 29587 05 29587 07 Existing table: Name Null? Type LIST

  • Does anyone know how to enable/disable a dropdown menu in Forms at runtime?

    Is there a way how to disable/enable dropdown menu for menus that have multiple sub levels in Oracle Forms? I am trying to use the set_menu_item_property function but this takes 'menu.item' format for the first argument. I am trying to enable/disable

  • Opening Bex analyser on Excel takes long time on some client

    Hi, We have installed BEx analyser on several PC client, on my PC BEx is opening in 2 or 3 seconds, on others it can takes 30 or 40 seconds. We have checked the configuration and the problem is the same on two PC having same parameters. Has anyone se

  • Making a flying 3D Composition match background movement

    Hi! I'm working on a school project and started on a task that actually was to big for me. Therefore, I need help. In AE I have a shot panning the cafeteria at the school, and I've made a  rough 3D Ice-tea. The Ice-tea is supposed to fly through the