Slow table creation after upgrade from 10.2.0.3 to 11.2.0.1 using DBUA

Question

I've recently completed a database upgrade from 10.2.0.3 to 11.2.0.1 using the DBUA.
I've since encountered a slowdown when running a script which drops and recreates a series of ~250 tables. The script normally runs in around 19 seconds. After the upgrade, the script requires ~2 minutes to run.
By chance has anyone encountered something similar?
The problem may be related to the behavior of an "after CREATE on schema" trigger which grants select privileges to a role through the use of a dbms_job call; between 10g and the database that was upgraded from 10G to 11g. Currently researching this angle.
I will be using the following table creation DDL for this abbreviated test case:
create table ALLIANCE  (
   ALLIANCEID           NUMBER(10)                      not null,
   NAME                 VARCHAR2(40)                    not null,
   CREATION_DATE        DATE,
   constraint PK_ALLIANCE primary key (ALLIANCEID)
           using index
       tablespace LIVE_INDEX
tablespace LIVE_DATA;When calling the above DDL, an "after CREATE on schema" trigger is fired which schedules a job to immediately run to grant select privilege to a role for the table which was just created:
create or replace
trigger select_grant
after CREATE on schema
declare
    l_str varchar2(255);
    l_job number;
begin
    if ( ora_dict_obj_type = 'TABLE' ) then
        l_str := 'execute immediate "grant select on ' ||
                                     ora_dict_obj_name ||
                                    ' to select_role";';
        dbms_job.submit( l_job, replace(l_str,'"','''') );
    end if;
end;
{code}
Below I've included data on two separate test runs.  The first is on the upgraded database and includes optimizer parameters and an abbreviated TKPROF.  I've also, included the offending sys generate SQL which is not issued when the same test is run on a 10g environment that has been set up with a similar test case.  The 10g test run's TKPROF is also included below.
The version of the database is 11.2.0.1.
These are the parameters relevant to the optimizer for the test run on the upgraded 11g SID:
{code}
SQL> show parameter optimizer
NAME                                 TYPE        VALUE
optimizer_capture_sql_plan_baselines boolean     FALSE
optimizer_dynamic_sampling           integer     2
optimizer_features_enable            string      11.2.0.1
optimizer_index_caching              integer     0
optimizer_index_cost_adj             integer     100
optimizer_mode                       string      ALL_ROWS
optimizer_secure_view_merging        boolean     TRUE
optimizer_use_invisible_indexes      boolean     FALSE
optimizer_use_pending_statistics     boolean     FALSE
optimizer_use_sql_plan_baselines     boolean     TRUE
SQL> show parameter db_file_multi
NAME                                 TYPE        VALUE
db_file_multiblock_read_count        integer     8
SQL> show parameter db_block_size
NAME                                 TYPE        VALUE
db_block_size                        integer     8192
SQL> show parameter cursor_sharing
NAME                                 TYPE        VALUE
cursor_sharing                       string      EXACT
SQL> column sname format a20
SQL> column pname format a20
SQL> column pval2 format a20
SQL> select sname, pname, pval1, pval2 from sys.aux_stats$;
SNAME                PNAME                     PVAL1 PVAL2
SYSSTATS_INFO        STATUS                          COMPLETED
SYSSTATS_INFO        DSTART                          03-11-2010 16:33
SYSSTATS_INFO        DSTOP                           03-11-2010 17:03
SYSSTATS_INFO        FLAGS                         0
SYSSTATS_MAIN        CPUSPEEDNW           713.978495
SYSSTATS_MAIN        IOSEEKTIM                    10
SYSSTATS_MAIN        IOTFRSPEED                 4096
SYSSTATS_MAIN        SREADTIM               1565.746
SYSSTATS_MAIN        MREADTIM
SYSSTATS_MAIN        CPUSPEED                   2310
SYSSTATS_MAIN        MBRC
SYSSTATS_MAIN        MAXTHR
SYSSTATS_MAIN        SLAVETHR
13 rows selected.
{code}
Output from TKPROF on the 11g SID:
{code}
create table ALLIANCE  (
   ALLIANCEID           NUMBER(10)                      not null,
   NAME                 VARCHAR2(40)                    not null,
   CREATION_DATE        DATE,
   constraint PK_ALLIANCE primary key (ALLIANCEID)
           using index
       tablespace LIVE_INDEX
tablespace LIVE_DATA
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          4           0
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.00       0.00          0          0          4           0
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 324
{code}
... large section omitted ...
Here is the performance hit portion of the TKPROF on the 11g SID:
{code}
SQL ID: fsbqktj5vw6n9
Plan Hash: 1443566277
select next_run_date, obj#, run_job, sch_job
from
(select decode(bitand(a.flags, 16384), 0, a.next_run_date,
  a.last_enabled_time) next_run_date,       a.obj# obj#,
  decode(bitand(a.flags, 16384), 0, 0, 1) run_job, a.sch_job  sch_job  from
  (select p.obj# obj#, p.flags flags, p.next_run_date next_run_date,
  p.job_status job_status, p.class_oid class_oid,      p.last_enabled_time
  last_enabled_time, p.instance_id instance_id,      1 sch_job   from
  sys.scheduler$_job p   where bitand(p.job_status, 3) = 1    and
  ((bitand(p.flags, 134217728 + 268435456) = 0) or
  (bitand(p.job_status, 1024) <> 0))    and bitand(p.flags, 4096) = 0    and
  p.instance_id is NULL    and (p.class_oid is null      or (p.class_oid is
  not null      and p.class_oid in (select b.obj# from sys.scheduler$_class b
                           where b.affinity is null)))   UNION ALL   select
  q.obj#, q.flags, q.next_run_date, q.job_status, q.class_oid,
  q.last_enabled_time, q.instance_id, 1   from sys.scheduler$_lightweight_job
  q   where bitand(q.job_status, 3) = 1    and ((bitand(q.flags, 134217728 +
  268435456) = 0) or         (bitand(q.job_status, 1024) <> 0))    and
  bitand(q.flags, 4096) = 0    and q.instance_id is NULL    and (q.class_oid
  is null      or (q.class_oid is not null      and q.class_oid in (select
  c.obj# from sys.scheduler$_class c                          where
  c.affinity is null)))   UNION ALL   select j.job, 0,
  from_tz(cast(j.next_date as timestamp),      to_char(systimestamp,'TZH:TZM')
  ), 1, NULL,      from_tz(cast(j.next_date as timestamp),
  to_char(systimestamp,'TZH:TZM')),     NULL, 0   from sys.job$ j   where
  (j.field1 is null or j.field1 = 0)    and j.this_date is null) a   order by
  1)   where rownum = 1
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        1      0.47       0.47          0       9384          0           1
total        3      0.48       0.48          0       9384          0           1
Misses in library cache during parse: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      1  COUNT STOPKEY (cr=9384 pr=0 pw=0 time=0 us)
      1   VIEW  (cr=9384 pr=0 pw=0 time=0 us cost=5344 size=6615380 card=194570)
      1    SORT ORDER BY STOPKEY (cr=9384 pr=0 pw=0 time=0 us cost=5344 size=11479630 card=194570)
194790     VIEW  (cr=9384 pr=0 pw=0 time=537269 us cost=2563 size=11479630 card=194570)
194790      UNION-ALL  (cr=9384 pr=0 pw=0 time=439235 us)
    231       FILTER  (cr=68 pr=0 pw=0 time=920 us)
    231        TABLE ACCESS FULL SCHEDULER$_JOB (cr=66 pr=0 pw=0 time=690 us cost=19 size=13157 card=223)
      1        TABLE ACCESS BY INDEX ROWID SCHEDULER$_CLASS (cr=2 pr=0 pw=0 time=0 us cost=1 size=40 card=1)
      1         INDEX UNIQUE SCAN SCHEDULER$_CLASS_PK (cr=1 pr=0 pw=0 time=0 us cost=0 size=0 card=1)(object id 5056)
      0       FILTER  (cr=3 pr=0 pw=0 time=0 us)
      0        TABLE ACCESS FULL SCHEDULER$_LIGHTWEIGHT_JOB (cr=3 pr=0 pw=0 time=0 us cost=2 size=95 card=1)
      0        TABLE ACCESS BY INDEX ROWID SCHEDULER$_CLASS (cr=0 pr=0 pw=0 time=0 us cost=1 size=40 card=1)
      0         INDEX UNIQUE SCAN SCHEDULER$_CLASS_PK (cr=0 pr=0 pw=0 time=0 us cost=0 size=0 card=1)(object id 5056)
194559       TABLE ACCESS FULL JOB$ (cr=9313 pr=0 pw=0 time=167294 us cost=2542 size=2529254 card=194558)
{code}
and the totals at the end of the TKPROF on the 11g SID:
{code}
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          4           0
Fetch        0      0.00       0.00          0          0          0           0
total        3      0.00       0.00          0          0          4           0
Misses in library cache during parse: 1
Misses in library cache during execute: 1
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse       70      0.00       0.00          0          0          0           0
Execute     85      0.01       0.01          0         62        208          37
Fetch       49      0.48       0.49          0       9490          0          35
total      204      0.51       0.51          0       9552        208          72
Misses in library cache during parse: 5
Misses in library cache during execute: 3
   35  user  SQL statements in session.
   53  internal SQL statements in session.
   88  SQL statements in session.
Trace file: 11gSID_ora_17721.trc
Trace file compatibility: 11.1.0.7
Sort options: default
       1  session in tracefile.
      35  user  SQL statements in trace file.
      53  internal SQL statements in trace file.
      88  SQL statements in trace file.
      51  unique SQL statements in trace file.
    1590  lines in trace file.
      18  elapsed seconds in trace file.
{code}
The version of the database is 10.2.0.3.0.
These are the parameters relevant to the optimizer for the test run on the 10g SID:
{code}
SQL> show parameter optimizer
NAME                                 TYPE        VALUE
optimizer_dynamic_sampling           integer     2
optimizer_features_enable            string      10.2.0.3
optimizer_index_caching              integer     0
optimizer_index_cost_adj             integer     100
optimizer_mode                       string      ALL_ROWS
optimizer_secure_view_merging        boolean     TRUE
SQL> show parameter db_file_multi
NAME                                 TYPE        VALUE
db_file_multiblock_read_count        integer     8
SQL> show parameter db_block_size
NAME                                 TYPE        VALUE
db_block_size                        integer     8192
SQL> show parameter cursor_sharing
NAME                                 TYPE        VALUE
cursor_sharing                       string      EXACT
SQL> column sname format a20
SQL> column pname format a20
SQL> column pval2 format a20
SQL> select sname, pname, pval1, pval2 from sys.aux_stats$;
SNAME                PNAME                     PVAL1 PVAL2
SYSSTATS_INFO        STATUS                          COMPLETED
SYSSTATS_INFO        DSTART                          09-24-2007 11:09
SYSSTATS_INFO        DSTOP                           09-24-2007 11:09
SYSSTATS_INFO        FLAGS                         1
SYSSTATS_MAIN        CPUSPEEDNW           2110.16949
SYSSTATS_MAIN        IOSEEKTIM                    10
SYSSTATS_MAIN        IOTFRSPEED                 4096
SYSSTATS_MAIN        SREADTIM
SYSSTATS_MAIN        MREADTIM
SYSSTATS_MAIN        CPUSPEED
SYSSTATS_MAIN        MBRC
SYSSTATS_MAIN        MAXTHR
SYSSTATS_MAIN        SLAVETHR
13 rows selected.
{code}
Now for the TKPROF of a mirrored test environment running on a 10G SID:
{code}
create table ALLIANCE  (
   ALLIANCEID           NUMBER(10)                      not null,
   NAME                 VARCHAR2(40)                    not null,
   CREATION_DATE        DATE,
   constraint PK_ALLIANCE primary key (ALLIANCEID)
           using index
       tablespace LIVE_INDEX
tablespace LIVE_DATA
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.01          0          2         16           0
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.01       0.01          0          2         16           0
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 113
{code}
... large section omitted ...
Totals for the TKPROF on the 10g SID:
{code}
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.02          0          0          0           0
Execute      1      0.00       0.00          0          2         16           0
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.00       0.02          0          2         16           0
Misses in library cache during parse: 1
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse       65      0.01       0.01          0          1         32           0
Execute     84      0.04       0.09         20         90        272          35
Fetch       88      0.00       0.10         30        281          0          64
total      237      0.07       0.21         50        372        304          99
Misses in library cache during parse: 38
Misses in library cache during execute: 32
   10  user  SQL statements in session.
   76  internal SQL statements in session.
   86  SQL statements in session.
Trace file: 10gSID_ora_32003.trc
Trace file compatibility: 10.01.00
Sort options: default
       1  session in tracefile.
      10  user  SQL statements in trace file.
      76  internal SQL statements in trace file.
      86  SQL statements in trace file.
      43  unique SQL statements in trace file.
     949  lines in trace file.
       0  elapsed seconds in trace file.
{code}
Edited by: user8598842 on Mar 11, 2010 5:08 PM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

Answer 1

So while this certainly isn't the most elegant of solutions, and most assuredly isn't in the realm of supported by Oracle...
I've used the DBMS_IJOB.DROP_USER_JOBS('username'); package to remove the 194558 orphaned job entries from the job$ table. Don't ask, I've no clue how they all got there; but I've prepared some evil looks to unleash upon certain developers tomorrow morning.
Not being able to reorganize the JOB$ table to free the now wasted ~67MB of space I've opted to create a new index on the JOB$ table to sidestep the full table scan.
CREATE INDEX SYS.JOB_F1_THIS_NEXT ON SYS.JOB$ (FIELD1, THIS_DATE, NEXT_DATE) TABLESPACE SYSTEM;The next option would be to try to find a way to grant the select privilege to the role without using the aforementioned "after CREATE on schema" trigger and dbms_job call. This method was adopted to cover situations in which a developer manually added a table directly to the database rather than using the provided scripts to recreate their test environment.
I assume that the following quote from the 11gR2 documentation is mistaken, and there is no such beast as "create or replace table" in 11g:
http://download.oracle.com/docs/cd/E11882_01/server.112/e10592/statements_9003.htm#i2061306
"Dropping a table invalidates dependent objects and removes object privileges on the table. If you want to re-create the table, then you must regrant object privileges on the table, re-create the indexes, integrity constraints, and triggers for the table, and respecify its storage parameters. Truncating and replacing have none of these effects. Therefore, removing rows with the TRUNCATE statement or replacing the table with a *CREATE OR REPLACE TABLE* statement can be more efficient than dropping and re-creating a table."