INSERT SQL Hangs sporadically

Question

Oracle Version:
Oracle Database 11g Release 11.2.0.1.0 - 64bit Production
OS Version:
Red Hat Enterprise Linux Server release 6.4
When this code runs successfully, it does so in 3-5 minutes:
Set Echo off
Set Head off
Set Space 0
Set Feedback ON
Set Pagesize 0
Set Termout OFF
Set Trimspool On
Set Linesize 200
Set Serveroutput On
set autotrace on
ALTER SESSION SET nls_date_format='yyyymmdd';
spool como.rs_create_kenshoo APPEND
truncate table DFS_DATA_A.WEB_KENSHOO_MDA;
INSERT INTO DFS_DATA_A.WEB_KENSHOO_MDA
select
       ken.ORDER_NUM,
       o.ORDER_DATE,
       o.DIVISION_CODE,
       CONV_DATE,
       CONV_TIME,
       CONV_DATETIME,
       KEYWORD,
       CHANNEL,
       CAMPAIGN,
       AD_GROUP,
       SEARCH_TERM,
       TYPE,
       DURATION,
       REVENUE,
       FILE_DATE,
       TO_DBASE_DATE,
       UPD_DBASE_DATE,
       SITE_ID
  from
      (select * from
        (select
               FILE_DATE,
               ORDER_NUM,
               T_CONV_DATE CONV_DATE,
               T_CONV_TIME CONV_TIME,
               T_CONV_DATETIME CONV_DATETIME,
               KEYWORD,
               CHANNEL,
               CAMPAIGN,
               AD_GROUP,
               SEARCH_TERM,
               TYPE,
               DURATION,
               T_REVENUE REVENUE,
               TO_DBASE_DATE,
               UPD_DBASE_DATE,
               SITE_ID,
               row_number () over (partition by order_num,t_conv_datetime,keyword,channel,campaign,ad_group,t_revenue,site_id order by file_date desc) rnum
          from
               DFS_STAGE.web_kenshoo_history
             ) ken_tmp
        where rnum = 1) ken
     left outer join
       (select order_num,
               order_date,
               division_code
          from orders where order_date >= '20080101'
        ) o
    on ken.order_num = o.order_num
spool off
exit
I am running it right now, and it hangs. I use the following to identify progress:
select
         case when v.piece = 0 then s.sql_exec_start else null end sql_exec_start,
         case when v.piece = 0 then s.osuser else null end osuser,
         case when v.piece = 0 then s.username else null end username,
         case when v.piece = 0 then s.sid else null end sid,
         case when v.piece = 0 then s.serial# else null end serial#,
         v.sql_text
from    gv$session s
          inner join
         gv$process p
          on p.addr = s.paddr
          and p.inst_id = s.inst_id
          left outer join
        sys.v_$sqltext v
          on s.sql_address = v.address
where    case when v.piece is null then 0 else 1 end = 1
        and to_char(sql_exec_start, 'YYYY-MM-DD HH24:MI:SS') < to_char(current_date - (1/24/60/60*1), 'YYYY-MM-DD HH24:MI:SS')
order by s.sql_exec_start,
         sid,
         s.serial#,
         v.piece
accept p_sid prompt 'Enter SID:'
accept p_serial prompt 'Enter Serial#:'
select   l.opname,
         l.target,
         to_char((1 - (l.totalwork - l.sofar) / totalwork) * 100, 9999.99) || '%' pct_complete,
         l.start_time,
         l.last_update_time,
         to_char(l.time_remaining / 60, '999,999,999') min_remain,
         to_char(l.elapsed_seconds / 60, '999,999,999') min_elapsd
from     v$session_longops l
where    l.sid = &p_sid
         and l.serial# = &p_serial
order by l.start_time,
        l.last_update_time
It read the table DFS_STAGE.web_kenshoo_history for about 30 minutes (100% according to my 2nd query), now no longer shows up in v$session_longops. I have verified there are no locks with:
SELECT
s1.username || '@' || s1.machine ||
' ( SID=' || s1.sid || ' )  is blocking '
|| s2.username || '@' || s2.machine || ' ( SID=' || s2.sid || ' ) ' AS
blocking_status
FROM v$lock l1, v$session s1, v$lock l2, v$session s2
WHERE
s1.sid=l1.sid and s2.sid=l2.sid
AND l1.BLOCK=1 and l2.request > 0
AND l1.id1 = l2.id1
AND l2.id2 = l2.id2
System load is not too high:
11:28:53 up 22 days,  7:04,  1 user,  load average: 3.00, 3.01, 2.92
Nothing out of the ordinary in the alert.log.
I am stumped. Any suggestions would be appreciated.

Answer 1

Oracle Version:
Oracle Database 11g Release 11.2.0.1.0 - 64bit Production
OS Version:
Red Hat Enterprise Linux Server release 6.4
When this code runs successfully, it does so in 3-5 minutes:
Set Echo off
Set Head off
Set Space 0
Set Feedback ON
Set Pagesize 0
Set Termout OFF
Set Trimspool On
Set Linesize 200
Set Serveroutput On
set autotrace on
ALTER SESSION SET nls_date_format='yyyymmdd';
spool como.rs_create_kenshoo APPEND
truncate table DFS_DATA_A.WEB_KENSHOO_MDA;
INSERT INTO DFS_DATA_A.WEB_KENSHOO_MDA
select
       ken.ORDER_NUM,
       o.ORDER_DATE,
       o.DIVISION_CODE,
       CONV_DATE,
       CONV_TIME,
       CONV_DATETIME,
       KEYWORD,
       CHANNEL,
       CAMPAIGN,
       AD_GROUP,
       SEARCH_TERM,
       TYPE,
       DURATION,
       REVENUE,
       FILE_DATE,
       TO_DBASE_DATE,
       UPD_DBASE_DATE,
       SITE_ID
  from
      (select * from
        (select
               FILE_DATE,
               ORDER_NUM,
               T_CONV_DATE CONV_DATE,
               T_CONV_TIME CONV_TIME,
               T_CONV_DATETIME CONV_DATETIME,
               KEYWORD,
               CHANNEL,
               CAMPAIGN,
               AD_GROUP,
               SEARCH_TERM,
               TYPE,
               DURATION,
               T_REVENUE REVENUE,
               TO_DBASE_DATE,
               UPD_DBASE_DATE,
               SITE_ID,
               row_number () over (partition by order_num,t_conv_datetime,keyword,channel,campaign,ad_group,t_revenue,site_id order by file_date desc) rnum
          from
               DFS_STAGE.web_kenshoo_history
             ) ken_tmp
        where rnum = 1) ken
     left outer join
       (select order_num,
               order_date,
               division_code
          from orders where order_date >= '20080101'
        ) o
    on ken.order_num = o.order_num
spool off
exit
I am running it right now, and it hangs. I use the following to identify progress:
select
         case when v.piece = 0 then s.sql_exec_start else null end sql_exec_start,
         case when v.piece = 0 then s.osuser else null end osuser,
         case when v.piece = 0 then s.username else null end username,
         case when v.piece = 0 then s.sid else null end sid,
         case when v.piece = 0 then s.serial# else null end serial#,
         v.sql_text
from    gv$session s
          inner join
         gv$process p
          on p.addr = s.paddr
          and p.inst_id = s.inst_id
          left outer join
        sys.v_$sqltext v
          on s.sql_address = v.address
where    case when v.piece is null then 0 else 1 end = 1
        and to_char(sql_exec_start, 'YYYY-MM-DD HH24:MI:SS') < to_char(current_date - (1/24/60/60*1), 'YYYY-MM-DD HH24:MI:SS')
order by s.sql_exec_start,
         sid,
         s.serial#,
         v.piece
accept p_sid prompt 'Enter SID:'
accept p_serial prompt 'Enter Serial#:'
select   l.opname,
         l.target,
         to_char((1 - (l.totalwork - l.sofar) / totalwork) * 100, 9999.99) || '%' pct_complete,
         l.start_time,
         l.last_update_time,
         to_char(l.time_remaining / 60, '999,999,999') min_remain,
         to_char(l.elapsed_seconds / 60, '999,999,999') min_elapsd
from     v$session_longops l
where    l.sid = &p_sid
         and l.serial# = &p_serial
order by l.start_time,
        l.last_update_time
It read the table DFS_STAGE.web_kenshoo_history for about 30 minutes (100% according to my 2nd query), now no longer shows up in v$session_longops. I have verified there are no locks with:
SELECT
s1.username || '@' || s1.machine ||
' ( SID=' || s1.sid || ' )  is blocking '
|| s2.username || '@' || s2.machine || ' ( SID=' || s2.sid || ' ) ' AS
blocking_status
FROM v$lock l1, v$session s1, v$lock l2, v$session s2
WHERE
s1.sid=l1.sid and s2.sid=l2.sid
AND l1.BLOCK=1 and l2.request > 0
AND l1.id1 = l2.id1
AND l2.id2 = l2.id2
System load is not too high:
11:28:53 up 22 days,  7:04,  1 user,  load average: 3.00, 3.01, 2.92
Nothing out of the ordinary in the alert.log.
I am stumped. Any suggestions would be appreciated.