SESSION EXPIRED and CPU Utilisation is 100% when called from Pro*C

Question

Dear Colleague
We are having a production system developed using Pro*c and PL/SOL packages in HP Unix.
The system is doing the following,
A file will be decoded using Proc and data will be load into 3 temporary tables (permanent table used for temporary) using SQLLoader. Then it calls a PL/SQL package which will process the data in the temporary tables and will store the summary information in Transaction table and will returns a unique id to the calling Pro*c module. The whole process is completed in a single process/oracle session.
The PL/SQL package, join the 3 Temporary tables and retrieve the data into an Oracle object(Collection) and process it. Then the processing summary will be loaded into Transaction table and return the Primary key.
Now the problem is, when the temporary tables are loaded with more than 200,000 rows, in some cases the system is running for hours before it finish. And during the process the session shows as EXPIRED in the database and CPU utilization for the respective ProcessId in Unix is almost 100%. Then after a while the session comes alive and finishes the process.
And for some other cases, with same row count, the entire process finished in seconds.
If the same process (PL/SQL package) when run directly using a separate session inside the database(and the temporary tables are still available), rather than calling from Pro*c, it finishes in seconds, where it took hours when called from Pro*c.
It will be highly appreciated, if anyone let me know, what actually is wrong. whether it is the resource allocation or should there any known problem in calling huge PL/SQL packages from Pro*C.
we are using Oracle 10g and HP-UX ed42 B.11.31 U System.
regards
Sajid

Answer 1

Dear Colleague
I just want to rephrase the problem again.
We have a Pro*C application. This will read the binary file and dump the data in ascii format to a data file. Then, within from the pro*C, 14 Dynamic tables will be created using EXEC SQL EXECUTE IMMEDIATE statement, each having almost a maximum of 45 columns. Also, created INDEXES to the tables in the same way from Pro*C.
And Control file will be created as follows
OPTIONS (SILENT=(FEEDBACK)) UNRECOVERABLE
LOAD DATA
INTO TABLE FILENAME_BCI_USED
FIELDS  TERMINATED BY '|'
TRAILING NULLCOLSColumns Listed
Then Sql loader will be called using system commands (from pro*C itself) to loead the data into the table, as follows.
   sprintf(syscommand,"sqlldr %s CONTROL=%s DATA=%s log=%s rows=5000 direct=true", glb_connect_string,ctl_file,data_file, log_file );
   ret_value = system(syscommand);data loaded successfully.
Then the Pro*C will call a PL/SQL package for processing the data.
The PL/SQL package, to process the data, dynamically construct the below query and use it for the reference cursor to retrieve data and load into a collection.
v_sql_query := 'SELECT '|| '/*+ index(b '||p_File_Name ||'_1) index(b '||p_File_Name ||'_2) index(s '||p_File_Name ||'_3) index(s '||p_File_Name ||'_4) index(s '||p_File_Name ||'_5) index(s '||p_File_Name ||'_6) index(e '||p_File_Name ||'_7)*/'   || '
         iot_call_record(b.rec_no ,
         b.rec_type ,
         substr(b.field1,1,15) ,
         nvl(substr((select s0.field1 from ' || SU_Table ||
                     ' s0 where s0.rec_no = s.rec_no and s0.sub_rec_type = 203) ,1, 25),
                             substr(b.field2, 1, 25)),
         substr(b.field3,1,25) ,
         b.field4 ,
         b.field5 ,
         s.field1 ,
         s.field2 ,
         (select sum(s1.field1) from ' || SU_Table ||
                     ' s1 where s1.field2 = s.rec_no and trim(s1.field2) = ''00'' ) ,
         s.field3 ,
         e.field2/power(10,e.field3) ,
         s.field4 ,
         s.field5 ,
         s.field6 ,
         s.field7 ,
         s.field8 ,
         s.field9 ,
         s.field10 ,
         nvl(b.field6,''F'') ,
         NULL ,
         s.field11 ,
         (select sum(s2.field11) from ' || SU_Table || ' s2 where s2.rec_no = s.rec_no)  ,
         NULL)
      FROM   ' || BCI_Table || ' b , ' || SU_Table || ' s, ' || EXCH_Table ||
                     ' e WHERE  b.filename = s.filename
        AND    b.rec_no = s.rec_no
        AND    (b.field7 = 0 OR b.field7 = 1)
        AND    TRIM(s.chg_type) = ''00''
        AND    (s.field1 = e.field2_Code )
        AND    not(s.field4 = ''V'' and s.field12 > 1)
        AND    not(s.field4 = ''W'' and s.field12 > 1)
        AND    not(b.rec_type = 75 and s.field12 > 1)
        AND    not(b.rec_type = 75 and s.field4 =''D'')
        AND    s.sub_rec_type <> 203
        and    (s.field12 = 1 or b.rec_type not in (20,30))';
    OPEN cur_call_events FOR v_sql_query;
    LOOP
      g_tab_call_events.DELETE;
      --      Execute immediate v_sql_query bulk collect into g_tab_call_events;
      FETCH cur_call_events BULK COLLECT
        INTO g_tab_call_events limit 5000;
      EXIT WHEN g_tab_call_events.COUNT = 0;
      BEGIN
        SAVEPOINT Block_Begin;
        process_records(p_file_name,
                        g_tab_call_events);
      EXCEPTION
        WHEN Severe_Error THEN
          Write_error('S');
          ROLLBACK TO Block_Begin;
        WHEN Warning THEN
          Write_error('W');
        WHEN NO_DATA_FOUND THEN
          Write_error('S');
          ROLLBACK TO Block_Begin;
      END;
    END LOOP;And the above module is behaving strangely.
With almost 150K or more rows in BCI_Table & SU_Table each and less than 10 rows in EXCH_Table, the application takes more than 100 minutes to complete the process.
When we checked the session activity, it is showing the same query for a very long time.
Where as files with 100K or less rows are getting processed in a minute.
And the performance for 150K+ rows is inconsistent, that when we isolate the package and run it directly calling from oracle, it is getting executed in less than 4 minutes. Whereas it takes 100+ minutes from Pro*C.
The execution plan for the above query is given below
                              Object Owner     Object Name          Cost     Cardinality     Bytes     CPU cost     IO cost
SELECT STATEMENT, GOAL = HINT: FIRST_ROWS                              494     2497          494406     143334576     483
TABLE ACCESS BY INDEX ROWID               SCHEMA1          FILENAME_SU_USED     4     1          21     30610          4
  INDEX RANGE SCAN                    SCHEMA1          FILENAME_3          3     1          21764     3
SORT AGGREGATE                                                  1     13          
  INDEX RANGE SCAN                    SCHEMA1          FILENAME_5          3     1          13     22064          3
SORT AGGREGATE                                                  1     7          
  TABLE ACCESS BY INDEX ROWID               SCHEMA1          FILENAME_SU_USED     4     1          7     30706          4
   INDEX RANGE SCAN                    SCHEMA1          FILENAME_3          3     1          21764     3
CONCATENATION                                   
  HASH JOIN                                                  355     2496          1307904     103068617     347
   TABLE ACCESS FULL                    SCHEMA1          FILENAME_EXCH_USED     3     2          26     35987          3
   HASH JOIN                                                  352     2496          813696     96532618     344
    VIEW                         SYS          VW_NSO_1          4     20          880     12549326     3
     HASH UNIQUE                                             4     20          220     12549326     3
      TABLE ACCESS FULL                    SCHEMA2          IOT_SERVICE_MATRIX     3     20          220     45207          3
    HASH JOIN                                                  347     15850          2234850     76145180     341
     TABLE ACCESS FULL                    SCHEMA1          FILENAME_SU_USED     130     16584          1111128     33625872     127
     TABLE ACCESS FULL                    SCHEMA1          FILENAME_BCI_USED     217     23757          1758018     31405896     214
  NESTED LOOPS                                                  139     1          198     40265959     136
   NESTED LOOPS                                                  135     1          154     27716633     133
    NESTED LOOPS                                             132     1          141     27680646     130
     TABLE ACCESS FULL                    SCHEMA1          FILENAME_SU_USED     129     1          67     27655992     127
     TABLE ACCESS BY INDEX ROWID          SCHEMA1          FILENAME_BCI_USED     3     1          74     24654          3
      INDEX RANGE SCAN                    SCHEMA1          FILENAME_1          2     1          15493     2
    TABLE ACCESS FULL                    SCHEMA1          FILENAME_EXCH_USED     3     1          13     35987          3
   VIEW                              SYS          VW_NSO_1          4     1          44     12549326     3
    SORT UNIQUE                                                  4     20          220     12549326     3
     TABLE ACCESS FULL                    SCHEMA2          IOT_SERVICE_MATRIX     3     20          220     45207          3Regards
Sajid
Edited by: user12039545 on Jul 11, 2010 12:05 AM
Edited by: user12039545 on Jul 11, 2010 12:15 AM
Edited by: user12039545 on Jul 11, 2010 12:32 AM
Edited by: user12039545 on Jul 11, 2010 12:34 AM
Edited by: user12039545 on Jul 11, 2010 12:37 AM