Commit operations in pipelined functions
Hi forum,
How to manage commit operations (eg: suppose I have 377 million records to load and I want to commit b/w any 1000 rows) fetched based on a pipelined function.How it is possible.?
Thanks ,
Aneesh A
Everything you said is true, except possibly the bit about more undo being used via the
loop, and from what I've seen of your other posts you may be right about that too.Well more undo is used.
From the demonstration in the link I posted.
The PL/SQL loop used 30 times more undo (rollback).
So you would have to commit at least thirty times over a million rows just to stay at the amount of undo that would have been used overall if no attempt at periodic commits had been made. There is no avoiding using more undo overall.
I'm all for the fewest commits possible - 1 is best. But its unpleasant to wait hours
when rollback/undo fill up first for the process to proceed to the error point,
then double the time for all the data to roll back.
In situations like this - only - are the periodic commits useful.But without the code to allow intermediate commits the process was 20 times faster, and the rollback was 200 times faster.
Not coincidentally, sites where periodic commits are needed are often the ones where the
DBA group can't/won't increase the rollback segments/undo eliminating the best option.Agreed except that they are used as a result but rarely needed.
Similar Messages
-
Pipelined function ignores DML changes on subqueries
Hello all,
I have a really specific issue when using a pipelined function used in a complex subquery where the function ignores the changes made on the current transaction. The problem is the hidden hint materialize sometimes used by the Oracle optimizer. I say sometimes because it depends mostly on the execution plan and the complexity of the query.
I can repeat the problem with a dummy scenario.
Let's say we have a dummy table with a simple record :
CREATE TABLE DUMMY ("NAME" VARCHAR2(50 BYTE));
INSERT INTO DUMMY VALUES('Original name');
We then create a package which will contain our pipelined function and its record object and collection:
CREATE OR REPLACE PACKAGE PKG_DUMMY AS
TYPE DUMMY_RECORD IS RECORD (NAME VARCHAR2(50 BYTE));
TYPE DUMMY_RECORDS IS TABLE OF DUMMY_RECORD;
FUNCTION FUNC_GET_DUMMY_NAME RETURN DUMMY_RECORDS PIPELINED;
END PKG_DUMMY;
CREATE OR REPLACE
PACKAGE BODY PKG_DUMMY AS
FUNCTION FUNC_GET_DUMMY_NAME RETURN DUMMY_RECORDS PIPELINED AS
BEGIN
FOR CUR IN ( SELECT * FROM DUMMY )
LOOP
PIPE ROW (CUR);
END LOOP;
END FUNC_GET_DUMMY_NAME;
END PKG_DUMMY;
With this SQL query, we can return the value of the table by the pipelined function :
WITH DUMMY_NAME AS
SELECT "NAME"
FROM TABLE(PKG_DUMMY.FUNC_GET_DUMMY_NAME())
SELECT "NAME"
FROM DUMMY_NAME
Result
Original name
If we modify the DUMMY table with a new name without a commit, and re-execute the query, we got the same result :
UPDATE DUMMY SET "NAME" = 'New name';
Result
New name
But if we add the materialize hint in the subquery (without doing a commit or rollback), we have the original value hence my issue :
WITH DUMMY_NAME AS
SELECT /*+ materialize */ "NAME"
FROM TABLE(PKG_DUMMY.FUNC_GET_DUMMY_NAME())
SELECT "NAME"
FROM DUMMY_NAME
Result
Original name
I know I can force my subquery to use an inline hint instead of the "materialize" hint chose by the optimizer but then the query lose a lot of performance. Is there a way to force Oracle to use current DML changes with the materialize hint on a pipelined funtion in a subquery?
This thread is also for this issue : http://stackoverflow.com/questions/1597467/is-using-a-select-inside-a-pipelined-pl-sql-table-function-allowedHi Eliante, Hi Dominic,
Very Interesting. Here what I can reproduce in Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
sql > truncate table dummy;
Table truncated.
sql >INSERT INTO DUMMY VALUES('Original name');
1 row created.Please pay attention that I didn't commit
sql > with dummy_name as
2 (
3 select "NAME"
4 from table(pkg_dummy.func_get_dummy_name())
5 )
6 select "NAME"
7 from dummy_name;
NAME
Original name
sql> start c:\dispcursor
PLAN_TABLE_OUTPUT
SQL_ID 838mtur4m74j2, child number 0
with dummy_name as ( select "NAME" from table(pkg_dummy.func_get_dummy_name()) ) select "NAME"
from dummy_name
Plan hash value: 117055
| Id | Operation | Name | Starts | A-Rows | A-Time | Buffers |
| 1 | COLLECTION ITERATOR PICKLER FETCH| FUNC_GET_DUMMY_NAME | 1 | 1 |00:00:00.01 | 15 |
Note
- rule based optimizer used (consider using cbo)
17 rows selected.
sql > with dummy_name as
2 (
3 select /*+ materialize */ "NAME"
4 from table(pkg_dummy.func_get_dummy_name())
5 )
6 select "NAME"
7 from dummy_name;
no rows selected
sql >start c:\dispcursor
PLAN_TABLE_OUTPUT
SQL_ID 9frx3wjk992rd, child number 0
with dummy_name as ( select /*+ materialize */ "NAME" from table(pkg_dummy.func_get_dummy_name()) ) select "NAME" from dummy_name
Plan hash value: 1359790764
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem |
| 1 | TEMP TABLE TRANSFORMATION | | 1 | | 0 |00:00:00.01 | 20 | | | |
| 2 | LOAD AS SELECT | | 1 | | 0 |00:00:00.01 | 19 | 1024 | 1024 | |
| 3 | COLLECTION ITERATOR PICKLER FETCH| FUNC_GET_DUMMY_NAME | 1 | | 0 |00:00:00.01 | 17 | | | |
| 4 | VIEW | | 1 | 8168 | 0 |00:00:00.01 | 0 | | | |
| 5 | TABLE ACCESS FULL | SYS_TEMP_0FD9D780C_BD7649E3 | 1 | 8168 | 0 |00:00:00.01 | 0 | | | |
16 rows selected.I can point out that the TABLE ACCESS FULL of the global temporary SYS_TEMP_0FD9D780C_BD7649E3 table created by Oracle in response to the -materialize hint is returning *0 rows* in operation 5.
Why?
It seems for me that the reason for that comes from the fact that the creation of this SYS_TEMP_0FD9D780C_BD7649E3 table is done via direct path read/direct path write and as far as
the insert of *'Original name'* has not been pushed yet into the disc then materializing the query will generate an empty temporary table (empty in this case).
This is why if I had committed I will not have seen such a kind of discrepancy between those two queries
What do you think?
Mohamed Houri
www.hourim.wordpress.com -
Use of Pipelined Function in Forms10g(10.1.0.2)...
Hi ,
I test the routines found in http://sheikyerbouti.developpez.com/recordset/record_set.htm
I have a problem in the sample routine in Forms10g regarding pipelined function.
Whereas , the whole routine database part and client side in SQL*PLUS works fine.... the client side on Forms10g does not.......
I used two versions in when-button-pressed trigger:
1)Declare
cur SYS_REFCURSOR ;
begin
Open cur for SELECT * FROM TABLE(Pkg_Cur_.Get_Raws2(CURSOR(SELECT * FROM EMP WHERE DEPTNO = 10)));
end;
The error message in compilation time is..
Error 606: The subquery cursor is invalid in the client side
and the second version is:
2)Declare
tab Pkg_Cur_.REC_EMP_;
begin
tab:=Pkg_Cur_.GET_RAWS2('SELECT * FROM EMP WHERE DEPTNO = 10');
end;whereas the database package is defined as:
CREATE OR REPLACE PACKAGE Pkg_Cur_
IS
TYPE REC_EMP_ IS TABLE OF EMP%ROWTYPE INDEX BY BINARY_INTEGER ;
FUNCTION Get_Raws RETURN REC_EMP_ ;
FUNCTION Get_Raws2 ( cur_lig IN SYS_REFCURSOR )
RETURN TYPE_TAB_REC_EMP_ PIPELINED ;
END Pkg_Cur_ ;and the database package body is as follows:
CREATE OR REPLACE PACKAGE BODY Pkg_Cur_
IS
FUNCTION Get_Raws RETURN REC_EMP_
IS
TAB REC_EMP_;
BEGIN
SELECT * BULK COLLECT INTO TAB FROM EMP ;
RETURN TAB ;
END ;
FUNCTION Get_Raws2 ( cur_lig IN SYS_REFCURSOR )
RETURN TYPE_TAB_REC_EMP_ PIPELINED
IS
Trec TYPE_REC_EMP_ := TYPE_REC_EMP_(NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL ) ;
Remp EMP%ROWTYPE ;
BEGIN
LOOP
FETCH cur_lig INTO Remp ;
EXIT WHEN cur_lig%NOTFOUND ;
-- Handling datas --
Trec.EMPNO := Remp.EMPNO ;
Trec.ENAME := Remp.ENAME ;
Trec.JOB := Remp.JOB ;
Trec.MGR := Remp.MGR ;
Trec.HIREDATE := Remp.HIREDATE ;
Trec.SAL := Remp.SAL * 1.1 ;
Trec.COMM := Remp.COMM ;
Trec.DEPTNO := Remp.DEPTNO ;
-- Return value --
PIPE ROW( Trec ) ;
END LOOP ;
RETURN ;
END ;
END Pkg_Cur_ ;
So , how the form trigger should be written...????
If somebody could help me i would be grateful...!!!!
Thanks a lot
SimonIn the second anonymous block you're trying to populate a record with a table. You would need to select from the function and either bulk collect into a table variable or fetch into a record variable in a loop. Bulk collect is also not supported on the client side.
I've been looking at pipelined function possibilities in another thread:
table functions
The problem you have is actually with the Cursor functions, not the Pipelined functions. -
PLS-00630: pipelined functions must have a supported collection return type
Hello, I created an TYPE of OBJECT and a PLSQL Function as shown below, but the function compilation errors with following. Not sure where is the issue?
PLS-00630: pipelined functions must have a supported collection return typeThis is on Oracle 10g r2
CREATE OR REPLACE TYPE cxs_plsql_profiler_object_type AS OBJECT (
cxs_object_name VARCHAR2 (128),
cxs_object_type VARCHAR2 (19),
cxs_object_status VARCHAR2 (7),
cxs_read_execution NUMBER,
cxs_buffer_gets NUMBER,
cxs_disk_reads NUMBER,
cxs_executions NUMBER,
cxs_sorts NUMBER,
cxs_sharable_mem NUMBER,
cxs_address NUMBER,
cxs_hashvalue NUMBER,
cxs_osuser VARCHAR2 (30),
cxs_username VARCHAR2 (30),
cxs_module VARCHAR2 (48),
cxs_machine VARCHAR2 (64),
cxs_status VARCHAR2 (8),
cxs_terminal VARCHAR2 (16),
cxs_percentconsume NUMBER,
cxs_percentrepeat NUMBER,
cxs_plan VARCHAR2 (120),
target_name VARCHAR2 (200),
referenced_name VARCHAR2 (200),
referenced_type VARCHAR2 (200),
targetowner VARCHAR2 (200),
refowner VARCHAR2 (200)
)and here is the API
FUNCTION CXS_GENERATE_PLSQL_PROFILER
RETURN cxs_plsql_profiler_object_type
PIPELINED IS
out_rec cxs_plsql_profiler_object_type ;
plsbatch plsql_batch;
skount integer;
dpendrec depend_tab;
dkount integer;
CURSOR objects
IS
SELECT object_name, object_type
FROM dba_objects
WHERE status = 'VALID'
AND owner NOT IN ('SYS', 'SYSTEM')
AND object_type IN ('PACKAGE', 'PROCEDURE', 'FUNCTION');
CURSOR apis (p_object dba_objects.object_name%TYPE)
IS
SELECT DISTINCT *
FROM (SELECT SUBSTR (a.sql_text, 1, 50) sql_text,
TRUNC
( a.disk_reads
/ DECODE (a.executions,
0, 1,
a.executions
) reads_per_execution,
a.buffer_gets, a.disk_reads, a.executions,
a.sorts, a.sharable_mem, a.address,
a.hash_value, b.osuser, b.username,
b.module, b.machine, b.status, b.terminal,
ROUND
(cxs_db_info.kompute_percentofsql
(a.sharable_mem),
5
) percentkonsume,
cxs_db_info.kount_repeat
(b.osuser,
b.terminal
) percentr,
c.operation explainplan
FROM v$sqlarea a, v$session b, v$sql_plan c
WHERE b.sql_hash_value = a.hash_value
AND b.sql_address = a.address
AND a.hash_value = c.hash_value
AND a.address = c.address
AND b.status = 'ACTIVE'
AND UPPER (a.sql_text) LIKE
'%' || p_object || '%'
AND c.ID = 0
ORDER BY 2 DESC)
WHERE ROWNUM <= 50; --profile option
BEGIN
skount := 0;
dkount := 0;
FOR i IN objects
LOOP
FOR j IN apis (i.object_name)
LOOP
skount := skount + 1;
plsbatch(skount).cxs_object_name := i.object_name;
plsbatch(skount).cxs_object_type := i.object_type;
plsbatch(skount).cxs_object_status := i.object_status;
plsbatch(skount).cxs_read_execution := j.reads_per_execution;
plsbatch(skount).cxs_buffer_gets := j.buffer_gets;
plsbatch(skount).cxs_disk_reads := j.disk_reads;
plsbatch(skount).cxs_executions := j.executions;
plsbatch(skount).cxs_sorts := j.sorts;
plsbatch(skount).cxs_sharable_mem := j.sharable_mem;
plsbatch(skount).cxs_address := j.address;
plsbatch(skount).cxs_hashvalue := j.hashvalue;
plsbatch(skount).cxs_osuser := j.osuser;
plsbatch(skount).cxs_username := j.username;
plsbatch(skount).cxs_module := j.module;
plsbatch(skount).cxs_machine := j.machine;
plsbatch(skount).cxs_status := j.status;
plsbatch(skount).cxs_terminal := j.terminal;
plsbatch(skount).cxs_percentconsume := j.percentconsume;
plsbatch(skount).cxs_percentrepeat := j.percentrepeat;
plsbatch(skount).cxs_plan := j.explainplan;
END LOOP;
FOR dd IN dpend (i.object_name)
LOOP
dkount := dkount + 1;
dependrec (dkount).target_name := dd.NAME;
dependrec (dkount).refname := dd.referenced_name;
dependrec (dkount).reftype := dd.referenced_type;
dependrec (dkount).target_owner := dd.owner;
dependrec (dkount).refowner := dd.referenced_owner;
END LOOP;
END LOOP;
for a in 1..skount loop
out_rec.cxs_object_type := plsbatch(a).object_type;
out_rec.cxs_object_status := plsbatch(a).object_status;
out_rec.cxs_read_execution := plsbatch(a).reads_per_execution;
out_rec.cxs_buffer_gets := plsbatch(a).buffer_gets;
out_rec.cxs_disk_reads := plsbatch(a).disk_reads;
out_rec.cxs_executions := plsbatch(a).executions;
out_rec.cxs_sorts := plsbatch(a).sorts;
out_rec.cxs_sharable_mem := plsbatch(a).sharable_mem;
out_rec.cxs_address := plsbatch(a).address;
out_rec.cxs_hashvalue := plsbatch(a).hashvalue;
out_rec.cxs_osuser := plsbatch(a).osuser;
out_rec.cxs_username := plsbatch(a).username;
out_rec.cxs_module := plsbatch(a).module;
out_rec.cxs_machine := plsbatch(a).machine;
out_rec.cxs_status := plsbatch(a).status;
out_rec.cxs_terminal := plsbatch(a).terminal;
out_rec.cxs_percentconsume := plsbatch(a).percentconsume;
out_rec.cxs_percentrepeat := plsbatch(a).percentrepeat;
out_rec.cxs_plan := plsbatch(a).explainplan;
PIPE ROW(out_rec);
end loop;
for b in 1..dkount loop
out_rec.target_name := dd.NAME;
out_rec.refname := dependrec (b).referenced_name;
out_rec.reftype := dependrec (b).referenced_type;
out_rec.target_owner := dependrec (b).owner;
out_rec.refowner := dependrec (b).referenced_owner;
PIPE ROW(out_rec);
end loop;
RETURN;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE(DBMS_UTILITY.format_error_backtrace);
DBMS_OUTPUT.PUT_LINE(SQLCODE);
DBMS_OUTPUT.PUT_LINE(SQLERRM);
END; and below are tradtional table types that are used in code above.
TYPE type_plsql_rec IS RECORD (
cxs_object_name VARCHAR2 (128),
cxs_object_type VARCHAR2 (19),
cxs_object_status VARCHAR2 (7),
cxs_read_execution NUMBER,
cxs_buffer_gets NUMBER,
cxs_disk_reads NUMBER,
cxs_executions NUMBER,
cxs_sorts NUMBER,
cxs_sharable_mem NUMBER,
cxs_address NUMBER,
cxs_hashvalue NUMBER,
cxs_osuser VARCHAR2 (30),
cxs_username VARCHAR2 (30),
cxs_module VARCHAR2 (48),
cxs_machine VARCHAR2 (64),
cxs_status VARCHAR2 (8),
cxs_terminal VARCHAR2 (16),
cxs_percentconsume NUMBER,
cxs_percentrepeat NUMBER,
cxs_plan VARCHAR2 (120)
TYPE plsql_batch IS TABLE OF type_plsql_rec
INDEX BY BINARY_INTEGER;
TYPE type_depend_tab IS RECORD (
target_name dba_dependencies.NAME%TYPE,
refname dba_dependencies.referenced_name%TYPE,
reftype dba_dependencies.referenced_type%TYPE,
target_owner dba_dependencies.owner%TYPE,
refowner dba_dependencies.referenced_owner%TYPE
TYPE depend_tab IS TABLE OF type_depend_tab
INDEX BY BINARY_INTEGER;
Thank you for your time in reading this post
RThank you Billy and Saubhik,
I have followed your guidelines and was able to resolve this error. Now, after successfully compiling the code, I attempted to execute it in a following way.
SELECT * FROM TABLE (cxs_generate_plsql_profiler);It gives following error: ORA-00904: "CXS_GENERATE_PLSQL_PROFILER": invalid identifier
I also tried putting in quotes like below
SELECT * FROM TABLE ('cxs_generate_plsql_profiler');Then, it gives following error:
ORA-22905: cannot access rows from a non-nested table item
Any Idea where I am doing wrong?
Thanks,
R -
How to write DML operation in a function
Hi
Its very urgent for me.
I am writing DML operation directly in a function and is being called from select statement, it is getting error as "DML Operations cannot be performed inside a query".
How to write a DML operation inside a function.
My objective is to call that function from select statement.
Please help me out.
ThankdNo no no. You're committing after each row! soany
other session running the same query will see the
changes you're making. Your session will equallysee
changes caused by running this query in those
sessions.Other session, yes, but current session will only see
the changes once it has completed the current
statement. Otherwise my "rn" column would not have
gone up sequentially in the above example. it would
have gone
1st row rn = 1 (all rows get updated by 1:-
2,3,4,5,6,7,8,9,10,11)
2nd row rn = 3 (all rows get update by 1:-
3,4,5,6,7,8,9,10,11,12)
3rd row rn = 5 (all rows get update by 1:-
4,5,6,7,8,9,10,11,12,13)
4th row rn = 7 (all rows get update by 1:-
5,6,7,8,9,10,11,12,13,14)
5th row rn = 9 (all rows get update by 1:-
6,7,8,9,10,11,12,13,14,15)
6th row rn = 11 (all rows get update by 1:-
7,8,9,10,11,12,13,14,15,16)
7th row rn = 13 (all rows get update by 1:-
8,9,10,11,12,13,14,15,16,17)
8th row rn = 15 (all rows get update by 1:-
9,10,11,12,13,14,15,16,17,18)
9th row rn = 17 (all rows get update by 1:-
10,11,12,13,14,15,16,17,18,19)
10th row rn = 19 (all rows get update by 1:-
11,12,13,14,15,16,17,18,19,20)
So the fact the commit happens each time the rows get
updated, isn't effecting the currently running select
statement.
No, actually you DO see the other session changes. This is because it is AUTONOMOUS transaction, and this a function.
Test by adding:
create or replace function incvals return number as
pragma autonomous_transaction;
v_val number;
begin
update t set rn = rn + 1;
select max(rn) into v_val from t;
dbms_lock.sleep(1); --add this line
commit;
return v_val;
end;
And test in two sessions.
You will NOT get sequential ascending.
>
Think about the effect of two parallel sessionsboth
running this query at the same time, and ask isthis
sensible?Gawd, no, of course not. Like I said, I'd never use
this sort of thing myself. I'm just wondering what
on earth the OP is trying to achieve.
:)Glad to hear it. -
DML Operations in Stored Function
Hi,
I have used Update Statement in a function. Which is giving an errror (ORA-14551: cannot perform a DML operation inside a query).
Can I use DML operations in a stored function ?
(I need help on locking master/transaction tables, i.e if a one user locks a master another user should not get modify access to the master/transactions).
Thanks
Ramesh GanjiSomeone who obviously din't read my previous post in this thread. You should pay attention, programming is all about the details.
ORA-14551: cannot perform a DML operation inside a query).Then it obviously does more than just "returns a Table Type object". Why are you doing DML in a function?
PLS-00653: aggregate/table functions are not allowed in PL/SQL scopeWe can only call pipelined functions from SQL queries. So you'll have to ditch the DML or make it an autonomous transaction. Be very careful if you adopt the latter approach.
Cheers, APC -
What exactly is parallelised in pipelined functions?
Hi All,
I have a slight confusion about pipelined table functions, which are parallelised (with PARALLEL_ENABLE clause).
Everywhere on the internet/documents it is mentioned that, we can use PARALLEL_ENABLE with pipelined table functions, I am not clear about exactly how this will work?
For example, I have a pipelines function, which is called by C# code to write data in file ( Oracle v11.2 on Linux 64bit). Function looks something like
FUNCTION extract_data (param1..., param2...., param3.... )
RETURN data_tab /* this data_tab is table of record type data_rec */
PIPELINED
IS
v_data_rec data_rec := data_rec ( null, null,....., null);
BEGIN
FOR i_rec IN ( select col1, col2, col3,.... FROM table1, table2 WHERE <all typical stuff here> )
LOOP
v_data_rec.col1 := i_rec.col1 ;
v_data_rec.col2 := i_rec.col2 ;
v_data_rec.col3 := i_rec.col3 ;
if v_data_rec.col1 = 'A' then
select col into v_data_rec.col4 from tablex where .... ;
elsif v_data_rec.col1 = 'B' then
select col into v_data_rec.col4 from tabley where .... ;
elsif v_data_rec.col1 = 'C' then
select col into v_data_rec.col4 from tablez where .... ;
elsif v_data_rec.col1 = 'D' then
select col into v_data_rec.col4 from tablet where .... ;
end if ;
/* some more bits of complex logic here */
PIPE ROW ( v_data_rec ) ;
END LOOP ;
END extract_data ;Now, if I add PARALLEL_ENABLE to my function definition, which bits will be parallelised? will anything be parallelised ?
Thoughts please.
Thanks in advance.i.e. this ( where the query might well be run in parallel, but the result processed serially by the pipelined function):
SQL> explain plan for
2 select *
3 from table(
4 f_serial_pipe
5 (cursor(select /*+ parallel (t1 4) */
6 *
7 from t1)));
Explained.
SQL>
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
Plan hash value: 1425181138
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 8168 | 1100K| 34 (3)| 00:00:01 |
| 1 | VIEW | | 8168 | 1100K| 34 (3)| 00:00:01 |
| 2 | COLLECTION ITERATOR PICKLER FETCH| F_SERIAL_PIPE | 8168 | | 34 (3)| 00:00:01 |
Note
- dynamic sampling used for this statement (level=4)
13 rows selected.
SQL> is different from thsi:
SQL> explain plan for
2 select *
3 from table(
4 f_parallel_pipe
5 (cursor(select /*+ parallel (t1 4) */
6 *
7 from t1)));
Explained.
SQL>
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
Plan hash value: 2029647211
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
| 0 | SELECT STATEMENT | | 8168 | 1100K| 34 (3)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 8168 | 1100K| 34 (3)| 00:00:01 | Q1,00 | P->S | QC (RAND) |
| 3 | VIEW | | 8168 | 1100K| 34 (3)| 00:00:01 | Q1,00 | PCWP | |
| 4 | COLLECTION ITERATOR PICKLER FETCH| F_PARALLEL_PIPE | 8168 | | 34 (3)| 00:00:01 | Q1,00 | PCWP | |
| 5 | PX BLOCK ITERATOR | | 798K| 97M| 2659 (1)| 00:00:02 | Q1,00 | PCWC | |
| 6 | TABLE ACCESS FULL | T1 | 798K| 97M| 2659 (1)| 00:00:02 | Q1,00 | PCWP | |
Note
- dynamic sampling used for this statement (level=4)
17 rows selected.
SQL> -
Pipelined function with huge volume
Hi all,
I have a table of 5 million rows with an average length of 1K for each row (dss system).
My SGA is 2G and PGA 1G.
I wonder if a pipelined function could support a such volume ?
Does anyone have already experienced a pipelined function with a huge volume ?
TIA
YangHello
Well, just over a month later and we're pretty much sorted. Our pipelined functions were not the cause of the excessive memory consumption and the processes are are now no longer consuming as much PGA as they were previously. Here's what I've learnt.
1. Direct write operations on partitioned tables require two direct write buffers to be allocated per partition. By default, these buffers are 256K each so it's 512K per partition. We had a table with 241 partitions which meant we inadvertently allocating 120MB of PGA without even trying. This is not a problem with pipelined functions.
2. In 10.2 the total size of the buffers will be kept below the pga_aggregate_target, or 64k per buffer, whichever is higher. This is next to useless though as to really constrain the size of the buffers at all, you need a ridiculously small pga_aggregate_target.
3. The size of the buffers can be as low as 32k and can be set using an undocumented parameter "_ldr_io_size". In our environment (10.2.0.2 Win2003 using AWE) I've set it to 32k meaning there will be 64k worth of buffers allocated to each partition significantly reducing the amount of PGA required.
4. I never want to speak to Oracle support again. We had a severity 1 SR open for over a month and it took the development team 18 days to get round to looking at the test case I supplied. Once they'd looked at it, they came back with the undocumented parameter which worked, and the ridiculous suggestion that I set the PGA aggregate target to 50MB on a production box with 4GB and 300+ dedicated connections. No only that, they told me that a pga_aggregate_target of 50MB was sensible and did so in the most patronising way. Muppets.
So in sum, our pipelined functions are working exceptionally well. We had some scary moments as we saw huge amounts of PGA being allocated and then 4030 memory errors but now it's sorted and chugging along nicely. The throughput is blistering especially when running in parallel - 200m rows generated and inserted in around 1 hour.
To give some background on what we're using pipelined functions for....
We have a list of master records that have schedules associated with them. Those schedules need to be exploded out to an hourly level and customised calendars need to be applied to them along with custom time-zone-style calculations. There are various lookups that need to be applied to the exploded schedules and a number of value calculations based on various rules. I did originally implement this in straight SQL but it was monsterous and ran like a dog. The SQL was highly complex and quite quickly became unmanageable. I decided to use pipelined functions because
a) It immensely simplified the logic
b) It gave us a very neat way to centralise the logic so it can be easily used by other systems - PL/SQL and SQL
c) We can easily see what it is doing and make changes to the logic without affecting execution plans etc
d) Its been exceptionally easy to tune using DBMS_PROFILER
So that's that. I hope it's of use to anyone who's interested.
I'm off to get a "pipelined fuinctions rule" tattoo on my face.
David -
Pipelined function in package example 11g
Hi all,
could you please show me example of using pipelined function in package with my sample data?
My need is to return the following data using pipelined function in package:
select 1 as t, 2 as y, 'u' as j, trunc(sysdate-1) as k from dual union all
select 3 as t, 4 as y, 'h' as j, trunc(sysdate-2) as k from dual Thanks ahead.p.s. if your problem is that you're not sure how to pipe multiple columns, take a look at this example (from my library of standard examples)...
SQL> CREATE OR REPLACE TYPE myemp AS OBJECT
2 ( empno number,
3 ename varchar2(10),
4 job varchar2(10),
5 mgr number,
6 hiredate date,
7 sal number,
8 comm number,
9 deptno number
10 )
11 /
Type created.
SQL> CREATE OR REPLACE TYPE myrectable AS TABLE OF myemp
2 /
Type created.
SQL> CREATE OR REPLACE FUNCTION pipedata(p_min_row number, p_max_row number) RETURN myrectable PIPELINED IS
2 v_obj myemp := myemp(NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3 BEGIN
4 FOR e IN (select *
5 from (
6 select e.*
7 ,rownum rn
8 from (select * from emp order by empno) e
9 )
10 where rn between p_min_row and p_max_row)
11 LOOP
12 v_obj.empno := e.empno;
13 v_obj.ename := e.ename;
14 v_obj.job := e.job;
15 v_obj.mgr := e.mgr;
16 v_obj.hiredate := e.hiredate;
17 v_obj.sal := e.sal;
18 v_obj.comm := e.comm;
19 v_obj.deptno := e.deptno;
20 PIPE ROW (v_obj);
21 END LOOP;
22 RETURN;
23 END;
24 /
Function created.
SQL> select * from table(pipedata(1,5));
EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO
7369 SMITH CLERK 7902 17-DEC-1980 00:00:00 800 20
7499 ALLEN SALESMAN 7698 20-FEB-1981 00:00:00 1600 300 30
7521 WARD SALESMAN 7698 22-FEB-1981 00:00:00 1250 500 30
7566 JONES MANAGER 7839 02-APR-1981 00:00:00 2975 20
7654 MARTIN SALESMAN 7698 28-SEP-1981 00:00:00 1250 1400 30
SQL> select * from table(pipedata(6,10));
EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO
7698 BLAKE MANAGER 7839 01-MAY-1981 00:00:00 2850 30
7782 CLARK MANAGER 7839 09-JUN-1981 00:00:00 2450 10
7788 SCOTT ANALYST 7566 19-APR-1987 00:00:00 3000 20
7839 KING PRESIDENT 17-NOV-1981 00:00:00 5000 10
7844 TURNER SALESMAN 7698 08-SEP-1981 00:00:00 1500 0 30
SQL> select * from table(pipedata(11,15));
EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO
7876 ADAMS CLERK 7788 23-MAY-1987 00:00:00 1100 20
7900 JAMES CLERK 7698 03-DEC-1981 00:00:00 950 30
7902 FORD ANALYST 7566 03-DEC-1981 00:00:00 3000 20
7934 MILLER CLERK 7782 23-JAN-1982 00:00:00 1300 10
SQL> -
Pipelined function..please help
hi all...
i need a fuction which will split the data n insert into the table...
i created a fuction like this :
CREATE OR REPLACE FUNCTION FN_SPLIT(text IN VARCHAR2 DEFAULT NULL,delimiter IN VARCHAR2 DEFAULT ' ')
RETURN SPLIT_TYPE_TABLE PIPELINED
IS
PRAGMA AUTONOMOUS_TRANSACTION;
TYPE ref0 is REF CURSOR;
myCursor ref0;
out_rec SPLIT_TYPE := SPLIT_TYPE(null, null);
--CURSOR MSTFRC IS SELECT * FROM TEMP_SPLIT;
index_ NUMBER(10,0);
BEGIN
BEGIN
/*index_ := -1;
cnt:=0;
l_str:=text;
loop
l_n := instr( l_str, delimiter );
exit when (nvl(l_n,0) = 0);
cnt:=cnt+1;
INSERT INTO TEMP_SPLIT (ID,NAME)
VALUES (cnt,ltrim(rtrim(substr(l_str,1,l_n-1))));
l_str := substr( l_str, l_n+1 );
end loop;*/
Load_Temp_Splitting(text,delimiter);
open myCursor for select * from temp_split;
LOOP FETCH myCursor into out_rec.ID,out_rec.NAME;
EXIT WHEN myCursor%NOTFOUND;
PIPE ROW(out_rec);
END LOOP;
CLOSE myCursor;
RETURN;
END;
END FN_SPLIT;
it created succesfully without errors but when i run this function it showing an error like cannot evaluate pipelined function..
my requirement is to split the data like 'as,af,er,yt' split this by comma n insert into the table with row id like
1 as
2 af
3 er
like...
please help friends....
thnks in advance...
lol
Neethu> when i run this function it showing an error like cannot evaluate pipelined
function.
And that is one of the most important pieces of the puzzle - the actual error displayed. What is it? (full error, i.e. number + message)
As for the code in the function. That looks a bit like an ugly hack to me. Why the INSERT into table? Why not simply use a local collection or array?
And why a pipelined table function specifically?
The following code demonstrates a string split function:
SQL> create or replace type TStrings as table of varchar2(4000);
2 /
Type created.
SQL>
SQL> create or replace function tokenise( cString varchar2, cSeparator varchar2 DEFAULT ',' ) return TStrings AUTHID CURRENT_USER is
2 strList TStrings;
3 str varchar2(4000);
4 i integer;
5 l integer;
6
7 procedure AddString( cLine varchar2 ) is
8 begin
9 strList.Extend(1);
10 strList( strList.Count ) := cLine;
11 end;
12
13 begin
14 strList := new TStrings();
15
16 str := cString;
17 loop
18 l := LENGTH( str );
19 i := INSTR( str, cSeparator );
20
21 if i = 0 then
22 AddString( str );
23 else
24 AddString( SUBSTR( str, 1, i-1 ) );
25 str := SUBSTR( str, i+1 );
26 end if;
27
28 -- if the separator was on the last char of the line, there is
29 -- a trailing null column which we need to add manually
30 if i = l then
31 AddString( null );
32 end if;
33
34 exit when str is NULL;
35 exit when i = 0;
36 end loop;
37
38 return( strList );
39 end;
40 /
Function created.
SQL>
SQL> select tokenise( 'as,af,er,yt' ) from dual;
TOKENISE('AS,AF,ER,YT')
TSTRINGS('as', 'af', 'er', 'yt')
SQL>
SQL> select * from TABLE( tokenise( 'as,af,er,yt' ) );
COLUMN_VALUE
as
af
er
yt
SQL> -
Pipeline function raised ORA-06519: active autonomous transaction detected
Hi All,
My name is John and I've got a problem which I need to share with all of you guru and experts. I've created the following pipeline function under the Oracle user ABC:
CREATE OR replace FUNCTION SomeFunction(p_from_date DATE, p_to_date DATE) RETURN T_TAB_A pipelined
IS
PRAGMA autonomous_transaction;
BEGIN
DELETE FROM temp_rcm;
INSERT INTO temp_rcm
SELECT * FROM int.facility fd,
int.capacity co
WHERE co.resource_name = fd.resource_name
AND co.trade_date = fd.trade_date
AND co.trade_date BETWEEN p_from_date AND p_to_date;
COMMIT;
FOR rec IN (SELECT co.*
FROM temp_rcm co
left join int.outage o
ON ( o.flag = 'Y'
AND o.reason_flag = 'F'
AND o.INTERVAL = co.INTERVAL
AND co.resource_name = o.resource_name )
ORDER BY co.INTERVAL,
co.name) LOOP
pipe ROW (T_A( rec.INTERVAL, rec.trade_date,
rec.resource_name,rec.day_of_week_long, rec.working_day, rec.peak));
END LOOP;
RETURN;
END SomeFunction;
I was able to compile and create the SomeFunction function successfully but when I executed it using the following command:
select * from table(SomeFunction(to_date('01/01/2010',to_date('01/01/2010')));
I was returned with the Oracle error - ORA-06519: active autonomous transaction detected and rolled back
I have searched through the web, such Oracle error occurs whenever the function has a missing 'COMMIT' or 'ROLLBACK' command inside an autonomous_transaction. But the fact is I have already included the 'COMMIT;' in the function. I suspected that the error was caused by the tables which I queried against (like int.facility and int.capacity) were all views that belonged to another schema called int. Or is that something that I miss in the function? Thank you for your time and assistance.
Regards,
Johnjohnwanng wrote:
Hi Guys,
Thank you for all your feedback. In addition to your reply, Bill, can you spare some time and provide us a simple example of the steps involved to implement the 'correct' implementation based on the queries that I've used. As I do not understand your vanilla approach. Much appreciated and thank you for the time again.
Regards,
JohnIf I had to guess, Billy may have meant something like this (untested):
CREATE OR REPLACE FUNCTION SomeFunction
( p_from_date IN int.facility.trade_date%TYPE
, p_to_date IN int.facility.trade_date%TYPE
RETURN SYS_REFCURSOR
AS
rcur SYS_REFCURSOR;
BEGIN
OPEN rcur FOR
SELECT co.interval
, co.trade_date
, co.resource_name
, co.day_of_week_long
, co.working_day
, co.peak
FROM int.capacity co
JOIN int.facility fd ON fd.resource_name = co.resource_name
AND fd.trade_date = co.trade_date
LEFT OUTER JOIN int.outage o ON o.interval = co.interval
AND o.resource_name = co.resource_name
WHERE co.trade_date BETWEEN p_from_date AND p_to_date
AND o.reason_flag = 'F'
AND o.flag = 'Y'
ORDER BY co.interval
, co.name
RETURN rcur;
END;
/I made the following modifications:
1. I set the input parameter data types to match that of the table column you are checking against. A good practice to get into.
2. Removed the autonomous transaction and inserting into a temp table. In Oracle it's a good practice to perform everything in a single SQL statement if possible.
3. Changed the return data type to a SYS_REFCURSOR
Hope this helps and provides a good example. -
Execute pipelined function within other function .
Hi,
I'm on 9.2.0.8 , and got GUI application which can accept select only statements but I need kind of insert functionality.
So I've come up with clever idea and used pipelined functions with autonymous transaction to do inserts.
All looks like:
select * from table(opened_cursors_list)
select * from table(pgastat_list)
select * from table(sqlcosts_list)
all I need now is running this 3 function 1 after other in such way I can use select run_all_three() from dual;
or other way but called via select so my application gui can swallow it .
I've tried:
CREATE OR REPLACE Function gather_test_data
RETURN number
IS
cnumber number := 1;
BEGIN
select * from table(opened_cursors_list);
select * from table(pgastat_list);
select * from table(sqlcosts_list);
RETURN cnumber;
EXCEPTION
WHEN OTHERS THEN
raise_application_error(-20001,'An error was encountered - '||SQLCODE||' -ERROR- '||SQLERRM);
END;but got error 6/2 PLS-00428: an INTO clause is expected in this SELECT statement
So look like I need some fake I dont know how its called 'collection' to deal with that .
I know thats whole think is kind of workaround but its working with that gui application :).
Any ideas how to call that pipelined functions ?
Regards.
GregGWell I did pipelined functions like that:
CREATE OR REPLACE function opened_cursors_list
return OPENED_CURSORS_table_type
pipelined
as
PRAGMA AUTONOMOUS_TRANSACTION;
begin
for l_rec in (select sysdate datetime, st.value , se.program, se.MODULE , se.username
from v$sesstat st, v$session se
where
st.statistic# = 3
and st.sid = se.sid
and value > 100
order by value desc
LOOP
PIPE ROW (OPENED_CURSORS_scalar_type(l_rec.datetime,
l_rec.VALUE ,
l_rec.PROGRAM ,
l_rec.MODULE ,
l_rec.USERNAME ));
insert into opened_cursors values(l_rec.datetime,
l_rec.VALUE ,
l_rec.PROGRAM ,
l_rec.MODULE ,
l_rec.USERNAME);
commit;
END LOOP;
RETURN;
END;
/And its working, but as You said I think I need some dummy record to select into it.
It doesn't really matters what kind of dummy record. Just to fullfil pl/sql requirements .
So how to define such dummy record to select into it from pipelined function ?
Regards.
Greg
Edited by: GregG on Apr 5, 2011 12:40 PM -
oracle db version 9.2.0.7
create or replace TYPE ENAME_T AS OBJECT (name varchar2(10));
create or replace TYPE ENAME_NT AS TABLE OF ename_t;
CREATE OR REPLACE FUNCTION GET_ENAME (comm IN number)
RETURN ename_nt PIPELINED
IS
v_comm number := comm;
v_ename_t ename_t;
BEGIN
for v_ename in (select ename from emp where comm > v_comm) loop
v_ename_t := ename_t(v_ename);
pipe row (v_ename_t);
end loop;
return;
END;
Error(10,18): PLS-00306: wrong number or types of arguments in call to 'ENAME_T'
I do not see why I am getting this message. What can I do to can the results of the sql "select ename from emp where comm > v_comm" in a pipeline function
thankstry
v_ename_t := ename_t(v_ename.ename);
should work -
Interactive Report using a View with a Pipelined Function
Hello fellow Apex people,
I'm Using Application Express 4.1.0.00.32
I've got an interactive report that references a view (STOCK) and a pipelined function
The basic query is listed below.
SELECT S.CHANGED_TIME "Changed Time"
, S.CHANGED_BY "Changed By"
, S.ID "Id"
, STKST_DESCRS.STOCK_STATUS_CODES "Stock Status Codes"
, STKST_DESCRS.STOCK_STATUS_DESCRS "Stock Status"
, S.ORIGINAL_CONTAINER "Original Container"
FROM STOCK S
, table(LWS_StkstStatus (S.ID)) STKST_DESCRS
ORDER BY S.CO_ID,
S.SEQUENCE_NUM;
When the page is first run all the data is displayed correctly,
If I define a filter, sort or a blank search the data from the pipelined function (STKST_DESCRS.) becomes null and isn't displayed.
Does anyone know what is happening?
Many ThanksI'm curious why you find this dangerous. I want a report that looks like this:
Opportunity X:
4 - 2-apr-2008 - Closed deal
3 - 1-mar-2008 - Called Joe again
2 - 12-feb-2008 - Called Joe
1 - 14-jan-2008 - Initial call with customer.
When you enter a new note, I want it to be numbered 5. The only problem I can imagine is someone deleting a note, which will almost never happen, and if it does, it just leaves a numbering gap. I don't see how using the function in a SELECT will accomplish this. -
Hi!
I have a problem with List View Report in mobile application (theme 50 in apex) after updating to apex 4.2.2. I created Report -> List View. I used select from pipelined function in Region Source. Then when page is running and submited three times (or refreshed three times) I get an error:
Error during rendering of region "LIST VIEW".
ORA-01007: variable not in select list
Technical Info (only visible for developers)
is_internal_error: true
apex_error_code: APEX.REGION.UNHANDLED_ERROR
ora_sqlcode: -1007
ora_sqlerrm: ORA-01007: variable not in select list
component.type: APEX_APPLICATION_PAGE_REGIONS
component.id: 21230833903737364557
component.name: LIST VIEW
error_backtrace:
ORA-06512: at "APEX_040200.WWV_FLOW_DISP_PAGE_PLUGS", line 4613
ORA-06512: at "APEX_040200.WWV_FLOW_DISP_PAGE_PLUGS", line 3220
I get this error only when I use select from pipelined function in Region Source (for example: "select value1, value2 from table(some_pipelined_function(param1, param2)) ").
You can check it on http://apex.oracle.com/pls/apex/f?p=50591 (login - demo, password - demo).
In this application:
- I created package TAB_TYPES_PKG:
create or replace PACKAGE TAB_TYPES_PKG IS
TYPE cur_rest_r IS RECORD (
STR_NAME VARCHAR2(128),
INFO VARCHAR2(128)
TYPE cur_rest_t IS TABLE OF cur_rest_r;
END TAB_TYPES_PKG;
- I created pipelined function TEST_FUNC:
create or replace
FUNCTION TEST_FUNC
RETURN TAB_TYPES_PKG.cur_rest_t PIPELINED IS
r_cur_rest TAB_TYPES_PKG.cur_rest_r;
BEGIN
r_cur_rest.STR_NAME := 'ROW 1';
r_cur_rest.INFO := '10';
PIPE ROW (r_cur_rest);
r_cur_rest.STR_NAME := 'ROW 2';
r_cur_rest.INFO := '20';
PIPE ROW (r_cur_rest);
r_cur_rest.STR_NAME := 'ROW 3';
r_cur_rest.INFO := '30';
PIPE ROW (r_cur_rest);
r_cur_rest.STR_NAME := 'ROW 4';
r_cur_rest.INFO := '40';
PIPE ROW (r_cur_rest);
r_cur_rest.STR_NAME := 'ROW 5';
r_cur_rest.INFO := '50';
PIPE ROW (r_cur_rest);
RETURN;
END TEST_FUNC;
- I created List View Report on Page 1:
Region Source:
SELECT str_name,
info
FROM TABLE (TEST_FUNC)
We can see error ORA-01007 after refresing (or submiting) Page 1 three times or more.
How to fix it?Hi all
I'm experiencing the same issue. Predictably on every third refresh I receive:
Error
Error during rendering of region "Results".
ORA-01007: variable not in select list
Technical Info (only visible for developers)
is_internal_error: true
apex_error_code: APEX.REGION.UNHANDLED_ERROR
ora_sqlcode: -1007
ora_sqlerrm: ORA-01007: variable not in select list
component.type: APEX_APPLICATION_PAGE_REGIONS
component.id: 6910805644140264
component.name: Results
error_backtrace: ORA-06512: at "APEX_040200.WWV_FLOW_DISP_PAGE_PLUGS", line 4613 ORA-06512: at "APEX_040200.WWV_FLOW_DISP_PAGE_PLUGS", line 3220
OK
I am running Application Express 4.2.2.00.11 on GlassFish 4 using Apex Listener 2.0.3.221.10.13.
Please note: this works perfectly using a classic report in my desktop application; however, no joy on the mobile side with a list view. I will use a classic report in the interim.
My region source is as follows:
SELECT description AS "DESCRIPTION", reference AS "REFERENCE" FROM TABLE(AUTOCOMPLETE_LIST_VIEW_FNC('RESULTS'))
The procedure:
FUNCTION AUTOCOMPLETE_LIST_VIEW_FNC(
p_collection_name IN VARCHAR2)
RETURN list_row_table_type
AS
v_tab list_row_table_type := list_row_table_type();
BEGIN
DECLARE
jsonarray json_list;
jsonobj json;
json_clob CLOB;
BEGIN
SELECT clob001
INTO json_clob
FROM apex_collections
WHERE collection_name = p_collection_name;
jsonobj := json(json_clob);
jsonarray := json_ext.get_json_list(jsonobj, 'predictions');
FOR i IN 1..jsonArray.count
LOOP
jsonobj := json(jsonArray.get(i));
v_tab.extend;
v_tab(v_tab.LAST) := list_row_type(json_ext.get_string(jsonobj, 'description'), json_ext.get_string(jsonobj, 'reference'));
END LOOP;
RETURN(v_tab);
END;
END AUTOCOMPLETE_LIST_VIEW_FNC;
Thanks!
Tim
Maybe you are looking for
-
Multiple instances of JDeveloper on Same Machine (JDev 11.1.2.3 on Windows)
Hello: I would like to setup two completely seperate instances of JDeveloper. So I installed JDev#1 in a seperate directory than JDev#2. Each seems to work ok. I would like to keep these two JDev installs completely seperate. JDev#1 I plan to modify
-
Data comparison between 2 models
hi, we've 2 models - consolidation and planning in our BPC 10.0 environment. we consolidate actuals and send data to planning using BADI embedded logic script. Some times, we are seeing differences between CONSOLIDATION model and PLANNING model data.
-
Oracle ADF for weblogic server
Hi, Can I extend weblogic domain to include Oracle ADF my version of weblogic is 10.3.0.0 Thanks, Charan
-
How much data does forwarding use?
Hi All, Just curious if people know how much data forwarding uses? I'm traveling to Europe next week and need to have my phone available until I meet someone, but then I want to forward it to my home number for the rest of the trip. I know forwarding
-
Cannot connect to the net using my wi fi on my blackberry curve
hi there, i cannot connect to the net using my wi fi on my blackberry curve 8520. wen i go in to settings and click on to manage connections it as a tick at the side of wi fi and says TALKTALK- 3B0BD4 and a tick at the end. if i turn my mobile netwo