SQL XML-Selection-Performance in one big xml-file

Hi!
I'm a xml-db novice and have now my trouble with a 30 MB XML-File like this:
<root>
<element>
<value amount="10.00"/>
</element>
<element>
<value amount="20.00"/>
</element>
12000 elements following
</root>
The xml-file i imported in an xmltype-Object-Table.
Now i wanted to make a checksum over all elements. I took that query i found in OTN:
select sum(to_number(extractValue(value(list_of_values),'//@amount'),'9999999999.99'))
from xml_table t,
table(xmlsequence(extract(value(t),'//root/element/value'))) list_of_values
This Query took a half hour! Should I create an Index? When yes, what type of index? Or is there another way to take the sum of all element-values? I can't live with that performance because i have to take some more checksums.
Bye,
Jens

Similar Messages

Why no more clips and just one big .mov file?

I had been working in iMovie and iDVD 4 for some time up until present when I upgraded to iLife 6. One of the things I like about ver. 4 was that the whole movie was saved as a file but there were also the individual clips saved as well. I could use these clips when creating my iDVD project. Now I see that those clips are gone and there's just one big .mov file to drop into my iDVD themes. Why did they do this??? Is there some benefit to this that I'm not understanding? Maybe I don't have something set correctly in the prefs.? Is there anyway around this so that I have control over what part of a movie gets dropped into my iDVD themes?
-Daryl

Thank you so much, Matt. That did the trick. I would've never thought of that. I was beginning to figure that the clips just had to be somewhere else that wasn't obvious to me. I couldn't believe that they would do away with that. Thanks again!
-Daryl

XML:selecting the value of a xml tag using plsql

Hi,
I have a table xx_table with one of the column xx_column which is of type XMLTYPE.I inserted the xml data into this column with xml data as below.
<userComments xmlns:ns1="http://xmlns.oracle.com/XXXX_AccountDistribution"
xmlns="http://xmlns.oracle.com/XXXXX_AccountDistribution">
<task:userComment
xmlns:task="http://xmlns.oracle.com/bpel/workflow/task">
<task:comment>*********** Rejecting invoice **********</task:comment>
<task:updatedBy>
<task:id>r</task:id>
<task:displayName></task:displayName>
<task:systemVersionFlag></task:systemVersionFlag>
</task:updatedBy>
<task:updatedDate>2010-05-19T14:09:15-07:00</task:updatedDate>
<task:displayNameLanguage></task:displayNameLanguage>
<task:action></task:action>
<task:systemVersionFlag></task:systemVersionFlag>
</task:userComment>
</userComments>
My question ,how do i select the value of the tag "comment" from the above xml data using plsql? Can we write a query to select the value of the tag "comment" to fetch "*********** Rejecting invoice **********" as value.
Regards,
Sandeep

If you have rights to modify tabular model then, you can create a measure in your Tabular model which returns previous week and then use this measure in Pivot Table.
Thanks,
Sagar K
(Blog: http://datamazik.blogspot.in/)

AudioFileOpenURL really bad performance when opening big *.m4a files

I did some testing for a project of mine and noticed that opening a big .m4a file with AudioFileOpenURL is really slow.
I tested opening about 25000 audio files.
All of them, except large mp4 podcasts like the engadget podcast, open really fast.
There seems to be a direct connection between the size of the file and the time it takes for AudioFileOpenURL to open it.
Files that are around 3MB have no noticable delay and files around 30MB will take seconds to open.
While opening AudioFileOpenURL consumes a lot of cpu.
I tried setting the correct fileTypeHint but that makes no difference at all.
Is there any explanation for this behaviour?
I can not continue with my project until i'm able to open all audio files in a reasonable time.
Maybe somebody has an idea how I can speed this up.

Certainly a reasonable concern...
The question at this point is, How are the reports going to be used and deployed once they are in production?
I'd look at it from that direction.
For example... They may be able to dump the data directly to another database on a separate server that would insulate the main enterprise server. This would allow the main server to run the necessary queries during off peak hours and would isolate any reporting activity to a "reporting database".
This would also keep the data secure and encrypted (it would continue to enjoy the security provided by an RDBMS). Text & csv files can be copied, emailed, altered & deleted by anyone who sees them. Placing them in encrypted .zip folders prevents them from being read by external applications.
<Hope you liked the sales pitch I wrote for you to give to the client... =^)
If all else fails and you're stuck using the csv files, at least see if they can get it all on one file. Joining the 2 files is killing you performance wise... More so than using 1 massive file.
Jason

Capturing One Big Video File

I can't seem to find the setting to let me capture one video file in PE7. It seems to create several and all I want is one. Which setting is it?
Thanks,
Brian

I'm with Hunt. I thought if you turned Scene Detection off, it would produce one big file. But I guess I'm assuming you're capturing DV-AVI from tape. If PRE is giving you fits, WinDV is a very nice (and free) capture utility that would produce one, big file for you.
I haven't yet worked (much) with HDD / flash card source assets, but I believe in that case the source would be in different files, no?
Just out of curiosity, why do you want one, big file versus several smaller?

Performance problem submitting big XML string parameter to the web service

We deployed a web service on the OC4J via oracle.j2ee.ws.StatelessJavaRpcWebService . This web service takes one string as a parameter. The string contains XML. Evrything works great UNLESS input XML string reaches 5Mb in size. When it happens OC4J does something with it for about 10 minutes (yes, minutes) before it calls the web service method. At this time java.exe consumes 100% of CPU.
WE tried to increase JVM heap size, stack size, etc, - no effect.
Please, help!
Thank you in advance,
Vlad.

Hi Sheldon,
What i feel is that it's not been handled in your webservice if the parameter is null or "" <blank> space
i just you to take care in webservice that if the parameter is null or "" pass the parameter null to the stored proc
Regards
Pavan

One big audio file

When I sync my iPhone, the memory used changes during sync by about 0.55 GB, every time. The song count changes by 1. So somewhere, I have asong that's 0.55 GB, but nothing that big shows up on my listing. Any suggestions where to look?

Insert the CD.
Don't Import it yet.
Select the CD in iTunes.
Select the consecutive tracks uyou want joined as one file.
Go to menu Advanced > Join CD tracks.
Do this for each set of tracks you want as one file.
Then Import the CD.
If the tracks are already in iTunes, no need to import them again.
Use this -> Join Together
"Join Together automates the process of joining the files of selected iTunes tracks together with QuickTime and exporting them as a single AAC Music or Audiobook file/track. Optionally, you can then create a "chapterized" audio file of the exported AAC file with pointers to the individually joined tracks."

Performance issue with big CSV files as data source

Hi,
We are creating crystal reports for a large banking corporation with CSV files as data source. For some reports, we need join 2 csv files. The problem we met now is that when the 2 csv files are very large (both >200M), the performance is very bad and it takes an hour or so to refresh the data in Crystal Reports designer. The same case for either CR 11.5 or CR 2008.
And my question is, is there any way to improve performance in such situations? For example, can we create index on the csv files? If you have ever created reports connecting to CSV, your suggestions will be highly appreciated.
Thanks,
Ray

Certainly a reasonable concern...
The question at this point is, How are the reports going to be used and deployed once they are in production?
I'd look at it from that direction.
For example... They may be able to dump the data directly to another database on a separate server that would insulate the main enterprise server. This would allow the main server to run the necessary queries during off peak hours and would isolate any reporting activity to a "reporting database".
This would also keep the data secure and encrypted (it would continue to enjoy the security provided by an RDBMS). Text & csv files can be copied, emailed, altered & deleted by anyone who sees them. Placing them in encrypted .zip folders prevents them from being read by external applications.
<Hope you liked the sales pitch I wrote for you to give to the client... =^)
If all else fails and you're stuck using the csv files, at least see if they can get it all on one file. Joining the 2 files is killing you performance wise... More so than using 1 massive file.
Jason

Best practice for optimizing processing of big XMLs?

All,
What is the best practice when dealing with large XML files ... (say couple of MBs).
Instead of having to read the file from the file system everytime a static method is run, what would be the best way in which the program reads the file once and then keeps it in memory. So the next time it would not have to read and parse it all over again?
Currently my code just read the file in the static method like ...
public static String doOperation(String path,...) throws Exception
try{
String masterFile = path+"configfile.xml";
Document theFile = (Document)getDocument(masterFile);
Element root = theFile.getDocumentElement();
NodeList nl = root.getChildNodes();
// ... operations on file
Optimization tips and tricks most appreciated :-)
Thanks,
David

The best practice for multi-megabyte XML files is not to have them at all.
However if you must, presumably you don't need all of the information in your XML, repeatedly. Or do you? If you need a little bit of it here, then another little bit of it there, then yet another little bit of it later, then you shouldn't have stored your data in one big XML.
Sorry if that sounds unhelpful, but I'm having trouble imagining a scenario when you need all the data in an XML document repeatedly. Perhaps you could expand on your design?
PC²

Sql:variable and XML query performance

Can someone help with sql:variable() in xml queries? It seems that when I attempt to reference variables with the sql:variable(...) function in an xpath function (exist or nodes) it comes up with a totally different query plan, possibly ignoring
my secondary indices like the ones for VALUE, PATH.
But if I replace sql:variable("@p_ObjectIdentifierForReference") with the literal (ie. "ord/p/ord0616.p") then it uses secondary indices more consistently.
Below you will see an unsuccessful attempt to get the query to "OPTIMIZE FOR" a specific literal value of @p_ObjectIdentifierForReference. But this doesn't give work. It doesn't give me a plan using the secondary index I expect.
Ideally there would be a way to get the sql:variable(...) function to give the same query plan as a literal. Not sure why that isn't the default behavior.
DECLARE
@p_ObjectIdentifierForReference
varchar(500);
SET
@p_ObjectIdentifierForReference
= 'ord/p/ord0616.p';
WITH
XMLNAMESPACES ('uri:schemas-progress-com:XREFD:0004'
as D)
SELECT
XREF_FileDataReference.XREF_FileData
AS XrefFileData,
InnerRowNode.value('/D:Reference[1]/D:File-num[1]',
'int')
AS FileNumber,
InnerRowNode.value('/D:Reference[1]/D:Line-num[1]',
'int')
AS LineNumber
FROM
(SELECT
XREF.XREF_FileData.XREF_FileData,
XREF.XREF_FileData.XREF_FileEntry,
InnerRow.query('.')
AS InnerRowNode
FROM
XREF.XREF_FileData
OUTER APPLY
DataXref.nodes('/D:Cross-reference/D:Source/D:Reference[@Object-identifier = sql:variable("@p_ObjectIdentifierForReference")
and @Reference-type = "RUN"]')
as T(InnerRow)
WHERE    DataXref.exist('/D:Cross-reference/D:Source/D:Reference[@Object-identifier
= sql:variable("@p_ObjectIdentifierForReference") and @Reference-type = "RUN"]')
= 1)
AS XREF_FileDataReference
INNER
JOIN XREF.XREF_MemberBuilt
ON XREF_MemberBuilt.XREF_FileData
= XREF_FileDataReference.XREF_FileData
INNER
JOIN XREF.XREF_FileEntry
ON XREF_FileEntry.XREF_FileEntry
= XREF_FileDataReference.XREF_FileEntry
WHERE
XREF_MemberBuilt.XREF_ProjectBuilt
= 69
OPTION(RECOMPILE,
OPTIMIZE FOR (@p_ObjectIdentifierForReference
= 'ord/p/ord0616.p')

I tried to create a "repro" of your query so we can work on it and try and improve it, but I got the best results by just adding text() and [1] to it, eg
SELECT
XREF_FileDataReference.XREF_FileData AS XrefFileData,
InnerRowNode.value('(/D:Reference/D:File-num/text())[1]', 'int') AS FileNumber,
InnerRowNode.value('(/D:Reference/D:Line-num/text())[1]', 'int') AS LineNumber
FROM (
In my main repro, even with a large piece of xml with 100,000 elements, there still wasn't much difference between the queries:
USE tempdb
GO
IF NOT EXISTS ( SELECT * FROM sys.schemas WHERE name = 'XREF' )
EXEC( 'CREATE SCHEMA XREF' )
GO
IF OBJECT_ID('XREF.XREF_FileData') IS NOT NULL DROP TABLE XREF.XREF_FileData
CREATE TABLE XREF.XREF_FileData
rowId INT IDENTITY,
DataXref XML,
XREF_FileData INT,
XREF_FileEntry INT,
CONSTRAINT PK_XREF_FileData PRIMARY KEY ( rowId )
GO
IF OBJECT_ID('XREF.XREF_MemberBuilt') IS NOT NULL DROP TABLE XREF.XREF_MemberBuilt
CREATE TABLE XREF.XREF_MemberBuilt
XREF_ProjectBuilt INT,
XREF_FileData INT
GO
IF OBJECT_ID('XREF.XREF_FileEntry') IS NOT NULL DROP TABLE XREF.XREF_FileEntry
CREATE TABLE XREF.XREF_FileEntry
XREF_FileEntry INT
GO
-- Create larger piece of xml for repro
;WITH XMLNAMESPACES ( DEFAULT 'uri:schemas-progress-com:XREFD:0004' ), cte AS (
SELECT TOP 100000 ROW_NUMBER() OVER ( ORDER BY ( SELECT 1 ) ) rn
FROM master.sys.columns c1
CROSS JOIN master.sys.columns c2
CROSS JOIN master.sys.columns c3
INSERT INTO XREF.XREF_FileData ( DataXref, XREF_FileData, XREF_FileEntry )
SELECT
SELECT
CASE rn WHEN 9999 THEN 'ord/p/ord0616.p' ELSE CAST( rn AS VARCHAR(20) ) END AS "@Object-identifier",
'RUN' AS "@Reference-type",
SELECT
rn AS "File-num",
rn * 10 AS "Line-num"
FOR XML PATH(''), TYPE
) AS "*"
FROM cte
FOR XML PATH('Reference'), ROOT('Source'), TYPE
).query('<Cross-reference xmlns="uri:schemas-progress-com:XREFD:0004">{.}</Cross-reference>'), 1, 100
INSERT INTO XREF.XREF_FileEntry ( XREF_FileEntry )
VALUES ( 100 )
INSERT INTO XREF.XREF_MemberBuilt ( XREF_ProjectBuilt, XREF_FileData )
VALUES ( 69, 1 )
GO
--SELECT * FROM XREF.XREF_FileData
--SELECT * FROM XREF.XREF_FileEntry
--SELECT * FROM XREF.XREF_MemberBuilt
--GO
-- Add primary XML index
CREATE PRIMARY XML INDEX xidx_XREF_FileData ON XREF.XREF_FileData (DataXref)
GO
-- Add value, property and path xml indexes
CREATE XML INDEX xvalidx_XREF_FileData ON XREF.XREF_FileData (DataXref)
USING XML INDEX xidx_XREF_FileData FOR VALUE
CREATE XML INDEX xpthidx_XREF_FileData ON XREF.XREF_FileData (DataXref)
USING XML INDEX xidx_XREF_FileData FOR PATH
CREATE XML INDEX xprpidx_XREF_FileData ON XREF.XREF_FileData (DataXref)
USING XML INDEX xidx_XREF_FileData FOR PROPERTY
GO
:exit
DBCC DROPCLEANBUFFERS
DBCC FREEPROCCACHE
GO
DECLARE @p_ObjectIdentifierForReference varchar(500);
SET @p_ObjectIdentifierForReference = 'ord/p/ord0616.p';
;WITH XMLNAMESPACES ('uri:schemas-progress-com:XREFD:0004' as D)
SELECT
XREF_FileDataReference.XREF_FileData AS XrefFileData,
InnerRowNode.value('/D:Reference[1]/D:File-num[1]', 'int') AS FileNumber,
InnerRowNode.value('/D:Reference[1]/D:Line-num[1]', 'int') AS LineNumber
FROM (
SELECT
XREF.XREF_FileData.XREF_FileData,
XREF.XREF_FileData.XREF_FileEntry,
InnerRow.query('.') AS InnerRowNode
FROM XREF.XREF_FileData
OUTER APPLY DataXref.nodes('/D:Cross-reference/D:Source/D:Reference[@Object-identifier = sql:variable("@p_ObjectIdentifierForReference") and @Reference-type = "RUN"]') as T(InnerRow)
WHERE DataXref.exist('/D:Cross-reference/D:Source/D:Reference[@Object-identifier = sql:variable("@p_ObjectIdentifierForReference") and @Reference-type = "RUN"]') = 1
) AS XREF_FileDataReference
INNER JOIN XREF.XREF_MemberBuilt ON XREF_MemberBuilt.XREF_FileData = XREF_FileDataReference.XREF_FileData
INNER JOIN XREF.XREF_FileEntry ON XREF_FileEntry.XREF_FileEntry = XREF_FileDataReference.XREF_FileEntry
WHERE XREF_MemberBuilt.XREF_ProjectBuilt = 69
OPTION( RECOMPILE, OPTIMIZE FOR (@p_ObjectIdentifierForReference = 'ord/p/ord0616.p') )
GO
DBCC DROPCLEANBUFFERS
DBCC FREEPROCCACHE
GO
DECLARE @p_ObjectIdentifierForReference varchar(500);
SET @p_ObjectIdentifierForReference = 'ord/p/ord0616.p';
;WITH XMLNAMESPACES ('uri:schemas-progress-com:XREFD:0004' as D)
SELECT
XREF_FileDataReference.XREF_FileData AS XrefFileData,
InnerRowNode.value('(/D:Reference/D:File-num/text())[1]', 'int') AS FileNumber,
InnerRowNode.value('(/D:Reference/D:Line-num/text())[1]', 'int') AS LineNumber
FROM (
SELECT
XREF.XREF_FileData.XREF_FileData,
XREF.XREF_FileData.XREF_FileEntry,
InnerRow.query('.') AS InnerRowNode
FROM XREF.XREF_FileData
OUTER APPLY DataXref.nodes('/D:Cross-reference/D:Source/D:Reference[@Object-identifier = sql:variable("@p_ObjectIdentifierForReference") and @Reference-type = "RUN"]') as T(InnerRow)
WHERE DataXref.exist('/D:Cross-reference/D:Source/D:Reference[@Object-identifier = sql:variable("@p_ObjectIdentifierForReference") and @Reference-type = "RUN"]') = 1
) AS XREF_FileDataReference
INNER JOIN XREF.XREF_MemberBuilt ON XREF_MemberBuilt.XREF_FileData = XREF_FileDataReference.XREF_FileData
INNER JOIN XREF.XREF_FileEntry ON XREF_FileEntry.XREF_FileEntry = XREF_FileDataReference.XREF_FileEntry
WHERE XREF_MemberBuilt.XREF_ProjectBuilt = 69
OPTION( RECOMPILE, OPTIMIZE FOR (@p_ObjectIdentifierForReference = 'ord/p/ord0616.p') )
GO
So I guess I'm saying I cannot reproduce your problem on SQL 2008 R2 or SQL 2012. Does anything about this repro stand out as different from your situation?
Looking at your query I would say you might consider the following:
are you really seeing big differences in query duration?
pretty much ignore estimated plan costs for xml queries
consider breaking it up; eg carve off the xml then do the joins? If poor cardinality estimation is part of the problem this might help
Understand what PATH, PROPERTY and VALUE are for, then only create the ones you need
do you really have the range of queries that requires all three?
this is still a great article on xml indexes:
http://technet.microsoft.com/en-us/library/ms191497.aspx
What's performance like with the primary xml index only?
If performance is that important, consider materialising the columns permanently
I think the buffer_descriptors stuff is a distraction - mostly your cache is warm right?
plan forcing could be a last resort
Selective XML indexes in SQL 2012 onwards are great : ) much less storage required for example but much more specific

Selective XML Index feature is not supported for the current database version , SQL Server Extended Events , Optimizing Reading from XML column datatype

Team , Thanks for looking into this ..
As a last resort on optimizing my stored procedure ( Below ) i wanted to create a Selective XML index ( Normal XML indexes doesn't seem to be improving performance as needed ) but i keep getting this error within my stored proc . Selective XML
Index feature is not supported for the current database version.. How ever
EXECUTE sys.sp_db_selective_xml_index; return 1 , stating Selective XML Indexes are enabled on my current database .
Is there ANY alternative way i can optimize below stored proc ?
Thanks in advance for your response(s) !
/****** Object: StoredProcedure [dbo].[MN_Process_DDLSchema_Changes] Script Date: 3/11/2015 3:10:42 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- EXEC [dbo].[MN_Process_DDLSchema_Changes]
ALTER PROCEDURE [dbo].[MN_Process_DDLSchema_Changes]
AS
BEGIN
SET NOCOUNT ON --Does'nt have impact ( May be this wont on SQL Server Extended events session's being created on Server(s) , DB's )
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
select getdate() as getdate_0
DECLARE @XML XML , @Prev_Insertion_time DATETIME
-- Staging Previous Load time for filtering purpose ( Performance optimize while on insert )
SET @Prev_Insertion_time = (SELECT MAX(EE_Time_Stamp) FROM dbo.MN_DDLSchema_Changes_log ) -- Perf Optimize
-- PRINT '1'
CREATE TABLE #Temp
EventName VARCHAR(100),
Time_Stamp_EE DATETIME,
ObjectName VARCHAR(100),
ObjectType VARCHAR(100),
DbName VARCHAR(100),
ddl_Phase VARCHAR(50),
ClientAppName VARCHAR(2000),
ClientHostName VARCHAR(100),
server_instance_name VARCHAR(100),
ServerPrincipalName VARCHAR(100),
nt_username varchar(100),
SqlText NVARCHAR(MAX)
CREATE TABLE #XML_Hold
ID INT NOT NULL IDENTITY(1,1) PRIMARY KEY , -- PK necessity for Indexing on XML Col
BufferXml XML
select getdate() as getdate_01
INSERT INTO #XML_Hold (BufferXml)
SELECT
CAST(target_data AS XML) AS BufferXml -- Buffer Storage from SQL Extended Event(s) , Looks like there is a limitation with xml size ?? Need to re-search .
FROM sys.dm_xe_session_targets xet
INNER JOIN sys.dm_xe_sessions xes
ON xes.address = xet.event_session_address
WHERE xes.name = 'Capture DDL Schema Changes' --Ryelugu : 03/05/2015 Session being created withing SQL Server Extended Events
--RETURN
--SELECT * FROM #XML_Hold
select getdate() as getdate_1
-- 03/10/2015 RYelugu : Error while creating XML Index : Selective XML Index feature is not supported for the current database version
CREATE SELECTIVE XML INDEX SXI_TimeStamp ON #XML_Hold(BufferXml)
FOR
PathTimeStamp ='/RingBufferTarget/event/timestamp' AS XQUERY 'node()'
--RETURN
--CREATE PRIMARY XML INDEX [IX_XML_Hold] ON #XML_Hold(BufferXml) -- Ryelugu 03/09/2015 - Primary Index
--SELECT GETDATE() AS GETDATE_2
-- RYelugu 03/10/2015 -Creating secondary XML index doesnt make significant improvement at Query Optimizer , Instead creation takes more time , Only primary should be good here
--CREATE XML INDEX [IX_XML_Hold_values] ON #XML_Hold(BufferXml) -- Ryelugu 03/09/2015 - Primary Index , --There should exists a Primary for a secondary creation
--USING XML INDEX [IX_XML_Hold]
---- FOR VALUE
-- --FOR PROPERTY
-- FOR PATH
--SELECT GETDATE() AS GETDATE_3
--PRINT '2'
-- RETURN
SELECT GETDATE() GETDATE_3
INSERT INTO #Temp
EventName ,
Time_Stamp_EE ,
ObjectName ,
ObjectType,
DbName ,
ddl_Phase ,
ClientAppName ,
ClientHostName,
server_instance_name,
nt_username,
ServerPrincipalName ,
SqlText
SELECT
p.q.value('@name[1]','varchar(100)') AS eventname,
p.q.value('@timestamp[1]','datetime') AS timestampvalue,
p.q.value('(./data[@name="object_name"]/value)[1]','varchar(100)') AS objectname,
p.q.value('(./data[@name="object_type"]/text)[1]','varchar(100)') AS ObjectType,
p.q.value('(./action[@name="database_name"]/value)[1]','varchar(100)') AS databasename,
p.q.value('(./data[@name="ddl_phase"]/text)[1]','varchar(100)') AS ddl_phase,
p.q.value('(./action[@name="client_app_name"]/value)[1]','varchar(100)') AS clientappname,
p.q.value('(./action[@name="client_hostname"]/value)[1]','varchar(100)') AS clienthostname,
p.q.value('(./action[@name="server_instance_name"]/value)[1]','varchar(100)') AS server_instance_name,
p.q.value('(./action[@name="nt_username"]/value)[1]','varchar(100)') AS nt_username,
p.q.value('(./action[@name="server_principal_name"]/value)[1]','varchar(100)') AS serverprincipalname,
p.q.value('(./action[@name="sql_text"]/value)[1]','Nvarchar(max)') AS sqltext
FROM #XML_Hold
CROSS APPLY BufferXml.nodes('/RingBufferTarget/event')p(q)
WHERE -- Ryelugu 03/05/2015 - Perf Optimize - Filtering the Buffered XML so as not to lookup at previoulsy loaded records into stage table
p.q.value('@timestamp[1]','datetime') >= ISNULL(@Prev_Insertion_time ,p.q.value('@timestamp[1]','datetime'))
AND p.q.value('(./data[@name="ddl_phase"]/text)[1]','varchar(100)') ='Commit' --Ryelugu 03/06/2015 - Every Event records a begin version and a commit version into Buffer ( XML ) we need the committed version
AND p.q.value('(./data[@name="object_type"]/text)[1]','varchar(100)') <> 'STATISTICS' --Ryelugu 03/06/2015 - May be SQL Server Internally Creates Statistics for #Temp tables , we do not want Creation of STATISTICS Statement to be logged
AND p.q.value('(./data[@name="object_name"]/value)[1]','varchar(100)') NOT LIKE '%#%' -- Any stored proc which creates a temp table within it Extended Event does capture this creation statement SQL as well , we dont need it though
AND p.q.value('(./action[@name="client_app_name"]/value)[1]','varchar(100)') <> 'Replication Monitor' --Ryelugu : 03/09/2015 We do not want any records being caprutred by Replication Monitor ??
SELECT GETDATE() GETDATE_4
-- SELECT * FROM #TEMP
-- SELECT COUNT(*) FROM #TEMP
-- SELECT GETDATE()
-- RETURN
-- PRINT '3'
--RETURN
INSERT INTO [dbo].[MN_DDLSchema_Changes_log]
[UserName]
,[DbName]
,[ObjectName]
,[client_app_name]
,[ClientHostName]
,[ServerName]
,[SQL_TEXT]
,[EE_Time_Stamp]
,[Event_Name]
SELECT
CASE WHEN T.nt_username IS NULL OR LEN(T.nt_username) = 0 THEN t.ServerPrincipalName
ELSE T.nt_username
END
,T.DbName
,T.objectname
,T.clientappname
,t.ClientHostName
,T.server_instance_name
,T.sqltext
,T.Time_Stamp_EE
,T.eventname
FROM
#TEMP T
/** -- RYelugu 03/06/2015 - Filters are now being applied directly while retrieving records from BUFFER or on XML
-- Ryelugu 03/15/2015 - More filters are likely to be added on further testing
WHERE ddl_Phase ='Commit'
AND ObjectType <> 'STATISTICS' --Ryelugu 03/06/2015 - May be SQL Server Internally Creates Statistics for #Temp tables , we do not want Creation of STATISTICS Statement to be logged
AND ObjectName NOT LIKE '%#%' -- Any stored proc which creates a temp table within it Extended Event does capture this creation statement SQL as well , we dont need it though
AND T.Time_Stamp_EE >= @Prev_Insertion_time --Ryelugu 03/05/2015 - Performance Optimize
AND NOT EXISTS ( SELECT 1 FROM [dbo].[MN_DDLSchema_Changes_log] MN
WHERE MN.[ServerName] = T.server_instance_name -- Ryelugu Server Name needes to be added on to to xml ( Events in session )
AND MN.[DbName] = T.DbName
AND MN.[Event_Name] = T.EventName
AND MN.[ObjectName]= T.ObjectName
AND MN.[EE_Time_Stamp] = T.Time_Stamp_EE
AND MN.[SQL_TEXT] =T.SqlText -- Ryelugu 03/05/2015 This is a comparision Metric as well , But needs to decide on
-- Peformance Factor here , Will take advise from Lance if comparision on varchar(max) is a vital idea
--SELECT GETDATE()
--PRINT '4'
--RETURN
SELECT
top 100
[EE_Time_Stamp]
,[ServerName]
,[DbName]
,[Event_Name]
,[ObjectName]
,[UserName]
,[SQL_TEXT]
,[client_app_name]
,[Created_Date]
,[ClientHostName]
FROM
[dbo].[MN_DDLSchema_Changes_log]
ORDER BY [EE_Time_Stamp] desc
-- select getdate()
-- ** DELETE EVENTS after logging into Physical table
-- NEED TO Identify if this @XML can be updated into physical system table such that previously loaded events are left untoched
-- SET @XML.modify('delete /event/class/.[@timestamp="2015-03-06T13:01:19.020Z"]')
-- SELECT @XML
SELECT GETDATE() GETDATE_5
END
GO
Rajkumar Yelugu

@@Version : ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Microsoft SQL Server 2012 - 11.0.5058.0 (X64)
   May 14 2014 18:34:29
   Copyright (c) Microsoft Corporation
   Developer Edition (64-bit) on Windows NT 6.2 <X64> (Build 9200: ) (Hypervisor)
(1 row(s) affected)
Compatibility level is set to 110 .
One of the limitation states - XML columns with a depth of more than 128 nested nodes
How do i verify this ? Thanks .
Rajkumar Yelugu

Slow extraction in big XML-Files with PL/SQL

Hello,
i have a performance problem with the extraction from attributes in big XML Files. I tested with a size of ~ 30 mb.
The XML file is a response of a webservice. This response include some metadata of a document and the document itself. The document is inline embedded with a Base64 conversion. Here is an example of a XML File i want to analyse:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <ns2:GetDocumentByIDResponse xmlns:ns2="***">
         <ArchivedDocument>
            <ArchivedDocumentDescription version="1" currentVersion="true" documentClassName="Allgemeines Dokument" csbDocumentID="***">
               <Metadata archiveDate="2013-08-01+02:00" documentID="123">
                  <Descriptor type="Integer" name="fachlicheId">
                     <Value>123<Value>
                  </Descriptor>
                  <Descriptor type="String" name="user">
                     <Value>***</Value>
                  </Descriptor>
                  <InternalDescriptor type="Date" ID="DocumentDate">
                     <Value>2013-08-01+02:00</Value>
                  </InternalDescriptor>
                  
               </Metadata>
               <RepresentationDescription default="true" description="Description" documentPartCount="1" mimeType="application/octet-stream">
                  <DocumentPartDescription fileName="20mb.test" mimeType="application/octet-stream" length="20971520 " documentPartNumber="0" hashValue=""/>
               </RepresentationDescription>
            </ArchivedDocumentDescription>
            <DocumentPart mimeType="application/octet-stream" length="20971520 " documentPartNumber="0" representationNumber="0">
               <Data fileName="20mb.test">
                  <BinaryData>
                    
                  </BinaryData>
               </Data>
            </DocumentPart>
         </ArchivedDocument>
      </ns2:GetDocumentByIDResponse>
   </soap:Body>
</soap:Envelope>
Now i want to extract the filename and the Base64 converted document from this XML response.
For the extraction of the filename i use the following command:
v_filename := apex_web_service.parse_xml(v_xml, '//ArchivedDocument/ArchivedDocumentDescription/RepresentationDescription/DocumentPartDescription/@fileName');
For the extraction of the binary data i use the following command:
v_clob := apex_web_service.parse_xml_clob(v_xml, '//ArchivedDocument/DocumentPart/Data/BinaryData/text()');
My problem is the performance of this extraction. Here i created some summary of the start and end time for the commands:
Start Time
End Time
Difference
Command
10.09.13 - 15:46:11,402668000
10.09.13 - 15:47:21,407895000
00:01:10,005227
v_filename_bcm := apex_web_service.parse_xml(v_xml, '//ArchivedDocument/ArchivedDocumentDescription/RepresentationDescription/DocumentPartDescription/@fileName');
10.09.13 - 15:47:21,407895000
10.09.13 - 15:47:22,336786000
00:00:00,928891
v_clob := apex_web_service.parse_xml_clob(v_xml, '//ArchivedDocument/DocumentPart/Data/BinaryData/text()');
As you can see the extraction of the filename is slower then the document extraction. For the Extraction of the filename i need ~01
I wonder about it and started some tests.
I tried to use an exact - non dynamic - filename. So i have this commands:
v_filename := '20mb_1.test';
v_clob := apex_web_service.parse_xml_clob(v_xml, '//ArchivedDocument/DocumentPart/Data/BinaryData/text()');
Under this Conditions the time for the document extraction soar. You can see this in the following table:
Start Time
End Time
Difference
Command
10.09.13 - 16:02:33,212035000
10.09.13 - 16:02:33,212542000
00:00:00,000507
v_filename_bcm := '20mb_1.test';
10.09.13 - 16:02:33,212542000
10.09.13 - 16:03:40,342396000
00:01:07,129854
v_clob := apex_web_service.parse_xml_clob(v_xml, '//ArchivedDocument/DocumentPart/Data/BinaryData/text()');
So i'm looking for a faster extraction out of the xml file. Do you have any ideas? If you need more informations, please ask me.
Thank you,
Matthias
PS: I use the Oracle 11.2.0.2.0

Although using an XML schema is a good advice for an XML-centric application, I think it's a little overkill in this situation.
Here are two approaches you can test :
Using the DOM interface over your XMLType variable, for example :
DECLARE
v_xml    xmltype := xmltype('<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
       <soap:Body>
          <ns2:GetDocumentByIDResponse xmlns:ns2="***">
             <ArchivedDocument>
                <ArchivedDocumentDescription version="1" currentVersion="true" documentClassName="Allgemeines Dokument" csbDocumentID="***">
                   <Metadata archiveDate="2013-08-01+02:00" documentID="123">
                      <Descriptor type="Integer" name="fachlicheId">
                         <Value>123</Value>
                      </Descriptor>
                      <Descriptor type="String" name="user">
                         <Value>***</Value>
                      </Descriptor>
                      <InternalDescriptor type="Date" ID="DocumentDate">
                         <Value>2013-08-01+02:00</Value>
                      </InternalDescriptor>
                      
                   </Metadata>
                   <RepresentationDescription default="true" description="Description" documentPartCount="1" mimeType="application/octet-stream">
                      <DocumentPartDescription fileName="20mb.test" mimeType="application/octet-stream" length="20971520 " documentPartNumber="0" hashValue=""/>
                   </RepresentationDescription>
                </ArchivedDocumentDescription>
                <DocumentPart mimeType="application/octet-stream" length="20971520 " documentPartNumber="0" representationNumber="0">
                   <Data fileName="20mb.test">
                      <BinaryData>
                        ABC123
                      </BinaryData>
                   </Data>
                </DocumentPart>
             </ArchivedDocument>
          </ns2:GetDocumentByIDResponse>
       </soap:Body>
    </soap:Envelope>');
domDoc    dbms_xmldom.DOMDocument;
docNode   dbms_xmldom.DOMNode;
node      dbms_xmldom.DOMNode;
nsmap     varchar2(2000) := 'xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns2="***"';
xpath_pfx varchar2(2000) := '/soap:Envelope/soap:Body/ns2:GetDocumentByIDResponse/';
istream   sys.utl_characterinputstream;
buf       varchar2(32767);
numRead   pls_integer := 1;
filename       varchar2(30);
base64clob     clob;
BEGIN
domDoc := dbms_xmldom.newDOMDocument(v_xml);
docNode := dbms_xmldom.makeNode(domdoc);
filename := dbms_xslprocessor.valueOf(
                docNode
              , xpath_pfx || 'ArchivedDocument/ArchivedDocumentDescription/RepresentationDescription/DocumentPartDescription/@fileName'
              , nsmap
node := dbms_xslprocessor.selectSingleNode(
            docNode
          , xpath_pfx || 'ArchivedDocument/DocumentPart/Data/BinaryData/text()'
          , nsmap
--create an input stream to read the node content :
istream := dbms_xmldom.getNodeValueAsCharacterStream(node);
dbms_lob.createtemporary(base64clob, false);
-- read the content in 32k chunk and append data to the CLOB :
loop
    istream.read(buf, numRead);
    exit when numRead = 0;
    dbms_lob.writeappend(base64clob, numRead, buf);
end loop;
-- free resources :
istream.close();
dbms_xmldom.freeDocument(domDoc);
END;
Using a temporary XMLType storage (binary XML) :
create table tmp_xml of xmltype
xmltype store as securefile binary xml;
insert into tmp_xml values( v_xml );
select x.*
from tmp_xml t
   , xmltable(
       xmlnamespaces(
         'http://schemas.xmlsoap.org/soap/envelope/' as "soap"
       , '***' as "ns2"
     , '/soap:Envelope/soap:Body/ns2:GetDocumentByIDResponse/ArchivedDocument/DocumentPart/Data'
       passing t.object_value
       columns filename    varchar2(30) path '@fileName'
             , base64clob clob         path 'BinaryData'
     ) x

How do i import XML-project from imovie with the clips being apart? And not imported as one big clip...

Hello!
I have a problem with my final cut express. I want to import my project from iMovie (multiple clips in one project) without the clips together as one big clip.
Important: XML-from imovie
This had worked before but one day a note showed up at the screen saying:
If i choose yes the project becomes one big clip and if i choose no this error shows up:
and the project wont be imported.
Help me please! This has never been a problem before.
Sorry for the bad english, i am from sweden.
Thanks! //Karl Ekdahl

Hello!
I have a problem with my final cut express. I want to import my project from iMovie (multiple clips in one project) without the clips together as one big clip.
Important: XML-from imovie
This had worked before but one day a note showed up at the screen saying:
If i choose yes the project becomes one big clip and if i choose no this error shows up:
and the project wont be imported.
Help me please! This has never been a problem before.
Sorry for the bad english, i am from sweden.
Thanks! //Karl Ekdahl

Building big XML file from scratch - Urgent

Oracle 8.1.7.3 on windows NT platform
What is the best way to generate a quiet big XML file from multiple tables ?
I have information stored in many relational tables from which I need to generate a XML flat file either stored in a CLOB field or in a text file in a system directory. This XML file will be then used
as an input to generate a report either in HTML using XSLT, or in a PDF file using apache-fop or in a MS Excel file using SoftArtisan ExcelWriter.
My XML file has many levels in it structure, I mean that it is composed of one root element with 2 children, each children has a 3 children. One on these 3 children has 2 children and so on. Actually there are more or less 10 nested levels.
To generate this XML file, I tried to use XSU for PL/SQL with the nested cursor() feature plus XSLT to transform the raw XML file to my requirements.
But obviously there are some limitations using this method. Particularly, if the inner cursor returns an empty set I get exhausted resultset java error... A TAR confirmed that limitation.
So I had to give up this method to use basic nested PL/SQL cursors. Each fetched row is then inserted into a table (varchar2) with a sequence number so that with a cursor like select xml_chunk from my_table order by sequence, I get the whole XML file that I save either in a flat file or in a CLOB (using append method).
This method works fine, but it takes time and it's not flexible at all as I have to construct each XML tag. I guest this way of proceeding is not the more efficient...
Using DOM method won't be better as I still need PL/SQL cursor to select each level of my XML structure and in addition I might for sure encounter a problem of memory.
So what solutions would you suggest to generate this XML file. It must be quiet fast. The XML file can be up to 2Mo big. My system is actually a kind of on-the-fly reports generation. I mean that the XML file needs to be created with up-to-date data many times during the working hours !
Quick answers or suggestions would be greatly appreciated. It's very urgent !!
Thanks

I looks like the best way is to using the SAX processing for your application? Do you know the DTD or XML schema of your output XML document?
Would you send me the sample code for the method "to use XSU for PL/SQL with the nested cursor() feature plus XSLT to transform the raw XML file" to reproduce the problem?

Loading big XML files using JDBC gives errors

Hi,
I've created a XMLType table using binary storage, with the restriction that any document stored has a (any) schema:
CREATE TABLE XMLBIN OF XMLTYPE
XMLTYPE STORE AS BINARY XML
ALLOW ANYSCHEMA;Then I use JDBC to store a relatively large document using the following code:
Class.forName("oracle.jdbc.driver.OracleDriver").newInstance();
String connectionString = "jdbc:oracle:thin:@host:1521:sid";
File f = new File("c:\\temp\\big.xml");
Connection conn = DriverManager.getConnection(connectionString, "username", "password");
XMLType xml = XMLType.createXML(conn,new FileInputStream(f));
String statementText = "INSERT INTO xmlbin VALUES (?)";
OracleResultSet resultSet = null;
OracleCallableStatement statement = (OracleCallableStatement)conn.prepareCall(statementText);
statement.setObject(1,xml);
statement.execute();
statement.close();
conn.commit();
conn.close();Loading a file of 61Mb (real Mb, in non-IT Mb (where 1Mb seems to be 10^6) it is 63.9Mb) or less doesn't give any errors, loading a file bigger then that gives the following error:
java.sql.SQLRecoverableException: Io exception: Software caused connection abort: socket write error
        at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
        at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:960)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1222)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3381)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3482)
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:3856)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1373)
        at jdbctest.Tester.main(Tester.java:60)A succesful insert of a 63Mb file takes about 23 seconds to execute. The 70Mb file fails already after a few seconds, so I'm ruling out any time outs.
I'm guessing there are some buffers that need to be enlarged, but don't have a clue which ones.
Anyone any idea what might cause the problem and how to resolve?
My server runs Oracle 11g Win32. The client is Windows running Sun Java 1.6, using ojdbc6.jar and Oracle 11g Client installed.
Cheers,
Harald

Hi Mark,
The trace log in the OEM shows me:
Errors in file d:\oracle11g\app\helium\diag\rdbms\helium\helium\trace\helium_ora_6948.trc (incident=7510): ORA-07445: exception encountered: core dump [__intel_new_memcpy()+613] [ACCESS_VIOLATION] [ADDR:0x0] [PC:0x6104B045] [UNABLE_TO_WRITE] [] If needed I can post the full contents (if I find out how, am still a novice :-))
Cheers,
Harald

SQL XML-Selection-Performance in one big xml-file

Similar Messages

Maybe you are looking for