Ingestion Performance with Out-of-line on Large Schema

I have a large and complex schema consisting of 5 XSD's and few hundred complex types and a few thousand elements. This schema will be used for loading files ranging from 50mb to 650mb.
Since there are circular references and self references, I have annotated out-of-line storage for a few of the global element references. The ingestion performance for a very small sample file of 800k has shown severe performance problems with the ingestion. By using a sql trace, I have been able to limit the problem to the out-of-line storage.
This 800k file should be able to load in under one second yet it has taken almost 2 minutes. The inserts to the out-of-line tables represent about 80%+ of the total time. Both of the entries I have included below are out-of-line tables.
From the Trace
1 session in tracefile.
2109 user SQL statements in trace file.
1336 internal SQL statements in trace file.
3445 SQL statements in trace file.
74 unique SQL statements in trace file.
148036 lines in trace file.
133 elapsed seconds in trace file.
INSERT /*+ NO_REF_CASCADE NESTED_TABLE_SET_REFS */ INTO
"<VENDOR>"."GSMRELATION_VSDATA" e (e.sys_nc_rowinfo$,e.sys_nc_oid$,
e.docid)
VALUES
(:1,:2,:3) RETURNING e.sys_nc_oid$,e.rowid INTO :4,:5
call count cpu elapsed disk query current rows
Parse 323 0.03 0.02 0 0 0 0
Execute 323 42.64 41.91 0 4178 3418 323
Fetch 0 0.00 0.00 0 0 0 0
total 646 42.67 41.94 0 4178 3418 323
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: 60 (recursive depth: 1)
INSERT /*+ NO_REF_CASCADE NESTED_TABLE_SET_REFS */ INTO
"<VENDOR>"."UTRAN_VSDATA" e (e.sys_nc_rowinfo$,e.sys_nc_oid$,e.docid)
VALUES
(:1,:2,:3) RETURNING e.sys_nc_oid$,e.rowid INTO :4,:5
call count cpu elapsed disk query current rows
Parse 140 0.04 0.01 0 0 0 0
Execute 140 18.75 18.38 0 1338 1429 140
Fetch 0 0.00 0.00 0 0 0 0
total 280 18.79 18.40 0 1338 1429 140
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: 60 (recursive depth: 1)
The data above was gathered from Solaris 9 running 10.2.0.2 with the latest patches available.
I have been able to reproduce this same error to a different degree using a personal instance of Windows XP 10.2.0.2.
Does anyone have any idea what is happening?
Thanks,
VJ

By the way, SR 5991118.993 has been created. The Oracle tech has suggested
Bug 4369117
Abstract: UNACCEPTABLE DIRECT PATH OUT-OF-LINE LOB INSERT PERFORMANCE
Would this be a problem even if I'm not using CLOB annotation?
Thanks,

Similar Messages

  • With header line & with out header line ?

    what is difference between with header line & without header line ?

    When you create an internal table object you can also declare a header line with the same name. You can use the header line as a work area when you process the internal table. The ABAP statements that you use with internal tables have short forms that you can use if your internal table has a header line. These statements automatically assume the header line as an implicit work area. The following table shows the statements that you must use for internal tables without a header line, and the equivalent statements that you can use for internal tables with a header line:
    Operations without header line
    Operations with header line
    Operations for all Table Types
    INSERT <wa> INTO TABLE <itab>.
    INSERT TABLE ITAB.
    COLLECT <wa> INTO <itab>.
    COLLECT <itab>.
    READ TABLE <itab> ... INTO <wa>.
    READ TABLE <itab> ...
    MODIFY TABLE <itab> FROM <wa> ...
    MODIFY TABLE <itab> ...
    MODIFY <itab> FROM <wa> ...WHERE ...
    MODIFY <itab> ... WHERE ...
    DELETE TABLE <itab> FROM <wa>.
    DELETE TABLE <itab>.
    LOOP AT ITAB INTO <wa> ...
    LOOP AT ITAB ...
    Operations for Index Tables
    APPEND <wa> TO <itab>.
    APPEND <itab>.
    INSERT <wa> INTO <itab> ...
    INSERT <itab> ...
    MODIFY <itab> FROM <wa> ...
    MODIFY <itab> ...
    Using the header line as a work area means that you can use shorter statements; however, they are not necessarily easier to understand, since you cannot immediately recognize the origin and target of the assignment. Furthermore, the fact that the table and its header line have the same name can cause confusion in operations with entire internal tables. To avoid confusion, you should use internal tables with differently-named work areas.
    The following example shows two programs with the same function. One uses a header line, the other does not.
    With header line:
    TYPES: BEGIN OF LINE,
    COL1 TYPE I,
    COL2 TYPE I,
    END OF LINE.
    DATA ITAB TYPE HASHED TABLE OF LINE WITH UNIQUE KEY COL1
    WITH HEADER LINE.
    DO 4 TIMES.
    ITAB-COL1 = SY-INDEX.
    ITAB-COL2 = SY-INDEX ** 2.
    INSERT TABLE ITAB.
    ENDDO.
    ITAB-COL1 = 2.
    READ TABLE ITAB FROM ITAB.
    ITAB-COL2 = 100.
    MODIFY TABLE ITAB.
    ITAB-COL1 = 4.
    DELETE TABLE ITAB.
    LOOP AT ITAB.
    WRITE: / ITAB-COL1, ITAB-COL2.
    ENDLOOP.
    Without header line:
    TYPES: BEGIN OF LINE,
    COL1 TYPE I,
    COL2 TYPE I,
    END OF LINE.
    DATA: ITAB TYPE HASHED TABLE OF LINE WITH UNIQUE KEY COL1,
    WA LIKE LINE OF ITAB.
    DO 4 TIMES.
    WA-COL1 = SY-INDEX.
    WA-COL2 = SY-INDEX ** 2.
    INSERT WA INTO TABLE ITAB.
    ENDDO.
    WA-COL1 = 2.
    READ TABLE ITAB FROM WA INTO WA.
    WA-COL2 = 100.
    MODIFY TABLE ITAB FROM WA.
    WA-COL1 = 4.
    DELETE TABLE ITAB FROM WA.
    LOOP AT ITAB INTO WA.
    WRITE: / WA-COL1, WA-COL2.
    ENDLOOP.
    The list, in both cases, appears as follows:
    1 1
    2 100
    3 9
    The statements in the program that does not use a header line are easier to understand. As a further measure, you could have a further work area just to specify the key of the internal table, but to which no other values from the table are assigned.
    Internal table with header line
    you can use anywhere except obkect oriented concept.
    Internal table without header line :
    You should use in Object oriented concept..
    Always try to use without header line,performance point of view it is best..
    Example :
    Without header line.
    Structure
    types : begin of ty_itab ,
    matnr type mara-matnr,
    end of ty_itab.
    Internal table
    data i_itab type standard table of ty_itab .
    Work area
    data wa_itab like line of i_itab
    With header line
    data : begin of i_itab occurs 0,
    matnr like mara-matnr,
    end of i_itab
    itab with header lines are obsolete, anyway it will work but not recommended. instead use work area or more effiecient is field symbols. so donot use itab with header line.
    i will explain use of itab w/o header line.
    Data: itab1 type standard table of mara with header line occurs 0,
            itab2 type standard table of mara,
            wa_itab2 type mara.
    loop at itab1.
    "This will work fine.
    endloop.
    loop at itab2.
    "This will give erro that itabd does not hav workarea
    endloop.
    "so write
    loop at itab2 into wa_itab2.
    "This will work
    endloop.
    <b>The difference between
    whih header line and with out heater line of internal table.
    ex:-
    a) Data : itab like mara occurs 0 with header line.
    b) Data: itab like mara occurs 0.
    -While adding or retrieving records to / from internal table we have to keep the record temporarily.
    -The area where this record is kept is called as work area for the internal table.
    -The area must have the same structure as that of internal table. An internal table consists of a body and an optional header line.
    -Header line is a implicit work area for the internal table. It depends on how the internal table is declared that the itab will have the header line or not.
    a) Data : itab like mara occurs 0 with header line.
    table is with header line
    b) Data: itab like mara occurs 0.
    table is without header line</b>
    regards,
    srinivas
    <b>*reward for useful answers*</b>

  • Internal table with out header line

    Hi friends,
    Can u send me code for internal table with out header line : how to declare ,how to populate data and how to access the data
    Regards,
    vijay

    Hi Vijay
    There are several ways to declare an internal table without header line:
    A) You can define a type table
    TYPES: BEGIN OF TY_ITAB OCCURS 0,
            INCLUDE STRUCTURE ZTABLE.
    TYPES: END   OF TY_ITAB.
    and then your intrnal table:
    DATA: ITAB TYPE TY_ITAB.
    B) DATA: ITAB TYPE/LIKE STANDARD TABLE OF ZTABLE.
    C) DATA: ITAB TYPE/LIKE ZTABLE OCCURS 0.
    All these ways create a STANDARD TABLE
    You can create other types of internal table, for example SORTED TABLE or HASHED TABLE.
    These kinds of table can allow to improve the performance because they use different rules to read the data.
    When it wants to manage a table without header line, it need a work area, it has to have the same structure of table.
    DATA: WA LIKE ZTABLE.
    DATA: T_ZTABLE LIKE STANDARD TABLE OF ZTABLE.
    A) To insert the record:
    If you use INTO TABLE option you don't need workarea
    SELECT * FROM ZTABLE INTO TABLE T_ZTABLE
                                      WHERE FIELD1 = 'Z001'
                                        AND FIELD2 = '2006'.
    but if you want to append a single record:
    SELECT * FROM ZTABLE INTO wa WHERE FIELD1 = 'Z001'
                                   AND FIELD2 = '2006'.
    APPEND WA TO T_ZTABLE.
    ENDSELECT.
    Now you need workarea.
    B) To read data: you need always a workarea:
    LOOP AT T_ZTABLE INTO WA WHERE ....
      WRITE: / WA-FIELD1, WA-FIELD2, WA-FIELD3.
    ENDLOOP.
    or
    READ T_ZTABLE INTO WA WITH KEY FIELD3 = '0000000001'.
    IF SY-SUBRC = 0.
    WRITE: / WA-FIELD1, WA-FIELD2, WA-FIELD3.
    ENDIF.
    Anyway if you want to know only if a record exists, you can use the TRANSPORTING NO FIELDS option, in this case it doesn't need a workarea.
    READ T_ZTABLE WITH KEY FIELD3 = '0000000001'
                                      TRANSPORTING NO FIELDS.
    IF SY-SUBRC = 0.
    WRITE 'OK'.
    ENDIF.
    C) To update the data: it always needs a workarea
    LOOP AT T_ZTABLE INTO WA WHERE FIELD3 = '0000000001'.
    WA-FIELD3 = '0000000002'.
    MODIF T_ZTABLE FROM WA.
    ENDLOOP.
    or
    READ T_ZTABLE INTO WA WITH KEY FIELD3 = '0000000001'.
    IF SY-SUBRC = 0.
    WA-FIELD3 = '0000000002'.
    MODIF T_ZTABLE FROM WA INDEX SY-TABIX
    ENDIF.
    AT the end you can use the internal table to update database:
    MODIFY/UPDATE/INSERT ZTABLE FROM T_ZTABLE.
    See Help online for key words DATA, you can find out more details.
    Max
    Message was edited by: max bianchi

  • Download alv report output to excel format with out header line

    Hi experts,
    i want to download a alv report output into excel formatt with out the header line but it has to download including field description. as this output will fed into another transaction, the downloaded excel file should be with out header line.
    fro eg:
    Report   : Zabc                      ABAP Development          Page  :     1
    Run Date : 12/14/06                                                     System: UD400 
    Run Time : 08:45:37
    this header details should not be downloaded into the excel file.
    could somebody help me please.
    thanks
    deepu

    hi jayanti,
    thanks for your response.
    i have delclared all the field types as character but still it is not downloading and it 's sy-subrc is 4... the code is as below.
    *field names
      lt_fieldnames-value = 'Material Number'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Plant'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Material Group'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Material Description'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'UOM'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Price Unit'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Material Type'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'X-Plant Status'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Valuation Class'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = lw_avmng.
      APPEND lt_fieldnames.
      lt_fieldnames-value = lw_avntp.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Latest PO Qty'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Latest PO Cost'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'PO Creation Date'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = lw_fcaqt.
      APPEND lt_fieldnames.
      lt_fieldnames-value = 'Prev. Yr. Std. Cost'.
      APPEND lt_fieldnames.
      lt_fieldnames-value = lw_stcst.
      APPEND lt_fieldnames.
      CALL FUNCTION 'MS_EXCEL_OLE_STANDARD_DAT'
        EXPORTING
          file_name                       = 'XLSHEET'
        CREATE_PIVOT                    = 0
        DATA_SHEET_NAME                 = ' '
        PIVOT_SHEET_NAME                = ' '
        PASSWORD                        = ' '
        PASSWORD_OPTION                 = 0
        TABLES
        PIVOT_FIELD_TAB                 =
          data_tab                        = t_output1
          fieldnames                      = lt_fieldnames
        EXCEPTIONS
          file_not_exist                  = 1
          filename_expected               = 2
          communication_error             = 3
          ole_object_method_error         = 4
          ole_object_property_error       = 5
          invalid_pivot_fields            = 6
          download_problem                = 7
          OTHERS                          = 8
      IF sy-subrc <> 0.
        MESSAGE e001 WITH 'Data could not be downloaded'.
      ENDIF.
    ENDFORM.                               " z_dwn_xl
    thanks
    deepu

  • How can we improve query performance with out indexes?

    Hello Experts,
    I have a problem with table(calc) which contain 3 crore records and table doesn't have index, on that table one of the view(View_A) created.
    When i use the view in  below query SELECT count(*)
    FROM
      Table A INNER JOIN Table B ON (A.a=B.b)
       LEFT OUTER JOIN View_A ON ( Table A.a=View_A.a)
       LEFT OUTER JOIN View_B ON (Table A.a=View_B.a)
    In above query View_A is causing the problem, view_A is created on Calc table. One more thing when i execute select statement on the view it was running fine.
    With out View_A query fetching data fine. Update stats of the table also fine. When i run cost plan for scanning only cost % is 90.
    Can any help me please?.
    Thank you all.
    Regards,
    Jason.

    Jason,
    Not sure what you are trying to do. But outer joins are bad for performance. So try to avoid them.
    You also say that you have a view on a calc table. What are you calculating? Are you using user defined functions maybe?
    Regards,
    Nico

  • Can analytics can be performed with out BI and analytics implemented on CRM

    Hi all,
    I want to configure CRM system with marketing, Sales, service scenarios only and i want to perform some Lead analysis, Campaing analysis etc. is it possible with out integration of BI, any installations of CRM analytics modules?
    Rgds,
    Rao

    Hi Rao
    With just ABAP reports you can do reporting like in any OLTP system be it R/3 or CRM. The OLAP systems were initially made in order to get in-depth reporting using the data warehousing techniques to store, extract, retrieve data for presentation so that there is no performance degradation on the OLTP system.
    Now CRM analytics is just technically like any ERP module analytics (be it CO, FI, SD etc.: if you put aside the diff. kinds of extraction techniques) and like them needs a BI system. The different modeling techniques have to be built normally on a separate BIW system and the results are applied back to business processes in CRM/ERP.
    Since netweaver 2004 the BI content as an addon as against the prior releases. Not many customers follow the idea of performing the BI related transactions on the same system as the CRM is unless you have a really robust hardware.

  • Very slow network performance with Intel NIC when TCP Large Send Offload is enabled

    I have a Windows 2008 server with two NIC's (Intel PRO/1000 PL and Intel PRO/1000 PM), and one virtual external switch connected to each NIC.
    With the 01/14/08 network driver, I get very low throughput on both NIC's (~10KB/sec when downloading from my server when it should be around 11MB/sec).
    I've found that disabling "TCP Large Send Offload (IPv4)" on the virtual switch and "Large Send Offload v2 (IPv4)" on the physical adapters solves the problem.
    Is this a bug in the driver?
    Thank you,
    Ricardo Costa

    2 years later, I was able to get that machine to work with Hyper-V under Windows Server 2008 R2, but only after disabling everything that had anything to do with TCP or IP offloading.  I think that I had to turn off 4 settings on one controller,
    and 5 settings on the other controller.  Once that was done, everything worked flawlessly.  It is very apparent that the Virtual Controllers do not properly use many of the features of non-Intel devices (not all Intel devices work either), and
    that Hyper-V does not completely uninstall if you remove it.
    If you have any problems, start turning off the features that your adapter supports, until it starts working.  You'll need to weight the cost of replacing the controller with a fully supported controller, of which there is no list, or using your existing
    controller at a capacity lower than it physically supports.  This has been an elusive issue with Microsoft's Virtual Server projects since they were first in beta, so don't expect a fix anytime soon.
    P.S. For those who don't know, the settings are on the actual device and can be changed in the Device Manager.  If you are still having trouble after turning off all the settings, then you will need to replace the controller(s) with a different
    make/model.
    Bill Bosacker's Blog @ http://www.openSourceC.org/

  • Accessing a field from a structure with out header line

    Hi Guys,
                 I am trying to assign a field from a structure type line of data to another field.
    This structure type line don't have a header line.
    here is the example.
    IT_EKKNU TYPE  MMPUR_EKKNU.
    when I use following statement..
    move IT_EKKNU-kostl to e_cekko-kostl.
    Iam getting error "IT_ENKKNU is not a structure or Internal table with header line.
    How to access the fields in structure IT_EKKNU.
    Thank U for ur time.
    Cheers
    S Kumar

    Here IT_EKKNU is an internal table without Header line.
    You can use a structure for assignment.
    DATA: IT_EKKNU TYPE MMPUR_EKKNU,
               wa_ekknu type    ekknu.
    read table it_ekknu into wa_ekknu........
    move wa_EKKNU-kostl to e_cekko-kostl.
    *modify the code for your conditions*.

  • Lt_item_type is a table with out header line and therefore has no component

    Dear frnds,
    I am new to webdynpro i got this error while creating a tree .can any body tell me what is the mistake i have done.
    Regards.
    siva

    Hi,
    As you are new to the tree concepts.Try to do this tutorial.It will give a idea.
    [Tree|http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/d075dbc5-3b33-2c10-5f9f-99bf2738fe6a?QuickLink=index&overridelayout=true]
    Regards,
    Karthik.R

  • I need a code for Advanced Search based on my custom multi select listbox and query through programatically (jquery)with out design

    Hi,
    I have a grid and advanced search link buttonand
    search button.
    if i click on Advanced Search link...open a panel with 3 list boxes(multi selection)
    if i select multi selected items in list boxes and click on search-->show the grid with selected items
    getting data from Sql server custom database.
    And the code i wil expect  is 'Programatically using jqury and asp.net code' with out single line of design.
    Could you please help on the above task.

    Hi,
    I have a grid and advanced search link buttonand
    search button.
    if i click on Advanced Search link...open a panel with 3 list boxes(multi selection)
    if i select multi selected items in list boxes and click on search-->show the grid with selected items
    getting data from Sql server custom database.
    And the code i wil expect  is 'Programatically using jqury and asp.net code' with out single line of design.
    Could you please help on the above task.

  • Intenal table with out headerline

    hi,
    i have a doubt,please clarify.
    data:itab like <databse table> occurs 0 with header line.
    or
    data:itab like <databse table>occurs 0,
            wa like line type of itab.
    in above two cases , in my view  no difference.
    in first case there will be one workarea will be created with same name itab,
    in second case we are explicitly creating work area.
    ok.
    here my doubt is in what cases it is madatory to create  internal table with  explicitly work area?
    thanks in advance.
    venu

    hi,
        abap object oriented programming in higher versions doesn't allow internal table with header line, so we can create internal table with out header line.
    for that one we follow two approches...
    1)
    TYPES: BEGIN OF LINE,
             COL1 TYPE I,
             COL2 TYPE I,
           END OF LINE.
    DATA :  ITAB TYPE STANDARD TABLE OF LINE,
                 WA TYPE LINE.
    2) In second case create line type(structure) and row type by using  SE11.
    SE11->select DATA ELEMENT object> here select STRUCTURE object and provide required fields>now select ROW TYPE object here provide LINE TYPE( STRCTURE which is created in previous)-> SAVE and activate.
    IN the above case STRCTURE acts as WORK-AREA AND ROW TYPE acts as body.
    object oriented programming doesn't support to using LIKE key word.
    regards,
    Ashok

  • MII Performance with Large XML

    HI,
    We are facing performance issue while parsing XML of large size of around 50MB.
    Initially XMII was crashing due to "out of heap memory error". However, with some changes the problem has been removed.But
    now, we are facing an issue with time taken by the Payload to execute. Its taking more than half-an-hour to execute the transaction.
    The solution tried so far has decreased the time to half-an-hour. earlier it was taking more than one and half hour to process.
    We have tried parallel processing by asynchronous call to the transactions
    Is there any way to further reduce the time of processing?
    Further, is it possible to find out which parser is used by MII internally for XML SAX or DOM Parser.
    Thanks
    Amit

    Hi Amit,
    Just some tips to improve performance of BLS.
    1. Use Xpath whenever possible.
    2. Remove unnecessary repeaters performing on BLS.
    3. Check the Like to Optimizing BLS Performance.
    Optimizing BLS Performance for XML Handling in SAP MII
    If you are storing the data in database. just pass whole xml to query and insert data using bulk insert.
    Thanks
    Anshul

  • Schedule lines should not be maintained with out release

    schedule lines should not be maintained with out release of schedule agreement......but system accepting to maintain the schedule lines.Please say how.
    regards,
    Sridhar kumar.a

    Hello,
    I have the same problem as you. Someone know something about it?
    kanyadhara, did yo solve this problem?
    Thanks a lot

  • I can not store all my Music on my internal Macbook pro hard drive so I am storing it on a large external drive connected to my airport extrem.  How do I get Itunes to search for the music here with out trying to copy it to my laptops hard drive??

    I can not store all my Music on my internal Macbook pro hard drive so I am storing it on a large external drive connected to my airport extreme (2 TB drive plugged into the USB port).  I see the drive on my laptop and I can add and delete files no problem.  How do I get Itunes to search for the music here with out trying to copy it to my laptop's hard drive?  I don't have enough space to do that.

    How did you move the music to the external drive?  What exactly is on the drive?  The entire iTunes folder or only music?  If it is the entire iTunes folder you can do the option+start suggestion earlier.  If you copied only music and did so by dragging it there then you need to delete it again and consolidate/organize it there instead so iTunes tracks the move.  iTunes 12 for Mac: Change where your iTunes files are stored - http://support.apple.com/kb/PH19507

  • I am using i 4 phone. recently I had a problem with my lap top and had formatted hard disk of it. Now I want to use sync data in my iphone back to itune n my lap top. how can I perform this task with out loosing data in my i phone.

    I am using i 4 phone. recently I had a problem with my lap top and had formatted hard disk of it. Now I want to sync data in my iphone back to itune on my lap top. how can I perform this task with out loosing data in my i phone.

    Hey floridiansue,
    Do you have an installed email program such as Microsoft Outlook?  If your email is through an online login, such as Gmail, etc, then one will have to create an email association with a program such as Microsoft Outlook on the PC for this Scan to Email system to function.
    -------------How do I give Kudos? | How do I mark a post as Solved? --------------------------------------------------------
    I am not an HP employee.

Maybe you are looking for