How to combine large number of key-value pair tables into a single table?

I have 250+ key-value pair tables with the following characteristics
1) keys are unique within a table but may or may not be unique across tables
2) each table has about 2 million rows
What is the best way to create a single table with all the unique key-values from all these tables? The following two queries work till about 150+ tables
with
  t1 as ( select 1 as key, 'a1' as val from dual union all
          select 2 as key, 'a1' as val from dual union all
          select 3 as key, 'a2' as val from dual )
, t2 as ( select 2 as key, 'b1' as val from dual union all
          select 3 as key, 'b2' as val from dual union all
          select 4 as key, 'b3' as val from dual )
, t3 as ( select 1 as key, 'c1' as val from dual union all
          select 3 as key, 'c1' as val from dual union all
          select 5 as key, 'c2' as val from dual )
select coalesce(t1.key, t2.key, t3.key) as key
,      max(t1.val) as val1
,      max(t2.val) as val2
,      max(t3.val) as val3
from t1
full join t2 on ( t1.key = t2.key )
full join t3 on ( t2.key = t3.key )
group by coalesce(t1.key, t2.key, t3.key)
with
  master as ( select rownum as key from dual connect by level <= 5 )
, t1 as ( select 1 as key, 'a1' as val from dual union all
          select 2 as key, 'a1' as val from dual union all
          select 3 as key, 'a2' as val from dual )
, t2 as ( select 2 as key, 'b1' as val from dual union all
          select 3 as key, 'b2' as val from dual union all
          select 4 as key, 'b3' as val from dual )
, t3 as ( select 1 as key, 'c1' as val from dual union all
          select 3 as key, 'c1' as val from dual union all
          select 5 as key, 'c2' as val from dual )
select m.key as key
,      t1.val as val1
,      t2.val as val2
,      t3.val as val3
from master m
left join t1 on ( t1.key = m.key )
left join t2 on ( t2.key = m.key )
left join t3 on ( t3.key = m.key )
/

A couple of questions, then a possible solution.
Why on earth do you have 250+ key-value pair tables?
Why on earth do you want to consolodate them into one table with one row per key?
You could do a pivot of all of the tables, without joining. something like:
with
  t1 as ( select 1 as key, 'a1' as val from dual union all
          select 2 as key, 'a1' as val from dual union all
          select 3 as key, 'a2' as val from dual )
, t2 as ( select 2 as key, 'b1' as val from dual union all
          select 3 as key, 'b2' as val from dual union all
          select 4 as key, 'b3' as val from dual )
, t3 as ( select 1 as key, 'c1' as val from dual union all
          select 3 as key, 'c1' as val from dual union all
          select 5 as key, 'c2' as val from dual )
select key, max(t1val), max(t2val), max(t3val)
FROM (select key, val t1val, null t2val, null t3val
      from t1
      union all
      select key, null, val, null
      from t2
      union all
      select key, null, null, val
      from t3)
group by keyIf you can do this in a single query, unioning all 250+ tables, then you do not need to worry about chaining or migration. It might be necessary to do it in a couple of passes, depending on the resources available on your server. If so, I would be inclined to create the table first, with a larger than normal percent free, then do the first set as a straight insert, and the remaining pass or passes as a merge.
Another alternative might be to use the approach above, but limit the range of keys in each pass. So pass one would have a predicate like where key between 1 and 10 in each branch of the union, pass 2 would have key between 11 and 20 etc. That way everything would be straight inserts.
Having said all that, I go back to my second question above, why on earth do you want/need to do this? What is the business requirement you are trying to solve. There might be a much better way to meet the requirement.
John

Similar Messages

  • Maximum number of key/value pairs in btree or hash database

    It's not clear to me from the documentation what the limit is on the number of key/value pairs in a Btree or hash database. The documentation on logical record numbers says that the db_recno_t type is a 32-bit unsigned type, "which limits the number of logical records in a Queue or Recno database, and the maximum logical record which may be directly retrieved from a Btree database, to 4,294,967,295". What does "directly retrieved" mean in this sentence? Does this mean that Btree is limited to 4,294,967,295 key/vaue pairs? If so, is this also the limit for a hash database?

    Hi mcgarvek,
    In case of Queue, Recno and Btree configured for logical record numbers - you can do that by setting DB_RECNUM flag: http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/db_set_flags.html#DB_RECNUM , the maximum number of records is 4,294,967,295 and after that limit, the next record will be inserted as record number 1.
    In case of Hash and Btree, the maximum database size depends on the page size selected by the application. Berkeley DB stores database file page numbers as unsigned 32-bit numbers and database file page sizes as unsigned 16-bit numbers. Using the maximum database page size of 65536, this results in a maximum database file size of 248 (256 terabytes).
    Database limits: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_misc/dbsizes.html
    Logical record numbers: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/logrec.html
    Regards,
    Bogdan Coman

  • Experience with creation of a large number of key figures

    Hi:
    I have a need to create a large number of Key Figures (at least about 50 of them) for DP use. Do you have any experience with automating this task if so, can you share your experience?  Thanks.
    Satish

    Hi,
    So far we have not come across a need to automate KF creation. And 50 KF is not a huge number, it would take may be a couple of hours to create them. Also, since it is a one time activity, developing an automation script for the same does not sound useful. You might want to consider creating them manually so that you can ensure all settings are proper.
    Hope this helps.
    Thanks & Regards
    Mani Suresh

  • How can we create a look-up in Enterprise Gateway.. like key value pair..???

    How can we create a look-up in Enterprise Gateway.. like key value pair..???

    Hi,
    You want to have a look at KPS, Key Property Store. Link: Key Property Stores
    Cheers,
    Stefan

  • Passing a large number of column values to an Oracle insert procedure

    I am quite new to the ODP space, so can someone please tell me what's the best and most efficient way to pass a large number of column values to an Oracle procedure from my C# program ? Passing a small number of values as parameters seem OK but when there are many, this seems inelegant.

    Passing a small number
    of values as parameters seem OK but when there are
    many, this seems inelegant.Is it possible that your table with a staggering amount of columns or method that collapses without so many inputs is ultimately what is inelegant?
    I once did a database conversion from VAX RMS system with a "table" with 11,000 columns to a normalized schema in an Oracle database. That was inelegant.
    Michael O
    http://blog.crisatunity.com

  • How to reconnect large number of photos?

    how to reconnect large number of photos?

    Click File >> Reconnect >> All Missing Files and it should automatic import. If it doesn't, Browse for yourself and locate the folder where media corresponding to these missing files is present.
    Thanks
    Andaleeb

  • How to block large number of scheduling agreements?

    How to block large number of scheduling agreements (Purchasing doc type 'LPA') ?

    Hi
    You can use T.code MASS and choose scheduling agreement and you see field LOEKZ there (deletion indicator) You can mark it for deletion and it can be undelete if you want in future.
    If you want to block it with source list, you need to do LSMW or BDC for ME01 and for your agreement block it in a validity period for a plant
    Regards
    Antony

  • CRM7 EH1 ICI/CTI: Add Key-Value Pair to Call Attached Data

    We are CRM 7 EHP1 integrated to Genesys CTI (Communication Mangement Software) through ICI.
    Is it possible in web_ui to add a Key-Value Pair (kvp) to the Call Attached Data (CAD) that it is sent back to the CMS?
    For example: When an agent clicks Transfer to transfer a call to a queue, just prior to the transfer we would like to add a new kvp to the CAD that is sent back to Genesys via ICI.
    Any advice on how to do this is appreciated.

    Hi shiva
    Thanks for your help,
    Can you check this coding and revert me back ASAP Please.
    REPORT BDS_GOS_CONNECTION.
    DATA : logical_system LIKE BAPIBDS01-log_system.
           CLASSNAME LIKE BAPIBDS01-CLASSNAME
           OBJKEY LIKE SWOTOBJID-objkey.
    PARAMETERS: pa_lo_sys BAPIBDS01-log_system,
                pa_class like BPIBDS01-CLASSNAME,
                pa_objkey like swotobjidobjkey.
    AT SELECTION-SCREEN.
    CALL FUNCTION 'BDS_GOS_CONNECTIONS_GET'
             EXPORTING
                  bor_id             = bor_id
             IMPORTING
                  logical_sytem      = pa_lo_sys.
                  classname          = pa_class.
                  objkey            = pa_objkey.
             EXCEPTIONS
                  no_objects_found     = 1
                  internal_error       = 2
                  internal_gos_error   = 3.
       IF sy-subrc <> 0.
         MESSAGE ID sy-msgid TYPE sy-msgty NUMBER sy-msgno
                 WITH sy-msgv1 sy-msgv2 sy-msgv3 sy-msgv4.
       ENDIF.
    clear v_attno1.
    i_object1-typeid = 'BUS2012'.
    i_object1-catid  = 'BO'.
    i_object1-instid = i_yItem-docno.
    call method cl_gos_attachment_query=>count_for_object
       exporting
        is_object = i_object1
        ip_arl    = space
       receiving
        rt_stat   = i_stat1.
    read table i_stat1 into wa_stat1 index 1.
    if sy-subrc eq c_0.
        move wa_stat1-counter to v_attno1.
    endif.             
    CALL METHOD cl_gos_attachment_query=>count_for_object
    EXPORTING
    is_object = object
    ip_arl =    space
    RECEIVING
    rt_stat = lt_stat.
    READ TABLE lt_stat INDEX 1 into ls_stat.
    count = ls_stat-counter.
    *The object has to be a concatenation of your document, like this:
    CONCATENATE object-instid tab-gjahr INTO object-instid.
    ELSE.
    CONCATENATE tab-bukrs tab-belnr tab-gjahr INTO
    object-instid.
    ENDIF.

  • Mapping unique elements to a Key Value Pair using XSLT in ESB

    Hi Guys,
    I am in need of a solution for mapping some of the response elements to a a key value pair in my target schema. How could I achieve this. It is very very urgent. How will the XSL look like
    Source
    <Source>
    *<Element1>One</Element1>*
    *<Element2>Two</Element2>*
    <Action>Manage</Action>
    </Source>
    Target
    <Target>
    <Action>Manage</Action>
    <AdditionalData>
    *<KeyValuePair>*
    *<key>Element1</key>*
    *<value>One</value>*
    *</KeyValuePair>*
    *<KeyValuePair>*
    *<key>Element2</key>*
    *<value>Two</value>*
    *</KeyValuePair>*
    </AdditionalData>
    </Target>
    Edited by: user13156113 on May 25, 2010 7:01 AM

    Below is the soultion which I finally did it by myself. Any other solutions would be welcome.
    <ns10:AdditionalData>
    <xsl:for-each select="//node()">
    <xsl:if test="text()">
    <ns16:KeyValuePair>
    <ns16:Key>
    <xsl:value-of select="xp20:upper-case(name(.))"/>
    </ns16:Key>
    <ns16:Value>
    <xsl:value-of select="."/>
    </ns16:Value>
    </ns16:KeyValuePair>
    </xsl:if>
    </xsl:for-each>
    </ns10:AdditionalData>

  • Can we achieve data conversion for my scenario key value pair

    My data 
    create table test
    ( id int,name varchar(10),value int, color varchar(20))
    I would like to have data in the following format 
    where each id will combination of data set for each row for a id and separated by delimiter from another set
    like a key value pair
    ID Value
    1 |a,10,green|a,15,blue|
    2 |b,11,red|b,12,yellow|
    Please let me know if this can be achived in SSIS using trasnformation or in SQL ?
    And if so what is the best approach to hande this conversion
    Mudassar

    Thanks for your help.
    I can assume that it can only be handled in SQL and this cant be done in SSIS transformations.
    EXAMPLE 4 - Using XML PATH & correlated subquery for sublist
    -- Create comma delimited sublist
    SELECT   Subcategory = ps.[Name],
             ColorList = Stuff((SELECT DISTINCT  ',
    ' + Color AS [text()]
                                FROM AdventureWorks2008.Production.Product
    p
                                WHERE p.ProductSubcategoryID = ps.ProductSubcategoryID
                                FOR XML PATH ('')),1,1,'')
    FROM     AdventureWorks2008.Production.ProductSubcategory
    ps
    ORDER BY Subcategory;
    GO
    Subcategory             ColorList
    Helmets                 Black, Blue, Red
    Hydration Packs         Silver
    Jerseys                 Multi, Yellow
    Mudassar

  • Application specific key-value pairs in jndi.properties

    Hello,
    Can I specify my application specific key-value pair in jndi.properties?
    I tried something like this
    java.naming.factory.initial=.jndi.WLInitialContextFactory
    java.naming.provider.url=t3://localhost:7001
    myVar=myVal
    When i tried looking up "myVar" from my client program, I got an error.
    The other parameters like weblogic.jndi.WLInitialContextFactory are picked up.
    Anyhelp will be appreciated
    Vasim

    We have a similar problem.
    We would like to configure our PROVIDER_URL for a specific web application - not
    for the entire server. Since the URL should be different in development, test
    and production environments, we would prefer to just set it in the deployment
    descriptor. And we have a lot of code that just uses
    ctx = new InitialContext();
    when looking up EJBs, queues etc.
    Actually, to take the problem one step further, it should be expected that later
    we will have EJB's deployed on different machines/clusters - so we will actually
    need specific urls for each EJB.
    Is there a good way to do this? Or will we have to custom-develop our own jndi
    configuration standard using application parameters to set which JNDI provider
    each EJB should be looked up with?
    Alternativaely, can we "import" the JNDI trees of the app server in the JNDI tree
    of the web servers?
    So, how should we go about this?
    Robert Patrick <[email protected]> wrote:
    Vasim wrote:
    Hi Robert,
    You are right. But The object "myVar" which I am trying to look upis not in
    the JNDI tree nor am I interesed in binding it . But my requirementis that
    I have one application specific variable which I am trying to lookup and I
    dont want to have a separare config file for this..and hence the question..So, put the properties you want in the jndi.properties file and load
    the properties
    file from your code by doing something like this:
    Properties props = new Properties();
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    if (cl == null)
    cl = System.getSystemClassLoader();
    InputStream is = cl.getResourceAsStream("jndi.properties");
    props.load(is);
    Personally, I would not use this file and would create an application-specific
    file
    or, as Daniel suggested, define your properties as a System property
    and use
    System.getProperty("myVar").
    btw, is jndi.properties only for those objects which are bound to jnditree?
    jndi.properties is only used for creating the JNDI InitialContext. The
    whole idea
    of this file is that in remote client code (without the jndi.properties
    file), you
    need to do something like this to tell the JNDI classes how to connect
    to the JNDI
    provider:
    Properties props = new Properties();
    props.put(Context.PROVIDER_URL, "t3://myservername:7001");
    props.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");
    InitialContext ctx = new InitialContext(props);
    but inside the server, you only need to do this because the server is
    the provider
    and already knows how to connect to itself:
    InitialContext ctx = new InitialContext();
    Therefore, the jndi.properties file allows you to externalize this property-setting
    code that sets up the properties to be passed to the InitialContext constructor
    so
    that the remote client code can now look exactly like the code inside
    the server.
    The InitialContext constructor will look for this jndi.properties file
    in your
    classpath and load it to get the necessary configuration information
    to determine
    how to connect to the JNDI provider.
    Hope this helps,
    Robert

  • Key value pair error

    Hello
    how can I fix this problem ?
    SAPNW2004sJavaSP9_Trial\SAP_NetWeaver_2004s_SR_1
    jdkversion 142_09 .
    ERROR 2008-07-09 23:56:30
    CJS-30051  Cannot insert a key value pair into the secure store fails; see output of log file SecureStoreInsert.log: SAP Secure Store in the File System - Copyright (c) 2003 SAP AG
    ERROR 2008-07-09 23:56:30
    FCO-00011  The step insertAdminDataInSecStore with step key |NW_Java_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|1|0|NW_CI_Instance|ind|ind|ind|ind|11|0|NW_CI_Instance_Configure_Java|ind|ind|ind|ind|3|0|insertAdminDataInSecStore was executed with status ERROR .
    Thanks
    sas

    Hi this the content of  SecureStoreInsert .
    com.sap.security.core.server.secstorefs.NoEncryptionException: Encryption or decryption is not possible because the full version of the SAP Java Crypto Toolkit was not found (iaik_jce.jar is required, iaik_jce_export.jar is not sufficient) or the JCE Jurisdiction Policy Files don't allow the use of the "PbeWithSHAAnd3_KeyTripleDES_CBC" algorithm.
    at com.sap.security.core.server.secstorefs.SecStoreFS.openExistingStore(SecStoreFS.java:1975)
    at com.sap.security.core.server.secstorefs.SecStoreFS.handleInsert(SecStoreFS.java:963)
    at com.sap.security.core.server.secstorefs.SecStoreFS.main(SecStoreFS.java:1276)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:324)
    at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)
    Caused by: java.lang.SecurityException: The provider IAIK may not be signed by a trusted party
    at javax.crypto.SunJCE_b.a(DashoA12275)
    at javax.crypto.Cipher.a(DashoA12275)
    at javax.crypto.Cipher.getInstance(DashoA12275)
    at com.sap.security.core.server.secstorefs.Crypt.<init>(Crypt.java:220)
    at com.sap.security.core.server.secstorefs.SecStoreFS.<init>(SecStoreFS.java:1346)
    at com.sap.security.core.server.secstorefs.SecStoreFS.handleInsert(SecStoreFS.java:954)
    ... 6 more

  • Is it possible to load 1 billion key-value pairs into BerkeleyDB database?

    Hello,
    I experiment with loading huge datasets into BerkeleyDB database. The procedure is as follows:
    1. Generate a dump-like file using a script. The file contains key-value pairs (on separate lines, exactly in the format of the dump file, that can be produced by db_dump). The index is hash.
    2. Use db_load to create a database. The OS is windows server 2003.
    Both key and values are 64-bit longs.
    Using this procedure, I succeeded to load 25 million pairs in the database. It took about 1-2 hours.
    Next, I tried to load 250 million pairs into an empty database. db_loader runs already 15 hours. It's memory consumption is very low: private bytes ~2M, working set ~2M, virtual size ~13M. db_loader already read all the pairs from disk, as IO is very low now: ~4M per second. I am not sure if db_loader will finish in next 24h hours.
    My goal is to load eventually 3 billion key-value pairs into one DB.
    I will appreciate if someone will advise me:
    1. If BerkeleyDB is capable of dealing with such database volume.
    2. Is my procedure good, how to optimize it. Is it possible to allocate more RAM to db_load? Are there other ways to optimize loading time?
    Thank you,
    Gregory.

    Hello Sandra,
    The version is: Berkeley DB 5.0.21: (March 30, 2010).
    The data: keys and values are random 64 bit numbers.
    The header of the "dump" file that I am trying to load is (there are 256 * 1e6 key-value pairs in the file):
    VERSION=3
    format=bytevalue
    type=hash
    h_nelem=512000000
    db_pagesize=8192
    HEADER=END
    The db_load allocates 1G memory cache.
    Thank you,
    Gregory.

  • Effective way to parse key-value pair.

    Hi,
    I've a PL/SQL process, in which I've lot of dynamic sql routines planned, to parse input data to a procedure.
    One of the parameter to call or that'd be passed to this procedure is a "key-value pair"
    e.g.
    procedure p ( id in number, state_name in varchar2 )
    Values could be
    20, 'vc1=AZ, vc2=WA, vc3=NY, vc4=NJ'
    My requirement is as follows
    <ol><li>Parse 'state_name', for values of state ( AZ, WA, NY, NJ ) - note that the list could be of variable length</li>
    <li>I'm not interested in 'key' per se ( vc1, vc2, vc3... )</li>
    <li>State names are important as ( as ) part of the process, I want to concatenate them to form tables-name,
         which (already) exists e.g. table_CA, table_AZ, table_WA, table_NJ - so basically I need to
         parse these names and have them in a way to be able to use them as part of other dynamic SQL's.
    </li>
    </ol>
    Please advice.
    Thanks.

    shankariyer wrote:
    Is it possible for me to populate the output of this regex_replace into an array of varchar2 ??
    Sure:
    CREATE OR REPLACE
      TYPE STATE_TBL_TYPE
        AS
          TABLE OF VARCHAR2(2);
    SET SERVEROUTPUT ON
    DECLARE
        v_value_pair VARCHAR2(30) := 'vc1=AZ, vc2=WA, vc3=NY, vc4=NJ';
        v_state_tbl  STATE_TBL_TYPE := STATE_TBL_TYPE();
    BEGIN
        FOR v_i IN 1..NVL(LENGTH(v_value_pair) - LENGTH(REPLACE(v_value_pair,',')) + 1,0) LOOP
          v_state_tbl.EXTEND;
          v_state_tbl(v_i) := LTRIM(REGEXP_SUBSTR(v_value_pair,'=[A-Z]+',1,v_i),'=');
        END LOOP;
        FOR v_i IN 1..v_state_tbl.COUNT LOOP
          DBMS_OUTPUT.PUT_LINE(v_state_tbl(v_i));
        END LOOP;
    END;
    SQL> SET SERVEROUTPUT ON
    SQL> DECLARE
      2      v_value_pair VARCHAR2(30) := 'vc1=AZ, vc2=WA, vc3=NY, vc4=NJ';
      3      v_state_tbl  STATE_TBL_TYPE := STATE_TBL_TYPE();
      4  BEGIN
      5      FOR v_i IN 1..NVL(LENGTH(v_value_pair) - LENGTH(REPLACE(v_value_pair,',')) + 1,0) LOOP
      6        v_state_tbl.EXTEND;
      7        v_state_tbl(v_i) := LTRIM(REGEXP_SUBSTR(v_value_pair,'=[A-Z]+',1,v_i),'=');
      8      END LOOP;
      9      FOR v_i IN 1..v_state_tbl.COUNT LOOP
    10        DBMS_OUTPUT.PUT_LINE(v_state_tbl(v_i));
    11      END LOOP;
    12  END;
    13  /
    AZ
    WA
    NY
    NJ
    PL/SQL procedure successfully completed.
    SQL> SY.

  • Cannot insert a key value pair into the secure store fails; see output of l

    Hi,
    how can I fix this problem ?
    SAPNW2004sJavaSP9_Trial\SAP_NetWeaver_2004s_SR_1
    jdkversion 142_09 .
    ERROR 2008-07-10 13:13:31
    CJS-30051  Cannot insert a key value pair into the secure store fails; see output of log file SecureStoreInsert.log: SAP Secure Store in the File System - Copyright (c) 2003 SAP AG
    Regds
    sas

    Hi Arzu,
    thank you for your replying.
    The current OS I am using is Microsoft Windows XP
    Service Pack 2.
    The very last installation was made with JDK version 142_12.
    However it was pointless. I can try to reinstall with
    the mentioned newest JCE policy files .
    Can tell me from where I can obtain these above
    JCE policy files ?
    Regards
    Erdem Sas

Maybe you are looking for