Distributed partitioning

Hi!
First of all thank you for the time you grant me... and sorry for my english too!
I currently work for my university on databases distribution and partitioning concepts.
I installed two oracle instances on two servers. Each instance contains one database (lets call them db_1 and db_2).
These two databases are linked to each other with oracle database links.
Each instance contains tables, and I managed to make queries on remote tables... everything seems fine on that side.
My goal then was to do table partitioning.
For example, a customer table in db_1:
- when client name is between A and F: customer.partition_1
- when client name is between G and Z: customer.partition_2
Everything seems fine on that side too :)
I suppose that, with partitioning metadatas, Oracle is able to optimize queries to consult only the partitions that might contain the data.
For example, in a " select from customer where name like 'A%'; " query, partition_2 should remain quietly at rest.
Now my question is: is it possible that customer.part_1 stay on db_1 but part_2 goes on db_2 ?
That is, make a mixture between distribution and partitioning.
db_1 will have a customer table, but the 2 partitions containing the data will be distributed.
The ultimate goal is to have a customer table on both databases, but with distributed partitions.
In this case, access to the remote partition will be made only when necessary.
I know it seems completely useless, but it's for my graduation project, and faculties projects are not intended to be useful, but interesting!
Having said that, from this idea, we can imagine distributed systems powerful enough...
If it's possible, I haven’t found this information in the oracle doc. Or I missed something...
Thanks in advance for any answer, if you need a fuller explanation, do not hesitate to ask!
Pedro

Oracle's partitioning option would not allow you to have one partition of the CUSTOMER table in database A and a second partition in database B (that technique is generally referred to as "sharding"). Oracle provides features like RAC, Active DataGuard, and various replication technologies in order to allow you to build distributed systems in the real world.
Since this is an academic exercise, however, you could build something like what you're describing using the (*very old-school*) techniques that were used prior to partitioning being available. In Oracle 7, for example, you could implement a form of poor-man's partitioning where you create, say, 26 different physical tables, your code inserts the row into the appropriate table (i.e. customers with a last_name of 'Cave' would go in the CUSTOMER_C table), and a view does the UNION ALL to combine all the tables.
CREATE TABLE customer_a (
  customer_id NUMBER PRIMARY KEY,
  first_name VARCHAR2(30),
  last_name VARCHAR2(30)
CREATE TABLE customer_b (
  customer_id NUMBER PRIMARY KEY,
  first_name VARCHAR2(30),
  last_name VARCHAR2(30)
CREATE VIEW customer
AS
SELECT *
  FROM customer_a
WHERE last_name LIKE 'A%'
UNION ALL
SELECT *
  FROM customer_b
WHERE last_name LIKE 'B%'
UNION ALL
SELECT *
  FROM customer_z
WHERE last_name LIKE 'Z%'Assuming you have a query like
SELECT *
  FROM customer
WHERE last_name = 'Cave'the optimizer should be able to identify that only the CUSTOMER_C table would need to be scanned and ignore the other 25 tables in the query. In this setup, CUSTOMER_A could be in one database and CUSTOMER_Z could be in another. This setup, though, is hugely more fragile than partitioning-- it's not something that people today would design. Getting all the setup exactly right so that the optimizer knows which tables it can ignore can be rather non-trivial (I doubt that this particular example would work without tweaking) and is a bit of a lost art. I'm sure there are papers from the mid-90's when this was a more common workaround to the lack of a partitioning option that discuss this technique in more detail. And I'm sure some of those papers could use a bit of updating given the huge changes that have been made to the optimizer over the years.
Justin

Similar Messages

  • SQL Server Distributed Partitioning Views how to add a new node online

    We are using distributed partitioning views in SQL Server 2012 Enterprise Edition for scaling out our data across more than one servers. Now we faced to question how to add a new node (server) into the scale outed db servers system without sending the servers
    down, so our users will be able to use them during the process as well.
    For example we have 4 servers with scaled out data. When we add the new empty server, the CHECKINGs for the partitioning columns should be reorganized. But during the process the partitioning views are not working.
    The High Availability, Always On or Failover Cluster approaches seem are not resolve the problems.
    So my question is how to add new node online?
    KH

    Thank you Erland for the reply.
    Yes, it's sounds as possible solution but has some not resolvable nuance in it. Let's say we copied some data from Node5 to new added Node6. Let's assume in Node5 we had data in Table1 with partitioning column's values 100,101,102,103,104,105,106.  Now
    we want to copy part of the rows with partitioning column's values 103,104,105,106 from Node5.Table1 into Node6.Table1. With this Node5 will contain less data and will work more quickly (less IO, less CPU usage etc), and the rest data will be contained on
    Node6. But because of Node5 is already in use, the Node5.Table1 contains CHECK CONSTRAINT = ParttionColumn should be from 100 up to 106. This is check for Node5. The Distributed Partitioning Views are already using the CHECKs to identify what server should
    be used to get data from.
    Now when we copied part of the Node5.Table1 rows to Node6.Table1 the views are still using the 103-106 rows from Node5.Table1 because the CHECK points there. Then we include the newest Node6.Table1 in the distributed partitioning views. OK, but we should
    set some CHECK on new Node6.Table1 which will be used by views. We can't set intersecting checking like Node5 has CHECK 100-106 and Node6 has CHECK 103-106. We also can't edit Node5 check and set it 100-102 untill the data will be removed in it. But this means
    that the data will not be available during the execution. 
    So, any ideas ?
    KH

  • Distributed on a PC

    From: ESDS01:: "MAIL-11 Daemon" 16-DEC-1996 18:18:02.76
    To: [email protected]
    CC:
    Subj: forte-users-digest V1 #169
    forte-users-digest Monday, 16 December 1996 Volume 01 : Number 169
    From: "Russell Francis" <[email protected]>
    Date: Mon, 16 Dec 1996 20:00:32 -0000
    Subject: Interface to Business Objects
    Hi all,
    We have a client who plans to use Business Objects as their adhoc reporting
    tool to accompany their Forte' application. There are a number a subsystems
    within their Forte' application that will contain fixed reports.
    We are considering doing all their reports in Business Objects. This would
    mean that the interface to start the report and provide any parameters
    required for the report would be part of the Forte' Application. The actual
    printing and previewing would be done by Business Objects using a
    predefined report, invisible to the user.
    Has anybody tried this before?
    Any information would be appreciated,
    Regards,
    Russell Francis.
    / __ \ X EXIX Limited
    | ______| I Excellence In
    \ \_____ Corporate Software
    \______/ X Solutions
    Russell Francis - Forte' Consultant
    [email protected]
    Tel: + 44 468 94 60 60
    George Nowakowski <[email protected]> wrote:
    <snip!>
    Can anyone please give me some hints as to how I might get Forte
    Distributed to work on a single PC? <snip!>
    What you need to do is start your PC as a server. Unfortunately you
    can't (according to page 87 of the system management guide) run a
    Macintosh or PC/Windows machine as a Forte server. Assuming that by "PC"
    you mean a Wintel machine, I see a couple alternatives:
    * Run NT. Forte supports NT as a server on an Intel platform, but you'll
    want a serious amount of memory (80 meg was enough to serve 4
    developers here), and probably a serious CPU as well. Our
    experience was on an NT server. I have NOT tried to make NT
    client into a Forte server, but given all the heat in
    comp.sys.whatever.nt.system about the lack of differences "under
    the hood" between NT server and NT client, I suspect it might
    work. You'll need to install TCP/IP (which doesn't happen by
    default).
    * Hack around with the windows kit. Judging by my Macintosh, Forte
    provides all the pieces; maybe you can persuate Windows to be a
    server.
    Tom Wyant The opinions expressed in this document are
    [email protected] those of the author, except for the
    [email protected] following, which is William Shakespeare's:
    "The first thing we do is, we'll kill
    all the lawyers."
    Henry VI, Part II, Act 4, Scene 2

    I have found it possible to do "stand-alone" development without a node
    manager or environment manager (under both Windows NT 3.51 & 4.0) by:
    1) Creating a shadow repository
    2) Detaching the shadow repository
    3) Running Forte stand-alone against the shadow (switched using Forte
    Control Panel)
    All application resources must be installed & running local (ODBC connection
    to databases, etc...)
    -- Greg
    =======================
    [email protected]
    www.sagesoln.com
    415.329.7243 x 305
    =======================
    George,
    There are a couple of ways you can do testing on a single PC.
    1. If you have NT 3.5.1 or 4.0 on your PC, you can run the
    environment manager (nodemgr -e YourEnvName). With an environment
    manager running, you can then run Forte Distributed, partition, and
    run distributed which will start server partitions on the NT box, etc.
    2. If you have Win 3.1 or Win 95, then you cannot run an environment
    manager or server partitions, but you can still connect to an ODBC database.
    Setting up an ODBC resource manager is very straight-forward. You do some
    of the work in the ODBC control panel (ie. mapping a name to a particular
    data source such as Access file, flat file, etc.). You then run Forte
    Standalone which will automatically use ODBC for any DBSessions or
    DBResourceMgr
    service objects that need to get created by a run. You don't need to
    change your service object to point to an ODBC Resource Mgr, the system
    does it automagically.
    Hope this helps,
    Bobby

  • Distributed Cache gets are very time consuming!!!!!

    We have an application that uses Coherence as a distributed partitioned cache server.
    Our problem is that the access to the near cache is very fast but the access to the remote servers is very cpu consuming (maybe for the deserialization of the objects retrieved) at such point that disabling the cache and reading the objects directly from the dbms (Oracle) improves the performances.
    We use coherence as the second level cache of hibernate so we have not much control on how hibernate serialize and deserialize it's objects (we cannot implement a custom externalization).
    Have you any experience of performance problems with the combination hibernate/tangosol and distributed cache?

    Hi Giancarlo,
    It's Jon, not Cameron ;)
    The Hibernate cache API (including 3.0) does not support bulk-fetches from cache, which means that these cache fetches are serial rather than parallel. Depending on how much of your performance problem is network related, Coherence may actually perform much better than straight JDBC as the application load increases (time per request will remain similar, but won't increase much under heavier load).
    The advantage to using the Coherence API (rather than Hibernate API) is that you have far better control over how your queries run. The drawback of course is that you have to code your own queries if they become more complex than search queries. As a result, it often makes sense to use Hibernate (or equivalent) for the bulk of your queries, then code specific queries directly against Coherence when performance really matters. A simple example of this is the ability to perform bulk gets (using <tt>NamedCache.getAll()</tt>). With queries that return hundreds or thousands of items, this can have a huge impact on performance (an order of magnitude or more depending on object complexity). You also get the ability to use XmlBean/ExternalizableLite and cluster-parallel search queries.
    As an aside, with the Coherence cache servers, you should avoid making your JVM heaps excessively large. As a general rule of thumb, we recommend a 512MB heap size for cache servers, using as many instances as required to reach your desired cache capacity. Also, from what I've seen, 64-bit JVMs are less efficient with memory usage than 32-bit (though performance seems a bit better on the x86-64 platforms vs the 32-bit versions).
    Jon Purdy
    Tangosol, Inc.

  • Data distribution in distributed caching scheme

    When using the distributed ( partitioned ) scheme in coherence , how the data distribution is happening among the nodes in data-grid..? Is there any API to control it or are there some configurations to control it ?

    Hi 832093
    A distributed scheme works by allocating the data to partitions (by default there are 257 of these, but you can configure more for large clusters). The partitions are then allocated as evenly as possible to the nodes of the cluster, so each node owns a number of partitions. Partitions belong to a cache service so you might have a cache service that is responsible for a number of caches and a particular node will own the same partitions for all those caches. If you have a backup count > 0 then a backup of each partition is allocated to another node (on another machine if you have more than one). When you put a value into the cache Coherence will basically perform a hash function on your key which will allocate the key to a partition and therfore to the node that owns that partition. In effect a distibuted cache works like a Java HashMap which hashes keys and allocates them to buckets.
    You can have some control over which partition a key goes to if you use key association to co-locate entries into the same partition. You would normally do this to put related values into the same location to make processing them on the server side more efficient in use-cases where you might need to alter or query a number of related items. For example in finanial systems you might have a cache for Trades and a cache for TradeValuations in the same cache service. You can then use key association to allocate all the Valuations for a Trade to the same partition as the parent Trade. So if a Trade was mapped to partition 190 in the Trade cache then all of the Valuations for that Trade would map to partition 190 in the TradeValuations cache and hence be on the same node (in the same JVM process).
    You do not really want to have control over which nodes partitions are allocated to as this could alter Coherence ability to evenly distribute partitions and allocate backups properly.
    JK

  • Setting up semantic partitions ASE

    Running  ASE 15.5
    We have a  non-partitioned table  with 640 million rows. We are looking at setting up range semantic partitions  by create date
    There are 200 + million rows for 2013 and 380 + million rows for 2014
    I am thinking about setting up following partitions by create date:
    Partitions 1  -  4 :  For 2015  By quarter   
    Partition 5 -  For year 2014
    Partition 6   -  2013 and and earlier
    Add new Partitons for each new  year  ...
    Only updating  current data  --  i.e. any data more than month old is no longer updated ..
    Is this a viable breakdown ?
    1st attempt at partitioning  ..

    Actually, I would like to comment that there are some nuances with partitioning and stats to be aware of.....but as far as your question goes, a lot depends on your version of ASE.   For pre-ASE 15.7, sampling works, but the best practice taught in classes was to do a full non-sampled stats first, then do 2-3 updates with sampling, then a full non-sampled stats in cycle - so if doing update stats weekly, the first run of the month would be a full non-sampled and other weeks in the month would be sampled.    However, what this is doing is trying to help you determine if stats sampling is working similarly to a non-sampled stats by virtue of the fact that you may have performance issues the latter weeks of the month using sampled stats vs. non-sampled.   How well this works often depends on how values are added to the different columns - e.g. scattered around evenly or monotonically increasing.    I personally have found that in later revs of 15.7 (e.g. sp100+) that running stats with hashing is much faster than stats with sampling and generates more accurate stats.   I know Mark has seen some issues - not sure where/why - but then I have seen problems with just update stats generically in which we have had to delete stats before re-running update stats....so not sure if the problem was caused by starting with non-hashed stats and then trying to update with hashed stats or not (I have always started with hashed stats).
    Now, there is an interesting nuance of update stats with partitioning.    Yes, you can run update stats on partition basis.......but it doesn't update table stats (problem #1) and it can also lead to stats explosion.   I am not saying don't run update stats on a partition basis - I actually encourage it  - but suggest you know what is going to happen.    For example, partitioning - especially range partitioning - works best as far as maintenance commands - when you get in the 30+ partition range - and especially in the 50-100 partition range - assuming evenly distributed partitions.  In your case, you will likely get the same effect on 2014 and 2015 partitions as they will be much smaller.  When you run update stats on each partition, (assuming the default histogram steps) you will get 20 steps PER partition.....which can mean 1000+ for the entire table (if 50 partitions).  Not necessarily a problem unless the query needs to hit all the partitions (or some significant number of them) at which point the query will need considerable proc cache to load those stats.  Sooo......when using partitions, keep in mind that you may need to increase proc cache to handle the increase use during optimization.    On the table stats perspective, what it means is that periodically you might want to run update statistics (not update index statistics) on the table.....however, in my experience this hasn't been as necessary as one would think....and might only be necessary if you see the optimizer picking a table/partition scan when you think it should be choosing an index.
    In your case, you might only have 20 steps for the whonking huge partition and then 20 steps for 2014 and 20 steps for each of the 2015 quarterly partition.    You might want to run the update stats for the 2013 and before partition with a larger step count (e.g. 100) and then run it with 20 or so for the other partitions.
    Using partitions the way you are doing is interesting in a different perspective.   The current data is extremely small and therefore fast access (fewer index levels) and you don't get quite the penalty for queries that span a lot of partitions - e.g. a 5 year query doesn't have to hit 20 partitions the way it would for complete quarterly partitions.   However, this assumes the scheme is:
    Partition 1 = data 2+ previous years
    Partition 2 = data for previous year
    Partitions 3-6 = data for current year by quarter
    Which means at the end (or so) of each year, you will need to merge partitions.   Whenever you merge partitions, you will then need to run update stats again.
    If the scheme instead is just to have historical partitions but going forward each year will simply have data by quarter, you might want to see what the impact on queries are - especially reports on non-unique indexes where the date range spans a lot of partitions or date range is not part of the query.

  • Azure SQL Federations Retired - Will Table Partitions replace?

    My latest project was designed around Azure SQL Federations. Needless the frustration of the  this feature leaving is high, I can't imagine the people that have already coded their projects around it.
    With Azure SQL Federations gone, is table partitioning going to be in the road map soon?

    Hi Mendokusai,
    The features of SQL Server that are typically used for partitioning data in on-premises solutions include: Replication, Table Partitioning, Partitioned Views, Distributed Partitioned Views, File group Strategies, and Cross-database queries (Distributed Queries).
    Azure SQL Database does not support all these features except Partitioned Views. Azure SQL Database does offer a scale out solution, which horizontally partitions data across multiple databases.
    Azure SQL Database supports two methods to implement partitions, one is custom sharding, another is federations.
    The current implementation of Federations will be retired with Web and Business service tiers.  Consider deploying custom sharding solutions to maximize scalability, flexibility, and performance.
    For more information, you can review the following articles.
    http://msdn.microsoft.com/library/azure/dn495641.aspx
    http://msdn.microsoft.com/en-us/library/jj156171.aspx
    Regards,
    Sofiya Li
    Sofiya Li
    TechNet Community Support

  • Unequal Distribution of Objects

    We are trying to use Coherence for caching large amount of business data (500GB). We are using 64 cache-servers running on 8 Linux boxes, and have defined multiple distributed caches. When we try to find the number of instances of each object in various cache-server nodes using JMX, we notice that while most of the nodes have a fair number of objects (3600), about 8 of them have very few objects (700). These nodes contains very small number of objects for every cache that we have in the cluster. Why is it so? Our understanding is that Coherence distributes data evenly. We are using the default Unit-calculator.
    Also, we are not able to find the which node is using which JMX port number. When we see that Node no 6 is caching very few objects, we want to examine its memory, but we can't figure which instance of cache-server is really Node 6. Fact that we are running 8 cache-servers per box makes it more difficult.

    user5606651 wrote:
    We are trying to use Coherence for caching large amount of business data (500GB). We are using 64 cache-servers running on 8 Linux boxes, and have defined multiple distributed caches. When we try to find the number of instances of each object in various cache-server nodes using JMX, we notice that while most of the nodes have a fair number of objects (3600), about 8 of them have very few objects (700). These nodes contains very small number of objects for every cache that we have in the cluster. Why is it so? Our understanding is that Coherence distributes data evenly. As Paul mentioned, Coherence does not distributed data evenly. It distributes partitions evenly.
    The distribution of data (i.e. keys) to partition depends on your cache key objects, usually the binary form, but you can influence this by providing a KeyPartitioningStrategy where you can override the placement of a key to a partition. The algorithm for coming up with a partition id in KeyPartitioningStrategy for a key must take only the serializable state of the key into account.
    Overriding this class would help altering the way your objects are distributed into partitions, as likely the distribution of your keys to partitions is not even.
    What class do you actually use as cache key?
    We are using the default Unit-calculator.
    AFAIK, unit calculators have nothing to do with partition distribution. Coherence assumes that the distribution of entries to partitions comes up with evenly distributed partition sizes.
    Also, we are not able to find the which node is using which JMX port number. When we see that Node no 6 is caching very few objects, we want to examine its memory, but we can't figure which instance of cache-server is really Node 6. Fact that we are running 8 cache-servers per box makes it more difficult.You can find out the process id of all nodes if you look at the Members attribute of the Coherence/Cluster MBean (it is referred to as process=... in the Member description). Starting from the process id you can find out what ports are used by which process by using netstat or lsof.
    Best regards,
    Robert

  • Re: Forte Environment on OpenVMS Cluster - doubling up forno reason?

    Hi,
    I tested Forte on OpenVMS more than one year ago. So, it's only
    what I remember and it was on Forte R2. Normally, it should work
    exactly as you described if you have one TCP alias for the two nodes.
    The test I made (one year ago) prouved that Forte does not
    know the active cluster of Digital.
    Now I work on RS6000 and we have planned to use HACMP (but the cluster
    is not active). We use one TCP alias for the two machines, and the same
    socket number. So if Machine 1 fails, HACMP will startup Forte and the
    environment on the Machine 2. The application is made to be able to
    reconnect
    to a new environment (and name server). For the deployment, we should
    deploy
    only once, because the environment sees only one alias and socket on the
    TCP
    level. On that Idea, you should try to start the same environement on
    the
    2 machines with one alias and the same socket. The deployment should be
    done
    only once, but you should need to load distrib twice. You may need to
    deploy
    (Appdist directory to have a local load) manually.
    I'm really interested on the result if you make the test.
    Hope it helps,
    Daniel Nguyen
    Freelance Forte Consultant
    Dirk Haben (Tel +61 (0)3 6230 5300) wrote:
    >
    G'Day
    Anyone out here using Forte on a OpenVMS CLUSTER?
    I'm running an environment over a cluster and want to know if I can cut
    down on any builds since both nodes use the same directory on the same
    disk(s) via the same controller etc.
    For a start I don't want Forte to "distribute" the new appdist and userapp
    to the clustered node - it's already "there".
    If I can cut down on the loading of the app and on the install that would
    be good to.
    BTW, so far I only replicate a loadbalanced EntityManager part to attach to RDB dbs on the two machines - 5 each node. Same database.
    Dirk
    [email protected]
    To unsubscribe, email '[email protected]' with
    'unsubscribe forte-users' as the body of the message.
    Searchable thread archive <URL:http://pinehurst.sageit.com/listarchive/>-
    To unsubscribe, email '[email protected]' with
    'unsubscribe forte-users' as the body of the message.
    Searchable thread archive <URL:http://pinehurst.sageit.com/listarchive/>

    I have found it possible to do "stand-alone" development without a node
    manager or environment manager (under both Windows NT 3.51 & 4.0) by:
    1) Creating a shadow repository
    2) Detaching the shadow repository
    3) Running Forte stand-alone against the shadow (switched using Forte
    Control Panel)
    All application resources must be installed & running local (ODBC connection
    to databases, etc...)
    -- Greg
    =======================
    [email protected]
    www.sagesoln.com
    415.329.7243 x 305
    =======================
    George,
    There are a couple of ways you can do testing on a single PC.
    1. If you have NT 3.5.1 or 4.0 on your PC, you can run the
    environment manager (nodemgr -e YourEnvName). With an environment
    manager running, you can then run Forte Distributed, partition, and
    run distributed which will start server partitions on the NT box, etc.
    2. If you have Win 3.1 or Win 95, then you cannot run an environment
    manager or server partitions, but you can still connect to an ODBC database.
    Setting up an ODBC resource manager is very straight-forward. You do some
    of the work in the ODBC control panel (ie. mapping a name to a particular
    data source such as Access file, flat file, etc.). You then run Forte
    Standalone which will automatically use ODBC for any DBSessions or
    DBResourceMgr
    service objects that need to get created by a run. You don't need to
    change your service object to point to an ODBC Resource Mgr, the system
    does it automagically.
    Hope this helps,
    Bobby

  • Re-caching issues

    Hello,
    I am running into a slight problem and hoping that some one here has some insight into what has become a day-long brick wall for me.
    I have a Near Cache that I am using and I am attempting to update entries in the cache so that soon I can add listeners to handle events when entries in the cache are changed. Basically I want to utilize the cache to notify me when something has been modified and this modification has replaced a previous entry in the cache.
    My problem is this:
    I have two keys, either created one after another - or at separate times (I have done both to attempt to determine the problem) and two different objects. Let us called these keys KeyA and KeyB
    KeyA.hashCode() returns an integer that is equal to KeyB.hashCode()
    KeyA.equals(KeyB) returns true
    KeyB.equals(KeyA) returns true
    However, if I execute the following code:
    cache.put(KeyA, SomeObject);
    cache.put(KeyB, SomeDifferentObject);
    and then examine the contents of the cache, I have both SomeObject and SomeDifferentObject cached under KeyA and KeyB, respectively. It is my goal that I should replace SomeObject with SomeDifferentObject so that Tangosol will create an event that I can listen for and react to appropriately.
    Does anyone know why this might be occurring? Is there something I am overlooking? Or do I need to take different steps to come to the point that I am shooting for? Also, the data in KeyA and KeyB are identical except for one string that is not used in either hashCode() or equals() (only in toString for identification purposes)
    Thanks for any help in advance!
    Kevek

    Hi Kevek,
    Also, the data in KeyA and KeyB are
    identical except for one string that is not used in
    either hashCode() or equals() (only in toString for
    identification purposes)This is likely the problem. In a distributed/partitioned cache, serialized forms of the keys are used for equality checking since the data is stored in serialized format in the cache. So if key1.equals(key2), then key1 serialized needs to equal key2 serialized. This does not appear to be the case for your key since you have a string that (I presume) is being serialized that is not taken into account when performing equals/hashCode.
    You can try one of the following:
    1) Make the string transient. This means that the value of the string may be lost once placed into the cache.
    2) Remove the string
    3) Include the string (and all non-transient variables) in equals/hashCode.
    If you have any questions/doubts, feel free to send us the key class, either on the forum or at support (at) tangosol.com
    Thanks,
    Patrick Peralta

  • How to avoid Cache misses?

    Hi,
    Before I explain the problem here's my current setup.
    - Distributed/partitioned cache
    - Annotated JPA classes
    - Backing map linked to an oracle database
    - Objects are stored in POF format
    - C++ extend client
    When I request an item that does not exist in the cache, the JPA magic forms a query and assembles the object and stores that inside the cache.
    However if the query returns no results then coherence sends back a cache miss. Our existing object hierarchy can request items that don't exist (this infrastructure is vast and entrenched and changing it is not an option). This blows any near cache performance out of the water.
    What I want to do is to intercept a cache miss and store a null object in the cache on that key (by null it will be 4 bytes in length). The client code can interpret the null object as a cache miss and everything will work as usual - however the null object will be stored in the near cache and performance will return.
    My problem is, as annotated JPA does all the 'magic', I don't get to intercept if the query returns an empty set. I've tried both map triggers and listeners, however as expected they don't get called as no result set is generated.
    Does anyone know of an entry point where I can return an object to coherence in the event of a query returning an empty set. I'd also like the ability to configure this behaviour on a per cache basis.
    Any help gratefully received.
    Thanks
    Rich
    Edited by: Rich Carless on Jan 6, 2011 1:56 PM

    Hi,
    If you are using 3.6 you can do this by writing a sub-class of JpaCacheStore that implements BinaryEntryStore or a more genric way (which would suit other people who have asked similar questions recently) would be to write an implementation of BinaryEntryStore that wraps another cache store.
    Here is one I knocked up recently...
    package org.gridman.coherence.cachestore;
    import com.tangosol.net.BackingMapManagerContext;
    import com.tangosol.net.CacheFactory;
    import com.tangosol.net.DefaultConfigurableCacheFactory;
    import com.tangosol.net.cache.BinaryEntryStore;
    import com.tangosol.net.cache.CacheStore;
    import com.tangosol.run.xml.XmlElement;
    import com.tangosol.util.Binary;
    import com.tangosol.util.BinaryEntry;
    import java.util.Set;
    public class WrapperBinaryCacheStore implements BinaryEntryStore {
        private BackingMapManagerContext context;
        private CacheStore wrapped;
        public WrapperBinaryCacheStore(BackingMapManagerContext context, ClassLoader loader, String cacheName, XmlElement cacheStoreConfig) {
            this.context = context;
            DefaultConfigurableCacheFactory cacheFactory = (DefaultConfigurableCacheFactory) CacheFactory.getConfigurableCacheFactory();
            DefaultConfigurableCacheFactory.CacheInfo info = cacheFactory.findSchemeMapping(cacheName);
            XmlElement xmlConfig = cacheStoreConfig.getSafeElement("class-scheme");
            wrapped = (CacheStore)cacheFactory.instantiateAny(info, xmlConfig, context, loader);
        @Override
        public void erase(BinaryEntry binaryEntry) {
            wrapped.erase(binaryEntry.getKey());
        @SuppressWarnings({"unchecked"})
        @Override
        public void eraseAll(Set entries) {
            for (BinaryEntry entry : (Set<BinaryEntry>)entries) {
                erase(entry);
        @Override
        public void load(BinaryEntry binaryEntry) {
            Object value = wrapped.load(binaryEntry.getKey());
            binaryEntry.updateBinaryValue((Binary) context.getValueToInternalConverter().convert(value));
        @SuppressWarnings({"unchecked"})
        @Override
        public void loadAll(Set entries) {
            for (BinaryEntry entry : (Set<BinaryEntry>)entries) {
                load(entry);
        @Override
        public void store(BinaryEntry binaryEntry) {
            wrapped.store(binaryEntry.getKey(), binaryEntry.getValue());
        @SuppressWarnings({"unchecked"})
        @Override
        public void storeAll(Set entries) {
            for (BinaryEntry entry : (Set<BinaryEntry>)entries) {
                store(entry);
    }Using the JPA example from the Coherence 3.6 Tutorial you would configure it like this...
    <distributed-scheme>
        <scheme-name>jpa-distributed</scheme-name>
        <service-name>JpaDistributedCache</service-name>
        <backing-map-scheme>
            <read-write-backing-map-scheme>
                <internal-cache-scheme>
                    <local-scheme/>
                </internal-cache-scheme>
                <cachestore-scheme>
                    <class-scheme>
                        <class-name>org.gridman.coherence.cachestore.WrapperBinaryCacheStore</class-name>
                        <init-params>
                            <init-param>
                                <param-type>com.tangosol.net.BackingMapManagerContext</param-type>
                                <param-value>{manager-context}</param-value>
                            </init-param>
                            <init-param>
                                <param-type>java.lang.ClassLoader</param-type>
                                <param-value>{class-loader}</param-value>
                            </init-param>
                            <init-param>
                                <param-type>java.lang.String</param-type>
                                <param-value>{cache-name}</param-value>
                            </init-param>
                            <init-param>
                                <param-type>com.tangosol.run.xml.XmlElement</param-type>
                                <param-value>
                                    <class-scheme>
                                        <class-name>com.tangosol.coherence.jpa.JpaCacheStore</class-name>
                                        <init-params>
                                            <init-param>
                                                <param-type>java.lang.String</param-type>
                                                <param-value>{cache-name}</param-value>
                                            </init-param>
                                            <init-param>
                                                <param-type>java.lang.String</param-type>
                                                <param-value>com.oracle.handson.{cache-name}</param-value>
                                            </init-param>
                                            <init-param>
                                                <param-type>java.lang.String</param-type>
                                                <param-value>JPA</param-value>
                                            </init-param>
                                        </init-params>
                                    </class-scheme>
                                </param-value>
                            </init-param>
                        </init-params>
                    </class-scheme>
                </cachestore-scheme>
            </read-write-backing-map-scheme>
        </backing-map-scheme>
        <autostart>true</autostart>
    </distributed-scheme>As you can see the WrapperBinaryCacheStore takes four cpnstructor parameters (set up in the init-params)
    <li>First is the Backing Map Context
    <li>Second is the ClassLoader
    <li>Third is the cache name
    <li>Fourth is the XML configuration for the cache store you want to wrap
    If the load method of the wrapped cache store returns null (i.e. nothing in the DB matches the key) then instead of returning null, the BinaryEntry is updated with a Binary representing null. Because the corresponding key is now in the cache with a value of null then the cache store will not be called again for the same key.
    Note If you do this and then subsequently your DB is updated with values for the keys thet were previously null (by something other than Coherence) then Coherence will not load them as it is never going to call load for those keys again.
    I have given the code above a quick test and it seems to work fine.
    If you are using 3.5 then you can still do this but you need to use the Coherence Incubator Commons library which has a version of BinaryCacheStore. The code and config will be similar but not identical.
    JK
    Edited by: Jonathan.Knight on Jan 6, 2011 3:50 PM

  • MapListener & MapEvent Network Details

    Hello
    What are the network details concerning MapListeners within a Coherence cluster?
    MapListeners aren't required to be serializable, so this indicates that MapListeners are maintained on the heap of the JVM that has registered the MapListener. I'm concerned with purely client-side MapListeners here, not backing map listeners or Extend listeners.
    What is flying around the cluster when I do a put(.., ..) on a distributed (partitioned, replicated, etc.) cache? The put() call can occur from any node in the cluster, even one that does not have a MapListener registered locally. Is the Cluster constantly generating and sending (possibly broadcasting) MapEvents in response to puts() and removes(), even in the case where there are no client-side MapListeners on any node in the cluster? Does the cluster infrastructure blindly send (possibly broadcast) a MapEvent (or some internal representation) to all cluster nodes?
    I ask out of interest in deepening my understanding of Coherence. The documentation is quite clear in terms of the API, but it makes no mention of what is happening under the covers, and I would like to know.
    Regards, Pete

    Hi Pete,
    What is flying around the cluster when I do a put(.., ..) on a distributed (partitioned, replicated, etc.) cache? The put() call can occur from any node in the cluster, even one that does not have a MapListener registered locally. Is the Cluster constantly generating and sending (possibly broadcasting) MapEvents in response to puts() and removes(), even in the case where there are no client-side MapListeners on any node in the cluster? Does the cluster infrastructure blindly send (possibly broadcast) a MapEvent (or some internal representation) to all cluster nodes?The behaviour is that, if there is no client-side MapListener registered, no event will be sent. And when there is a client-side MapListener registered, the event/message is sent basically on UDP only to the nodes registered with MapListener.
    Hope this clear your queries.
    Regards,
    Rock
    Edited by: Rock Ching on Jan 20, 2011 12:57 AM

  • Can coherence compress data before storing in heap ?

    Hi,
    I have to store large amount of data into cache, just wanted to know if coherence provide compression facility which could help me store more data without adding extra hardware.
    I know about Elastic data concepts , but I don't want to go for this straight way. I am not sure how performent it will be if backup count is 0. Well I would first like to discover if there is anything related to compression already support by coherence which can be enable/disable by config.

    If you are planning to store this large amount of data in a distributed (partitioned) cache, then look at making it POF format. I've found that POF serialised data can take 1/4 of the space of regular Java serialised objects. Use a tool like JConsole/VisualVM to have a look at the before and after results for your data objects.
    Cheers,
    Steve

  • How to accelerate by partitioning drives & how to distribute data among 'em

    Dear forum,
    I have read guide to storage acceleration and guides to phototoshop acceleration, but they always warn that the best solution depends on the work i do, the hardware i have, and the hardware i think i can afford to buy. I'm hoping that if i tell you what photoshop work i do, what hardware i have, and what hardware i'm intending to buy, you can tell me how to accelerate by partitioning my drives and how to distribute data among them. My biggest questions are about how big the volumes should be, and what should go on each volume. It sounds vague here, but I get more specific below:
    THE PHOTOSHOP WORK I DO:
    *wet-mount raw scans of 6x7 cm film using silverfast software on microtek artixscan 120tf 4000dpi scanner: resulting 16-bit TIFF file is typically 550 MB in size.
    *working in Photoshop CS2 on same file, adding multiple layers makes file 1 GB to 1.4 GB in size
    *my system's limitations show up most painfully when I OPEN a file (this can take five minutes) SAVE a file (this can take more than ten minutes!), when i FLATTEN the image's layers for printing (this can take 5 minutes), and when i CONVERT the file from 16-bit to 8-bit (this can take 5 minutes). most other operations in Ps CS2 are fast enough (not snappy, but fast enough) for me to stay with my current processor for the time being.
    THE HARDWARE I HAVE:
    *Power Mac G5 dual 1.8GHz, made in 2004, with only 4 slots for RAM (not 8 slots).
    (I'm told this has quite limited bus speed, as compared with other dual-processor G5s, and that this hardware will not benefit much at all from adding a RAID array.)
    *one internal seagate 80GB 7200rpm SATA drive. this is half-full (it has 39 GB on it): it holds my OS and my Users folder, but NOT my photoshop image files.
    *one internal Western DIgital 400 GB 7200rpm SATA drive. this holds my photoshop image files, but not my user folder.(This WD drive turns out to cause the G5 to hang up occasionally, requiring a re-boot; to avoid this, i recently learned, i can connect it with a host card adapter [see below].)
    *two 500 GB external firewire drives
    *two 300GB external USB drives
    *I have 2.25 GB of RAM, but I'm about to buy 2 more GB to max out at 4GB.
    THE HARDWARE I'M INTENDING TO BUY:
    *2GB of RAM, of course.
    *two Hitachi T7K500 500 GB SATAII HD 16MB Cache 7200rpm drives to occupy both internal drive slots in the G5
    *a 2-drive external enclosure to hold my old seagate 80GB drive and my old WD400GB drive.
    *a seritek host card adaptor for connecting the external enclosure to the G5.
    THE PLAN:
    What follows is a combination of suggestions I have received about what I could do and my speculation about how I could do it. Please see my Questions, embedded in the lines below: I'd be very grateful for any amendments or directions you can offer on this topic.
    Drive A: first newly internal Hitachi 500GB drive:
    partition into 2 volumes:
    first (faster) volume, "volume A1," of 100GB to hold OS and Users folder but NOT photoshop image files.
    (Question: how much space should I leave free on volume A1 for optimum performance? is 50% free of 100GB optimal? is 60% free of 100GB better? Is 50% free of 150GB better still? or does that cut into the other volume's space too much (indirectly cutting into the space of "volume B1" on Drive B, which is to be the WorkDisk/ScratchDisk)?
    second (slower) volume, "volume A2" of remainder GB (almost 400GB) as backup for 400GB "volume B1" of the OTHER internal Hitachi Drive, a.k.a. Drive B.
    Drive B: second newly internal Hitachi 500GB drive:
    partition into 2 volumes:
    first (faster) volume, "volume B1" of almost 400GB as designated WorkDisk/ScratchDisk for large photoshop image files;
    second (slower) partition "volume B2" (exactly 100GB) as backup for 100GB volume 1 (OS volume) of the OTHER internal Hitachi Drive, a.k.a. Drive A.
    (Question: how much space should I leave free on this WorkDisk/ScratchDisk for optimum performance? is 50% free of almost 400GB optimal? is 60% free of almost 400GB better? Is 50% free of 300GB just as good, with the additional advantage of indirectly allowing "volume A1" on Drive A to be 150+GB?
    Drive C: old Seagate 80GB drive, in external enclosure: disk designated for running the Photoshop Application? How would I set this up? any pitfalls to watch out for? should i partition this drive, or leave the whole thing for Photoshop? or is it better to run photoshop off Drive D?
    Drive D: old WD 400 GB Drive: second scratch disk? Storage disk? Both storage and scratch disk? how large should an empty volume on this disk be in order to be useful as a scratch disk? volume 1 or volume 2? if i run the Photoshop Application off of this drive, how large should the volume for that be? should it be volume 1, the faster, outside volume, leaving volume 2 for scratch disk space? or vice versa?
    External Firewire and USB drives: i guess i'll just use them for storage/archiving and extra backup? or am i much safer buying more SATAs and Enclosures? or are the external firewire and USB drives plenty safe (so long as i double-back up), since i'll only power them up for the data transfer, and then power them back down?
    Given that the large Photoshop files are not in my User folder, does it matter whether i keep the User folder (with its MS Word docs and a bunch of PDFs and so on) on my OS volume, "volume A1"? would it speed things up when I'm using photoshop if i moved the Users folder to another drive? what if i'd like to play iTunes while also working on photoshop? my iTunes music folder (with all the song data) is already on an external firewire drive. but the iTunes Library and iTunes application are, of course, in my User folder, which is on the OS drive. would moving the Users folder to another drive make much difference when i use photoshop and iTunes simultaneously?
    But I wonder whether it makes sense to be using volume A2 on Drive A as a backup drive: wouldn't it make more sense to back up my working files to two external drives that can be traded out, one on-site and one off-site, back and forth (not so convenient when one of the backup drives is internal!)? and after all, why would i devote a 400GB volume to the task of backing up another 400GB volume that will never be more than half full? I need to leave a WorkDisk/ScratchDisk half empty for efficient use, but i can back up that 200GB of working files on a 200GB volume, right? so for a backup drive, I might as well use a slow, inexpensive external USB drive that will only be tuned on for backup and will then stay powered off, a drive that's easily transportable on and off site, right? or am i misunderstanding something?
    by the way, what backup software do you recommend for backing up back and forth between Drive A and Drive B? I've been using Carbon Cpy Cloner. do you recommend that? or something that does more archiving of progressive states of data?
    Thank you for any help you can offer!
    Sincerely,
    Mark Woods
    Dual 1.8 GHz PowerPC G5 (2.2), 512 KB L2 Cache per CPU, w/ 4 RAM slots   Mac OS X (10.3.9)   2.25 GB DDR SDRAM (2x128MB plus 2x1GB)

    Crossposted: Re: How to use Oracle partitioning with JPA @OneToOne reference?

  • Make parallel query (e.g. partition-wise join) evenly distributed cross RAC

    How To Make Parallel Query's Slaves (e.g. full partition-wise join) Evenly Distributed Across RAC Nodes?
    Environment
    * 4-node Oracle 10gR2 (10.2.0.3)
    * all instances are included in the same distribution group
    * tables are hash-partitioned by the same join key
    * 8-CPU per node, 48GB RAM per node
    Query
    Join 3 big tables (each has DOP=4) based on the hash partition key column.
    Problem
    The QC is always on one node, and all the slaves are on another node. The slave processes are supposed to be distributed or allocated to multiple nodes/instances, but even the query spawns 16 or more slaves, these slaves are running from only one node. And the QC process is never running on the same node! ? ! ? !
    The other 2 nodes are not busy during this time. Is there any configuration wrong or missing here? Why can't the RAC distribute the slaves better, or at least run some slaves together with QC?
    Please advise.
    Thank you very much!
    Eric

    Hi,
    If your PARALLEL_INSTANCE_GROUP and LOAD_BALANCING is set properly it means the oracle is assuming that intra node parallelism is more beneficial than inter node parallelism, i mean parallelism across multiple nodes. This is very true in scenarios where partition wise joins are involved. intra node parallelism avoids unnecessary interconnect traffic.

Maybe you are looking for

  • Error Message while sending a photo message

    While trying to send a photo message the following failed error message occurs   1xDNS not responding.   Phone being used is a Casio GzOne

  • Mail doesn't work properly on my brand new MacPro

    I receive mails with hours delay - if at all. Reading the Support Communities apparently it is not only me. I have hotmail, Gmail, exchange, ICloud mailadresses, same problem with all of them. I downloaded airmail - no problems!!!! Can anyone help me

  • PO bill to address not appearing

    Hi, In one my the user's purchase order when we click on print preview we have a section called please bill to which contains the billing address. In this case why is the section not appearing? while other sections like vendor ,deliver to appears cor

  • RTSI pulse using a CAN frame

    Hi, I'm currently using the ncWriteMult function (we can't use the ncWrite function because our application simulates problems of transmission on the CAN i.e. stopping an exact amount a frames or sending frames with the wrong DLC => we need a good ac

  • Photoshop CC already installed, CC still offers to download/install

    I have been using Photoshop CC (Photography bundle) for several months now. When I use the Update option, it sends me to the Adobe Application Manager, which offers me updates on... CS6 applications!!! Then, if I open CC, I am offered to install Phot