Message filtering in propagation process (stream replication environment)

Hi!
We have fine configured stream replication in star topology:
ORCL2 <=> ORCL1 <=> ORCL3
Where the ORCL1 is "headquarters" and there is no message flow between ORCL2 and ORCL3.
For some reason we want to filter messages in propagation processes, e.g. DML captured on ORCL1 should be replicated only to ORCL2 or only to ORCL3. There is one propagation process for each "satellite" database.
To solve this problem I have written function:
FUNCTION Replicate_Lcr (
p_lcr IN SYS.lcr$_row_record)
RETURN VARCHAR2 IS
which will be making a decision whether to pass the message (return 'Y') or not (return 'N').
But there is problem: rule is evaluated and function is executed (there is insert into 'stream_log_lcr' table) but value of the expression seems to be 'FALSE' and message (LCR) is not beeing sent to ORCL2 (or to ORCL3).
When I remove function 'Replicate_Lcr' from propagation rule condition then every message captured by capture process on ORCL1 reaches destination database (ORCL2 or ORCL3).
The second observation is that, if I run the same code on ORCL2 or ORCL3 then everything seems to be OK: there is insert into 'stream_log_lcr' table and DML captured on ORCL2 (or ORCL3) appears in ORCL1 ("headquarters").
I suppose that this could be problem with other version of database on "headquarters" (ORCL1) or configuration issues.
I will appreciate every suggestion.
Databases:
ORCL1: 64-bit Windows, ver. 10.2.0.4.0, Windows 2008 server
ORCL2: 32-bit Windows, ver. 10.2.0.1.0, Windows XP
ORCL3: 32-bit Windows, ver. 10.2.0.1.0, Windows XP
SQL code run on ORCL1:
CREATE TABLE stream_log_lcr (
data DATE NOT NULL,
msg SYS.lcr$_row_record NOT NULL
-- simplified, always return 'Y'
CREATE OR REPLACE FUNCTION Replicate_Lcr (
p_lcr IN SYS.lcr$_row_record)
RETURN VARCHAR2 IS
PRAGMA AUTONOMOUS_TRANSACTION;
BEGIN
IF p_lcr IS NOT NULL
THEN
INSERT INTO stream_log_lcr
VALUES (SYSDATE,
p_lcr);
COMMIT;
END IF;
RETURN 'Y';
END;
-- create propagation process with above function in rule condition
BEGIN
DBMS_STREAMS_ADM.add_schema_propagation_rules
(schema_name => 'data_schema',
streams_name => 'primary_to_secondary2',
source_queue_name => 'strmadmin.capture_primary',
destination_queue_name => 'strmadmin.from_primary@ORCL2',
include_dml => TRUE,
include_ddl => TRUE,
source_database => 'ORCL1',
and_condition => ' strmadmin.Replicate_Lcr(:dml) = ''Y'' ',
inclusion_rule => TRUE,
queue_to_queue => TRUE);
END;
-- check if function 'Replicate_Lcr' was evoked:
SELECT * FROM stream_log_lcr ORDER BY data;

hi porzer,
In Propagation process ( source) also 0 errors. But in apply ( dest ) under statistics under server status is displaying as IDLE. And Coordinator status is APPLYING. In Capture (source) no error. In Applying ( dest ) no error. what else i can do please?
up to now what i did i am telling:
I had 2 databases had one table same table. one database i changed the mode to ARCHIVELOG mode. Another database is in NOARCHIVELOG mode only. In first database setup streams i run. And i run manually 2 .sql files and one .dat file like this :
SQL>@e:\oracle\product\10.2.0\client_2\sysman\report\OTEST_ADMIN_NON_OMS_SETUP.sql
SQL>host e:\oracle\product\10.2.0\client_2\sysman\report\OTEST_ADMIN_NON_OMS_exportimport.bat
SQL>@e:\oracle\product\10.2.0\client_2\sysman\report\OTEST_ADMIN_NON_OMS_startup.sql
Any thing else i can do? i didn't have metalink registration.i hope i am not boring you.
Thanks in advance.

Similar Messages

  • Dequeue Messages queued by caprture process

    RDBMS Version: 10.1.0.4
    Operating System and Version: WINDOWS 2003 SERVER
    Error Number (if applicable):
    Product (i.e. SQL*Loader, Import, etc.): Oracle Streams
    Hi,
    Can I do the following.
    I had set up couple of databases for replication using ORACLE STREAMS in hub configuration(One central and many local). My problem is that sometimes some of the connection between them will not work for a significant period of time. That's why I need some tool to move the data.
    So I decide to dequeue the which are captured by the CAPTURE process and to process them with a PL/SQL program, save their payload in a file or external table and then trough some way to bring this data to the master server and the other way round.
    But I can't dequeue messages because the agent created for the propagation process has no name and I can't dequeue messages if I don't have the CONSUMER.
    All examples that I saw in Metalink use already propagated messages with a propagation process.
    So I need some other way to propagate my messages different then propagation process! The task is even more complicated because at the central DB some of the messages has a multiple consumers...
    Any suggestions?

    Wel in 10g it is just as simple as: create a apply-process by adding a table/schema rule that matches the enqueued messages. If you have a captureprocess for several tables off a schema, then a schema rule for that schema owning those tables would do. Then with the set_enqueue_destination api you can add a queue on this rule. Which means that the apply process won't apply those messages but reroutes them on your user-queue. This queue you can create by yourself but it must be of type 'sys.anydata'.
    The messages enqueued on that queue by the apply process can be dequeued by a user process.
    So just put-in an apply process that dequeues the captured LCR's for you and enqueues them on your user-queue. And then you can dequeue them.
    Regards,
    Martien

  • Can you use streams replication to replicate an advanced queue?

    We need to be able to support failover of an Advanced Queue between a primary database instance and one or more alternate instances. To insure consitancy across the multiple-database instances, every enqueued and dequeued message must be replicated in the event of a failure on the primary node. TAF is used to automatically failover the App to an alternate database instance. Without getting into the details, Data Guard/Standby/RAC are NOT options.
    Questions:
    Is replication of an Advanced Queue supported via Streams Replication?
    Are there guidlines/recommendations on how this should be done/setup?

    No. AQ is not supported by Oracle Streams. User defined and Sys.AnyData are not supported types.
    You can create AQ propagation process from source to backup site. But you will need to dequeue both sites simultaniously.
    Or you can create schadow table (

  • Streams Replication:Source database Physical or Logical Standby DB

    Can the source database in streams replication be a physical or logical standby database ? If so, is the process of configuring streams the same as a regular database ? Are there any best practices or different configuration if the source is Logical or Physical standby DB ?
    Thanks in advance.

    Never done it, but I don't see any reason why it should not work.
    Streams, at capture site, is only a data dictionary game and in a logical standby your data dictionary is open read write.
    Streams, at capture site, never touch the source tables, in fact they may even not exists from Streams point of view,
    as it deals only with the redo that are generated.
    So Streams horizon is limited to the data dictionary, the log buffer, the archives and, in SYSAUX tablespace, all the LOGMNR_% tables. All these structures are read write in the logical standby. However, for the capture/propagation you may have to set to true the 'include_tagged_lcr' parameters.

  • Production moves to another server with existing streams replication

    Hi All,
    We have oracle bi directional streams replication setup between 2 server(us and uk). Everything is working as expected.
    But,We are going to move UK server from (ex: cam19 to cam29).
    Will it impact on replication setup.
    we will restore new server via cold backup.Also we will stopped all the process on both the sides ,before cold backup.
    Can you please suggest? What should we do before doing that.
    How we can avoid any problem while moving server.
    so it will not impact on replication.
    Thanks in advance!!!!
    Many Thanks
    nick

    Thanks a lotz Anurag..
    I have one small question not on same topic, Its regarding table add in existing replication setup.
    What happened ,When we add a new table to existing replication setup if any reason table is not replicating between two database then we have to remove the rules for that particular table and setup again. Some time what happened we got error "queue has errors" i dont know the ORA number.
    in that case what cases when apply process ABORETED and when we try to start the process it gives same error and ABORTED again.
    then we have remove the whole replication Manually and setup again. It's very horriable....
    Could you please help that before drop the rules for particular table then wht should we do ? Do we need to unscheduled the propagation process and then drop the rules becuase i read on metalink that negative rules drops while propagation process using the same rule set.
    Please Suggest!!!!!!!!!!
    Many Thanks

  • How to debug the propagation process?

    Hi,
    I'm trying to figure out why my streams setup is not dequeueing the capture queue and propagating.
    Please refer to the following thread for some history of the setup.
    Capture and Propagation process interaction
    I'm actually getting quite pissed off with all this as it should "JUST WORK". What i'm trying to do is prob the simplest stream setup. So either i'm stupid and done something fundamentally wrong or streams is sh!t. I'm hoping/assuming its the former for the time being.
    Message was edited by:
    JA

    There is a way to trace propagation process. But it would not be very easy to undestand output.
    You can find propagation job sql from what column in dba_jobs.
    You can enable trace for regular sqlplus session and execute propagation.
    I am not sure if it would be beneficial fo you to use this method.
    I would recomend to check propagation rules one more time. Question: Do you really need propagation rules? In most of time it is make sense to remove rule set from propagation process.
    Regards

  • Replication environment

    I'm trying to create replication environment with one master site and one materialized view site in my test databases on the same node. I followed step-by-step the example in the Replication Management API Reference book ( from June 2001 ). Replication environment and replication object is created successfully but replication not runing properly and dont send any error message. My ORACLE Server 9i (9.0.1.1) Enterprise Edition for Windows NT/2000/XP Pro runing on Win2000 Advanced server.
    Thanks.
    Attila Fintor

    Have you set the system variable job_queue_processes to be higher than 0
    e.g. alter system set job_queue_processes = 2;
    I had this problem and searched for a while till i found the reason.
    If this is not set correctly, manual refresh will work but no automatic
    refresh may run since no backgraound processes are allowed!

  • Extensive IO after Configuring DB Streams Replication

    Hi all,
    <br><br>
    I'm learning on configuring and managing Oracle Streams Replication.
    <br><br>
    I managed to configure a bidirectional Database replication using the PRE_INSTANTIATION_SETUP and POST_INSTANTIATION_SETUP procedures in the DBMS_STREAMS_ADM package between two databases (DB1 and DB2). I used RMAN DUPLICATE command for instantiation.
    <br><br>
    Here is my question:
    <br><br>
    Once the configuration is finished, <b>I noticed an extensive IO activities on the hardisks in both DB1 and DB2</b>, although there were no DML or DDL against any of the databases by users. Is that normal in Oracle Streams environment? What is the reason of it? May be it is this way only in the first few hours of finishing this configuration and eventually the IO will become stable.
    <br><br>
    Oracle 10g R2 on Windows XP SP2

    Could be the writing of data dictionary to redo logs on target and the mining of it on the source.
    Did it stop after a while?

  • Streams replication in Oracle XE.

    Hi,
    I'm planning an implementation of Oracle Streams replication. Some of the considerations and facts are stated below.
    1. DDL Changes will be maintained.
    2. Replication will be Bi-Directional.
    3. The operating system on the source and downstream sites are the same.
    4. The hardware and operating system architecture on the source and downstream sites are the same.
    5. Both the source database and the downstream database are be running Oracle Database 10g (details below).
    Source Database
    Oracle Database 10g Express Edition Release 10.2.0.1.0 - Beta
    PL/SQL Release 10.2.0.1.0 - Beta
    CORE 10.2.0.1.0 Beta
    TNS for 32-bit Windows: Version 10.2.0.1.0 - Beta
    NLSRTL Version 10.2.0.1.0 - Beta
    Downstrean Database
    Personal Oracle Database 10g Release 10.1.0.2.0 - Production
    PL/SQL Release 10.1.0.2.0 - Production
    CORE 10.1.0.2.0 Production
    TNS for 32-bit Windows: Version 10.1.0.2.0 - Production
    NLSRTL Version 10.1.0.2.0 - Production
    Here I have found that Oracle XE (source) is not supporting the local capture process.
    My questions:
    1. Can I consider downstream capture in this scenario? (As I have Personal Oracle Database 10g Release 10.1.0.2.0 as downstream database)
    2. In case of archive log downstream capture what will be the most efficient way of redo log transportation.
    Thanks in advance.

    1. In a RAC cluster? The name I would put in the scripts is "insanity." Why? You are going to bring this instance to a crawl.
    2. No problem in doing a transformation. The problem was in your first question.
    3. I wouldn't worry much about monitoring it. I'd just get a stopwatch and see how many seconds before it crashes.
    If the data is in one schema in the database then just make it available, via object privs, to the others. Replicating it, as you have described, is likely going to be a painful experience in far more ways than just wasted storage.

  • Tracing execution and standard Forte message filters

    Hello,
    During the process of tuning an application, or just of making it work, it
    is useful to trace the flow of processing.
    There are two ways of doing it :
    1) add your own trace instructions (calls to the LogMgr) at the appropriate
    places,
    2) use the traces of the Forte Interpreter !
    If one relies on solution 1, then it relies on what the developers have
    written (and thus sometime forgotten to write !) in their code.
    If one relies on solution 2, then potentially it can get as much information
    as is available to the interpreter (and the debugger ?), in a fully
    independant way since it uses directly the code itself and not added trace
    instructions.
    So solution 2 seems quite interesting.
    Unfortunately there are some potential problems :
    a) I haven't found in the documentation an exhaustive description of the
    logs the Forte tools do. The only and very short description is on page 148
    of the System management Guide. It's far from being exhaustive. So it
    requires playing with the filters.
    I recommand trying trc:in:1-63. I guess "in" stands for the interpreter.
    - level 1 seems to give the call tree,
    - level 255 seems to give almost the code !
    b) since the flags are not documented by Forte, how reliable and stable will
    they be in future versions ?
    c) what happens for compiled partitions is not clear to me (I have not tried
    it yet).
    So my question : are the message filters used by the Forte Tools like the
    interpreter described somewhere, i.e. in some Tech Note (I don't have access
    to them yet) ?
    best regards,
    Pierre Gelli
    ADP GSI
    Payroll and Human Resources Management
    72-78, Grande Rue, F-92310 SEVRES
    phone : +33 1 41 14 86 42 (direct) +33 1 41 14 85 00 (reception desk)
    fax : +33 1 41 14 85 99

    From: Pierre Gelli <[email protected]>
    Subject: tracing execution and standard Forte message filters
    Hello,
    During the process of tuning an application, or just of making it work, it
    is useful to trace the flow of processing.
    There are two ways of doing it :
    1) add your own trace instructions (calls to the LogMgr) at the appropriate
    places,
    2) use the traces of the Forte Interpreter !
    So solution 2 seems quite interesting.
    Unfortunately there are some potential problems :
    a) I haven't found in the documentation an exhaustive description of the
    logs the Forte tools do. The only and very short description is on page 148
    of the System management Guide. It's far from being exhaustive. So it
    requires playing with the filters.
    I recommand trying trc:in:1-63. I guess "in" stands for the interpreter.
    - level 1 seems to give the call tree,
    - level 255 seems to give almost the code !
    b) since the flags are not documented by Forte, how reliable and stable will
    they be in future versions ?
    Pierre Gelli,
    level 255 is the most detailed you are right on tracing..... As for documentation
    you will want to get ahold of several good tech notes that your Forte
    rep or support can get you which provide alot of the info you are after.
    Let me know if you can't do this and I can send you some of this info, but you are
    best to get the latest and greatest directly from Forte.
    Len Leber
    ATG Partners

  • Streams Replication: Logical Standby DB as source

    I don't have much experience with streams replications. So, the following might sound like a silly questions to Oracle Gurus:
    If the source database is a logical standby database, can stream replication be used to replicate some of the tables in the source database to the target DB ?
    The source and the target DB are 11g R2.
    Thanks in adavance:
    - Sanjay
    Edited by: sanjayku on Dec 8, 2010 4:38 PM

    Never done it, but I don't see any reason why it should not work.
    Streams, at capture site, is only a data dictionary game and in a logical standby your data dictionary is open read write.
    Streams, at capture site, never touch the source tables, in fact they may even not exists from Streams point of view,
    as it deals only with the redo that are generated.
    So Streams horizon is limited to the data dictionary, the log buffer, the archives and, in SYSAUX tablespace, all the LOGMNR_% tables. All these structures are read write in the logical standby. However, for the capture/propagation you may have to set to true the 'include_tagged_lcr' parameters.

  • Weblogic Messaging Bridge - T3S Enabling Process

    Dear All,
    I need your help to resolve an issue in Weblogic Messaging Bridge - T3S enabling process.
    I'm doing a POC to forward messages between 2 different domains (from wl 10.3.2 to wl 10.0) using Messaging Bridge with T3S Protocol.
    I enabled SSL Port, Custom Identity and Custom Trust in those domains.
    In my laptop (single JVM), WL Messaging Bridge in 10.3.2 is able to forward messages between domains using T3S.
    My Testing Scenario:
    A Sample web application use the https/t3s protocol to put message(s) in the source jms queue                    
    Weblogic Messaging Bridge use the t3s protocol to get message(s) from source jms queue & forward the message(s) to the destination jms queue                    
    But, When I tried the same POC in sepearate boxes (WL 10.3.2 in one box, WL 10.0 in another box). I got the below exceptiion.
    Any one please throw some lights here to resolve the issue.
    ####<Oct 12, 2011 4:11:49 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1318416109085> <BEA-000000> <Exception:
    javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://172.16.2.25:4936: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination]
         at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
         at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:783)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:367)
         at weblogic.jndi.Environment.getContext(Environment.java:315)
         at weblogic.jndi.Environment.getContext(Environment.java:285)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at weblogic.jms.adapter.JMSBaseConnection.getInitialContext(JMSBaseConnection.java:2253)
         at weblogic.jms.adapter.JMSBaseConnection.startInternal(JMSBaseConnection.java:261)
         at weblogic.jms.adapter.JMSBaseConnection.start(JMSBaseConnection.java:247)
         at weblogic.jms.adapter.JMSManagedConnectionFactory.createManagedConnection(JMSManagedConnectionFactory.java:190)
         at weblogic.connector.security.layer.AdapterLayer.createManagedConnection(AdapterLayer.java:802)
         at weblogic.connector.outbound.ConnectionFactory.createResource(ConnectionFactory.java:91)
         at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:395)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:315)
         at weblogic.connector.outbound.ConnectionPool.reserveResource(ConnectionPool.java:596)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:309)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnectionInfo(ConnectionManagerImpl.java:352)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:307)
         at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)
         at weblogic.jms.adapter.JMSBaseConnectionFactory.getTargetConnection(JMSBaseConnectionFactory.java:123)
         at weblogic.jms.bridge.internal.MessagingBridge.getConnections(MessagingBridge.java:860)
         at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1051)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    Caused By: java.net.ConnectException: t3s://172.16.2.25:4936: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination
         at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
         at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:153)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:352)
         at weblogic.jndi.Environment.getContext(Environment.java:315)
         at weblogic.jndi.Environment.getContext(Environment.java:285)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at weblogic.jms.adapter.JMSBaseConnection.getInitialContext(JMSBaseConnection.java:2253)
         at weblogic.jms.adapter.JMSBaseConnection.startInternal(JMSBaseConnection.java:261)
         at weblogic.jms.adapter.JMSBaseConnection.start(JMSBaseConnection.java:247)
         at weblogic.jms.adapter.JMSManagedConnectionFactory.createManagedConnection(JMSManagedConnectionFactory.java:190)
         at weblogic.connector.security.layer.AdapterLayer.createManagedConnection(AdapterLayer.java:802)
         at weblogic.connector.outbound.ConnectionFactory.createResource(ConnectionFactory.java:91)
         at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:395)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:315)
         at weblogic.connector.outbound.ConnectionPool.reserveResource(ConnectionPool.java:596)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:309)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnectionInfo(ConnectionManagerImpl.java:352)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:307)
         at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)
         at weblogic.jms.adapter.JMSBaseConnectionFactory.getTargetConnection(JMSBaseConnectionFactory.java:123)
         at weblogic.jms.bridge.internal.MessagingBridge.getConnections(MessagingBridge.java:860)
         at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1051)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    Caused By: java.rmi.ConnectException: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:464)
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:315)
         at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:251)
         at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:194)
         at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
         at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
         at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:153)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:352)
         at weblogic.jndi.Environment.getContext(Environment.java:315)
         at weblogic.jndi.Environment.getContext(Environment.java:285)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at weblogic.jms.adapter.JMSBaseConnection.getInitialContext(JMSBaseConnection.java:2253)
         at weblogic.jms.adapter.JMSBaseConnection.startInternal(JMSBaseConnection.java:261)
         at weblogic.jms.adapter.JMSBaseConnection.start(JMSBaseConnection.java:247)
         at weblogic.jms.adapter.JMSManagedConnectionFactory.createManagedConnection(JMSManagedConnectionFactory.java:190)
         at weblogic.connector.security.layer.AdapterLayer.createManagedConnection(AdapterLayer.java:802)
         at weblogic.connector.outbound.ConnectionFactory.createResource(ConnectionFactory.java:91)
         at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:395)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:315)
         at weblogic.connector.outbound.ConnectionPool.reserveResource(ConnectionPool.java:596)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:309)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnectionInfo(ConnectionManagerImpl.java:352)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:307)
         at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)
         at weblogic.jms.adapter.JMSBaseConnectionFactory.getTargetConnection(JMSBaseConnectionFactory.java:123)
         at weblogic.jms.bridge.internal.MessagingBridge.getConnections(MessagingBridge.java:860)
         at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1051)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    >
    ####<Oct 12, 2011 4:11:49 PM IST> <Warning> <Connector> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416109086> <BEA-190032> << eis/jms/WLSConnectionFactoryJNDINoTX > ResourceAllocationException thrown by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): "javax.resource.ResourceException: ConnectionFactory: failed to get initial context (InitialContextFactory =weblogic.jndi.WLInitialContextFactory, url = t3s://172.16.2.25:4936, user name = null) ">
    ####<Oct 12, 2011 4:11:49 PM IST> <Info> <MessagingBridge> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416109089> <BEA-200043> <Bridge "Bridge-0" failed to connect to the target destination and will try again in 45 seconds. This could be a temporary condition unless the messages repeat several times. (java.lang.Exception: javax.resource.ResourceException: ConnectionFactory: failed to get initial context (InitialContextFactory =weblogic.jndi.WLInitialContextFactory, url = t3s://172.16.2.25:4936, user name = null)
         at weblogic.jms.adapter.JMSBaseConnection.throwResourceException(JMSBaseConnection.java:1665)
         at weblogic.jms.adapter.JMSBaseConnection.throwResourceException(JMSBaseConnection.java:1645)
         at weblogic.jms.adapter.JMSBaseConnection.startInternal(JMSBaseConnection.java:460)
         at weblogic.jms.adapter.JMSBaseConnection.start(JMSBaseConnection.java:247)
         at weblogic.jms.adapter.JMSManagedConnectionFactory.createManagedConnection(JMSManagedConnectionFactory.java:190)
         at weblogic.connector.security.layer.AdapterLayer.createManagedConnection(AdapterLayer.java:802)
         at weblogic.connector.outbound.ConnectionFactory.createResource(ConnectionFactory.java:91)
         at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:395)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:315)
         at weblogic.connector.outbound.ConnectionPool.reserveResource(ConnectionPool.java:596)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:309)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnectionInfo(ConnectionManagerImpl.java:352)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:307)
         at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)
         at weblogic.jms.adapter.JMSBaseConnectionFactory.getTargetConnection(JMSBaseConnectionFactory.java:123)
         at weblogic.jms.bridge.internal.MessagingBridge.getConnections(MessagingBridge.java:860)
         at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1051)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    -------------- Linked Exception ------------
    javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://172.16.2.25:4936: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination]
         at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
         at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:783)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:367)
         at weblogic.jndi.Environment.getContext(Environment.java:315)
         at weblogic.jndi.Environment.getContext(Environment.java:285)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at weblogic.jms.adapter.JMSBaseConnection.getInitialContext(JMSBaseConnection.java:2253)
         at weblogic.jms.adapter.JMSBaseConnection.startInternal(JMSBaseConnection.java:261)
         at weblogic.jms.adapter.JMSBaseConnection.start(JMSBaseConnection.java:247)
         at weblogic.jms.adapter.JMSManagedConnectionFactory.createManagedConnection(JMSManagedConnectionFactory.java:190)
         at weblogic.connector.security.layer.AdapterLayer.createManagedConnection(AdapterLayer.java:802)
         at weblogic.connector.outbound.ConnectionFactory.createResource(ConnectionFactory.java:91)
         at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1180)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:395)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:315)
         at weblogic.connector.outbound.ConnectionPool.reserveResource(ConnectionPool.java:596)
         at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:309)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnectionInfo(ConnectionManagerImpl.java:352)
         at weblogic.connector.outbound.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:307)
         at weblogic.connector.outbound.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:115)
         at weblogic.jms.adapter.JMSBaseConnectionFactory.getTargetConnection(JMSBaseConnectionFactory.java:123)
         at weblogic.jms.bridge.internal.MessagingBridge.getConnections(MessagingBridge.java:860)
         at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1051)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    Caused by: java.net.ConnectException: t3s://172.16.2.25:4936: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination
         at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
         at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:153)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:352)
         ... 28 more
    Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090477]Certificate chain received from 172.16.2.25 - 172.16.2.25 was not trusted causing SSL handshake failure.; No available router to destination
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:464)
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:315)
         at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:251)
         at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:194)
         at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
         at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
         ... 31 more
    )>
    ####<Oct 12, 2011 4:11:49 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416109479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:50 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416110055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:50 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416110479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:51 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416111055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:51 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416111479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:52 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416112055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:52 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416112479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:53 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416113055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:53 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416113479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:54 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416114055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:54 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416114479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:55 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416115051> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:55 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416115479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:56 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416116055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:56 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416116479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:57 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416117055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:57 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416117479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:58 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416118055> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    ####<Oct 12, 2011 4:11:58 PM IST> <Debug> <MessagingBridgeRuntimeVerbose> <GLLNX01> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1318416118479> <BEA-000000> <Bridge Bridge-0 doTrigger(): state = 2 stopped = false>
    Thanks in advance.

    Think this can help you on the way - http://blog.eisele.net/2010/06/weblogic-server-ssl-httpst3s-and-java.html

  • Replication environment data inconsistency...

    Gurus,
    I have multi master replication environment there is one schema SGI on two databases LLU and BER data in LLU is correct and data in BER is incorrect.
    I need to drop SGI schema in BER and import export dump from LLU, but I don't want to break replication environment and rebuild it.
    I have checked DBMS_RECTIFIER_DIFF given in metalink but somehow could not find concrete example that would help me use it correctly.
    Does anyone have any idea about it? Or any other suggestion on how can I achieve my desired goal?
    Please advice.

    Gurus,
    I have multi master replication environment there is
    one schema SGI on two databases LLU and BER data in
    LLU is correct and data in BER is incorrect.
    I need to drop SGI schema in BER and import export
    dump from LLU, but I don't want to break replication
    environment and rebuild it.
    If you drop the schema you will destroy your replication environment anyway. In that case you have more work to do in order to cleanup the replication environment at your other master site.
    I don't think you want to do that!
    My personal opinion is that you should just remove from the replication environment the master replication objects that belong to the schema you want to refresh. Refresh the data (using import utility) and then create the replication objects again.
    I don't say it is easy, but I think it is the right way to go. Don't be lazy. Follow the rules written in the docs and you should be fine.
    The high level picture of the steps you have to do is:
    - Make sure there are no deferred transactions waiting to be applied. If there are, deal with them first.
    - From your master definition site suspend the replication activity (using dbms_repcat.suspend_master_activity)
    - From your master definition site, remove the master rep objects you want to refresh. (using dbms_repcat.drop_master_repobject)
    - Import the data in SGI BER
    - From your master definition site, create the master rep objects again using (DBMS_REPCAT.CREATE_MASTER_REPOBJECT and make sure you use copy_rows => FALSE and use_existing_object => TRUE)
    - Resume the master activity (using dbms_repcat.resume_master_activity)
    and that is it.
    And again, make sure you don't have deferred transactions before you start this.
    Read the docs until you fully understand what you have to do.
    In order to feel more comfortable, I would suggest you to test this approach first.
    I hope my advice will help you to find the right approach.
    P.S. Hermant's suggestion is also worth considering (the link he provided). The approach that I mentioned may require more work from your side, but it also gives you more control and flexibility so you would know what is going on in every moment.
    Message was edited by:
    tekicora

  • Oracle 10g streams replication problem.

    Hi,
    Can someone help me oracle replication troubleshooting ?
    I have source and target databases. I have setup one-way stream replication successfully. Whatever data I insert into table A, it is replicated to target database succesfully.
    However, if I delete the records from the destination database and then it does not apply old values of the source database to destination database.
    I reset the START_SCN in the source database and then I setup destination SCN number using DBMS_APPLY_ADM.SET_SCHEMA_INSTANTIATION_SCN and started capture process on the source database.
    Can somebody help how should I recover the destination database ?

    Hi Basu,
    Thanks a lot for response
    I set up streams between between source and destination database using schema instantiation and setup SCN number for destination after which apply process apply all the changes and all it is woking file.
    Now I deleted records from table of the same schema in the destination database. Now how do I get all the records in this table using streams.
    I looked at the oracle doc and found that I need to do build using dbms_streams_capture package and I did that and after that I reset START_SCN to the old value on the source database. I also reset the SCN at the SCHEMA label on the destination database for apply process to re-apply the old changes. But all this did not help me to get old data.
    Do you know specific steps which I need to do to get old data ?
    Thanks a lot

  • Trying to create a Photomerge in Bridge, when the files open in Photoshop ( small jegs ) , I get a error message of " Unable to process latte files " / " null is not an object "

    Trying to create a Photomerge in Bridge, when the files open in Photoshop ( small jegs ) , I get a error message of " Unable to process latte files " / " null is not an object "
    Please help

    Wait a second, Photoshop can make lattes now?  Personally I prefer my coffee black, but man have I been under-utilizing this program.

Maybe you are looking for