Change Data Capture, insert+delete instead of update

Hello everyone,
I'm having an odd issue, that I need some help understanding. I've got a number of tables enabled for Change Data Capture, but I've noticed some behavior that doesn't make sense.
I am utilizing CDC to populate an ODS via SSIS packages, and have noticed a few instances where I'm getting a delete operation come through, but the source record still exists. In the source system, I can query 'cdc.fn_cdc_get_net_changes_...'
and see combinations of 1 (delete) and 4 (update new) for the __$operation.
__$start_lsn, __$operation, TableId, ...
0x000BB75700002F250005, 1, 1938241
0x000BB75700002F250005, 4, 1938241
Also, if I query the cdc.<Table>_CT table directly, I see pairs of 1 and 2 (insert) for the same __$start_lsn and __$seqval. How can this happen? If I get *net* changes, shouldn't I only get one record back per the primary
key of the source table?
__$start_lsn, __$seqval, __$operation, __$update_mask, TableId, ...
0x000BB75700002E7C0005, 0x000BB75700002E7C0003, 1, 0x0FFF, 1938241
0x000BB75700002E7C0005, 0x000BB75700002E7C0003, 2, 0x0FFF, 1938241
0x000BB75700002F250005, 0x000BB75700002F250003, 1, 0x0FFF, 1938241
0x000BB75700002F250005, 0x000BB75700002F250003, 2, 0x0FFF, 1938241
Let me say, I've got a trace going to see if I can find the command(s) that is causing the duplicate results, but has anyone ever seen this before?
Thanks in advance for your time and help!
Jarret

This is called a deffered update. It is the way that some tables are updated - ie a delete followed by an insert.
See if the behavior still occurs when using trace flag 8201, ie
DBCC TRACEON (8207, -1)
looking for a book on SQL Server 2008 Administration?
http://www.amazon.com/Microsoft-Server-2008-Management-Administration/dp/067233044X looking for a book on SQL Server 2008 Full-Text Search?
http://www.amazon.com/Pro-Full-Text-Search-Server-2008/dp/1430215941

Similar Messages

Change data capture - ignore delete?

Hello,
I'm trying to solve an issue I met on a synchronous change data capture : I created a CDC table with the DBMS_LOGMNR_CDC_PUBLISH.CREATE_CHANGE_TABLE procedure:
BEGIN
   DBMS_LOGMNR_CDC_PUBLISH.CREATE_CHANGE_TABLE (
      OWNER             => 'APPO',
      CHANGE_TABLE_NAME => 'CT_TEMP_H1',
      CHANGE_SET_NAME   => 'SYNC_SET',
      SOURCE_SCHEMA     => 'APPO',
      SOURCE_TABLE      => 'TEMP_H1',
      COLUMN_TYPE_LIST => 'ISSUE VARCHAR2(10), TITOLO VARCHAR2(150)',
      CAPTURE_VALUES    => 'BOTH',
      RS_ID             => 'Y',
      ROW_ID            => 'N',
      USER_ID           => 'N',
      TIMESTAMP         => 'N',
      OBJECT_ID         => 'N',
      SOURCE_COLMAP     => 'Y',
      TARGET_COLMAP     => 'Y',
      OPTIONS_STRING    => NULL)
END;
Now, it works correctly, and in the CT_TEMP_H1 table I can find all the information about changes that occourr in the TEMP_H1 table
What I'd like to know is if there is a way to avoid the capture "trigger" to fire in the event of DELETE records: due to the application business logic, I'm interested just in insert and updates on the source table, and I don't mind about delete(s), so I don't want them to consume resources (both disk space and CPU) on the server.
Is there a supported way to do so?
Thanks in advance,
- Paolo

Thanks for your answer, Damorgan,
But the CAPTURE_VALUES parameter only manages the UPDATE issues, to have one row containing the row values before the update occurred and another row containing the row values after the update occurred.
What I'm interested in is aviod logging in the change table all the delete operations occourring in the source tables (a sort of "afert indert or update" trigger, instead of the "insert, update or delete"...)
Any hint?
- Paolo

Using Change Data Capture in SSIS - how to handle schema changes

I was asked to consider change data capture for a database recently. I can see that from the database perspective, its quite nice. When I considered how I'd do this in SSIS, it seemed pretty obvious that I might have a problem, but I wanted to
confirm here.
The database in question changes the schema about once per month in production. We have a lot of controls in our environment, so everytime a tables schema is changed, I'd have to do a formal change request to deal with a change to my code
base, in this case my SSIS package; it can be a lot of work. If I wanted to track the data changes for inserts, update and deletes using an SSIS package to send the data changes to the destination tables, would I have to change my SSIS package
with every schema change, or is there a way to keep the exact same SSIS package with CDC without having to change it every month?
Thanks,
Keith

Hi Keith,
What is your exact requirement?
If you want to capture the object_created, object_deleted or object_altered informations, you can try using
Extended events .
As mentioned in your OP:
"If I wanted to track the data changes for inserts, update and deletes using an SSIS package to send the data changes to the destination tables, would I have to change my SSIS package with
every schema change, or is there a way to keep the exact same SSIS package with CDC without having to change it every month?"
If you want the databases in two different environments to be in sync, then take periodic
backup and apply(restore) on the another destination DB.
(or)
you can also try with
SQL Server replication if it is really needed.
As I understand from your description, if you want the table data & schema to be in sync in two different database:
then create job [script that will drop the destination DB table & create the copy of source DB table ] as per your requirement:
--CREATE DATABASE db1
--CREATE DATABASE db2
USE db1
GO
CREATE TABLE tbl(Id INT)
USE db2
GO
IF EXISTS (SELECT * FROM SYS.OBJECTS WHERE name = 'tb1' and TYPE = 'u')
DROP TABLE dbo.tb1
SELECT * INTO db2.dbo.tb1 FROM db1.dbo.tbl
SELECT * FROM dbo.tb1
--DROP DATABASE db1,db2
sathya - www.allaboutmssql.com ** Mark as answered if my post solved your problem and Vote as helpful if my post was useful **.

Change Data Capture in SQL Server

Post Author: rtamanji
CA Forum: Data Integration
I am having primary key constraint problems whenever there is a failure during the data flow from the source to the target, and I have to restart the process. It looks like check points only occurs if the data flows completely ends. How often are these check points initiated ? The AL table keeps track of this value, is there a similarly table to determine the last record read ? Or does DI reads the entire Repcmd table using the last xact_seqno stored ? Thanks for any information.

It is not using triggers. The source is the transaction log.
Related link:
Basics of Change Data Capture
...The source of change data for change data capture is the SQL Server transaction log. As inserts, updates, and deletes are applied to tracked source tables, entries that describe those
changes are added to the log. The log serves as input to the change data capture capture process. This reads the log and adds information about changes to the tracked table’s associated change table. Functions are provided to enumerate
the changes that appear in the change tables over a specified range, returning the information in the form of a filtered result set. The filtered result set is typically used by an application process to update a representation of the source in some external
environment. ...
Kalman Toth, SQL Server & Business Intelligence Training; SQL Server 2008 Training

CDC (Change Data Capture) with 2 Tables

Hello,
i want to use the change data capture for the typical Data Warehousing progress (bring the new or updatet table from source to the staging area)
So i have implemented a mapping with to tables. The source table is S_Account, the target or staging table is W_Account_DS.
I have connected/mapped the tables and in the next step i goes into property inspector of W_ACCOUNT_DS into the Area Change Data Capture --> Change Data Capture Filter and put into the follow expression Builder
Select *
from S_Acccount
where INOUTGRP1.LAST_UPD_DT = CURRENT_DATE (<-- Validation Succesful)
and the next Step i make Enabeled is true in the Change Data Capture Area. So i think it should ready for one test case.
I set in SQL Developer with an update statement the LAST_UPD_DT at 28.03.2012 in this case he should not update/insert anything, right? But he insert the full table.
I set in SQL Developer with an update statement the LAST_UPD_DT by 2 Columms to 28.03.2012 in this case he should update he should update the full table - 2 columns with the date 28.03.2012, right?
So what make i wrong or have i forgotten anything? (must set keys by the tables? I think not! If yes why?)
Must i set anything properties else? Why doesnt´work it?
I look forward for your replies :)

Setting up CDC is a fairly complex process with different options. Setting just the filter in OWB is only a very small part.
There is a blog post below on how to use code templates to do CDC which gives some insight;
http://www.rittmanmead.com/2009/10/changed-data-capture-and-owb11gr2/
Plus and older one illustrating how to use Oracle logs;
http://www.rittmanmead.com/2006/04/asynchronous-hotlog-distributed-change-data-capture-and-owb-paris/
Cheers
David

Change data capture from DRM to Target system

Hi All,
I have a situation where client wants Hyperion DRM to be single source of truth and pass Hierarchy information from DRM to subscribing target systems. There are 3 target systems and the database is MS SQL server.
Now in the export profile ,I am using database export where I have mapped the DRM node/properties to target table's columns. For the initial load it is a new record in the database and the data flows as it is a 1:1 mapping, however if few of the property value changes and I wan to update only the corresponding columns what will the architecture for push the change data in to target.
Can we achieve this without have any staging tables and capture the change data ?
Do need to use ODI ?
Does DRM has any work around without putting extra effort hrs by building CDC tables ?
I will really appreciate your suggestions and comments on this......

Thanks for the quick reply. However I was trying to resolve the Change data capture in Hierarcy through as" Database Table" but it seems when we choose the DEVICE option as database in the target tab of export it can only export or insert new records into the tables but cannot update and existing record.
PLease correct me if my understnding is wrong ?????

Can you help me about change data captures in 10.2.0.3

Hi,
I made research about Change Data Capture and I try to implement it between two databases for two small tables in 10g release 2.MY CDC implementation uses archive logs to replicate data.
Change Data Capture Mode Asynchronous autolog archive mode..It works correctly( except for ddl).Now I have some questions about CDC implementation for large tables.
I have one senario to implement but I do not find exactly how can I do it correctly.
I have one table (name test) that consists of 100 000 000 rows , everyday 1 000 000 transections occurs on this table and I archive the old
data more than one year manually.This table is in the source db.I want to replicate this table by using Change Data Capture to other stage database.
There are some questions about my senario in the following.
1.How can I make the first load operations? (test table has 100 000 000 rows in the source db)
2.In CDC, it uses change table (name test_ch) it consists of extra rows related to opearations for stage table.But, I need the orjinal table (name test) for applicaton works in stage database.How can I move the data from change table (test_ch) to orjinal table (name test) in stage database? (I don't prefer to use view for test table)
3.How can I remove some data from change table(name test_ch) in stage db?It cause problem or not?
4.There is a way to replicate ddl operations between two database?
5. How can I find the last applied log on stage db in CDC?How can I find archive gap between source db and stage db?
6.How can I make the maintanence of change tables in stage db?

Asynchronous CDC uses Streams to generate the change records. Basically, it is a pre-packaged DML Handler that converts the changes into inserts into the change table. You indicated that you want the changes to be written to the original table, which is the default behavior of Streams replication. That is why I recommended that you use Streams directly.

Yes, it is possible to capture changes from a production redo/archive log at another database. This capability is called "downstream" capture in the Streams manuals. You can configure this capability using the MAINTAIN_* procedures in DBMS_STREAMS_ADM package (where * is one of TABLES, SCHEMAS, or GLOBAL depending on the granularity of change capture).

A couple of tips for using these procedures for downstream capture:
 1) Don't forget to set up log shipping to the downstream capture database. Log shipping is setup exactly the same way for Streams as for Data Guard. Instructions can be found in the Streams Replication Administrator's Guide. This configuration has probably already been done as part of your initial CDC setup.
 2) Run the command at the database that will perform the downstream capture. This database can also be the destination (or target) database where the changes are to be applied.
 3) Explicitly define the parameters capture_queue_name and apply_queue_name to be the same queue name. Example:
 capture_queue_name=>'STRMADMIN.STREAMS_QUEUE'
 apply_queue_name=>'STRMADMIN.STREAMS_QUEUE'

Change Data Capture How to Tell which you are running Sync or Asyn

Hi ,
I am taking over a new system that has change data capture running, but Im really confused how this is running. Most of the CDC is set up using Sync(triggers), but I have about 5 tables that DO NOT have system generated triggers on them. I know Streams is NOT running/configured. I know Capture is not running/configured (because nothing in DBA_CAPTURE table). I can tell that these 5 tables are still getting updated in the change table schema. I can not figure out how the 5 tables that DO NOT have triggers on them are updating the change data set tables.
I had thought the the 5 tables, must be configured with HOTLOG, but when I look at the CHANGE_SETS table they all (including these 5 tables) are set to CHANGE SOURCE NAME = SYNC SOUCE. I would expected that to be HOTLOG_SOURCE. So I "assume" they aren't set up using Asnc. hot log mode. So maybe the other Async modes are used, but not pushed to another database? Is that possible?
Any other ideas on how to figure out how the CDC is set up for these 5 tables?
thanks for your help.

Thanks for the reply, but I think I must have not stated the problem clearly. I dont WANT to set the source I want to figure out how this CDC is working. I see ALL the sources are currently set to SYNC_SOURCE. Almost all of the tables are set up with system triggers on them, but 5 dont have system triggers, yet the source says SYNC_SOURCE. I did validate that the change tables are getting updated for these tables. my question is how are they getting updated? I "assume" since they dont have system triggers ont eh table they aren't synchoronus cdc (like the other tables are). yet the source says SYNC_SOURCE. What am I missing? How can I tell if the redo log is populating those changes tables? Im pretty sure it is (Becuase there aren't triggers or jobs running), but Im curious if there is a way to tell for sure.
Thanks,

Error While enahling CDC(Change data capture) on Table.

I am enabling Change data capture (CDC) on SQL server 2012 Enterprise edition(11.0.2100.60)
. I am able to enable it on Database level with below SQL, but failed to enable on Table level.
Use DatabaseName
GO
Exec sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table @source_schema = N'dbo',
@source_name = N'TableName', @role_name = NULL
GO
Got Error like,
'msg 22832, Level 16, State 1, Procedure sp_cdc_enable_table_internal, Line 623
Could not update the metadata that indicates table [dbo].[TableName] is enabled for Change Data Capture.
The failure occurred when executing the command '[sys].[sp_cdc_add_job] @job_type = N'capture''.
The error returned was 22836: 'Could not update the metadata for database DatabaseName to indicate that a Change Data Capture job has been added.
The failure occurred when executing the command 'sp_add_jobstep_internal'.
The error returned was 14234: 'The specified '@server' is invalid (valid values are returned by sp_helpserver).'.
Use the action and error to determine the cause of the failure and resubmit the request.'.
Use the action and error to determine the cause of the failure and resubmit the request.'
Would anyone help me to out of this?
Thanks in advance..!!

Related thread:
http://social.technet.microsoft.com/Forums/sqlserver/en-US/fa0c2a52-63b5-4a39-9f35-fe6f0eb21d1d/change-data-capture-on-table?forum=sqldatawarehousing
Make sure SQL Server Agent is running.
Kalman Toth Database & OLAP Architect
SQL Server 2014 Design & Programming
New Book / Kindle: Exam 70-461 Bootcamp: Querying Microsoft SQL Server 2012

Change Data Capture: Unsubscribed column results in new change set record

I have an installation using Oracle 11g Change Data Capture in synchronous mode. As of writing, I have a single publication and a single subscription. The setup is pretty basic:
PUB_TABLE
PUB_ID*
PUB_NAME*
PUB_USELESS_FLAG
* columns are included in both the PUBLICATION and the SUBSCRIPTION. The final column is of no consequence and is excluded from both the PUBLICATION and SUBSCRIPTION.
My problem is that any time PUB_USELESS_FLAG gets updated (even if no other columns are changed), a record is written to not only the change table but the change table view. The table in question has a significant number of transactions logged against it, but only a subset of those changes are of note to the subscriber (which is why the subscriber does not listen to every column). However, since the subscription is receiving records for every change to the source table, the result is an enormous amount of overhead as the subscriber is forced to sift through numerous records that are of no consequence.
I read the Change Data Capture manual for Oracle 11g; I could not find any reference to the observed functionality. Is this a consequence of using synchronous vs. asynchronous modes of capture? Is there an option to only receive changes on subscribed columns?
Thanks in advance

Bring this to the top because I have the same question. This website (http://www.bikinfo.com/HTML/CDC/CDC_Article.html) makes it sound like this problem is unavoidable.
Edited by: nik_m on Nov 10, 2011 8:04 AM
Edited by: nik_m on Nov 10, 2011 8:05 AM

Change data capture-staging table

hi,
while using change data capture in oracle 11g,r2.I used source table and target table in the mapping.[ both source and target are oracle].
Do i have to use staging table for the mapping ,instead of target table.??
if yes, then where and how do i create a staging table and put it in the code template mapping.
Do i have to enable cdc in property editor for both the source and target tables?

Could you explain your requirement?
Where does change take place?

Change Data Capture (CDC) - Best Approach

Hi,
I am new to BODI and currently trying to design a change data capture approach.
In other ETL tools i have used, the last session run timestamp is stored as a parameter and can be called within a mapping. So the mapping updates this parameter everytime it runs. Therefore when implementing CDC the load_date on the table has to be compared to this parameter.
Is there a similar concept in BODI?
I appreciate your help.
Thanks
Azeem

Yes, there is such a concept, you have to build it yourself. e.g. https://wiki.sdn.sap.com:443/wiki/display/BOBJ/Inititialize-End+Script
When you use a CDC enabled datastore however, the database wants to keep track of the data read, that's part of their CDC API. So it is not DI is querying all changes since yesterday, it is a "select * from CDC table" and the database returns just the rows since the last read. Push vs. Pull.

Change Data Capture for XML

We have an XML file being created every week on the mainframe. This file is loaded to Oracle Database. Initially we were performing a refresh of a table, due to business reasons, we need to load only the changes from this XML file to our stage database.
Then, the changes incorporated to the stage needs to be applied to our warehouse database.
Is this possible? We cannot implement Change Data Capture on the Mainframe side.

>
they are not oracle DBs anyway
>
Then why are you posting in an Oracle forum and using Oracle terms like 'change data capture'?
>
3. don't want to serialize my inserts because since I am not sure I can keep up with the insert rate
>
That doesn't even make any sense. Oracle can generate sequence numbers faster than you can use them. Just put a trigger on the tables.
You need to provide an example and explain how the data gets into a DB and how it gets out.

Enabling Change data capture on database

While Enabling CDC on database I am getting below error, Can any body help me please...
Msg 22830, Level 16, State 1, Procedure sp_cdc_enable_db_internal, Line 193
Could not update the metadata that indicates database edFusionODS is enabled for Change Data Capture. The failure occurred when executing the command 'SetCDCTracked(Value = 1)'. The error returned was 15404: 'Could not obtain information about Windows NT
group/user 'CSD\bhaskary', error code 0x6e.'. Use the action and error to determine the cause of the failure and resubmit the request.
Regards Vikas Pathak

Where is this stored procedure State 1, Procedure sp_cdc_enable_db_internal
try to open this procedure and see what it is doing, it seems it is not able to find the user CSD\bhaskary
Mark ANSWER if this reply resolves your query, If helpful then VOTE HELPFUL
INSQLSERVER.COM
Mohammad Nizamuddin
Hallo Mohammad,
that's not a good advice because the stored procedure is a system procedure and not visible :).
Try to find it in master or the database itself - it will fail!
MCM - SQL Server 2008
MCSE - SQL Server 2012
db Berater GmbH
SQL Server Blog (german only)
Agree, this was just for troubleshooting and understanding the internals to see what might be causing the issue.
@Vikas: make sure you are connected in network (it is a domain account right), or you can try with local admin account see if it works fine
Mark ANSWER if this reply resolves your query, If helpful then VOTE HELPFUL
INSQLSERVER.COM
Mohammad Nizamuddin

Oracle 11g Change data Capture

We have setup a change data capture on Oracle 11gR2. the records that get changed in the source tables will get captured in the change tables. I have some unnecessary rows in the source tables, which i would like to eliminate them without replicating the changes to target table. i know the key values to identify the rows that got populated in the change table and can manually delete it. Is there any known/unknown side affects to eliminate the records from the change table.

Hi,
you can use dbms_streams.set_tag to hide transactions from replication.
see: http://docs.oracle.com/cd/E11882_01/server.112/e10705/rep_tags.htm#STREP390
sample:
BEGIN
DBMS_STREAMS.SET_TAG( tag => HEXTORAW('1D'));
delete from my_table where column1 = 1;
commit;
DBMS_STREAMS.SET_TAG( tag => null);
END;
the apply tag has to be another than is set to your apply process.
therefore first determine the actual tag for your session:
SET SERVEROUTPUT ON
DECLARE
raw_tag RAW(2048);
BEGIN
raw_tag := DBMS_STREAMS.GET_TAG();
DBMS_OUTPUT.PUT_LINE('Tag Value = ' || RAWTOHEX(raw_tag));
END;
first try with a sample...
kind regards
johann.

Change Data Capture, insert+delete instead of update

Similar Messages

Maybe you are looking for