Configuration Dataset = 90% of Data Warehouse - Event Errors 31552

Question

Hi All,
I'm currently running SCOM 2012 R2 and have recently had some problems with the Data Warehouse Data Sync. We currently have around 800 servers in our production environment, no Network devices, we use Orchestrator for integration with our call logging system
and I believe this is where our problems started. We had a runbook which got itself into a loop and was constantly updating alerts, it also contributed to a large number of state changes. We have resolved that problem now, but I started to receive alerts
saying SCOM couldn't sync Alert data under event 31552.
Failed to store data in the Data Warehouse.
Exception 'SqlException': Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. 
One or more workflows were affected by this.  
Workflow name: Microsoft.SystemCenter.DataWarehouse.StandardDataSetMaintenance 
Instance name: Alert data set 
Instance ID: XX
Management group: XX
I have been researching problems with syncing alert data, and came across the queries to manually do the database maintenance, I ran that on the alert instance and it took around 16.5 hours to run on the first night, then it ran fast (2 seconds) most the
day, when it got to about the same time the next day it took another 9.5 hours to run so I'm not sure why that's giving different results.
Initially it appeared all of our datasets were out of sync, after the first night all appear to be in sync bar the Hourly Performance Dataset. Which still has around 161 OutstandingAggregations. When I run the Maintenance on Performance it doesn't appear
to be fixing it. (It runs in about 2 seconds, successfully)
I recently ran DWDatarp on the database to see how the Alert Dataset was looking and to my surprise I found that the Configuration Dataset has blown out to take up 90% of the DataWarehouse, table below. Does anyone have any ideas on what might cause this
or how I can fix it?
Dataset name                   Aggregation name     Max Age     Current Size, Kb
Alert data set                 Raw data                 400       132,224 (  0%)
Client Monitoring data set     Raw data                  30             0 (  0%)
Client Monitoring data set     Daily aggregations       400            16 (  0%)
Configuration dataset          Raw data                 400   683,981,456 ( 90%)
Event data set                 Raw data                 100    17,971,872 (  2%)
Performance data set           Raw data                  10     4,937,536 (  1%)
Performance data set           Hourly aggregations      400    28,487,376 (  4%)
Performance data set           Daily aggregations       400     1,302,368 (  0%)
State data set                 Raw data                 180       296,392 (  0%)
State data set                 Hourly aggregations      400    17,752,280 (  2%)
State data set                 Daily aggregations       400     1,094,240 (  0%)
Microsoft.Exchange.2010.Dataset.AlertImpact Raw data      
7     0 (  0%)
Microsoft.Exchange.2010.Dataset.AlertImpact Hourly aggregations        
3     0 (  0%)
Microsoft.Exchange.2010.Dataset.AlertImpact Daily aggregations      
182     0 (  0%)
Microsoft.Exchange.2010.Reports.Dataset.Availability Raw data                 400           176 (  0%)
Microsoft.Exchange.2010.Reports.Dataset.Availability Daily aggregations       400             0 (  0%)
Microsoft.Exchange.2010.Reports.Dataset.TenantMapping Raw data 7             0 (  0%)
Microsoft.Exchange.2010.Reports.Dataset.TenantMapping Daily aggregations       400             0 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ActiveUserMailflowStatistics.Data Rawdata                   3        84,864 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ActiveUserMailflowStatistics.Data Hourly aggregations        7       407,416 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ActiveUserMailflowStatistics.Data Daily aggregations       182       143,128 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ServerMailflowStatistics.Data Raw data                   7         6,088 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ServerMailflowStatistics.Data Hourly aggregations       31        20,056 (  0%)
Microsoft.Exchange.2010.Reports.Transport.ServerMailflowStatistics.Data Daily aggregations       182         3,720 (  0%)
I have one other 31553 event showing up on one of the Management servers as follows,
Data was written to the Data Warehouse staging area but processing failed on one of the subsequent operations.
Exception 'SqlException': Sql execution failed. Error 2627, Level 14, State 1, Procedure ManagedEntityChange, Line 368, Message: Violation of UNIQUE KEY constraint 'UN_ManagedEntityProperty_ManagedEntityRowIdFromDAteTime'. Cannot insert duplicate key in
object 'dbo.ManagedEntityProperty'. The duplicate key value is (263, Aug 26 2013  6:02AM). 
One or more workflows were affected by this.  
Workflow name: Microsoft.SystemCenter.DataWarehouse.Synchronization.ManagedEntity 
Instance name: XX 
Instance ID: XX
Management group: XX
which from my readings means I'm likely in for an MS support call.. :( But I just wanted to see if anyone has any information about the Configuration Dataset as I couldn't find much in my searching.

Answer 1

Hi All,
The results of the MS Support call were as follows, I don't recommend doing these steps without an MS Support case, any damage you do is your own fault these particular actions resolved our problems:
1. Regarding the Configuration Dataset being so large. 
This was caused by our AlertStage table which was also very large, we truncated the alert stage table and ran the maintenance tasks manually to clear this up. As I didn't require any of the alerts sitting in the AlertStage table we simply did a straight truncation
of the the table. The document linked by MHG above shows the process of doing a backup & restore on the AlertStage Table if you need to. It took a few days of running maintenance tasks to resolve this problem properly. As soon as the truncation had taken
place the Confirguration Dataset dropped in size to less than a gig.
2. Error 31553 Duplicate Key Error
This was a problem with duplicate keys in the ManagedEntityProperty table. We identified rows which had duplicate information, which could be gathered from the Events being logged on the Management Server.
We then updated a few of these rows to have a slightly different time to what was already in the Database. We noticed that the event kept logging with a different row each time we updated the previous row. We ran the following query to find out how many rows
actually had duplicates:
select * from ManagedEntityProperty mep
inner join ManagedEntity me on mep.ManagedEntityRowId = me.ManagedEntityRowId
inner join ManagedEntityStage mes on mes.ManagedEntityGuid = me.ManagedEntityGuid
where mes.ChangeDateTime = mep.FromDateTime
order by mep.ManagedEntityRowId
This returned over 25,000 duplicate rows. Rather than replace the times for all the rows, we removed all duplicates from the database. (Best to have MS Check this one out for you if you have a lot of data)
After doing this there was a lot of data moving around the Staging tables (I assume from the management server that couldn't communicate properly), so once again we truncated the AlertStage table as it wasn't keeping up. Once this was done everything worked
properly and all the queues stayed under control.
To confirm things had been cleared up we checked the AlertStage table had no entries and the ManagedEntityStage table had no entries. We also confirmed that the 31553 events stopped on the Management server.
Hopefully this can help someone, or provide a bit more information on these problems.