InternalCatalogException in Reporting Services 2008 - no reports rendering

Question

Hi all,
We've had an unexpected outage on two of our reporting services server overnight and we're hoping someone can point us in the right direction.  We have a Windows Server 2008 machine running about 5 instances of reporting services in standalone, non-IIS
mode.
Two of these instances have started hitting exceptions when rendering any reports.  I include an excerpt from the log below.
The first evidence of a problem in the log is an InternalCatalogException, of which I can find many unrelated hits when searching the forums so I don't think this is specific enough to trace.  I do note a reference to "segmentedChunkStore" in the
text which seems like a better clue.
I've eliminated the idea of a machine/OS issue due to several instances not affected by this problem.  There is some possibly related info which may help:
-This machine is a HyperV VM which has been moved between hosts within the last few weeks. In the process, all Report Server databases were backed up and restored at the new site.
- It uses a Custom Authentication extension authored by us.
- Within the last few days, users of three RS instances advised that a role permission they had prior to the move (Consume Reports Task) was no longer granted.  I had to re-grant this to the role.  The two
RS instances now experiencing the InternalCatalogException issue raised an error at the time about not being able to find a group when I granted the task.  I added the group, granted again, and it granted without error so I thought nothing more
of it.
- These two RS instances have been working fine since the HyperV host move right up until last night.
- The machine had several Windows Updates queued up which were installed last night.  Since completion of the updates and subsequent reboot, these two Reporting Services instances now raise this InternalCatalogException error when trying to render any
reports from Report Manager, Report Server, or previous from Report Viewer.
At this point, I'm suspecting some sort of data corruption has occurred in the ReportServer databases, initially causing the loss of the Consume Reports permission and, after a reboot, these exeptions. But any suggestions on how to troubleshoot further greatly
appreciated. Here is the excerpt from the log.
library!ReportServer_0-1!8d4!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/IncConseqCountYTD).
library!ReportServer_0-1!12f0!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/IncCount13Months).
library!ReportServer_0-1!f98!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/my30DayActionList).
library!ReportServer_0-1!10c0!09/19/2012-20:55:53:: Call to GetItemTypeAction(/Risk Manager/Dashboard Module/Medsys/org30DayStatusPanel).
library!ReportServer_0-1!830!09/19/2012-20:55:54:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/org30DayStatusPanel')
library!ReportServer_0-1!830!09/19/2012-20:55:54:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.,
segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!830!09/19/2012-20:55:55:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson
chunks!ReportServer_0-1!830!09/19/2012-20:55:55:: w WARN: Rolling back shared chunk transaction for snapshot '49663ae9-6692-4ea2-8a5d-210de4729653', Permanent=False.
library!ReportServer_0-1!830!09/19/2012-20:55:55:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/my30DayStatusPanel')
library!ReportServer_0-1!8d4!09/19/2012-20:55:55:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/triCountPanel')
library!ReportServer_0-1!8d4!09/19/2012-20:55:55:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
chunks!ReportServer_0-1!ff8!09/19/2012-20:55:55:: w WARN: Rolling back shared chunk transaction for snapshot '68d6456c-1ed0-46eb-8bec-37b04f94065a', Permanent=False.
library!ReportServer_0-1!ff8!09/19/2012-20:55:55:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!8d4!09/19/2012-20:55:56:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson
chunks!ReportServer_0-1!8d4!09/19/2012-20:55:56:: w WARN: Rolling back shared chunk transaction for snapshot '464d6ec5-f256-4633-97d9-792c700931a0', Permanent=False.
library!ReportServer_0-1!8d4!09/19/2012-20:55:56:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!b90!09/19/2012-20:55:56:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/my30DayActionList')
library!ReportServer_0-1!b90!09/19/2012-20:55:56:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!1598!09/19/2012-20:55:57:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/IncCount13Months')
library!ReportServer_0-1!b90!09/19/2012-20:55:57:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson
chunks!ReportServer_0-1!b90!09/19/2012-20:55:57:: w WARN: Rolling back shared chunk transaction for snapshot '94bb87a7-bece-454e-b418-78b18b9eacb1', Permanent=False.
library!ReportServer_0-1!b90!09/19/2012-20:55:57:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!8d4!09/19/2012-20:55:57:: i INFO: RenderForNewSession('/Risk Manager/Dashboard Module/Medsys/IncConseqCountYTD')
library!ReportServer_0-1!1598!09/19/2012-20:55:57:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!8d4!09/19/2012-20:55:58:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., segmentedChunkStore;
 Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
chunks!ReportServer_0-1!8d4!09/19/2012-20:55:58:: w WARN: Rolling back shared chunk transaction for snapshot 'e254656f-6c22-49e6-b473-5ed986e0a730', Permanent=False.
library!ReportServer_0-1!8d4!09/19/2012-20:55:58:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!ReportServer_0-1!1598!09/19/2012-20:55:58:: i INFO: Exception InternalCatalogException dumped to: C:\Program Files\Microsoft SQL Server\MSRS10.TESTIMPACRS\Reporting Services\Logfiles flags= ReferencedMemory, AllThreads, SendToWatson
chunks!ReportServer_0-1!1598!09/19/2012-20:55:58:: w WARN: Rolling back shared chunk transaction for snapshot 'e15cc0b9-c8b1-446a-a3bf-da2ceba6273d', Permanent=False.
library!ReportServer_0-1!1598!09/19/2012-20:55:58:: w WARN: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
library!WindowsService_0!14a0!09/19/2012-21:01:07:: i INFO: Call to CleanBatch()
library!WindowsService_0!14a0!09/19/2012-21:01:13:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings.
Regards,
Michael

Answer 1

There are some more objects that reference the hardcoded temp DB of the ReportingServer:
- dbo.ExtendedDataSources (VIEW)
- dbo.ExtendedDataSets (VIEW)
- dbo.ExtendedCatalog (FUNCTION)
I've created a TSQL Script which does the job for you. The script below creates a script that can be executed on a renamed ReportingServer DB. 
Just set appropriate values for the variables @oldTempDB and
@newTempDB 
SET NOCOUNT ON
GO
IF EXISTS (SELECT * FROM tempdb..sysobjects WHERE id=OBJECT_ID('tempdb..#tmp_objects')) DROP TABLE #tmp_objects
GO
DECLARE @oldTempDB varchar(150), @newTempDB varchar(150)
SET @oldTempDB='ReportServer$TempDB'
SET @newTempDB='MY_ReportServer$TempDB'
SELECT
CASE type_desc
WHEN 'VIEW'
THEN 'VIEW'
WHEN 'SQL_STORED_PROCEDURE'
THEN 'PROC'
WHEN 'SQL_TRIGGER'
THEN 'TRIGGER'
WHEN 'SQL_SCALAR_FUNCTION'
THEN 'FUNCTION'
WHEN 'SQL_INLINE_TABLE_VALUED_FUNCTION'
THEN 'FUNCTION'
WHEN 'SQL_TABLE_VALUED_FUNCTION'
THEN 'FUNCTION'
END AS objectType
, m.definition AS SQLCommand
, s.name AS SchemaName
, o.name as ObjectName
, s.name + '.' + o.name AS objectFullName
, 0 AS done
, 0 AS success
, NEWID() AS ID
, LEN(m.definition) AS Length
INTO #tmp_objects
FROM sys.objects o
INNER JOIN sys.all_sql_modules m
ON m.object_id=o.object_id
INNER JOIN sys.schemas s
ON s.schema_id=o.schema_id
WHERE type_desc IN (
'VIEW'
, 'SQL_STORED_PROCEDURE'
, 'SQL_TRIGGER'
-- , 'USER_TABLE'
, 'SQL_SCALAR_FUNCTION'
, 'SQL_TABLE_VALUED_FUNCTION'
, 'SQL_INLINE_TABLE_VALUED_FUNCTION'
AND PATINDEX('%' + @oldTempDB + '%', m.definition)>0
AND PATINDEX('%' + @newTempDB + '%', m.definition)=0
PRINT 'BEGIN TRANSACTION'
PRINT 'SET NUMERIC_ROUNDABORT OFF'
PRINT 'GO'
PRINT 'SET ANSI_PADDING, ANSI_WARNINGS, CONCAT_NULL_YIELDS_NULL, ARITHABORT, QUOTED_IDENTIFIER, ANSI_NULLS ON'
PRINT 'GO'
DECLARE @SQLDefinition VARCHAR(MAX), @ObjectName VARCHAR(200), @ObjectType VARCHAR(50), @CommandID VARCHAR(50), @ObjectFullName VARCHAR(350), @SchemaName VARCHAR(50), @searchPattern varchar(250), @reverseSearchPattern varchar(250)
DECLARE @LineBreakPos INT, @Pos INT, @patternCounter INT
WHILE EXISTS(
SELECT *
FROM #tmp_objects
WHERE done=0
BEGIN
SELECT TOP 1
@CommandID=ID
FROM #tmp_objects
WHERE done=0
ORDER BY objectName
/*-----------------------create output-----------------------------------------*/
SELECT
@SQLDefinition=t.SQLCommand
, @ObjectName=t.objectName
, @SchemaName=t.SchemaName
, @ObjectFullName=t.objectFullName
, @ObjectType=t.objectType
FROM #tmp_objects t
WHERE
t.ID=@CommandID
SET @patternCounter=0
/************replace create by alter and oldTempDB by newTempDB***********/
WHILE @patternCounter<4
BEGIN
IF @patternCounter=0 SET @searchPattern='%CREATE %' + @ObjectType +'% %' + @SchemaName + '.' + @ObjectName + '%'
IF @patternCounter=1 SET @searchPattern='%CREATE %' + @ObjectType +'% %[' + @SchemaName + '].' + @ObjectName + '%'
IF @patternCounter=2 SET @searchPattern='%CREATE %' + @ObjectType +'% %[' + @SchemaName + '].[' + @ObjectName + ']%'
IF @patternCounter=3 SET @searchPattern='%CREATE %' + @ObjectType +'% %' + @SchemaName + '.[' + @ObjectName + ']%'
SET @reverseSearchPattern=REPLACE(REVERSE(@SearchPattern),'[','[[]')
SET @SearchPattern=REPLACE(@SearchPattern,'[','[[]')
IF PATINDEX(@searchPattern, @SQLDefinition)>0
BEGIN
SET @patternCounter=99
SET @SQLDefinition= REPLACE(
SUBSTRING(
SUBSTRING(@SQLDefinition,
1,
LEN(@SQLDefinition) - PATINDEX(@ReverseSearchPattern, REVERSE(@SQLDefinition))+1
PATINDEX(@searchPattern, @SQLDefinition),
LEN(@SQLDefinition)
'CREATE ',
'ALTER '
+ SUBSTRING(
@SQLDefinition, LEN(@SQLDefinition) - PATINDEX(@ReverseSearchPattern, REVERSE(@SQLDefinition))+2
, LEN(@SQLDefinition)
SET @SQLDefinition=REPLACE(@SQLDefinition, @oldTempDB, @newTempDB)
END
ELSE
BEGIN
SET @patternCounter=@patternCounter+1
END
END
IF @patternCounter=99 /*found pattern*/
BEGIN
/************END replace create by alter*******/
PRINT ''
PRINT ''
PRINT '/******ALTER ' + @ObjectType + ' ' + @ObjectFullName + '******/'
PRINT 'PRINT ''******ALTER ' + @ObjectType + ' ' + @ObjectFullName + '*******'''
PRINT 'GO'
SET @LineBreakPos=CHARINDEX(CHAR(13)+CHAR(10), LEFT(@SQLDefinition,8000))
IF @LineBreakPos>0 AND LEN(@SQLDefinition)>=8000
BEGIN
SET @Pos=1
WHILE @LineBreakPos>0
BEGIN
PRINT SUBSTRING(@SQLDefinition, @Pos, @LineBreakPos-1)
SET @Pos=@Pos + @LineBreakPos+1
SET @LineBreakPos=CHARINDEX(CHAR(13)+CHAR(10), SUBSTRING(@SQLDefinition, @Pos, 8000))
END
END
ELSE
BEGIN
PRINT @SQLDefinition
END
PRINT ''
PRINT 'GO'
PRINT 'IF @@ERROR>0 BEGIN ROLLBACK TRANSACTION ''Errors occured, script abborded'' SET NOEXEC ON END'
PRINT 'GO'
/*---------------------------end output----------------------------------------------*/
END
UPDATE TG
SET TG.done=1
, TG.success=(CASE WHEN @patternCounter=99
THEN 1
ELSE 0
END
FROM #tmp_objects TG
WHERE
TG.ID=@CommandID
END
PRINT 'IF @@TRANCOUNT>0 BEGIN COMMIT TRANSACTION PRINT ''Successful'' END '
PRINT 'GO'
PRINT 'SET NOEXEC OFF'
Select *
FROM #tmp_objects
WHERE success=0
DROP TABLE #tmp_objects
Execute this script in MSSQL Management Studio or in a similar tool and copy the created script from the "Messages" pane. If any object fails a table containing them will appear in the results pane.
Make sure you check the resulting script first before executing to avoid accidential altering of an object.
have fun with it! ...and thanks to microsoft for making us an extra loop...
Claude