Applying transaction log corrupted database?

Question

Hi experts,
Symptoms:
  I built a standby system this Monday and I completed a full DB restore without errors. I used log shipping to apply logs but an error occurred: SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 20:5746350; actual 0:0). It occurred during a read of page (20:5746350) in database ID 5 at offset 0x00000af5d5c000 in file 'l:\TCPDATA4\TCPDATA19.ndf'.
  I run dbcc page and found page(20:5746350) did corrupt on standby system. However, I ran dbcc page on production and it is ok. I restored this full db to another standby and ran dbcc page and it is ok. Also, Checkalloc, Checkcatalog, Checktable on production are ok.
Questions:
  Does checktable detect secondary indexes corruption? I am re-compressing COSP indexes and running full checkdb on production right now to find out what's going on in our DB. How do I fix this issue? Any information will be really appreciated.
Error message:
2011-08-23 17:26:00.19 spid52      The log shipping secondary database TCCDBM1.TCP has restore threshold of 45 minutes and is out of sync. No restore was performed for 1 minutes. Restored latency is 720 minutes. Check agent log and logshipping monitor information.
2011-08-23 17:26:39.62 Backup      Log was restored. Database: TCP, creation date(time): 2011/07/03(05:36:16), first LSN: 184193:4075900:1, last LSN: 184193:7288500:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {'w:\LogB\TCP_20110822213010.trn'}). This is an informational message. No user action is required.
2011-08-23 17:28:00.88 spid52      Error: 14421, Severity: 16, State: 1.
2011-08-23 17:28:00.88 spid52      The log shipping secondary database TCCDBM1.TCP has restore threshold of 45 minutes and is out of sync. No restore was performed for 2 minutes. Restored latency is 716 minutes. Check agent log and logshipping monitor information.
2011-08-23 17:28:34.03 Backup      Log was restored. Database: TCP, creation date(time): 2011/07/03(05:36:16), first LSN: 184193:7288500:1, last LSN: 184193:10880343:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {'w:\LogB\TCP_20110822213511.trn'}). This is an informational message. No user action is required.
2011-08-23 17:28:53.06 spid65      Error: 824, Severity: 16, State: 2.
2011-08-23 17:28:53.06 spid65      SQL Server detected a logical consistency-based I/O error: incorrect pageid (expected 20:5746350; actual 0:0). It occurred during a read of page (20:5746350) in database ID 5 at offset 0x00000af5d5c000 in file 'l:\TCPDATA4\TCPDATA19.ndf'.  Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.
2011-08-23 17:30:00.54 spid52      Error: 14421, Severity: 16, State: 1.
2011-08-23 17:30:00.54 spid52      The log shipping secondary database TCCDBM1.TCP has restore threshold of 45 minutes and is out of sync. No restore was performed for 2 minutes. Restored latency is 713 minutes. Check agent log and logshipping monitor information.
2011-08-23 17:30:35.94 spid53      Error: 824, Severity: 16, State: 2.

Answer 1

Hi Dennis!
I understand your reasonable doubt, but think that physical corruptions are 99,999...% caused by a hardware failure e.g. a defective RAM module, or a related, low-level failure as e.g. an outdated NIC driver, etc.
Then, in this case it is clear that you have a platform problem only on your standby system, and not in the one running productively right now.
Maybe you already checked your platform, but you should take into account that in most cases just reading the logs is not enough; stress tests on all of the components are usually necessary. Most of the time, when we analyze hardware related problems, the problem is only reproduced when special tests are performed. Furthermore it is possible that more than one component is weak or broken.
My recommendation is that you check the components in this order:
   - disks (complete surface) (while SAP and SQL Server are shut down)
   - disk controller
   - Memory (stress test)
   - CPU
   - all other components
Consider that restoring a backup and running in a damaged platform would imply further problems. You would be waiting until the next problem arises and that would imply a system halt or corruptions, maybe even with data loss.
Once the hardware checked and -if necessary- fixed, you can proceed as explained in SAP notes 142731 and 1420452. Of course you can also restore in a different hardware, but remember checking the platform as it is also exposed to hardware failures.
Cheers!!
--Jesú