Content distribution issue with message "Found package properties updated notification for package '00000123'"

Question

I've got strange issue with 1 of my primary sites that started just recently.  I've updated the content of 1 package and this now needs re-distribution to all DPs.  I've got 3 primary sites and on 2 of those the update succeeded and my DP's got
updated correctly.  One primary site it does not work and checking distrmgr.log I can see following message every xx minutes
Found package properties updated notification for package '000008B7' SMS_DISTRIBUTION_MANAGER 6/21/2014 12:00:31 AM 5836 (0x16CC) ==> 
Checking Pkgxfermgr.log does not show anything either,  I've tried with 2nd application but again the same result it only shows the the notification message.
I saw similar issue yesterday but with restart of the sms_exec service it eventually distributed the content, obviously this is not something I like to do too often.
does anyone have idea what this could be, suggestions are welcome.   Thx.

Answer 1

Jason,  I've checked the flow from the CAS down to the primary below is quick overview.  The package I'm testing is just small package that we use to check replication is working in our environment it's less then 1MB in size and we run it
1/week basically update content and then check monitoring-content distribution that it has been received on our 260 DP's around the globe
CAS (Located in the US EST Time zone)
Distrmgr.log 11:25am update package and needs to be send to 3x primary sites (A00; B00; C00)
sender.log  11:30am all 3 primary sites have received the content.
C00 (Primary site in Singapore having the issue)
distrmgr.log 11:27pm received notification of the package
despool.log   11:32pm packaged content extracted successfully
distrmg.log started to see message Found notification for package '000008B7' and followed by next message
Package '000008B7' (priority 2) is already in queue.
This goes one until 1:21am (Singapore time)
Start updating the package 000008B7... SMS_DISTRIBUTION_MANAGER 6/21/2014 1:21:45 AM 8296 (0x2068)
The Package Action is 1, the Update Mask is 160 and UpdateMaskEx is 0. SMS_DISTRIBUTION_MANAGER 6/21/2014 1:21:45 AM 8296 (0x2068)
Use drive G for storing the compressed package. SMS_DISTRIBUTION_MANAGER 6/21/2014 1:21:45 AM 8296 (0x2068)
Successfully created/updated the package 000008B7 SMS_DISTRIBUTION_MANAGER 6/21/2014 1:21:45 AM 8296 (0x2068)
STATMSG: ID=2311 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=SG-INTRA100.na.pg.com SITE=C00 PID=10132 TID=8296 GMTDATE=Fri Jun 20 17:21:45.848 2014 ISTR0="000008B7" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7=""
ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="000008B7"
After that nothing anymore.
When I check monitoring - content distribution I can see the package being updated/distributed to 80% of my DP's and last 20% are the dp connected below this Primary site.
To add to the confusion, there was another package distributed around the same time, and then one actually made it to the remote DP's.
This is first time I see this issue.