Please ensure that the applicationhost.config file is correct and recommit the last configuration changes made

Question

I am receiving event ID 5189 on Server 2008 R2, IIS 7.5.  I experience this error across a number of servers, each of which is running only the default web site, but have 600+ applications running on the default web site.  Each application
has its own application pool.  My point is, the applicationhost.config is rather large.
I have seen a number of posts on this error message, but my issue itself seems to differ somewhat.  I am aware of a similar post: http://social.technet.microsoft.com/Forums/en-US/3128ee93-1319-4cc2-9257-003512caf476/event-id-5189-microsoftwindowswas?forum=exchangesvrgeneral
which points to KB http://technet.microsoft.com/en-us/library/cc734988%28v=ws.10%29.aspx
However, in my case, it doesn't seem that the extreme measures for this resolution are required.  The error usually indicates a corrupted applicationhost.config that must be manually fixed, or otherwise the WAS service needs to be restarted, or the
server itself, etc.
In my case, the issue can usually be fixed by an application pool recycle, or at worst a stop/start.  I have an application which is sensitive to recycles, and so we do not have periodic recycle events configured for this application, which is the bulk
of the applications on the servers.  It seems to me that this issue is usually caused by the application being in use when there is an applicationhost.config change.
For instance, we have one application that we deploy using msdeploy.  We found that when we ran an installation for this application via msdeploy, we were receiving this event, and we would subsequently get 500 errors in the application (sometimes the
application pool corresponds to that which is identified in the event log, sometimes it doesn't).    It seems like applications that were being hit at the time the applicationhost.config was modified by msdeploy were affected.  In an attempt
to correct this, we moved from msdeploy to a batch file containing appcmd commands.  This also did not resolve the issue.
I have another application which is installed using PowerShell commandlets.  We started to notice that this script also caused the issue to occur on occasion.  As I continued to identify different processes that resulted in the same issue, I came
to the conclusion that this is likely due to any change in the applicationhost.config.  I assume the issue is that the applicationhost.config is modified during a read operation to the file by the application pool, or WAS service, and that the recycle
event, which corrects the issue, forces a re-read of the applicationhost.config, resolving the bad-read issue.
Has anyone else experienced this, or do you have any recommendations?  My current work-around is to not install any applications during peak hours, and performing an IISRESET after installs are completed, to ensure application stability.
Message              : The Windows Process Activation Service failed to generate an application pool config file for application pool 'myAppPool'. The error type is '7'. To resolve this issue,
please ensure that the applicationhost.config file is correct and recommit the last configuration changes made. The data field contains the error number.
Id                   : 5189
Version              : 0
Qualifiers           : 49152
Level                : 2
Task                 : 0
Opcode               : 0
Keywords             : 36028797018963968
RecordId             : 324108
ProviderName         : Microsoft-Windows-WAS
ProviderId           : 524b5d04-133c-4a62-8362-64e8edb9ce40
LogName              : System
ProcessId            : 0
ThreadId             : 0
MachineName          : MyComputer
TimeCreated          : 3/3/2014 9:03:24 PM
ContainerLog         : system
MatchedQueryIds      : {}
Bookmark             : System.Diagnostics.Eventing.Reader.EventBookmark
LevelDisplayName     : Error
KeywordsDisplayNames : {Classic}
Properties           : {System.Diagnostics.Eventing.Reader.EventProperty, System.Diagnostics.Eventing.Reader.EventProperty, System.Diagnostics.Eventing.Reader.EventProperty}

Answer 1

As an update to this issue, after doing some research, it appears the behavior is related to a new feature in IIS7 called Application Pool Isolation (http://technet.microsoft.com/en-us/library/dd163536.aspx).
This new feature prevents cross-application spillage by isolating the configuration for each application pool in IIS.  So in reality, an application pool never actually reads the applicationhost.config - that is just a master file, and each time it
is update, it triggers a change to the individual application pool configuration files, located in C:\inetpub\temp\apppools.
We did try setting Advanced Settings > Disable Recycling for Configuration Change to true, but after an IIS reset we were still able to reproduce the issue - worse actually, because in our test program, which continually hits the web
service, and also continually makes apphost.config changes, when the two events collide, the 500.19 issue precipitates; but after an additional change to the apphost.config, it seems that the additional changes will actually correct the issue (seemingly forcing
another recycle, which is also how we resolve the issue manually); after making this setting adjustment, once the 500.19 issue precipitates, additional changes to the apphost.config does not correct the problem, only a manual recycle corrects it.  This
is the recycling.disallowRotationOnConfigChange property of the app pool.  We also set the setting for recycling.disallowOverlappingRotation to True as well, with the same result.
We ARE able to prevent this issue by setting the C:\inetpub\temp\apppools\myapppool\myapppool.config to read-only, but for obvious reasons, this is not a desired work-around.
It seems like what is happening is that any change to the ApplicationHost.config is resulting in each of the C:\inetpub\temp\apppools temp application pool config files being updated as well, in addition to triggering an application pool recycle.  It
seems to me the desired behavior would be that these files are only updated if the applicationhost.config changes are specific to that application pool - but that doesn't seem to be the behavior.
After turning on failed request tracing, I was able to narrow the error down more:
ModuleName
AnonymousAuthenticationModule
Notification
2
HttpStatus
500
HttpReason
Internal Server Error
HttpSubStatus
19
ErrorCode
2147942413
ConfigExceptionInfo
\\?\C:\inetpub\application\application\web.config ( 9) :Failed to decrypt attribute 'password'
Notification
AUTHENTICATE_REQUEST
ErrorCode
The data is invalid. (0x8007000d)
This seems to be pretty static.  Trying to look into possible causes of this issue.