TMS 14.3.2 - No HTTPS Response from devices (again) - Looks like permissions issues (but possibly down to NetBIOS over TCP/IP)

Question

Hi All,
Sometime ago I posted regarding an issue I was seeing on our TMS , where most of the remote devices seemingly showed up as loosing remote connection (no HTTPS Response - https://supportforums.cisco.com/discussion/12290956/tms-lost-https-communciation-devices-not-really). However, I could not figure out the root cause of the error, and now I have managed to break things again - probably due to a Java upgrade which broke TMSPE (see https://supportforums.cisco.com/discussion/12404166/looks-latest-update-java-8-update-31-will-break-tmspe-11).
However, I thought I would look at little more closely this time to see if I could figure out the problem.
I am seeing a lot of entries in the Security Log in TMS that seem to indicated that my the account used from a TMSPE connection has a bad or invalid password (this is set in "Administrative Tools --> Configuration --> Provisioning Extension Settings", but this password is correct (I can log into TMS with this account, and reset and restarted services etc).
The computer attempted to validate the credentials for an account.
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: tmspeconnect
Source Workstation: TMS
Error Code: 0xc0000064
and
An account failed to log on.
Subject:
Security ID: IIS APPPOOL\TMSNet40AppPool
Account Name: TMSNet40AppPool
Account Domain: IIS APPPOOL
Logon ID: 0x214b6
Logon Type: 8
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: tmspeconnect
Account Domain: TMS
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xc000006d
Sub Status: 0xc0000064
Process Information:
Caller Process ID: 0x860
Caller Process Name: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Network Information:
Workstation Name: TMS
Source Network Address: 127.0.0.1
Source Port: 62114
Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
This event is generated when a logon request fails. It is generated on the computer where access was attempted.
The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).
The Process Information fields indicate which account and process on the system requested the logon.
The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.
The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
I just wanted to check this like file/folder NTFS permissions, or any specific IIS permission . Is there a list of NTFS folder permissions for which the TMS required with regard to the "IIS APPPOOL", and perhaps what may need to be check in IIS?
Oddly, the error list above refers the our "TMSPEConnect" account from the TMS domain, however this is incorrect - TMS is the machine name. Still we ARE seeing similar error which reference the actual domain name, such as:
An account failed to log on.
Subject:
Security ID: IIS APPPOOL\TMSNet40AppPool
Account Name: TMSNet40AppPool
Account Domain: IIS APPPOOL
Logon ID: 0x49e55
Logon Type: 8
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: tmspeconnect
Account Domain: mydomain
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xc000006d
Sub Status: 0xc000006a
Process Information:
Caller Process ID: 0xb18
Caller Process Name: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Network Information:
Workstation Name: TMS
Source Network Address: 1.1.1.1 (changed external IP)
Source Port: 47620
Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
This event is generated when a logon request fails. It is generated on the computer where access was attempted.
The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).
The Process Information fields indicate which account and process on the system requested the logon.
The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.
The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
Idea welcome.
Cheers
Chris

Answer 1

Hi all,
After a long a fraught weekend trying to fix things, I am not that much further on, but I am beginning to understand what it isn't........ I think :(
It looks as though the "DatabaseScanning" service, or anything that runs under the credentials of the "nt authority\network service" seems to have a problem.
The Audit log show lot of entries where services such as"TMSWebPublic" running under the ISS Application pool "tmsnet40apppool", and actions launched under our own credential appear to communicate just fine with the device, only for the 'network service' to time out the connection a short while later. The same goes for some (not all) of the Phone book updates, which also looks as though it runs under the credentials of the 'Network Service' account.
26/01/2015 08:30:54
nt authority\network service
Update
System
our-ctrl1
System status
NoResponse
TMSDatabaseScannerService
26/01/2015 08:30:54
nt authority\network service
Update
System
our-ctrl1
Connection status
NoHTTPSResponse
TMSDatabaseScannerService
26/01/2015 08:29:06
tmsnet40apppool
Update
System
our-ctr1l
System status
Idle
TMSWebPublic
26/01/2015 08:29:06
tmsnet40apppool
Update
System
our-ctrl1
Connection status
OK
TMSWebPublic
However, I still don't know at this point is this is related to the actual TMS front end, something to do with the domain and policies that have been applied to TMS, or the actual database, or even just communication with the back end database??? I did try running the services under a domain user (service) account, but then ran into other issues as I couldn't log into TMS.
BTW, removing TMS, and IIS and reinstalling did fix my second issue relating to the 'Communication Errors'.
Chris