AlwaysON cluster issues

Hi, we've been having some serious issues with our AlwaysON cluster configuration.
This is the short description of the scenario:
- Windows Server 2008R2 (Active Directory and DNS) (Subnet 1)
- Windows Server 2012R2 (SQLServer 2012 Cluster Node 1) (Subnet 2)
- Windows Server 2012R2 (SQLServer 2012 Cluster Node 2) (Subnet 3)
First of all, the cluster hadn't enrolled in the DNS server from it's creation.
Right after cluster creation we've been having' a "DNS server failure" error showing up
in cluster's log every 10-15 minutes.
After manually create the DNS record and giving the apropiate permissions to the CNO and
node accounts, the error changes to 'Handle is invalid' and is still showing up every
10-15 minutes.
Anyway, it seems that the AlwaysON feature is working somehow, and the application can access
the database through the Availability Group listener. But, recently, the SQL Server Log has
started to show the following message :"The Service Broker endpoint is in disabled or stopped state."
and since then, we are having intermitent access to the database with no apparent pattern
of ocurrence.
Any advice?

The DNS record has to be created by the virtual computer object (VCO). One reason why you are getting the error message "Handle is invalid" is because the encryption keys used on the domain controller is different from the one on the failover cluster.
Do you see Event ID 1228 and 1196 in the Windows event log? You can apply this
hotfix on your Windows Server 2012 R2 cluster nodes but you still need to let the DNS record be created by the VCO. Also, for multi-subnet clustering, you need to configure the RegisterAllProvidersIP and HostRecordTTL parameter values to meet your recovery
objectives.
Edwin Sarmiento SQL Server MVP | Microsoft Certified Master
Blog |
Twitter | LinkedIn
SQL Server High Availability and Disaster Recover Deep Dive Course

Similar Messages

AlwaysOn AG issue during backup

Hi everyone,
I've a strange issue on our AlwaysOn Cluster Sql Server 2012 Enterprise multi-instance, I'm evaluating to open a support case to MS but I would share the issue with the community before going on.
Sql server are dedicated to host Sharepoint 2010 farm content databases.
Scenario
n.2 VM (hypervisor ESX 5.1) Windows Server 2008 R2; each vm has 2 virtual disk, and 4 raw-mapped disk
each VM has MSFC role installed; quorum model is "file share majority"; no shared disk are mapped to clustered services.
each vm host a Sql Server 2012 enterprise installation; each VM host n.3 SQL named instance (Development, Testing, Production)
there are 3 Availability Group configured, one for each named-instance.
Raw-Mapped disk are configured in this way: raw1(500GB) for sql data, raw2(500GB) for t-log; raw3 and raw4 are dynamic disk configured in spanning mode (500+500GB) as sql backup repository.
Issue
note: This issue came out recently (while entire enviroment is on production since 1 year)
Our full backup job include a backup checksum and VERIFY.
Since a couple of weeks, this verify on a particular contentdb (CrawlStoreDB, around 80GB) cause the availability group to go in an unhealty state (RESOLVING).
From CLUSTER DIAGNOSTIC EXTENDED EVENT LOG I see this entries:
info_message   2013-05-27 15:44:42.4845706   [hadrag] SQLFetch() returns -1 with following information
info_message   2013-05-27 15:44:42.4845706   [hadrag] ODBC Error: [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not serialize the data for node 'filePath' because it contains a character (0x0000)
which is not allowed in XML. To retrieve this data convert it to binary, varbinary or image data type (6842)
info_message   2013-05-27 15:44:42.4845706   [hadrag] No more diagnostics results
info_message   2013-05-27 15:44:42.4845706   [hadrag] Discard the pending result sets
info_message   2013-05-27 15:44:42.4845706   [hadrag] ODBC Error: [24000] [Microsoft][SQL Server Native Client 11.0]Invalid cursor state (0)
From AlwaysOn health event file
availability_replica_state_change   2013-05-27 15:45:02.6615881   PRIMARY_NORMAL   RESOLVING_NORMAL   CABD99D4-8591-4B37-9160-6738DAE9C851
From Failover Cluster Manager Events
Cluster resource 'AGPROD' in clustered service or application 'AGPROD' failed.
The Cluster service failed to bring clustered service or application 'AGPROD' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
this situation remains "looping" for 1 hour, unless forcing a manual failover in SMSS; during this hour, the availability group is unreachble (RESOLVING)
If I deactivate the VERIFY on backup job, the issue doesn't happen.
Any idea to point out the problem?

The cluster is running a query that fails with SQL Error 6842 (Could not serialize the data for node 'filePath' because it contains a character (0x0000) which is not allowed in XML. To retrieve this data convert it to binary, varbinary or image data type
(6842)".
My guess is that the cluster is running the SQL Instance health check procedure
sp_server_diagnostics which returns a VARCHAR(MAX) column filled with XML, and that some diagnostic data has a column 'filePath' that has an embedded null value (0x000), which is causing
an XML conversion error. As to why it happens on backup verification, it's probably because the diagnostics returns information about the backup process, and one of the files involved has the embedded null value.
You can try to run this and see if it repros the issue.
declare @t table
create_time datetime,
component_type varchar(200),
component_name varchar(200),
state int,
state_desc varchar(20),
data xml
insert into @t
exec sp_server_diagnostics
select * from @t
But this is probably a case for support.
David
David http://blogs.msdn.com/b/dbrowne/

SQL 2012 AlwaysOn cluster IP not moving after failover, causing database to be read-only

SQL Server Cluster Name: SQLDAG01
SQL Server Cluster IP: 10.0.0.50
Cluster Listener IP: 10.0.0.60
Node 1 Name: SQL01
Node 1 IP: 10.0.0.51
Node 2 Name: SQL02
Node 2 IP: 10.0.0.52
Everything is fine when SQL01 is the primary. When failing over to SQL02, everything looks fine in the dashboard but for some reason the cluster IP, 10.0.0.50, is stuck on node 1. The databases are configured to provide secondary read access. When executing
a query on SQLDAG01, I get an error that the database is in read-only mode. Connectivity tests verify that SQLDAG01, 10.0.0.50, connects to SQL01 even though SQL02 is now the primary.
I've been Googling this for the better part of the day with no luck. Any suggestions? Is there a Powershell command force the cluster IP to move to the active node or something? Also I'm performing the failover as recommended, from Management Studio connected
to the secondary node.

This was the answer, it had been setup to use the cluster name instead of the application name. Whoever installed Sharepoint connected it to SBTSQLDAG01 instead of SHAREPOINT01. Once we changed Sharepoint to connect to SHAREPOINT01, the failover worked as
expected. We did have a secondary issue with the ARP cache and had to install the hotfix from http://support.microsoft.com/kb/2582281 to resolve it. One of the Sharepoint app servers was failing to
ping the SQL node after a failover, the ARP entry was stuck pointing to the previous node. This article actually helped a lot resolving that: http://blog.serverfault.com/2011/05/11/windows-2008-and-broken-arp/
One thing I did notice is that the SQL failover wizard does not move cluster groups "Available Storage" and "Cluster Group", I had to move those through the command line after using the wizard. I'm going to provide the client with a Powershell script that
moves all cluster groups when they need to do a manual failover. This also happens to be why the Sharepoint issue started, "Cluster Group" is what responds to the cluster name SBTSQLDAG01. Moving that group over to the node that has the active SQL cluster
group also made it work properly, but using the application name is the correct method.
Thanks everyone for all your help. Although the nitpicking about terminology really didn't help, that was a pointless argument and we really could have done without it. Yeah I know 2008 called is "Failover Cluster Manager" and MSCS is the "2003 term" but
really, they're basically the same thing and we don't really need to derail the conversation because of it. Also, If you look at the screenshot below you can clearly see "AlwaysOn High Availability" in SQL Management Studio. That's what it's called in SQL,
that's where you do all the work. Trying to tell me it's "not a feature" is wrong, pointless, and asinine, and doesn't get us anywhere.
Sorry it took so long to get back, I was off the project for a couple weeks while they were resolving some SAN issues that caused the failover to happen in the first place.

AlwaysOn Cluster reboot due to file share witness unavailability

Hi Team,
Anyone came across this scenario in AlwaysOn Availability Group (two node), file share witness times out and RHS terminate and cause the cluster node to reboot. File share witness is for continuous
failover and if the resource is unavailable my expectation was that it should go offline and should not impact Server or Sql Server. But its rebooting the cluster node to rectify the issue.
Configuration
Windows Server 2012 R2 (VMs) - two node, file share witness (nfs)
Sql Server 2012 SP2
Errors
A component on the server did not respond in a timely fashion. This caused the cluster resource 'File Share Witness' (resource type 'File Share Witness', DLL 'clusres2.dll') to
exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify
that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.
The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource.
Refer to the System event log to determine which resource and resource DLL is causing the issue.
Thanks,
-SreejitG

Thanks Elden, We were using DFS name for the file share! We gave the actual file share name and looks good now.
Few interesting facts, the failure happens exactly between window 12:30 PM to 1:30 AM and it never recorded any error specific to DFS! Not sure if there was any daily maintenance or task happens during the window pertaining to DFS.
+ DFS is not supported or recommended by Microsoft
Do not use a file share that is part of a Distributed File System (DFS) Namespace.
https://technet.microsoft.com/en-us/library/cc770620%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
Thanks,
-SreejitG

Oracel 11gR1 RAC Cluster issue

We have 2-node Oracle 11gR2 RAC on HP-UX 11.31 environment. It was running lase 2 month without any issue.
We got some netconfig issue, and node-1 got rebooted today. after the reboot cluster didn't not start on node-1, database is running on node-2.
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=301, loc=kgfokge
AMDU-00301: Unable to open file tmp-AMIPOCR01.ocr
AMDU-00204: Disk N0002 is in currently mounted diskgroup AMIPOCR01
AMDU-00201: Disk N0002: '/dev/rdisk/ora_OCR
] [8]
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ olsnodes -n
hublhp4 1
hublhp5 2
any idea please.
Edited by: ManoRangasamy on Jul 5, 2011 6:38 PM

Hi,
Please post the alertlog ASM from node 1, crsd.log and ocssd.log from node 1
It might be because node 1 can't see asm disk or permission accidentally changed when the node rebooted
Cheers

Server 2012 SMB 3.0 file cluster issues with Streaming media services 2008 R2

So I have a file server setup with the always available feature turned off on this brand new cluster. Everything seems to function normally and I can hit files from the shares no problem.
My issue I built a brand new 2008 R2 machine and created an on-demand publishing point to point to my 2012 file share cluster. I created a windows account for my windows media service to run as and applied read only to my video share. Media services see
the video files on the server in the GUI so I know its got connectivity, but when I go to test locally or externally it will not play the stream at all. It complains of access even know I know that is not the issue. I have verified the setup and the ACL
stuff is turned off on both the root and the publishing point. I can login to the Media server as my service account and play files from the share no problem. IIS media services can stream the files with no problem and yeah everything seems more like something
possibly with SMB 3.0 and something in media services.
I have a country rule where my data is not allowed in other countries so I have multiple video services on a domain with tunnels and what not my US Streaming services is working great but has a server 2008 R2 file cluster server. If I take that one to my
out of the country publishing point to test it streams great as they use the same services account. If I take my out of the country file share to the US I get the same result just will not work. I have poke prodded and banged my head and since I get this result
I have concluded it must be something with the new security of SMB 3.0.
Does anyone have this working or tested? Also I am using the exact same video file to test all of this. Or is there secret I missed to either make smb 3.0 even more compatible or make streaming services play nice.
Anything helps please let me know.
Thanks,
Adam

I'd recommend calling in for support on this one. Things aren't adding up. Probably the most useful thing to do would be to do a network trace of the WMS server connecting to the remote server over SMB.

Inexplainable 2008 Failover Cluster Issues

Hi,
We have a 2008 Failover Node & Disk Majority SQL 2005 cluster.
There are 2 nodes in the cluster with 2008 Ent 64-bit SP2 installed.
At around 00:20 each morning we see various FailoverClustering errors in the event logs on both servers.
EventID: 1135, 1069, 1177
Before the FailoverClustering events are seen, 2 informational events appear regarding the 'Microsoft Failover Clustering Virtual Adapater'
EventID: 4201 'The system detected that network adapter Local Area Connection* 9 was connected to the network, and has initiated normal operation.'
This is causing the resources to failover to the secondary node.
I have run the Cluster Validation Wizard and everything passes. I have disabled the Windows Firewall service on both nodes.
We are presenting the storage via NetApp and the nodes have 3 nics installed
NIC1 - Server Vlan - Speed/Duplex Set to 1000Mb Full
NIC2 - Storage Vlan - Speed/Duplex Set to 1000Mb Full
NIC3 - Heartbeat - Speed/Duplex Set to 100Mb Full
Please can anyone help me troubleshoot these issues ?
Thanks
Scott

Hi Scott,
Event ID 1135 — Cluster Service Startup
http://technet.microsoft.com/en-us/library/dd353973(WS.10).aspx
Event ID 1069 — Clustered Service or Application Availability
http://technet.microsoft.com/en-us/library/dd353893(WS.10).aspx
Event ID 1177 — Quorum and Connectivity Needed for Quorum
http://technet.microsoft.com/en-us/library/dd353872(WS.10).aspx
Event ID 4201 — TCP/IP Network Interface Connectivity
http://technet.microsoft.com/en-us/library/dd392958(WS.10).aspx
Hope it helps.
Tim Quan - MSFT

Deploying a WAR to a cluster issue

Hi,
My question is that, in cluster environment after a WAR file is deployed, I use iAS Admin tool to locate the WAR file. Then I highlight the specific servlet and click on the "Servlet Component Properties" button. A dialog box popups that list out all registered server (in cluster environment) with this servlet. Is there a quick way so that going to each WAR and registering each servlet can be avoided?
The reason I ask this is because everytime I deploy a WAR I first remove it before deploying it. So the
registry should be clean.
What I found is that after the a deployment of WAR1 to a iAS, say iAS1, I can see in Admin tool that the servlet within WAR1 are registered with the IP of iAS1. Ok, now I deploy the same WAR to iAS2. I go into Admin tool again and found that the WAR is now registered with the IP of iAS2, the IP for iAS1 was overwritten. What I have to do is MANUALLY add IP of iAS1 back and click "Apply" to save the change. If this action have to be done individually then it'll take ages to finish the change for my 100 more servlets, especailly when a clean deployment is required (which is what I have to do very soon).
Any idea to avoid the tedious work? I am using iAS6 SP2. Thanks.

Hi,
In my knowledge there was no such issue with sp2 related to your problem. Please check that your cluster is working fine by first testing the cluster Servlet example bundled with iAS. Please ensure that your cluster if properly configured (iasconfig name is same with a common LDAP). Also it is a feature of iAS that if you have deployed an application on all the servers of a cluster and after that you want to disable/undeploy it, you just need to undeploy it on any of the servers. It will remove the common data from the registry and make the app disable for other servers.
You don't have to undeploy the app before redeploying it again, as while redeploying, it will overwrite the previous registry contents (only if you do not change module names and GUIDs otherwise it is as good as a new different application).
Please feel free to ask further questions.
Sanjeev,
Developer Support, Sun ONE Application Server-India.

Transport to a cluster Issues?

All,
We have 2 application servers in our cluster. When we do the file export / import, we do it only on one.
How does this get replicated to the other server? We are seeing some intermittent issues - and im seeing if there is a place where you can check whether both the servers are in sync with respect to the objects.
Thanks
RKK

Do you know what basis did specifically to fix this problem?
Thanks,
Dan

Oracle on Windows Cluster issues

Hi All,
Oracle 11gr2 on Windows 64 Bit
In a active-passive setup on 2-Nodes with MS Cluster and Oracle Fail Safe,
1. Oracle Binaries are installed on local disk
2. Oracle Data files located on local disk (system, sysaux, redo, control, undo, temp)
3. Password file, Parameter files are on local disk
4. Application Data files on SAN
The issue is the cluster is unable to do a failover. But windows MSCS services are running fine. Both the Nodes are Up.
Is the reason because of the Oracle Data files not in shared SAN disk or a problem with Oracle Fail Safe.

It was quite a while ago but all i did was follow Oracle's documentaion on fail safe which can be found here
http://www.oracle.com/technetwork/documentation/failsafe-086865.html
What edition of Windows are you using 2003 or 2008. If its 2008 have you processed a validation report? In either case take both the Oracle service and TNS listner offline, then just try a fail over of the group, this will prove if the cluster is in a position to fail over (ip, virtual name, disks). If this succeeds the cluster group is good, what is left is just the Oracle service.

Sp5 Cluster issues

Has anyone had problems with Clustered managed servers in Weblogic Sp5.
Reason I ask is that we have been having problems like Multicast outOfmemory errors & Cluster members failing one after other for no reason.
If aware of any such issues, please let me know.
Thx
Anil

If you have memory issues starting up managed servers it might be that you have not allocated enough memory to the servers. You can increase the memory setting on you admin console or alternatively it might be that you need to remove old log files.

Win2003r2 connecting to win2012 with SQL2012r2 Cluster - issue

Dear TechNet members,
i have a issue which i cannot find any solution on, so trying my luck here :-)
I have a couple of Windows 2003 r2 Terminal Services servers where my application is connecting today to a 2-node Clustered Windows 2008r2 with SQL 2008r2 and working fine..
And then i have made a couple of new Windows 2012 R2 Terminal Servers and a 2-node Windows 2012 r2 with SQL 2012r2 on it.
My problem is that i cannot connect from my application on Windows 2003 r2 to the clustered new SQL 2012r2 server,
and get this issue:
"SQL State:1, SQL Error Code: -1, Level: 16, Procedure: , Line: 0
SQL Server Network Interface: Error Locating Server/Instance Specified [xFFFFFFF]"
however - my application Works Again on second attempt! and then suddenly it does not Work and then suddenly it Works...
this problem does not happend from a Windows Server 2012 Terminal Server running same application toward the clustered SQL 2012r2 Server, is it compatibility issue from win2k3 to SQL12r2? any fix?
Another strange thing that i have noticed: if i create a None-clustered Windows 2012 r2 with SQL2012r2 (same ISO's) and same instance name "SQL01" (none default) so it looks like the clustered setup but in a none clustered mode then
my application Works fine from all Windows 2003r2 terminal servers connecting to it.
So it seems like issue happens as soon as i enable "clustered services with SQL2012r2" on Windows 2012r2 and my application connecting from Windows 2003r2 Terminal Server.
Obs. i have installed latest SQL native client on win2003r2 servers which is SQL2008r2 native client, but still problems..
any idea?

Hi Daniel,
According to your description, you fail to connect to SQL Server 2012 named instance when it is a failover cluster instance. As described in this similar
KB article, the issue could be caused by that Windows Firewall drops the UDP response packet which is returned by SQL Server Browser.
To work around this issue, in the connection string from your application, please specify the TCP port number or the named pipe name to connect to the named instance and check if it is successful.
Thanks,
Lydia Zhang
Lydia Zhang
TechNet Community Support

BOE Cluster issue

Hi
We have 2 BOE servers installed standalone.
SERVER1:
OS is Windows 2003. BO XI 3.1. We have setup SSO with Windows AD and everything is working fine.
SERVER2:
OS is HP-UX. BO XI 3.1 is already installed standalone and want to use this for load balancing. CMS and other servers running and point to the same CMS DB as SERVER1.
We just added the SERVER2 node in SERVER1. Now they seem to be clustered. By default the cluster name reamins SERVER1. Now when people login into Infoview or Designer tools and use SERVER1 they are actually going against the cluster. When the request goes to SERVER2 it throws out an plugin error saying that the secWinAD is not found since it is HP-UX. When the request goes to SERVER1 everything works fine.
How do I auhenticate only against SERVER1 using AD and still be able to use the load balancing for other serversa?
If the authentication failes on SERVER2 is it possible to reroute the request to SERVER1? Another option I guess is to custom reinstall on HP-UX and deselect CMS and just install other servers. Will this work?
Thanks

Muralidhar Kolapalli wrote:
How do I auhenticate only against SERVER1 using AD and still be able to use the load balancing for other serversa?
If the authentication failes on SERVER2 is it possible to reroute the request to SERVER1?
Another option I guess is to custom reinstall on HP-UX and deselect CMS and just install other servers. Will this work?
Thanks
The AD plugin is not available on any unix/linux server. You just have to stop the CMS and then delete it from the CMC. At that point you will not have a cluster.
What type of SSO are you using (kerberos, trusted auth, etc)?
Regards,
Tim

3 Node HA cluster issues.

Hi,
We are having a tough time in setting up a 3 node HA cluster (with sun clusterware 3.2 on Solaris 10 systems). 2 systems are joining the cluster fine but the third system is not; I changed the netconfig and rebooted all the three systems. After the reboot, again only 2 systems are joining the cluster (scstat is hanging on these two systems for a while but is working fine after couple hours or so). The third system is never get connected to cluster and scstat is hanging on this system with message:
libsecurity: create of rpc handle to program rgmd_receptionist (100141) failed, will keep trying
libsecurity: create of rpc handle to program rgmd_receptionist (100141) failed,will keep trying
Any debugging hints to resolve this would be greatly appreciated. Thanks.

Check the following entry:
http://blogs.sun.com/TF/entry/secure_by_default_and_sun
do the basic checks like OS version, patch level etc.
Best Regards,
Madhan Kumar

Cisco Unity Connection cluster issue

Hello,
I have a Cisco Untiy Connection cluster of two servers, one publisher and one subscriber. They are running software: 7.1.3ES43.33034-43. Yesterday at about 8:43 I lost connection to the subscriber server. I could still ping it but was unable to get to it via GUI nor SSH. I spoke to TAC yesterday and they told me to simply reboot the server. I was a little concerned about this because I don't want to cause some kind of split-brain effect. I work in a hospital and it is imperative that the voicemail system stay up. My only other option would be to do this at 4:00 am on a Sunday. Any suggestions?
Thanks

Hi David and all,
we have a unity connection 9.x pair. after an IP address change on the subscriber we are getting the error message:
‘Communication is not functioning correctly between the servers in the Cisco Unity Connection cluster’
Both servers respectively show its own status as a publisher but lost communication with the other server (if we check cluster management from either server).
show tech network hosts – displays correct info.
show perf query class "Number of Replicates Created and State of Replication"
==>query class :
- Perf class (Number of Replicates Created and State of Replication) has instances and values:
ReplicateCount -> Number of Replicates Created = 603
ReplicateCount -> Replicate_State = 2
On both servers the replication is showing as 2 – so its good.
Various reboots have been done, services are up and working and there are no firewalls between server’s .
Any advice appreciated.
TIA…Jeff

AlwaysON cluster issues

Similar Messages

Maybe you are looking for