Exchange Performance - Submission Queue

Good afternoon,
We've been having issues recently with the Submission queue filling up on our sole Exchange 2010 mail server running Windows Server 2008 R2; emails are still being processed, just at a slower pace than they are being received, causing delays between
when someone sends an email to us and when the email is received. In addition just being on the mail server everything feels slow and sluggish. Sometimes the mail server will be maxed out on memory but other times memory usage will be lower (like
right now for example memory usage is at 46%), however CPU usage is never too high and network usage doesn't seem too high to me either, yet even still at the moment we have 1100 messages in the Submission queue. I'd like to provide our server specifications
and what we're running on the server below and would appreciate it if someone could tell me if the server should be able to handle this load adequately (on a side note I realize we may have a lot running on this one server and that this is not optimal):
Server specs:
- 2 Xeon 5650 Processors/12 Cores total @ 2.67 GHz
- 32 GB DDR3 RAM @ 1333 MHz
We have 121 mailboxes on the server, running all 3 main Exchange roles (Hub Transport, Mailbox, and Client Access). We have maybe 40 phones and iPads connecting as well. We also have the following running on the server:
- GFI MailEssentials (anti-spam software)
- GFI FaxMaker (for sending and receiving faxes through Outlook)
- IIS hosting our main website (~200 hits a day), 3 web applications that are what I would consider pretty lightly used, the MailEssentials configuration site and Exchange stuff like OWA
- Hyper-V hosting a virtual machine running Windows Server 2008 R2 running nothing but Blackberry Enterprise Server (assigned 2 GB memory, no processor reserve set)
We are a small to medium-size business and I feel as if our server should be able to handle the above, can anyone provide any input on this? I appreciate the help!

Thanks for the reply, I've never noticed any peformance issues with either of our domain controllers but I have never tried rebooting them when this happens so I'll have to try this next week and report back. We have two domain controllers on
slightly-lower spec machines than our mail server; the domain controller I consider the "primary" (though I know it doesn't work that way anymore) is also our "primary" internal DNS server, DHCP server, WSUS server, and hosts a file share
storing dat files for users to download McAfee updates from, so not doing too much (in my opinion). The other domain controller (our "backup") is also our SQL Server (supporting 4 applications and tables for a variety of Access databases),
"backup" internal DNS server, and hosts a virtual machine which hosts our accounting software; I've noticed small performance issues with this machine once in awhile but nothing major. For reference we only have 124 user accounts
and 150 computer accounts total in Active Directory. Could it be that our Exchange server is connecting to this slower "backup" domain controller and this is what's slowing it down? Thanks again for your help!

Similar Messages

Messages stuck in "Retry" in Submission Queue - Exchange 2013

Hello,
We have 2013 Exchange consisting of 2 CASs and 2 MBX servers spanning two locations via VPN, 1 of each server in each location. We are hosting exchange for an outside domain, and have just transferred them into our organization within the last month. There
are two users who consistently have messages stuck in the "Retry" status in the submission queue. Not all of their messages, but usually 2-4 are stuck and eventually kicked back to them.
I have read other forums about a potential issue with conversion to RTF. I have forced the queues empty and attempted this fix, but within a week the users had messages stuck again.
Has anybody run into this and possibly found a fix?

Thank you for all your replies.
It appears this issue has been resolved. At first I made the following change:
Get-RemoteDomain | Set-RemoteDomain -TNEFEnabled $False
Restarted Transport services on both mailbox DB Servers and the messages remained in the queue and more messages were getting stuck.
Then I made this change:
Get-RemoteDomain | Set-RemoteDomain -TNEFEnabled $True
Restarted Transport services on both mailbox DB Servers and the messages cleared from the queues and it seems no more messages are getting stuck as they were before.
I also changed the message timeout from 2 days to 2 hours. I'd rather know sooner than later that a message was stuck. I plan to continue monitoring this.
Thanks again.
Stop Stop Stop, this isn't a solution! Every single post I run into this answer, but it is a real crappy "solution". Simple: there is no solution yet, if everyone take this for an answer there will never we a good answer! This is a work around,
we got mails stucked in the queue for 2 months and it start with no reason. We have every update, enough disk space etc. Settinf the remote domain to true brings you the problem with Apple users and Gmail, hotmail etc. They will receive an email.dat or no
attachment when you send HTML mail. The False or Null is a future that make mail HTML so you can use signatures, pictures etc. It always worked, now only for 2 days and we must reboot again. Set de Remote domain to true we have problems with customers who
doens't receive their email correct!.
Microsoft please solve this problem and stop making crappy excuses or crappy workarounds, a solution is what we need!!!

#550 4.4.7 QUEUE.Expired; message expired ## With LastError "A storage transient failure has occurred during content conversion." In submission Queue. (Exchange 2013)

Greetings,
We seem to be having a problem with some users who are attempting to send e-mails from within the organisation to an external domain. Not all users are affected, and not all outgoing e-mails have this issue.
Some e-mails get stuck in the submission queue. This is the error message in Last Error : "A storage transient failure has occurred during content conversion."
Days later, the internal user who send the message gets a #550 4.4.7 QUEUE.Expired; message expired ## NDR.
We did have some initial configuration issues, but these were fixed more than a week ago :
- The external FQDN during EHLO was set to the wrong address, now pointing to the correct one.
- SPF record was updated with new IP adress.
Here is some additional information on the issue :
- Not on any blacklists - checked using dnsbl.info
- Telnet to remote servers works from exchange server, connections are accepted and can send mail.
- Outbound SMTP test ran using Microsoft Remote Connectivity Analyser : Passed with both External (Static) and Smarthost IP.
- This seems to happen only with emails that have an attachment and that are transfered, but only for the affected users.
- If content from these e-mails is manually copied over to a new email, email is sent to destination without problem.
Configuration information :
- Exchange 2013 running on Windows 2012 Datacenter with all latest updates.
- Outgoing e-mail is sent via smarthost. Only one outbound transport rule is active.
- Using internal DNS server.
- There is only one mailbox database.
Thank you for taking the time to read this!

On Wed, 16 Jan 2013 15:31:14 +0000, Ipigi wrote:
>Sorry, I often get some terms mixed up when I explain things as our users use the French version of outlook.
>
>E-mails are not transferred, but forwarded manually from their outlook. Message format in outlook is set to HTML and not Rich Text when they foward the e-mail.
Do they forward the message as an attachment?
>When forwarded internally, this is in the internet headers :
>
>Content-Type: application/ms-tnef; name="winmail.dat" Content-Transfer-Encoding: binary
Within your organization I'm prety sure that messages will use TNEF.
What does the message contain at the external recipient's side?
>It really seems to me that Exchange is not converting RTF to Plain Text. The first link you provided states in it's final paragraph that Exchange should be doing this conversion.
If you can, try creating a mail-enabed Contact for one of the external
recipients and set the message format on that.
>If I disabled TNEF as that link suggests, offending messages will get stuck in the submission queue again.
>
>I thank you for your help so far. This is not an issue I've had with any previous installations/migrations of Exchange that I have done.
>
>Please let me know if you need any additional information.
Have you tried UNsetting TNEF on the remote domain?
Set-RemoteDomain Default -TNEFEnabled $null
That should leave it up to the client to determine the format. It's
probably not what you're after, but see it makes a difference in the
format.
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP

Mail Stuck in Mail Submission Queue

We have an Exchange 2010 Server SP1 which is running the Hub Transport, Client Access and Mailbox Server roles all on one physical box.
The issue we encountered the other day is that ALL mail flow effectivly stopped and we have over 1000 messages sitting in the Mail Submission Queue. This meant that even internal mail being routed had stopped and was polling in this queue.
We first tried restarting Exchange Services such as the Information Store and Transport service. This did not work and mail was still polling and increasing in the Mail Submission Queue.
I then read somewhere that it could be caused by IIS so I performed an iisreset to restart the IIS services.
After IIS came back online sure enough the Mail Submission Queue started dropping and all mail started flowing again and delivering to mailboxes.
We run on the Exchange Server the built in anti-spam agents and Symantec EndPoint Protection Anti-Virus only. We have Messagelabs that scan our external mail for spam before delivering to the mail server.
So my question is.... Why is that restarting IIS caused the Mail Submission Queue to start processing normally? I am very concerned about this and would like some guidance on what the issue could be...
Thanks in advance.

Does anyone have any further ideas ?
Hi,
I would start with investigate free disk space on the disk drives.
If the logdrive containing the Transaction logs has less then 1 GB of free space, no emails would be submitted to the database. Hence the Mailbsubmission Queue.
See the section: Low Disk Space on Database Logs or Database Drives
http://technet.microsoft.com/en-us/library/bb331958.aspx#Low
Make sure that your backup is running as it should and logs are beeing flushed. An easy way to check when the last fullbackup was run is to run
get-mailboxdatabase - status | fl Name, *backup*
Your problem could also be caused by Hub Transport Back Pressure
Understanding Back Pressure
http://technet.microsoft.com/en-us/library/bb201658.aspx
Martina Miskovic
Hi,
The disk drives have more than enough free space. Im not sure what you mean with the backups as we use a third party application to backup the data to a remote site. I will investigate the back pressure but unsure if this will explain why restarting IIS
resolved this issue.

Some Journaled Calendar Requests get stuck in Submission queue - 550 5.6.0 - storage error in content conversion

I am having an issue on hub transport servers running Exchange 2010 SP3 RU6.
Where some journaled meeting request related messages (acceptances, etc.) get stuck in the Submission queue.
The error is:
400 4.4.7 The server responded with: 550 5.6.0 M2MCVT.StorageError; storage error in content conversion. The failure was replaced by a retry response because the message was marked for retry if rejected.
I raised the issue with our mail archiving vendor and they indicated this was an Exchange Server problem which should have been fixed in Exchange 2010 SP3 RU3, however we are running SP3 RU6 and the issue continues to occur.
I realize that we could just delete the stuck messages from queue but has anyone run in to this before and know how to prevent it from happening?
Thank You.

Hi Mike,
If you temporary disable\bypass third party Archiving , then what happens in the Queue.
Thanks , Prakash
Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

TNEF Issue - Winmail.dat or messages stuck in submission queue

Hi guys,
I've looked around the forums and have found people that have had submission queue issues when they set TNEF to $null or $false, and have found that the unified answer is to set TNEF to $true.
I also see folks having issues with recipients in remote domains receiving winmail.dat files in some instances (with a .pdf or .xls file attached) if TNEF is set to $true.
I have both issues. If I set TNEF to $true, I encounter the winmail.dat file problem. If I set TNEF to $false or $null, I encounter the submission queue issue (submission queue states "A transient storage failure occured", and users get a queue
expiration bounce). Neither of these states are acceptable. I cannot have messages stuck in the queue only to be expired, and I cannot have recipients in remote domains failing to receive attachments in emails.
Does anyone have a possible workaround that might resolve this particular situation? It seems that this has been an issue in Exch 2010.
Current Environment Configuration:
I am running Exchange 2013 SP1 Standard on both of my CAS/MBX servers, running on Server 2012 R2 patched to the latest updates. I have an inbound spam filter, but mail outbound is not filtered, therefore the spam filter on our side should not play a role.
I do have not made any changes to the exchange default malware filter either. Other than this issue, mail is flowing correctly and as expected.

I also see folks having issues with recipients in remote domains receiving winmail.dat files in some instances (with a .pdf or .xls file attached) if TNEF is set to $true.
Hi,
According to your description, my understanding is that all the remote domain's recipients would receive winmail.dat files when you send the message to them if TNEF is set to $true, right?
The Winmail.dat file is used to preserve Rich Text formatting. Outlook uses it when sending a Rich Text-formatted message. During transport, the content of the message may be changed, preventing the receiving client from being able to read the formatting
instructions. In other cases, the receiving client does not use or recognize the winmail.dat file.
I recommend that you refer to the following article to solve the issue:
http://support.microsoft.com/kb/278061
Method 1: Change the default message format
The sender can change the format of the email messages that they send by using the following steps:
On the Tools menu, click Options, and then click
Mail Format.
In Compose in this message format, click to select Plain Text, and then click
OK.
NOTE: To send to certain recipients that use RTF format and others recipients that use plain text format, the sender must set the option for the recipient in either the Personal Address Book or the recipient's contact record.
Method 2: Modify the recipient's entry in the Personal Address Book
The sender can use the following steps to remove the RTF format from the recipient attribute in the Personal Address Book:
On the Tools menu, click Address Book.
In Show Names From, click the Personal Address Book.
Select the addressee that you want to set as plain text, and then click
Properties on the File menu.
In the SMTP-General tab, click to clear the Always send to this recipient in Microsoft Exchange rich text format check box, and then click
OK.
Method 3: Change the specific contact format
The sender can use the following steps to set plain text in the recipient's contact record:
Open the recipient's record in the Contacts folder.
Double-click the recipient's e-mail address.
In the E-Mail Properties dialog box, click Send Plain Text only under
Internet Format.
Method 4: Set the Outlook Rich Text Format Internet e-mail setting
In Outlook 2003 and Outlook 2007, click Options on the Tools menu.
Click the Mail Format tab.
Click Internet Format.
Under Outlook Rich Text options, click either Convert to HTML format or
Convert to Plain Text format.
Hope this helps!
Thanks.
Niko Cheng
TechNet Community Support
Niko,
The Winmail.dat issue is only part of the problem, as described in my original post. For these messages (which it's a different set of messages every time, and it's becoming impossible for me to track them all down), if I set exchange or outlook to encode
the message in Rich Text, the message gets encoded and sent to the recipient. Some of our recipients will reject the message because it is rich text, others receive a winmail.dat file.
However, if I set exchange to not encode messages (TNEF= $false or $null), the message becomes stuck in the submission queue and eventually times out with an error resembling "Storage Transient Failure".

Messages Stuck in Submission Queue after Database failover to DR site

Hi Everyone,
Yesterday Night we had a Database failover to DR site but emails were stuck in the submission queue to resolve we failed back the Database. We have Exchange 2010 SP3 Ru4 with Active-Active DAG across 2 AD sites.
After checking the connectivity logs on the Transport server i found out that the Transport server was trying to connect to the mailbox server where this database was residing directly instead of using the Transport servers in the 2nd site.
We have 4 more databases on the same server but for those databases there was no issues and the mail flow was working fine.
Only thing i checked is that we have about 700 GB of log files for this database. Could that can cause this issue.

Hi,
Thanks for your response.
I did not restart transport service. I assume restarting transport service might have resolved the issue.
But, i want to know the reason why this happened as we have stretch DAG so this can cause major outage in case more mailbox database fails over.
Removing the logs files is what i am thinking to do maybe within this week.

Exchange performance and Data Execution Prevention

Hi
Is there any impact on Exchange server performance related to (DEP) Data Execution Prevention ? Should I exclude Exchange processes from DEP ?
I searched for any blog/forum post that would discuss this subject but had no luck finding any.
Regard,

Hi Grzegorz,
Data Execution Prevention is a Windows feature, not an Exchange server feature.
DEP is enabled on computers that are running Microsoft Windows Server by default.
In our lab, we often turn it on. Sometimes it can resulting that messaging performance is slower than expected. Besides, there is no official articles or blogs to explain how the Data Execution Prevention affect Exchange performance.
Hope my clarification is helpful.
Best regards,
Amy
Amy Wang
TechNet Community Support

Need to perform both Queue and stack operations in Visual C# code

Need to perform both Queue and stack operations, is any Data Structure available for this. or how can i custom create the structure for this?

Hi,
In this structure contains base logics of Queue and Stack. Well stack plays major role here with operations of PUSH, POP with additional operation of Queue i.e., ENQUEUE and DEQUEUE. Here the stack has enqueue and dequeue
https://code.msdn.microsoft.com/The-Stacked-Queues-An-11f3703a
Regards,
Selva Ganapathy K

SCOM 2012 SP1 exchange edge transport queue is showing as not monitored state.

Hi,
We have exchange servers which is being monitored by both SCOM 2007 and SCOM2012 SP1, in SCOM 2007 R2 exchange edge queue is getting discovered, but in SCOM 2012 SP1 it is showing as not monitored state.
Please help me in resolving this issue.

Hi,
What kind of MP did you import on SCOM 2012? Meanwhile, please try the following suggestions.
1. Start the Remote Registry service.
2. Enable File and Printer Sharing
3. Check the network connectivity between the management server and the Exchange Server and ensure the FQDN of the Exchange Server can be resolved.
4. Clear the HealthService queue on the Exchange Server:
a. Stop System Center Management service.
b. Go to C:\Program Files\System Center Operations Manager 2007\, and rename the “Health Service State” folder.
c. Restart System Center Management service.
Niki Han
TechNet Community Support

Exchange Performance load balancing

We have a single Exchange Server with 2 mailbox databases, MB01 and MB02.
Each user is split among the 2 databases. and Each database has its own hard drive location, e:\maildatabase\mb01 and f:\maildatabase\mb02. now the 2 drives are at 85% capacity and the server;s 32Gb memory is in 95-98% physical and commit memory.
I want to introduce a 2nd Server with same specs to reduce the load on the 1st, how do I go about this?
1. do I bring up the 2nd exchange server and move some mailbox over? and force users to only look for the 2nd?
2. Do I do DAG but it means only 1 server is active which would not help my cause
Note: I have 400 users and 78% are OWA users.

Hi MAS,
thanks for the suggestions, actually the log files are in a separate drive and the mailbox database as well.
Yes I am going to deploy another Exchange Server, my challenge is
1. do I split up the database even further into small database per department and spread the mailbox database between the 2 exchange?
Currently the 2 database are about 600GB each
break them into 300GB each and move 2 300GB to new server
Have certain IP/Subnet connect to either servers to reduce work load
DAG the Servers as a way to keep backup. However the resources required would have
dropped
These are just some of my thought,
Question are these doable and SMART?

"Message Queued for submission" performance counter high for Exchange Server 2010 Database

Hi,
We are receiving SCOM Alert as "Mail Submission service is experiencing issue: Messages Queued for Submission-sustained for 15 mins-RED (>25)".
After checking the "Messages queued for Submission" counter on the repective server it was found that the value is 66 but when we check the queue viewer we dont observe any messages stuck in Submission queue.
Would like to know what aspects can be checked here, in order to troubleshoot the issue?
The counter shows 0 value in perfmon for all other mailbox databases.
Thanks & Regards.

Hi,
This is a known issue with Exchange Server 2007 and Exchange Server 2010. We do not recommend that you use this counter for health alerts.
I recommend you refer to the following article:
http://support.microsoft.com/kb/2921142/en-us
It is caused by the MailSubmission service submits a message for delivery multiple times, the counter is incremented two times but decremented only one time. This situation can leave the counter with a positive value even though there
is no message in the queue. When you restart the service, the counter is reset.
Here is an similar thread for your reference:
http://social.technet.microsoft.com/Forums/en-US/c9b485a5-ef0e-4b9a-851e-28c4a4c0606a/msexchangeis-mailbox-messages-queued-for-submission-keeps-increasing?forum=exchangesvradminlegacy
Thanks.
Niko Cheng
TechNet Community Support

How do I improve performance while doing pull, push and delete from Azure Storage Queue

Hi,
I am working on a distributed application with Azure Storage Queue for message queuing. queue will be used by multiple clients across the clock and thus it is expected that it would be heavily loaded most on the time in usage. business case is typical as in
it pulls message from queue, process the message then deletes the message from queue. this module also sends back a notification to user indicating process is complete. functions/modules work fine as in they meet the logical requirement. pretty typical queue
scenario.
Now, coming to the problem statement. since it is envisaged that the queue would be heavily loaded most of the time, I am pushing towards to speed up processing of the overall message lifetime. the faster I can clear messages, the better overall experience
it would be for everyone, system and users.
To improve on performance I did multiple cycles for performance profiling and then improving on the identified "HOT" path/function.
It all came down to a point where only the Azure Queue pull and delete are the only two most time consuming calls outside. I can further improve on pull, which i did by batch pulling 32 message at a time (which is the max message count i can pull from Azure
queue at once at the time of writing this question.), this returned me a favor as in by reducing processing time to a big margin. all good till this as well.
i am processing these messages in parallel so as to improve on overall performance.
pseudo code:
//AzureQueue Class is encapsulating calls to Azure Storage Queue.
//assume nothing fancy inside, vanila calls to queue for pull/push/delete
var batchMessages = AzureQueue.Pull(32); Parallel.ForEach(batchMessages, bMessage =>
//DoSomething does some background processing;
try{DoSomething(bMessage);}
catch()
//Log exception
AzureQueue.Delete(bMessage);
With this change now, profiling results show that up-to 90% of time is only taken by the Azure Message delete calls. As it is good to delete message as soon as processing is done, i remove it just after "DoSomething" is finished.
what i need now is suggestions on how to further improve performance of this function when 90% of the time is being eaten up by the Azure Queue Delete call itself? is there a better faster way to perform delete/bulk delete etc?
with the implementation mentioned here, i get speed of close to 25 messages/sec. Right now Azure queue delete calls are choking application performance. so is there any hope to push it further.
Does it also makes difference in performance which queue delete call am making? as of now queue has overloaded method for deleting message, one which except message object and another which accepts message identifier and pop receipt. i am using the later
one here with message identifier nad pop receipt to delete message from queue.
Let me know if you need any additional information or any clarification in question.
Inputs/suggestions are welcome.
Many thanks.

The first thing that came to mind was to use a parallel delete at the same time you run the work in DoSomething. If DoSomething fails, add the message back into the queue. This won't work for every application, and work that was in the queue
near the head could be pushed back to the tail, so you'd have to think about how that may effect your workload.
Or, make a threadpool queued delete after the work was successful. Fire and forget. However, if you're loading the processing at 25/sec, and 90% of time sits on the delete, you'd quickly accumulate delete calls for the threadpool until you'd
never catch up. At 70-80% duty cycle this may work, but the closer you get to always being busy could make this dangerous.
I wonder if calling the delete REST API yourself may offer any improvements. If you find the delete sets up a TCP connection each time, this may be all you need. Try to keep the connection open, or see if the REST API can delete more at a time
than the SDK API can.
Or, if you have the funds, just have more VM instances doing the work in parallel, so the first machine handles 25/sec, the second at 25/sec also - and you just live with the slow delete. If that's still not good enough, add more instances.
Darin R.

Exchange 2013 Performance counters

Hi All
Can I have performance counters which need to be monitored regularly for below
CAS - Outlook & OWA Connectivity
Mail flow
Database & DAG health
ActiveSync
SAH

Hi SAH,
Tuning a system for optimum performance is an iterative process. We must take the time to understand all the variables that affect your system, including user profile, architecture, and hardware.
More details about Understanding Exchange Performance, please refer to:
https://technet.microsoft.com/en-us/library/dd351192(v=exchg.141).aspx
Besides, we can use Microsoft Exchange Jetstress and Microsoft Exchange Load Generator to evaluate system performance before deploy Exchange evnironment, for your reference:
http://technet.microsoft.com/en-us/library/dd335108(v=exchg.141).aspx
Also, we can run Exchange Best Practices Analyzer to monitor Exchange server:
http://technet.microsoft.com/en-us/library/aa997090(v=exchg.80).aspx
Note: we can download
this tool for Exchange 2013.
Best Regards,
Allen Wang

Turning off 504 non reporting performance collection rules in the Exchange 2010 MP

Hello
I read the following MS article about tuning the Exchange 2010 Management Pack
http://support.microsoft.com/kb/2592561
It advises to turn off 504 non reporting performance collection rules, then only turn back on the ones of interest.
However it does not state how to identify / locate these 504 rules. I would prefer to find/disable them via PowerShell rather than manually.
Can someone please advise how to locate these 504 rules please and disable on mass.
Thanks
AAnotherUser__
AAnotherUser__

you’ll need a script that performs the following steps:
Retrieve the management pack in which to store the overrides
Retrieve the class that will be targeted by the override
Retrieve the rules or monitors that will be disabled
Disable the rule or monitor
To disable bulk of rule, Here is the sample that disables all rules matching the “*events/sec*” filter.
$MP =
Get-SCOMManagementPack -displayname
"Exchange 2010 Overrides" |
`
where {$_.Sealed
-eq $False}
$Class =
Get-SCOMClass -DisplayName
"Exchange Performance rule"
$Rule =
Get-SCOMRule -DisplayName
"*Events/sec"
Disable-SCOMRule
-Class $Class -Rule
$Rule -ManagementPack
$MP -Enforce
Also, you can refer below link
http://www.systemcentercentral.com/opsmgr-2012-disabling-rules-and-monitors-in-bulk-in-powershell/
Please remember, if you see a post that helped you please click "Vote As Helpful" and if it answered your question, please click "Mark As Answer"

Exchange Performance - Submission Queue

Similar Messages

Maybe you are looking for