Azure (Website) WebJobs Performance

Question

I have azure web jobs (Continuous, Always on, Standard Large Instances) that scale out and is just an exe that does processing (lots of it). We've done a lot of perf work locally and testing multiple instances. However, after made the job go from triggered
to continuous the job does nothing.
It runs great processing 30-100 items / second for the first 30 seconds and then it just drops to 0/s. I can reproduce this every time by restarting the website. Locally this runs for days without issue (eating up lots of cpu / network too).
None of the jobs or iis in the process list are using any cpu...
It's worth noting that after 30 seconds my app stops writing to the console or writes out 0/s a few times and then nothing after that.
-Blake Niemyjski (Software Development Engineer)

Answer 1

It's now failing on triggered:
[02/04/2015 22:04:29 > 7af6cf: ERR ] Command 'cmd /c run.bat' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue.
cmd /c run.bat
[02/04/2015 22:04:29 > 7af6cf: SYS INFO] Status changed to Failed
[02/04/2015 22:04:29 > 7af6cf: SYS ERR ] System.AggregateException: One or more errors occurred. ---> Kudu.Core.Infrastructure.CommandLineException: Command 'cmd /c run.bat' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue.
cmd /c run.bat
at Kudu.Core.Infrastructure.IdleManager.WaitForExit(IProcess process)
at Kudu.Core.Infrastructure.ProcessExtensions.<Start>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Kudu.Core.Infrastructure.Executable.<ExecuteAsync>d__1d.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args)
at Kudu.Core.Infrastructure.Executable.ExecuteReturnExitCode(ITracer tracer, Action`1 onWriteOutput, Action`1 onWriteError, String arguments, Object[] args)
at Kudu.Core.Jobs.BaseJobRunner.RunJobInstance(JobBase job, IJobLogger logger, String runId)
---> (Inner Exception #0) ExitCode: -1, Output: Command 'cmd /c run.bat' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue., Error: Command 'cmd /c run.bat' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue., Kudu.Core.Infrastructure.CommandLineException: Command 'cmd /c run.bat' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue.
cmd /c run.bat
at Kudu.Core.Infrastructure.IdleManager.WaitForExit(IProcess process)
at Kudu.Core.Infrastructure.ProcessExtensions.<Start>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Kudu.Core.Infrastructure.Executable.<ExecuteAsync>d__1d.MoveNext()<---
I enabled debugging and have the latest azure sdk and vs2013 and I can't attach the debugger.. 
The job seems to get throttled..
-Blake Niemyjski (Software Development Engineer)