Report service not processing any new jobs 10.1.2.0.2

Question

I seem to have a job that is 'stuck' in the report queue when I start the report service through opmnctl.
Windows Server 2003
Server version is 10.1.2.0.2
I set the trace options to trace_all, which has given me the following snippet from the rwserver.trc file:
[2008/6/3 3:2:59:8] Debug 50103 (JobStore:writePersistFile): Purge persistent file done
[2008/6/3 3:2:59:133] Warning 50103 (EngineManager:registerEngine): REP-55103: API URLEngine:getEngineEnvs not applicable to URL engine
[2008/6/3 3:2:59:133] Info 56026 (EngineManager:registerEngine): Reports Server started up engine rwURLEng-0
[2008/6/3 3:2:59:149] Debug 50103 (EngineManager:updateEngineState): Engine rwURLEng-0 status is 1
[2008/6/3 3:2:59:149] State 56004 (EngineInfo:setState): Engine rwURLEng-0 state is: Ready
[2008/6/3 3:2:59:227] State 56012 (IdleThread:run): Reports Server is ready
[2008/6/3 3:2:59:227] State 50103 (IdleThread:run): ons notifier started
[2008/6/3 3:2:59:227] Debug 50103 (JobManager:runCurrentJobs): Job 34 is a sync job :false
[2008/6/3 3:2:59:227] Debug 50103 (JobManager:findDuplicatedJob): Found no duplicated job for job 34
[2008/6/3 3:2:59:227] Debug 50103 (JobManager:firstToRun): job 34 is first to run
[2008/6/3 3:2:59:227] Info 56020 (EngineManager:spawnEngine): Launching engine rwEng-0
[2008/6/3 3:2:59:227] Debug 50103 (EngineManager:spawnEngine): Start engine command line = D:\oracle\10gappr2\jdk\jre\bin\javaw -server -cp "D:\oracle\10gappr2\j2ee\home\lib\ojsp.jar;D:\oracle\10gappr2\reports\jlib\rwrun.jar;D:\oracle\10gappr2\jlib\zrclient.jar" -Duser.language=en -Duser.region=US -Xmx256M oracle.reports.engine.RWEngine name=rwEng-0 server=repperthpmstr01opera ORACLE_HOME=D:\oracle\10gappr2 engineimplclass=oracle.reports.engine.EngineImpl traceopts=trace_all tracefile=D:\oracle\10gappr2\reports\logs\repPERTHPMSTR01OPERA\rwEng-0.trc tracemode=trace_replace cacheDir=D:\oracle\10gappr2\reports\cache keepConnection="no" server_ior=D:\oracle\Temp\repperthpmstr01opera_16094127_1212476579227
[2008/6/3 3:2:59:227] Info 56021 (EngineManager:spawnEngine): Engine rwEng-0 has been launched
[2008/6/3 3:2:59:227] State 56004 (EngineInfo:setState): Engine rwEng-0 state is: Initial
[2008/6/3 3:3:0:227] Debug 50103 (JobManager:runCurrentJobs): Job 34 is a sync job :false
[2008/6/3 3:3:0:227] Debug 50103 (JobManager:findDuplicatedJob): Found no duplicated job for job 34
[2008/6/3 3:3:1:227] Debug 50103 (JobManager:runCurrentJobs): Job 34 is a sync job :false
[2008/6/3 3:3:1:227] Debug 50103 (JobManager:findDuplicatedJob): Found no duplicated job for job 34
The final two lines then repeat continuously.
If another job is submitted through the application, the application displays a message that printing is in progress, but the job is never picked up by an engine (the conf file specifies 8 engines). The following is then written to the trace file:
[2008/6/3 3:6:28:404] Debug 50103 (ConnectionImpl:runJob): Job queue for jobid = 48 is 0
[2008/6/3 3:6:28:404] Debug 50103 (ConnectionImpl:runJob): jobid = 48 is in current queue
[2008/6/3 3:6:28:435] Info 56013 (ConnectionManager:release): Connection 0 is released
[2008/6/3 3:6:29:279] Debug 50103 (JobManager:runCurrentJobs): Job 34 is a sync job :false
[2008/6/3 3:6:29:279] Debug 50103 (JobManager:findDuplicatedJob): Found no duplicated job for job 34
Again, those last 2 lines continue to repeat.
I have deleted the row with job_id 34 from rw_server_job_queue, restarted the report process, but the engine still tries to run this job.
This problem is currently in a test environment. Our support department has encountered this recently twice and resolved it by re-installing the software. They can do this in 40 minutes, as opposed to spending hours trying to diagnose with potentially no solution. I'd like to be able to fix this in the test environment so they can be better prepared if it should happen sometime in production.
Can anybody offer a suggestion what I should be looking into next? Please let me know if I haven't included enough info.
Many thanks.

Answer 1

After some trial and error, we found that a particularly badly written report was taking 10 minutes to process (it really should have only taken a few seconds). The users were getting impatient, killing the application from their terminal, and resubmitting the report, over and over and over....
We re-wrote the query in the report, and with the report processing in an acceptable time, we did not re-experience the issue.
Edited by: petesh on Oct 29, 2008 10:25 PM