12.1.3 - FNDLIBR's keep dying

Question

I am in the midst of an R12.1.3 implementation on RH Linux. Things were working nicely until today in our User Acceptance instance.
Noticed some of the Concurrent Mangers were not running. App tier has been shut down and restarted a few times. Bounced the database and listener also but after successfully starting they start to die (defunct). The log file will show "Found dead process:" and then "Starting STANDARD Concurrent Manager" again.
If I look at the Standard Manager and it shows Actual=0 where target =8 and Pending is 23.
I tried shutting down and running Concurrent Manager Recovery' but it seemed to think there were two pid's still running that 'ps -ef' didn't show.
I've tried shutting down and running cmclean now also but to no avail.
Any other things to try before I roll back to a previous backup?
This is not Prod but basically pre-prod.
I've looked at: Why Are Dead Process Found In The Standard Manager Log File? [ID 246584.1]
What about this?
update fnd_concurrent_requests
set phase_code = 'C',
status_code = 'D'
where phase_code = 'P' and
requested_by != 0;
Help!

Answer 1

Here's what I've found.
Example of ORA- errors in alert log. Every 5 minutes.
Thu Dec 29 10:07:18 2011
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP2
Thu Dec 29 10:12:20 2011
ORA-1652: unable to extend temp segment by 16 in tablespace TEMP1
It seems like the TEMP error is being caused by this OEM select statement:
SELECT end_time, status, session_key, session_recid, session_stamp, command_id, start_time, time_taken_display, input_type, output_device_type, input_bytes_display, output_bytes_display, output_bytes_per_sec_display FROM (SELECT end_time, status, session_key, session_recid, session_stamp, command_id, start_time, time_taken_display, input_type, output_device_type, input_bytes_display, output_bytes_display, output_bytes_per_sec_display FROM v$rman_backup_job_details WHERE status NOT LIKE 'RUNNING%' ORDER BY NVL(end_time, start_time) DESC) WHERE rownum = 1
But I'm not sure what the root cause is.
Also should my database have two temporary tablespaces called TEMP1 and TEMP2 (like it does now)?
Shouldn't it just be a TEMP tablespace like in my old 11i rdbms?
select tablespace_name,file_name,bytes/1024/1024 from dba_temp_files;
TABLESPACE_NAME FILE_NAME BYTES/1024/1024
TEMP1 /u10/oradata/xxxxx/db/apps_st/data/temp01.dbf 4000
TEMP2 /u10/oradata/xxxxx/db/apps_st/data/temp02.dbf 4000
Check 11i.
TABLESPACE_NAME FILE_NAME BYTES/1024/1024
TEMP /u10/oradata/xxxxx/temp01.dbf 2000
TEMP /u10/oradata/xxxxx/temp02.dbf 2000
TEMP /u10/oradata/xxxxx/temp03.dbf 2000
TEMP /u10/oradata/xxxxx/temp04.dbf 2000
TEMP /u10/oradata/xxxxx/temp05.dbf 2000
Looks like I have a Temporary Tablespace Group called TEMP so maybe it is the same thing.
Edited by: user6445925 on Dec 29, 2011 8:28 AM
Edited by: user6445925 on Dec 29, 2011 9:11 AM