"Waiting for output formatter"

Printing to network printers through a spool server has stopped working. Local front end printing still works fine but anything sent to one of the network printers waiting for spool server with the status text "Waiting for output formatter". There are plenty of spool processes waiting and no errors. Its a system recently upgraded to ERP2005 but has worked since for a month.

Chris Soden wrote:>
> One thing I have noticed in the spool system settings in SPAD the last spool scan on the spool server was yesterday just before the printing stopped.
Hi Chris,
I don't think this is the reason. Can you please check the logs of the spool requests and post them here?
Regards
Juan

Similar Messages

Printer delay - Waiting for output formatter

Hello,
since 2 days, our print outs out of SAP take up to 10 minutes before the actually get printed out.
In SP01 it says "waiting for output formatter".
- I checked table TSP01and it only has 800 entries.
- When I do a reprint for the delayed print out, it is printing out right away with no delay.
Any suggestions?
Thanks
Anne

We found the issues, so for the record:
A couple of print jobs have been sent to one printer and got stuck. The person using this printer is on vacation so nobody noticed. However after the printer has been turned off and back on, the print outs came out and the print speed for all other prints went back to normal.
Seems like this one printer was holding up all other prints.

Job is "Waiting for Output"

An RMAN backup-arc job is scheduled to run every day will fail to run after a week or 2. The status of the job shows as "Running", however, the command step in the logs shows that the job is "Waiting for Output". At this point the job stops running and will not execute anymore. This happens with other jobs as well.
There is BUG (4349948) which has been open for a year with no results.
Does anyone know a work around? How do I reliably run repetative jobs if I cannot count on OEM?
The job was scheduled through the Recovery wizard in OEM Grid 10.1.0.4

Thank you for the info. I do not have this EV defined on my Windows2003 server and I did a search of the oracle documentation 'EMAGENT_RECYCLE_DAYS' and found nothing. I am going to try it and hope I do not have to re-create the RMAN Jobs each week. If so I think I will fall back to Windows Task Manager and batch jobs.
Thanks again.

Calling another program and waiting for output

Hi,
I'm writing a program that passes a file created for an application called Net Sim but has to wait for the output. Net Sim lies in another directory from the java files and I don't know how to call the program with the file created as a parameter, such as:
ns2 network1.tcl
where 'ns2' is the dos command to run net sim and 'network1.tcl' is a file passed as a parameter to the program.
I've tried
try{
Runtime r = Runtime.getRuntime();
r.exec(" <directory> ");
}catch(IOException e){
System.out.println("Error " +e);
This doesn't work though and I could use some help, as well as how to save the file created by the program in another directory.
Finally, I need to know how to make the program wait for the output from net sim which is a file I intend to use. Can I do this with a while loop etc.?
I thank you in advance for any help,
Regards,
Gary.

Here's an article from JavaWorld describing some of the pitfalls when using Runtime.exec() that might help you:
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

HSDIO conditionally fetch hardware compare sample errors (script trigger to flag whether or not to wait for software trigger)

I am moderately new to Labview and definitely new to the HSDIO platform, so my apologies if this is either impossible or silly!
I am working on a system that consists of multiple PXI-6548 modules that are synchronized using T-CLK and I am using hardware compare. The issue I have is that I need to be able to capture ALL the failing sample error locations from the hardware compare fetch VI... By ALL I mean potentially many, many more fails than the 4094 sample error depth present on the modules.
My strategy has been to break up a large waveform into several subsets that are no larger than 4094 samples (to guarantee that I can't overflow the error FIFO) and then fetch the errors for each block. After the fetch is complete I send a software reference trigger that is subsequently exported to a scriptTrigger that tells the hardware it is OK to proceed (I do this because my fetch routine is in a while loop and Labview says that the "repeated capbility has not yet been defined" if I try to use a software script trigger in a loop).
This works fine, but it is also conceivable that I could have 0 errors in 4094 samples. In such a case what I would like to do is to skip the fetching of the hardware compare errors (since there aren't any) and immediately begin the generation of the next block of the waveform. That is, skip the time where I have to wait for a software trigger.
I tried to do this by exporting the sample error event to a PFI and looping that PFI back in to generate a script trigger. What I thought would happen was that the script trigger would get asserted (and stay asserted) if there was ever a sample error in a block, then I could clear the script trigger in my script. However, in debug I ended up exporting this script trigger back out again and saw that it was only lasting for a few hundred nanoseconds (in a case where there was only 1 injected sample error)... The sample error event shows up as a 1-sample wide pulse.
So, my question is this: is there a way to set a flag to indicate that at least one sample error occurred in a given block that will persist until I clear it in my script? What I want to do is below...
generate wfmA subset (0, 4094)
if scriptTrigger1
clear scriptTrigger1
wait until scriptTrigger0
end
clear scriptTrigger0
generate wfmA subset (4094, 4094)
I want scriptTrigger1 to be asserted only if there was a sample error in any block of 4094 and it needs to stay asserted until it is cleared in the script. scriptTrigger0 is the software trigger that will be sent only if a fetch is performed. Again, the goal being that if there were no sample errors in a block, the waiting for scriptTrigger0 will not occur.
I am probably going about it all wrong (obviously since it doesn't work), so any help would be much appreciated!

Please disregard most of my previous post... after some more debug work today I have been able to achieve the desired effect at slower frequencies. I did straighten out my script too:
generate wfmA
if scriptTrigger1
clear scriptTrigger0
wait until scriptTrigger0
end if
generate wfmA
scriptTrigger1 = sample error event flag
scriptTrigger0 = software trigger (finished fetching error backlog in SW)
However, I am still having a related issue.
I am exporting the Sample Error Event to a PFI line, looping that back in on another PFI line, and having the incoming version of the Sample Error Event generate a script trigger. My stimulus has a single injected sample error for debug. For additional debug I am exporting the script trigger to yet another PFI; I have the sample error event PFI and the script trigger PFI hooked up to a scope.
If I run the sample clock rate less than ~133MHz everything works... I can see the sample error event pulse high for one clock period and the script trigger stays around until it is consumed by my script's if statement.
Once I go faster than that I am seeing that the script trigger catches the sample error event occasionally. The faster I go, the less often it is caught. If I widen out the error to be 2 samples wide then it will work every time even at 200MHz.
I have tried PFI0-3 and the PXI lines as the output terminal for the sample error event and they all have the same result (this implies the load from the scope isn't the cause).
I don't know what else to try? I can't over sample my waveform because I need to run a true 200MHz. I don't see anything that would give me any other control over the sample error event in terms of its pulsewidth or how to export it directly to a script trigger instead of how I'm doing it.
Any other ideas?

Cups and "waiting for printer to become available"

Hi all,
i have a problem with cups (i don't know since when),
when i try to print something i receive the message:
"waiting for printer to become available"
And i'm aware that this problem happens when you have usblp loaded. So first of all i checked if the usblp module is loaded and here the result:
usblp 9311 0
usbcore 119004 8 usblp,uvcvideo,uas,usb_storage,usbhid,ohci_hcd,ehci_hcd
But if i go to check the /etc/rc.conf file the MODULES line is:
MODULES=(lib80211_crypt_tkip wl radeon vboxdrv vboxnetflt powernow-k8 cpufreq_ondemand !snd-pcm-oss !b43 !ssb !usblp)
And it seems that usblp module is not loaded.
Now the question is:
What's wrong?
I did something wrong? I forgot somethin?
Thanks,
Ivan

Oddly enough...
I couldn't find usblp in my lsmod output but I was also getting this "cups waiting for printer". My file utility is version 5.18 and although I considered downgrading it, that just felt too much like thrashing about. (My printer is a USB HP LaserJet 1020)
In the end, I did something else that felt like almost as much thrashing about. I followed the "changes to module blacklisting" page with usblp, restarted cups and cycled power on the printer. Now it's working again. I wish I entirely new why.
I think what happened was the automatic power up of the printer (or of the computer) loaded usblp but the fooz file for LaserJet was unloading it so (a) I never saw it in the lsmod output and (b) it was in the system long enough to snarl things, followed by (c) if only it was blacklisted at the time that the system saw the printer it would not be there and not foul things up. Does that sound like a reasonable chain of events.
one more reason to resent (grumble, grumble, grr grr) systemd over the older (admittedly clunkier, more manual -- but more CONTROLLABLE!) initscripts. (Okay, I'm over 45)
cheers...ank
Last edited by ansak (2014-04-27 03:43:41)

BackupExec: A timeout occurred waiting for NDMPD.NLM to init

"Error: 1004 A timeout occurred waiting for NDMPD.NLM to initialize."
A very common and well documented issue but every fix I have found has not worked. The list is extensive so I will start by describing the issue, then the environment and finally what I have tried so far.
The issue
I can not load BackupExec 9.1 or 9.2 on NetWare 6.5 SP8. The loader starts and when it gets to loading NDMPD.NLM the CPU hits 99% (reported by monitor.nlm) and BackupExec stops loading for about 80-120 seconds. The loader eventually gives up after a non-critical abend. From this point on I can not unload becdm.cdm and I generally reboot the server to clear the abend.
BackupExec has been working on the server for about 2 years, I can not link any changes to the time it stopped working.
It used to load reliably every time for about 1.5 years and then it started to exhibit this issue when loading 1 in 10 times. It has gradually gotten worse and now will not load at all.
Before you say it, yes I know Symantec dropped BackupExec for NetWare years ago.
It seems to me this can only be a hardware issue now but I'm keen to hear any other ideas?
The environment
* The physical server is a HP DL385 G5 (SKU 411360-371), the tape drive is an external half height HP Ultrium 448 LTO 2 (SKU DW086A) using SAS to connect to a HP SC44Ge SAS HBA (SKU 416096-B21). It has 18GB of RAM, Two quad-core AMD Opteron 2352 2.1 GHz CPU's and 5 x direct attached SAS hard drives running on the internal P400 RAID controller configured with RAID 5.
* I can provide firmware versions if needed, the BIOS is HP A09 2009-07-11. Don't have the tape drive or HBA versions to hand.
* The server is running VMware ESXi 4.1.0 build 260247, using a free license i.e. stand alone.
* The VMware server uses about 50% of RAM, 70% of storage and 12% of CPU.
* The guest is NetWare 6.5 SP8 (overlay) running as a guest with 4GB of vRAM and a single CPU. The tape drive is presented as a generic SCSI device.
* I'm running BackupExec for NetWare 9.2.1401.5 build 287839 with the NetWare Open File Option and a Windows Remote Agent.
* I'm running the storage driver LSIMPTNW.HAM version 5.02, 5 Dec 2007, which is default with SP8. BackupExec 9.2 device driver becdm.cdm version 7.50, 7 Feb 2006.
* The NetWare server sees the tape drive as an "Unbound Device Object" until becdm.cdm is loaded at which point its recognsied as "HP Ultrium 2-SCSI T65D"
* The tape drive is listed on the BackupExec HCL - "StorageWorks Ultrium 448 LTO2" except mine is SAS not SCSI. Enterprise Support - Symantec Corp. - Symantec Backup Exec for NetWare Servers (tm) 9.x (9.0, 9.1, 9.2) Hardware Compatibility List. Includes HCL information for supported drives, libraries, virtual tape devices, fibre-channel HBAs, switches, and rou
What have I tried so far?
* We have a rental tape drive identical to our own as the original one was over-heating, so I have tried two physical tape drives. Both using the same server, HBA and SAS tape drive cable.
* I have used tcpcon.nlm to check that port 10,000 is unused.
* I have tried the parameter -!X on the line of bestart.ncf that loads BackupExec to stop it checking if port 10,000 is free.
* I have added a set statement to c:\nwserver\startup.ncf "set auto load of cdm modules = off" so it will not try to load unwanted CDM's.
* I have renamed all copies of nwtape.cdm to nwtape.cdx on the C: so they can not be auto loaded.
* There are no load statements in the startup.ncf for other tape drive device drivers. Just scsihd.cdm and idecd.cdm.
* There are no tape libraries or robots involved. Its just a basic tape drive.
* The physical and virtual server have been rebooted numerous times to ensure its not a memory issue etc...
* I've loaded the server with no other significant modules loaded. Just the basics like eDirectory etc... and the same issue still occurs.
* In addition to the NW65 SP8 default version of lsimptnw.ham I have tried version 5.03.01, 23 Jan 2008, from the LSI website.
* I have deleted the original BackupExec folder from the server (sys:\bkupexec) and re-installed BackupExec numerous times.
* I have powered off the tape drive a number of times
* I have changed the tape drives virtual SCSI ID, presented by VMware to the NetWare guest, to be a higher number on the same virtual HBA as the virtual disk. I have also tried allocating the tape drive a virtual SCSI ID on a dedicated virtual HBA, SCSI 1:0.
* I have deleted the SMDR config file (sys:\etc\sms\smdr.cfg) and re-created it using the command "load smdr.nlm new"
* I have deleted the NetWare servers backup queue and the SMS SMDR Group from eDirectory and re-created the queue using the command "load qman.nlm new". Note: as its NW65SP8 the SMS RPC object does not exist and the SMS SMDR group is not re-created.
* I have deleted the two BackupExec objects from eDirectory and re-created them by loading BackupExec using "bestart.ncf".
* I have checked the health of eDirectory.
* I have tried the LSI SAS and LSI Parallel virtual HBA's in the VMware guest machine.
* I have tried adding the parameter qtags=off to the load command for lsimptnw.ham in startup.ncf.
* I have tried the parameter -b on the line of bestart.ncf that loads BackupExec
* I have tried loading ipxspx.nlm to stop any public symbol error messages when loading BackupExec
* I have also tried BackupExec version 9.1 build 0306.12 and 9.1.1158.10 build 289707
* I have built a second NetWare 6.5 SP8 server (SP8 overlay DVD, MD5 verified ISO media) using the Backup server profile during the GUI install stage. I have loaded afreecon.nlm, installed VMware tools (vmwtool.nlm) and BackupExec for NetWare 9.2.1401.5 build 287839. The same issue occurs
System output
1) Excerpt from "list storage adapters"
0x04 [V358-A3] SAS1068:00008 [slot 8]
0x06 [V358-A3-D0:0] HP Ultrium 2-SCSI T65D
2) Backup Exec SureStart Console
Module NETDB.NLM is already loaded
Module SMDR.NLM is already loaded
Module TSAFS.NLM is already loaded
Loading module TSANDS.NLM
Module BECDM.CDM is already loaded
Loading module B2D.NLM
Module NWIDK.NLM is already loaded
Module NSS.NLM is already loaded
Loading module OFM.NLM
Loading module NRLTLI.NLM
Loading module AD_ASPI.NLM
Loading module NDMPD.NLM
Error: 1004
A timeout occurred waiting for NDMPD.NLM to initialize.
Press <Alt+Esc> to return to the Server Console. After you resolve
the error condition(s), re-execute the BESTART command.
3) Abend log
Novell Open Enterprise Server, NetWare 6.5
PVER: 6.50.08
Server halted Sunday, 6 May 2012 5:38:23.196
Abend 1 on P00: Server-5.70.08: Page Fault Processor Exception (Error code 00000002)
Registers:
CS = 0008 DS = 0010 ES = 0010 FS = 0023 GS = 0023 SS = 0010
EAX = 00000000 EBX = 00204B58 ECX = 896B77E0 EDX = 8A26157C
ESI = 00204B58 EDI = 00000F9B EBP = 8A261DD4 ESP = 8A261DA8
EIP = 8A1E03FD FLAGS = 00010082
8A1E03FD C680B00E000000 MOV [EAX+00000EB0]=?, 00
EIP in LSIMPTNW.HAM at code start +000013FDh
Access Location: 0x00000EB0
The violation occurred while processing the following instruction:
8A1E03FD C680B00E000000 MOV [EAX+00000EB0], 00
8A1E0404 8A45FC MOV AL, [EBP-04]
8A1E0407 8845F8 MOV [EBP-08], AL
8A1E040A 8A45F8 MOV AL, [EBP-08]
8A1E040D 89EC MOV ESP, EBP
8A1E040F 5D POP EBP
8A1E0410 5F POP EDI
8A1E0411 5E POP ESI
8A1E0412 5B POP EBX
8A1E0413 C3 RET
Running process: Server 10 Process
Thread Owned by NLM: SERVER.NLM
Stack pointer: 8A261F5C
OS Stack limit: 8A25A020
Scheduling priority: 67371008
Wait state: 50500F0 Waiting for work
Stack: --8A261E01 ?
--00000001 (LOADER.NLM|KernelAddressSpace+1)
--00951740 ?
8A1E0414 (LSIMPTNW.HAM|(Code Start)+1414)
--0097BAA0 ?
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--01900190 ?
--157C157C ?
--0D000001 ?
--0097BACC ?
--8A261D01 ?
--8A261E18 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A1DFD6B (LSIMPTNW.HAM|(Code Start)+D6B)
--0097BAA0 ?
--0097BAC0 ?
--00000001 (LOADER.NLM|KernelAddressSpace+1)
--0097BA50 ?
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--00F20000 ?
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--00200000 ?
--0097BAA0 ?
--0097BAC0 ?
--8A261EB0 ?
--8A261E44 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A1FAA64 ?
--0097BA50 ?
--00000F03 (LOADER.NLM|KernelAddressSpace+F03)
--00000286 (LOADER.NLM|KernelAddressSpace+286)
--00951740 ?
8972FA00 (NWPA.NLM|HAI_Enable_Real_Mode_Access+34)
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--8A261E70 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A1FAC47 ?
--0097BA50 ?
--00204B58 ?
--0097BA50 ?
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--01080000 ?
--01080000 ?
--8A261E90 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A1FAC95 ?
--00952180 ?
--010824A0 ?
--02058001 ?
--8A261EB8 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A207658 ?
--010824A0 ?
--01091240 ?
--00951740 ?
--00008100 (LOADER.NLM|KernelAddressSpace+8100)
--00008134 (LOADER.NLM|KernelAddressSpace+8134)
--8A261EEC ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
8A208ED6 ?
--010824A0 ?
--00000003 (LOADER.NLM|KernelAddressSpace+3)
--00000046 (LOADER.NLM|KernelAddressSpace+46)
--02010002 ?
--010824A0 ?
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--00951740 ?
--00000001 (LOADER.NLM|KernelAddressSpace+1)
--00204B60 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
89729AE5 (NWPA.NLM|NPAThreadRoutine+3D)
--00010002 ?
-896E5110 (MM.NLM|PostponeEvent+0)
8973173F (NWPA.NLM|CDM_MountActivate_Message+EB)
--00000001 (LOADER.NLM|KernelAddressSpace+1)
--00000000 (LOADER.NLM|KernelAddressSpace+0)
--802171E4 ?
--802171E4 ?
80327F6B (MM.NLM|RetryPostponedMessages+237)
--01081A00 ?
--00204B60 ?
--00000F9B (LOADER.NLM|KernelAddressSpace+F9B)
--00204B58 ?
--00204B58 ?
003682B2 (SERVER.NLM|CallAESRoutineWithEsiSet+A)
Additional Information:
The CPU encountered a problem executing code in LSIMPTNW.HAM. The problem may be in that module or in data passed to that module by a process owned by SERVER.NLM.
Thanks,
MC

Well the problem is now resolved. It was either faulty tapes (about 12 of) or two tape drives with faulty heads. We had the heads replaced in one of the drives and tested it with a new tape, it works perfectly again.
MC

Waiting for a constructed image

I am having great difficulty with an image constructed from others. Essentially, I create an empty image and fill it with a series of drawImage() commands on the graphics context. However, I simply cannot get execution to wait for the image to be finished before it's displayed.
There are a couple of places where I may need to wait- when the images are first loaded (they are stored so only one disk access is needed for each image) and when the images are scaled before drawing them to the main image.
However, despite packing my code to the gills with MediaTrackers, nothing seems to force it to wait. I've been using:
MediaTracker tracker=new MediaTracker(this);
tracker.addImage(someImage,width,height,1); //or without the scale parameters
try{
tracker.wairForAll();
catch(Exception e) { System.err.println("Tracker error");}
//then use the image as requiredinside the JLabel that's doing the drawing, and then inside the class that loads the images. It simply doesn't work. Refreshing the output shows that the amount that gets drawn varies. Where is the delay? How can I force display to wait for the delay? It's driving me spare.

http://forums.adobe.com/thread/1195540

Waiting for udev uevents to be processed hangs for 30s. |SOLVED|

Hello. This is my first post.
Just installed ArchLinux with KDE 4.7,everything is working perfectly but something is really annoying.
When booting up the laptop (toshiba satellite) it takes too much loading this "waiting for udev uevents to be processed", like 30 or 40 secs,and after that all is fine.
i have tried a lot of things with DAEMONS AND MODULES, also tried blacklisting some modules just like this post said.
https://bbs.archlinux.org/viewtopic.php?id=133032
But nothing has worked.
Maybe this could help or at least be related:
dmesg | grep fail
[ 0.496066] pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND), returned control mask: 0x1d
[ 0.499488] pci0000:3f: ACPI _OSC request failed (AE_NOT_FOUND), returned control mask: 0x1d
It looks just like the same thing that happened to TuxLyn in the post i showed before.
I have an ATI card HD5400 so the other solutions in the Forum like this:
https://bbs.archlinux.org/viewtopic.php?id=92189 didn't work at all.
Btw, ArchLinux rules!, i really liked it so decided to leave Debian!
Greets!
Last edited by imJorgeMerinoS (2012-01-25 02:51:04)

Welcome to arch. Great first post - all the right info, and it shows you've looked around for your own answer.
However, it seems you did miss the one thread you needed to find. The title was something like "Udev hangs for 30 seconds on boot." If you google that (with site:bbs.archlinux.org) you should find it quickly.
The gist of it is that there are certain modules that need to be added to the MODULES line in rc.conf for now. It seems that the vast majority (if not all) of these are the wireless modules - brcmsmac in my case, but it would be different in yours.
If you don't know your wireless module you can find it in the output of "lspci -v" and look for the line "kernel driver in use" under your wireless card.

NiDCPower Wait For Event

Hello,
I am using the PXIe-4139 SMU to source a current sequence that starts when a trigger is received and takes about 40 seconds to complete.
After the niDCPower Initiate VI I have inserted the niDCPower Wait For Event VI with a timeout set to 300 seconds to allow for enough time to receive the trigger and complete the sequence. This VI waits for the Sequence Engine Done Event.
Now this works fine, however, when the user wishes to stop the programm for whatever reasons, they have to wait the full 300 seconds until the Wait For Event VI produces a timeout error and stops the programm. This is rather inconvenient. How can I change the programm so that the user can stop it even when the Wait For Event VI hasn't reached its timeout?
I tried this:
But this doesn't work. For some reason, I get a timeout error with code -200474 but the boolean operator returns FALSE.
Any other ideas?

If there an output on the wait for event VI that flags a timeout? If so, set the timeout short (like 100 msec) and wait for 3000 timeouts before continuing on.
Oh, and use an event structure. Timeout event (timeout=0) checks the wait for event, value change event for button breaks out of loop.
Mike...
Certified Professional Instructor
Certified LabVIEW Architect
LabVIEW Champion
"... after all, He's not a tame lion..."
Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

Query waiting for objects...

Hi,
I have a Pl/Sql query which when run is taking a long time to execute.
When i put a trace on it, i found that it is waiting for 2 objects and as soon as the wait is over the query produces the output immediately.
I tried puuting index on tables which had large records as i thought initially that it might be waiting due to full table scan, but its not of much help.
Earlier this query used to run within 5 minutes some 3 months back, but now it is taking about 1 hour daily and when the trace is on, it shows again it is waiting for 2 objects.
What im thinking is that, earlier when the query used to run within 5 minutes the data involved was not that heavy, but since the last 2 months the data is in millions every month.
Could you suggest me a possible solution to this problem of waiting for objects.
Thanks.
Database version being used - Oracle 10g

Hi,
When i put a trace on it, i found that it is waiting for 2 objects and as soon as the wait is over the query produces the output immediately.'it is waiting for 2 objects?' What exactly do you mean by that?
If you've done a trace then why didn't you post the tkprof output?
Please post the tkprof output (including wait events) and your PL/SQL procedure (or a representative piece of sample code)
Put this tag => before and after your tkprof output/code examples, so it says readable.

Query "Stuck" in CXPacket wait with no other waits for that session id in the blocking session id of sys.dm_os_waiting_tasks

So I have a query that is running with a large number of CXPacket waits. There are no other tasks in the sys.dm_os_waiting_tasks that are blocking this session id with a wait other than CXPacket Wait. This is taking place on a SQL Server 2008
R2 machine. My question is there something else to look at? I've gone ahead and posted the output of the entire sys.dm_os_waiting_tasks. Any thoughts?
waiting_task_address   session_id   exec_context_id   wait_duration_ms   wait_type   resource_address   blocking_task_address   blocking_session_id
blocking_exec_context_id   resource_description
0x0000000004E88988   NULL   NULL   13525   FT_IFTS_SCHEDULER_IDLE_WAIT   NULL   NULL   NULL   NULL   NULL
0x0000000004E89288   NULL   NULL   64128606   DISPATCHER_QUEUE_SEMAPHORE   NULL   NULL   NULL   NULL   NULL
0x0000000004ED6748   2   0   19502   XE_TIMER_EVENT   NULL   NULL   NULL   NULL   NULL
0x0000000004450508   3   0   2779501   XE_DISPATCHER_WAIT   NULL   NULL   NULL   NULL   NULL
0x000000000446A748   4   0   838   LAZYWRITER_SLEEP   NULL   NULL   NULL   NULL   NULL
0x000000000448E508   5   0   438   REQUEST_FOR_DEADLOCK_SEARCH   0x000000000B9301F8   NULL   NULL   NULL   NULL
0x0000000004E88508   6   0   1304946379   KSOURCE_WAKEUP   NULL   NULL   NULL   NULL   NULL
0x0000000004484508   7   0   146961   LOGMGR_QUEUE   0x0000000003597248   NULL   NULL   NULL   NULL
0x00000000804942C8   9   0   3366   SQLTRACE_INCREMENTAL_FLUSH_SLEEP   NULL   NULL   NULL   NULL   NULL
0x000000000448E748   10   0   929   SLEEP_TASK   NULL   NULL   NULL   NULL   NULL
0x0000000004450988   11   0   26214727   BROKER_EVENTHANDLER   NULL   NULL   NULL   NULL   NULL
0x0000000004EBC508   12   0   1304959888   ONDEMAND_TASK_QUEUE   0x0000000003575508   NULL   NULL   NULL   NULL
0x0000000004EA2508   13   0   146959   CHECKPOINT_QUEUE   0x0000000003590C70   NULL   NULL   NULL   NULL
0x0000000004EA2748   14   0   1304947282   BROKER_TRANSMITTER   NULL   NULL   NULL   NULL   NULL
0x0000000004484988   16   0   1304947282   BROKER_TRANSMITTER   NULL   NULL   NULL   NULL   NULL
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x0000000255EFEBC8   154   19   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x0000000258D26748   154   17   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x0000000258DB0E08   154   18   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x000000025A392748   154   21   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x0000000258DB2E08   154   20   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x000000025C694508   154   23   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x000000025A675708   154   24   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x000000000446AE08   154   0   19974433   CXPACKET   0x0000000804ECC1D0   0x00000001D3F902C8   154   22   exchangeEvent
id=Port80263100 WaitType=e_waitPortOpen nodeId=4
0x0000000004EBCE08   154   1   18269246   CXPACKET   0x000000080C705200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6220e00 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004485288   154   2   18269246   CXPACKET   0x000000080CD4F200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6220f00 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004EA3B88   154   3   18269246   CXPACKET   0x000000080CCDD200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6221000 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004ED7948   154   4   18269246   CXPACKET   0x000000080CC8B200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6220f80 WaitType=e_waitPipeGetRow nodeId=33
0x000000000448F708   154   5   18269242   CXPACKET   0x000000080CD41200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6220e80 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004451B88   154   6   18269245   CXPACKET   0x000000080CE31200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6221080 WaitType=e_waitPipeGetRow nodeId=33
0x0000000004E89048   154   7   18269246   CXPACKET   0x000000080D12B200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6221100 WaitType=e_waitPipeGetRow nodeId=33
0x000000000446ABC8   154   8   18269243   CXPACKET   0x000000080CE7D200   0x000000000448FB88   154   13   exchangeEvent
id=Pipe2f6221200 WaitType=e_waitPipeGetRow nodeId=33
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258D26748   154   17   18270546   CXPACKET   0x000000025C763020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247880 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB0E08   154   18   19974425   CXPACKET   0x00000008064DD020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247900 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000255EFEBC8   154   19   18269474   CXPACKET   0x0000000806355020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247800 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x0000000258DB2E08   154   20   18270502   CXPACKET   0x0000000805CC1020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247a00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322346   CXPACKET   0x0000000806395020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A392748   154   21   18322347   CXPACKET   0x0000000806395020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247980 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x00000001D3F902C8   154   22   18270621   CXPACKET   0x000000025CE67020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247c00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025C694508   154   23   18270541   CXPACKET   0x000000025C75D020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247a80 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004EBCE08   154   1   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x000000000448F708   154   5   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004485288   154   2   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004ED7948   154   4   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004EA3B88   154   3   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004451B88   154   6   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x0000000004E89048   154   7   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17
0x000000025A675708   154   24   18269840   CXPACKET   0x000000025C767020   0x000000000446ABC8   154   8   exchangeEvent
id=Pipe9c247b00 WaitType=e_waitPipeGetRow nodeId=17

Hello,
First, I would take a look at what your MAXDOP setting is, that's seems a bit high unless you have around 80 processors - stick to multiples of the number of processors in a single numa node and don't go over half of the numa nodes if this is for an OLTP
system. If it's for a DSS or DW system then having a very high maxdop should be fine.
When dealing with CXPACKET the wait means that it's waiting for all of the parallel threads to complete before it can merge the results and move on. If any single thread has to process more than any other (or waits longer for data) the whole process has
to wait. I would investigate to see if there is a large skew in statistics and one thread is processing much more than another. I'd also try creating an extended events session to grab all of the waitinfo for that session.
Sean Gallardy | Blog |
Twitter

Valid actions to prevent "Waiting for in-flux to clear"

Hi all,
In our application we use to do heavy processing with some connections to Timesten instance (about 20).
Sometimes, we got this in-flux message on syslog:
<wait not in flux> Waiting for in-flux to clear (now reason 2=connect pid 14515 nwaiters 87 count 597
ds /tt02/ttvl2/timesten/RT_02/ds/mta4350_64_02)
While these messages are printed on syslog, we're unable to create new connections to this instance. Normally we rename this instance entry on ODBC.ini to avoid new connections and wait until there's no in-flux messages with a so high value of nwaiters. This may take several hours...
What is the real meaning of this message? Perhaps some bootleneck or invalid configuration...
And what is the actions we can adopt to avoid this?
Thanks in advance,

Hi jsPalmer,
We use different disks for datastore and trasaction log files. And there's not a specific date time when the problems occurs. But for sure, this hapens always with one of our binaries (pm1RerateExtract).
We are a telecom company (Claro - Brazil) and this is part of a rerating process (developed by Amdocs). Every time we close a cycle for rerating and billing, this job is started. Today We have 31 Billing Cycles (effective: 26). Each one of this cycles have a datastore configured in TT (balance between 2 machines).
The cycle I posted the informations bellow (14) is big in terms of customers and calls.
I ran the monitor command you said, but there's no specific "LOG /BUFFER WAIT" attribute. I think this old version doesn't give this kind of output:
By now, we're running that process that uses to cause the locks. So far it's processing, but there's 90% of chance in the next hours it hangs and keep the ttcserver connections alive, locking new connections.
I'm collecting as much information as I can to open the SR
Some of the informations I've collected now:
# monitor output (Datastore Rt14_64S)
Command> monitor;
TIME_OF_1ST_CONNECT: Mon Feb 7 10:22:14 2011
DS_CONNECTS: 3915
DS_DISCONNECTS: 3807
DS_CHECKPOINTS: 5993
DS_CHECKPOINTS_FUZZY: 5993
DS_COMPACTS: 0
DS_MAX_SIZE: 8125440
DS_ALLOCATED_SIZE: 8125440
DS_IN_USE_SIZE: 4809078
PERM_ALLOCATED_SIZE: 7065600
PERM_IN_USE_SIZE: 4557324
PERM_IN_USE_HIGH_WATER: 4569842
TEMP_ALLOCATED_SIZE: 1059840
TEMP_IN_USE_SIZE: 251754
TEMP_IN_USE_HIGH_WATER: 571970
XACT_BEGINS: 577428492
XACT_COMMITS: 576843988
XACT_D_COMMITS: 2181311
XACT_ROLLBACKS: 584488
LOG_FORCES: 2189718
DEADLOCKS: 273
LOCK_TIMEOUTS: 37285
LOCK_GRANTS_IMMED: -1332342540
LOCK_GRANTS_WAIT: 234757
LOCK_DENIALS_COND: 1714469
CMD_PREPARES: 2291678
CMD_REPREPARES: 9430
CMD_TEMP_INDEXES: 0
LAST_LOG_FILE: 50354
REPHOLD_LOG_FILE: -1
REPHOLD_LOG_OFF: -1
SYS1: 0
SYS2: 0
SYS3: 0
SYS4: 0
SYS5: 0
SYS6: 0
SYS7: 13328
SYS8: 734735768
SYS10: 1
SYS11: 0
SYS12: 0
SYS13: 0
SYS14: 0
SYS15: 0
SYS16: 0
SYS17: 0
SYS9:
# dssize output:
Command> dssize m;
DS_MAX_SIZE: 7935
DS_ALLOCATED_SIZE: 7935
DS_IN_USE_SIZE: 4710
PERM_ALLOCATED_SIZE: 6900
PERM_IN_USE_SIZE: 4450
PERM_IN_USE_HIGH_WATER: 4462
TEMP_ALLOCATED_SIZE: 1035
TEMP_IN_USE_SIZE: 260
TEMP_IN_USE_HIGH_WATER: 558
# This specific datastore settings (note: LogBuffSize in KB; LogFileSize in MB )
[Rt14_64S]
Driver=/opt/TimesTen4.3/64/lib/libtten.sl
DataStore=/tt01/ttvl15/timesten/RT_14/ds/mta4350_64_14
LogDir=/tt01/ttvl95/timesten/RT_14/log
#BackupDir=/tt01/ttvl99/timesten/backup/RT_14
#ArchDir=/tt01/ttvl98/timesten/archive/RT_14
PermSize=6900
TempSize=1035
PermWarnThreshold=90
TempWarnThreshold=80
LogFileSize=32
LogBuffSize=256
Authenticate=0
AutoCreate=0
DurableCommits=0
ExclAccess=0
Isolation=1
LockLevel=0
Logging=1
LogPurge=0
MatchLogOpts=0
MemoryLock=1
OverWrite=0
Preallocate=0
SMPOptLevel=3
Temporary=0
ThreadSafe=1
WaitForConnect=1
UID=oper
# ttStatus output
Data store /tt01/ttvl15/timesten/RT_14/ds/mta4350_64_14
There are 41 connections to the data store
Data store is in shared mode
Shared Memory KEY 0x191cadb8 ID 281160715
Subdaemon pid 27914 context 0x800000010001ba90 connected (KEY 0x191cadb8)
Process pid 1581 context 0x800000010001aba0 connected (KEY 0x191cadb8)
Process pid 25769 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 15438 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 21735 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 9661 context 0x8000000100027920 connected (KEY 0x191cadb8)
Process pid 9797 context 0x8000000100027f60 connected (KEY 0x191cadb8)
Process pid 16965 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 16966 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22023 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22128 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22317 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22489 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 18162 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22607 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22686 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22863 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 948 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 22946 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 23053 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 23104 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 1876 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 1963 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 23608 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 23888 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 23936 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24270 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24746 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24820 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24828 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24883 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 24900 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 25051 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 25050 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 7432 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 7448 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 15509 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 15771 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 11407 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 11568 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
Process pid 18106 context 0x80000001001c63a0 connected (KEY 0x191cadb8)
# Processes running now
us1oper 22480 22353 253 10:13:49 ? 11:18 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_8 /us01/datavl01/
us1oper 28635 21612 0 10:19:33 ? 0:02 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_3 /us01/datavl01/
us1oper 15970 21994 0 11:12:44 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_5 /us01/datavl01/
us1oper 21417 21246 252 10:13:16 ? 23:43 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_2 /us01/datavl01/
us1oper 19989 19592 254 10:13:06 ? 23:09 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_0 /us01/datavl01/
us1oper 107 21417 0 11:09:58 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_2 /us01/datavl01/
us1oper 22799 22668 249 10:14:00 ? 25:13 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_10 /us01/datavl01
us1oper 21994 21854 255 10:13:33 ? 24:09 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_5 /us01/datavl01/
us1oper 14822 19989 0 11:11:29 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_0 /us01/datavl01/
us1oper 22159 22035 253 10:13:39 ? 27:35 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_6 /us01/datavl01/
us1oper 15399 20881 0 11:12:01 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_1 /us01/datavl01/
us1oper 21802 21654 253 10:13:28 ? 19:50 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_4 /us01/datavl01/
us1oper 10148 22480 0 10:20:19 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_8 /us01/datavl01/
us1oper 12357 22799 0 11:10:44 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_10 /us01/datavl01
us1oper 22327 22202 255 10:13:44 ? 24:24 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_7 /us01/datavl01/
us1oper 20881 20416 252 10:13:12 ? 25:13 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_1 /us01/datavl01/
us1oper 21612 21442 253 10:13:22 ? 15:25 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_3 /us01/datavl01/
us1oper 22646 22511 253 10:13:55 ? 19:04 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_9 /us01/datavl01/
us1oper 27886 21612 0 10:18:29 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_3 /us01/datavl01/
us1oper 12492 22480 2 10:21:28 ? 0:01 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_8 /us01/datavl01/
us1oper 17857 22646 0 10:27:24 ? 0:02 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_9 /us01/datavl01/
us1oper 13697 21417 0 11:11:05 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_2 /us01/datavl01/
us1oper 15812 22159 0 11:12:34 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_6 /us01/datavl01/
us1oper 17433 22646 0 10:26:22 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_9 /us01/datavl01/
us1oper 15228 22799 0 11:11:49 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_10 /us01/datavl01
us1oper 15282 21802 0 11:11:53 ? 0:00 pm1RerateExtract INSTANCE=RERATEREX_14_3_2011_4 /us01/datavl01/
# Connections made for some of the processes above
[64] /home/12022149/temp > cat output_connections_TT_pids_22646_27886_21612_C14_1155.txt
Mar 12 10:13:44 brux0147 TimesTen Server 4.3[22317]: EventID=16| Connect succeeded from client: brux0147 (brux0147); IP address: 10.18.49.111; Client PID: 21612; DSN: Rt14_64S; UID: oper
Mar 12 10:14:16 brux0147 TimesTen Server 4.3[*23053*]: EventID=16| Connect succeeded from client: brux0147 (brux0147); IP address: 10.18.49.111; Client PID: 22646; DSN: Rt14_64S; UID: oper
Mar 12 10:15:33 brux0147 TimesTen Server 4.3[24270]: EventID=16| Connect succeeded from client: brux0147 (brux0147); IP address: 10.18.49.111; Client PID: 21612; DSN: Rt14_64S; UID: oper
Mar 12 10:16:17 brux0147 TimesTen Server 4.3[*25050*]: EventID=16| Connect succeeded from client: brux0147 (brux0147); IP address: 10.18.49.111; Client PID: 22646; DSN: Rt14_64S; UID: oper
# Partial truss output (system calls) - PID *25050*
us1oper@brux0147:us1oper> time /usr/local/bin/truss -p 25050
( Attached to process 25050 ("/opt/TimesTen4.3/64/bin/ttcserver -p 14303") [64-bit] )
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 0
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 1
recv(7, "\0\001c5170103j \t1d0f05S _ 4 1 ".., 4000, 0) = 453
time(0x800003ffe8c85438) = 1299947882
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
semop(474869790, 0x800003ffe8c88e88, 1) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
semop(474869790, 0x800003ffe8c8c4e8, 1) = 0
semop(474869790, 0x800003ffe8c8aa98, 1) = 0
semop(474869790, 0x800003ffe8c8be48, 1) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
semop(474869790, 0x800003ffe8c87908, 1) = 0
send(7, "\0\00ed3170203j \t\v\00f\005\004".., 3795, 0) = 3795
time(0x800003ffe8c85450) = 1299947882
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 1
recv(7, "\0\0\014170103k 160f05S _ 4 1 3 ".., 4000, 0) = 20
time(0x800003ffe8c85438) = 1299947882
send(7, "\0\0\0\n170203k 16\n", 10, 0) = 10
time(0x800003ffe8c85450) = 1299947882
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 1
recv(7, "\0\0019a170103l \t1d0f05S _ 4 1 ".., 4000, 0) = 410
time(0x800003ffe8c85438) = 1299947882
semop(474869790, 0x800003ffe8c87cd8, 1) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
semop(474869790, 0x800003ffe8c88e88, 1) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
semop(474869790, 0x800003ffe8c8c4e8, 1) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
sigprocmask(0x1c8cb0, NULL, 0x80000001001c8db0) = 0
send(7, "\0\0\r9d170203l \t\v\00f\005\004".., 3485, 0) = 3485
time(0x800003ffe8c85450) = 1299947882
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 1
recv(7, "\0\0\014170103m 160f05S _ 4 1 4 ".., 4000, 0) = 20
time(0x800003ffe8c85438) = 1299947882
send(7, "\0\0\0\n170203m 16\n", 10, 0) = 10
time(0x800003ffe8c85450) = 1299947882
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
( Detaching from process 25050 ("/opt/TimesTen4.3/64/bin/ttcserver -p 14303") )
real 0m5.99s
user 0m0.02s
sys 0m0.15s
# Another truss output (PID *23053*)
us1oper@brux0147:us1oper> time /usr/local/bin/truss -p 23053
( Attached to process 23053 ("/opt/TimesTen4.3/64/bin/ttcserver -p 14303") [64-bit] )
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 0
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 0
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 0
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) = 0
select(8, 0x800003ffe8c85208, NULL, NULL, 0x800003ffe8c85308) [sleeping]
( Detaching from process 23053 ("/opt/TimesTen4.3/64/bin/ttcserver -p 14303") )
real 0m19.07s
user 0m0.01s
sys 0m0.04s
I also got many other informations (ttStatus verbose output, truss from other processes, lsof, etc). I'll tar everything, make a tar.gz file and attach to the SR.
Forgot to say these processes are running the same machine where datastore resides, But we've not configured the "ttLocalHost" parameters to use unix domain sockets instead of tcp connections. May this be a problem or a bottleneck?
Thanks for your help
Edited by: pstglia on 12/03/2011 08:19
Edited by: pstglia on 12/03/2011 08:34
Edited by: pstglia on 12/03/2011 08:41

DG端不定期报waiting for async I/Os failed错误

大家好，我这边数据库主库是RAC，物理DG端也是RAC，都采用的ASM，存储，DG端最近总是时不时的报错：
Errors in file /u01/oracle/diag/rdbms/racstandby/rac1/trace/rac1_arcr_11381.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: Read Failed. group:2 disk:0 AU:247 offset:0 size:1048576
WARNING: failed to read mirror side 1 of virtual extent 19 logical extent 0 of file 277 in group [2.1197980967] from disk DATA_0000 allocation unit 247 reason error; if possible, will try another mirror side
Errors in file /u01/oracle/diag/rdbms/racstandby/rac1/trace/rac1_arcr_11381.trc:
ORA-00333: redo log read error block 38912 count 2048
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
有时就报一两次，有时一下报一天，但看DG进程还都在，数据也都实时追着，不知道有什么影响，甚至有一次MRP0自己DOWN掉不知道和这个有没有关系，实际去ASM中查看文件可以访问，检测坏块也没有，删掉重建redo仍有此问题，删掉整个库从新做主从仍然有此问题，希望大家帮看看，感激
TRACE文件内容：
Trace file /u01/oracle/diag/rdbms/racstandby/rac1/trace/rac1_arcr_11381.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/oracle/11.2.0.3/product
System name: Linux
Node name: racstandby1
Release: 2.6.18-274.el5
Version: #1 SMP Fri Jul 22 04:43:29 EDT 2011
Machine: x86_64
Instance name: rac1
Redo thread mounted by this instance: 1
Oracle process number: 62
Unix process pid: 11381, image: oracle@racstandby1 (ARCR)
*** 2012-10-24 22:00:56.861
*** SESSION ID:(1714.1) 2012-10-24 22:00:56.861
*** CLIENT ID:() 2012-10-24 22:00:56.861
*** SERVICE NAME:(SYS$BACKGROUND) 2012-10-24 22:00:56.861
*** MODULE NAME:() 2012-10-24 22:00:56.861
*** ACTION NAME:() 2012-10-24 22:00:56.861
Initial buffer sizes: read 1024K, overflow 832K, change 805K
ORA-27061: waiting for async I/Os failed
*** 2012-10-25 10:04:19.253
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: Read Failed. group:2 disk:0 AU:247 offset:0 size:1048576
path:/dev/oracleasm/disks/VOL_DATA1
incarnation:0xefd74ddb asynchronous result:'I/O error'
subsys:System iop:0x2b3b7a33e780 bufp:0x2b3b7b42d000 osderr:0x0 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 19 logical extent 0 of file 277 in group [2.1197980967] from disk DATA_0000 allocation unit 247 reason error; if possible, will try another mirror side
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00333: redo log read error block 38912 count 2048
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 505856
Additional information: -1
WARNING: Read Failed. group:2 disk:0 AU:247 offset:0 size:1048576
path:/dev/oracleasm/disks/VOL_DATA1
incarnation:0xefd74ddb synchronous result:'I/O error'
subsys:System iop:0x2b3b7a33e780 bufp:0x2b3b7b42d000 osderr:0x0 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 19 logical extent 0 of file 277 in group [2.1197980967] from disk DATA_0000 allocation unit 247 reason error; if possible, will try another mirror side
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-15081: failed to submit an I/O operation to a disk
ORA-00333: redo log read error block 38912 count 2048
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-15081: failed to submit an I/O operation to a disk
Incomplete read from log member '+DATA/racstandby/onlinelog/group_7.277.797513635'. Trying next member.
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 251904
Additional information: -1
WARNING: Read Failed. group:3 disk:0 AU:123 offset:0 size:1048576
path:/dev/oracleasm/disks/VOL_FRA1
incarnation:0xefd74ddd synchronous result:'I/O error'
subsys:System iop:0x2b3b7a33e780 bufp:0x2b3b7b42d000 osderr:0x0 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 19 logical extent 0 of file 325 in group [3.1197980968] from disk FRA_0000 allocation unit 123 reason error; if possible, will try another mirror side
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 7 thread 1: '+FRA/racstandby/onlinelog/group_7.325.797513637'
ORA-15081: failed to submit an I/O operation to a disk
ORA-00333: redo log read error block 38912 count 2048
ORA-00312: online log 7 thread 1: '+FRA/racstandby/onlinelog/group_7.325.797513637'
ORA-15081: failed to submit an I/O operation to a disk
Incomplete read from log member '+FRA/racstandby/onlinelog/group_7.325.797513637'. Trying next member.
*** 2012-10-25 10:04:19.372 4320 krsh.c
ARCr: All Archive destinations made inactive due to error 333
*** 2012-10-25 10:04:19.398 2932 krsi.c
krsi_dst_fail: dest:1 err:333 force:0 blast:1
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: Read Failed. group:2 disk:0 AU:248 offset:0 size:1048576
path:/dev/oracleasm/disks/VOL_DATA1
incarnation:0xefd74ddb asynchronous result:'I/O error'
subsys:System iop:0x2b3b7a33e500 bufp:0x2b3b7b53d000 osderr:0x0 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 20 logical extent 0 of file 277 in group [2.1197980967] from disk DATA_0000 allocation unit 248 reason error; if possible, will try another mirror side
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 7 thread 1: '+FRA/racstandby/onlinelog/group_7.325.797513637'
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 7 thread 1: '+DATA/racstandby/onlinelog/group_7.277.797513635'
ORA-00312: online log 7 thread 1: '+FRA/racstandby/onlinelog/group_7.325.797513637'
*** 2012-10-25 10:04:19.546 4481 kcrr.c

ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 505856
15081, 00000, "failed to submit an I/O operation to a disk"
// *Cause: A submission of an I/O operation to a disk has failed.
// *Action: Make sure that all the disks are operational.
ORA-15081一般与存储磁盘有关，请把/var/log/messages、dmesg日志发给我
在11gR2中ORA-15081 可能导致instance crash
有问题请去OTN中文论坛开个帖子我会回复地址：http://www.otncn.org
如果需要发送附件，可以直接发邮件到 [email protected]

Timed out waiting for the CRS stack to start on Second node.

Hi,
I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
/u01/app/11.2.0/grid/log/node1/alertnode2.log
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:10.314
[crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:11.217
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:12.422
[crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:13.327
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:14.514
[crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:15.439
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:15.440
[ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
root.sh output
[root@node2 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2014-01-21 21:37:55: Parsing the host name
2014-01-21 21:37:55: Checking for super user privileges
2014-01-21 21:37:55: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p
arams
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node2'
CRS-2676: Start of 'ora.asm' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node2'
CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node2'
CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
Timed out waiting for the CRS stack to start.
Regards,
007

Hi,
Are you trying to install on Vmware Workstation or Vmware Server?
If you are using vmware server add below line in your .vmx file for both node
scsi1.sharedBus = "VIRTUAL"
First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition,

"Waiting for output formatter"

Similar Messages

Maybe you are looking for