Problem of SIGPOLL(SI_NOINFO) in latest Solaris9 kernel patch

Question

Hi,
We are facing a rather strange problem with the latest kernel patch on Solaris 9. (Generic_112233-08). We had not faced this problem with any of the other kernel patches of Solaris 9.
Our application has a main thread and a single child thread (pthread). The main thread schedules aio_writes() on the raw disk interface and lets the child thread block on sigwaitinfo() to listen to the signal completion notification. This is communicated to it via the SI_ASYNCIO code of SIGPOLL. The child thread then informs the main thread by writing to a bi-directional pipe. Since the main thread has registered for read interest on the bi-directional pipe (via /dev/poll) it is informed of the completion of the aio_write() without having to block itself. Under normal circumstances, the child thread receives SIGPOLL with SI_ASYNCIO code.
This application has been running fine on all the previous builds of Solaris (Generic, Generic_112233-04, Generic_112233-06) on sparc platform expect with the latest kernel patch. The child thread now keeps receiving SIGPOLL with SI_NOINFO code. There has been no change in our application and we are perplexed to the reason of this behaviour. Since it is SI_NOINFO there is not much debugging information we can get.
We have been able to replicate this behaviour using a small stand-alone program. We are attaching it at the end of the email. We tried this program on a couple of different Sparc systems and were able to reproduce this behaviour on one of them but not on the other.
Has anybody faced problems with regard to SIGPOLL in the latest kernel patch of Solaris 9 for sparc systems ?
Thanks
Regards
Raj Pagaku
proxy-24:~ >uname -a
SunOS proxy-24 5.9 Generic_112233-08 sun4u sparc SUNW,Ultra-5_10
proxy-24:~ >gcc -v
Reading specs from /usr/local/lib/gcc-lib/sparc-sun-solaris2.9/3.2/specs
Configured with: ../configure with-as=/usr/ccs/bin/as with-ld=/usr/ccs/bin/ld --disable-nls
Thread model: posix
gcc version 3.2
Compiled this program using the following command : gcc -g kernel_bug.c -lrt -lpthread
#include <stdio.h>
#include <aio.h>
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <signal.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <sys/types.h>
#define min(x,y) (((x)<=(y))?(x):(y))
#define DISPLAY_COUNT 10000
typedef struct DiskInfoCallOut {
void (*func_ptr)(void *);
void *data_ptr;
} DiskInfoCallOut;
typedef struct DiskInfo {
struct aiocb di_aiocb;
DiskInfoCallOut di_callout;
off_t di_currOffset;
int di_scheduled;
} DiskInfo;
typedef struct Disk {
int fd;
char *buffer;
int bufferLen;
} Disk;
static sigset_t aioSignalSet;
int aioSigFD[2];
int glob_scheduled = 1;
int glob_respond = 1;
Disk disk;
static void LaunchDiskOperation(DiskInfo *di);
char BUFDATA[4096] = {'a'};
char rawDeviceName[256] = "/dev/rdsk/";
static void
InitializeDisk()
int fd;
if ((fd = open(rawDeviceName, O_RDWR, 0)) == -1) {
fprintf(stderr, "Unable to open raw device \n");
exit(-1);
disk.fd = fd;
disk.buffer = BUFDATA;
disk.bufferLen = sizeof(BUFDATA);
static void
AIOSignalHandler(int sigNum, siginfo_t* si, void* context)
fprintf(stderr, "WARN: got signal %d in AIOSignalHandler!\n", sigNum);
/* Function implementing the slave thread */
static void*
AIOSignalThread(void *arg)
struct sigaction sa;
siginfo_t info;
sigset_t ss;
int sig_num;
int retVal;
/* Initialize the signal set*/
sigemptyset(&ss);
sigaddset(&ss, SIGPOLL);
if ((retVal = pthread_sigmask(SIG_SETMASK, &ss, NULL))) {
fprintf(stderr, "pthread_sigmask failed in AIOSignalThread \n");
exit(-1);
sa.sa_handler = NULL;
sa.sa_sigaction = AIOSignalHandler;
sa.sa_mask = aioSignalSet;
sa.sa_flags = SA_SIGINFO;
if (sigaction(SIGPOLL, &sa, NULL)) {
fprintf(stderr, "sigaction in AIOSignalThread \n");
exit(-1);
/* Wait infinitely for the signals and respond to the main thread */
while (1) {
sig_num = sigwaitinfo(&aioSignalSet, &info);
if (sig_num != SIGPOLL) {
fprintf(stderr, "caught unexpected signal %d in AIOSignalThread \n",
sig_num);
exit(-1);
if (info.si_code != SI_ASYNCIO){
fprintf(stderr, "ERROR: siginfo_t had si_code != SI_ASYNCIO, si_code = %d \n", info.si_code);
continue;
/* Write the stored pointer value in the pipe so that the main thread can process it */
if (write(aioSigFD[1], &(info.si_value.sival_ptr), sizeof(info.si_value.sival_ptr)) !=
sizeof(info.si_value.sival_ptr)) {
perror("Couldn't write the whole pointer");
exit(-1);
return (NULL);
static void
Init()
pthread_attr_t aioAttr;
pthread_t aioThread;
int retVal = 0;
/* Create a bidirectional pipe */
if (pipe(aioSigFD)) {
perror("pipe failed");
exit(-1);
/* Initialize to prevent other threads from being interrupted by
SIGPOLL */
sigemptyset(&aioSignalSet);
sigaddset(&aioSignalSet, SIGPOLL);
if ((retVal = pthread_sigmask(SIG_BLOCK, &aioSignalSet, NULL))) {
fprintf(stderr, "pthread_sigmask failed in Init\n");
exit(-1);
InitializeDisk();
if ((retVal = pthread_attr_init(&aioAttr)))
fprintf(stderr, "pthread_attr_init failed \n");
if ((retVal = pthread_attr_setdetachstate(&aioAttr, PTHREAD_CREATE_DETACHED)))
fprintf(stderr, "pthread_attr_setdetachstate failed \n");
if ((retVal = pthread_attr_setscope(&aioAttr, PTHREAD_SCOPE_SYSTEM)))
fprintf(stderr, "pthread_attr_setscope failed in \n");
if ((retVal = pthread_attr_setstacksize(&aioAttr, 2*1024*1024)))
fprintf(stderr, "pthread_attr_setstacksize failed \n");
if ((retVal = pthread_create(&aioThread, &aioAttr,
AIOSignalThread, NULL)))
fprintf(stderr, "pthread_create failed \n");
static void
UpdateDiskWriteInformation(DiskInfo *di)
di->di_currOffset += disk.bufferLen;
di->di_scheduled = 0;
static void
DiskOpCompleted(void *ptr)
DiskInfo di = (DiskInfo )ptr;
if (aio_error(&di->di_aiocb))
perror("aio_error");
if (aio_return(&di->di_aiocb) < 0)
perror("aio_return ");
UpdateDiskWriteInformation(di);
glob_respond++;
static void
LaunchDiskOperation(DiskInfo *di)
int res;
di->di_callout.func_ptr = DiskOpCompleted;
di->di_callout.data_ptr = di;
memset(&di->di_aiocb, 0, sizeof(di->di_aiocb));
di->di_aiocb.aio_fildes = disk.fd;
di->di_aiocb.aio_buf = disk.buffer;
di->di_aiocb.aio_nbytes = disk.bufferLen;
di->di_aiocb.aio_offset = di->di_currOffset;
di->di_scheduled = 1;
di->di_aiocb.aio_sigevent.sigev_notify = SIGEV_SIGNAL;
di->di_aiocb.aio_sigevent.sigev_signo = SIGPOLL;
di->di_aiocb.aio_sigevent.sigev_value.sival_ptr = &di->di_callout;
res = aio_write(&di->di_aiocb);
if (res == -1) {
perror("aio op error");
static void
HandleSignalResponses()
int fd;
#define DISKINFO_CALLOUT_MAX 64
DiskInfoCallOut* callout[DISKINFO_CALLOUT_MAX];
struct stat pipeStat;
int numCompleted;
int bytesToRead;
int sz;
int i;
fd = aioSigFD[0];
while (1) {
/* Find whether there is any data in the pipe */
if(-1 == fstat(fd, &pipeStat)) {
perror("fstat");
exit(-1);
if (pipeStat.st_size < sizeof(DiskInfoCallOut *))
break;
numCompleted = min((pipeStat.st_size/sizeof(DiskInfoCallOut *)),DISKINFO_CALLOUT_MAX);
bytesToRead = numCompleted * sizeof(DiskInfoCallOut *);
if ((sz = read(fd, callout, bytesToRead)) != bytesToRead) {
perror("Error reading from pipe");
exit(-1);
for (i = 0; i < numCompleted; i++)
(*callout[i]->func_ptr)(callout[i]->data_ptr);
int main(int argc, char *argv[])
DiskInfo *di;
FILE *logPtr1 = NULL;
FILE *logPtr2 = NULL;
FILE *logPtr3 = NULL;
struct rusage ru;
struct timeval t1, t2;
long timeTaken = 0;
int writeCount = 0;
int i;
char logFileName1[1024];
char logFileName2[1024];
char logFileName3[1024];
if (argc < 2) {
fprintf(stderr, "Usage : %s <partition_name> \n", argv[0]);
exit(-1);
strcat(rawDeviceName, argv[1]);
writeCount = 1;
printf("Partition selected = %s \n", rawDeviceName);
di = calloc(writeCount, sizeof(DiskInfo));
sprintf(logFileName1, "%s.log1", argv[0]);
if ((logPtr1 = fopen(logFileName1, "w+")) == NULL) {
fprintf(stderr, "Unable to create file test_pgm \n");
exit(-1);
sprintf(logFileName2, "%s.log2", argv[0]);
if ((logPtr2 = fopen(logFileName2, "w+")) == NULL) {
fprintf(stderr, "Unable to create file test_pgm \n");
exit(-1);
sprintf(logFileName3, "%s.log3", argv[0]);
if ((logPtr3 = fopen(logFileName3, "w+")) == NULL) {
fprintf(stderr, "Unable to create file test_pgm \n");
exit(-1);
Init();
for (i = 0; i < writeCount; i++) {
di.di_currOffset = (1 << 18) * (i + 1);
di[i].di_scheduled = 0;
gettimeofday(&t1, NULL);
while (1) {
int curScheduled = 0;
/* Schedule the disk operations */
for (i = 0; i < writeCount; i++) {
if (di[i].di_scheduled == 0) {
LaunchDiskOperation(&di[i]);
glob_scheduled++;
curScheduled++;
/* Handle the responses */
HandleSignalResponses();
if ((curScheduled) && (glob_respond % DISPLAY_COUNT == 0)) {
gettimeofday(&t2, NULL);
timeTaken = ((t2.tv_sec * 1000000 + t2.tv_usec) -
(t1.tv_sec * 1000000 + t1.tv_usec))/1000;
printf("Scheduled = %d, Responded = %d, Time Taken = %ld ms \n",
glob_scheduled, glob_respond, timeTaken);
fprintf(logPtr1, "Scheduled = %d, Responded = %d, Time Taken = %ld ms \n",
glob_scheduled, glob_respond, timeTaken);
fprintf(stderr,"wrote to logPtr1 ..\n");
fprintf(logPtr2, "Scheduled = %d, Responded = %d, Time Taken = %ld ms \n",
glob_scheduled, glob_respond, timeTaken);
fprintf(stderr,"wrote to logPtr2 ..\n");
fprintf(logPtr3, "Scheduled = %d, Responded = %d, Time Taken = %ld ms \n",
glob_scheduled, glob_respond, timeTaken);
fprintf(stderr,"wrote to logPtr3 ..\n");
t1 = t2;

Answer 1

Hi @cooldog ,
I hit this same LVM2 snapshot kernel oops on several Oracle Linux 6.5 servers running UEK R3 kernel version 3.8.13-16.3.1.  I have Linux Premier Support so I opened a Service Request.  Oracle Support got back to me with the following notes.
Hello Matt, 
Bug 17487738 : EXT4: STRESS TESTING WITH SUSPEND/RESUME FS ACCESS CAUSES FS ERRORS This bug is fixed in kernel version: 3.8.13-18. This kernel will be available quite soon for download.
You may upgrade the kernel once its available. ~Siju 
Update
Dear Matt, Latest available UEK3 kernel version 'kernel-uek-3.8.13-26.el6uek.x86_64' incorporates the required bugfix. [root@server1 tmp]# rpm -q --changelog -p kernel-uek-3.8.13-26.el6uek.x86_64.rpm | grep -i 17487738
warning: kernel-uek-3.8.13-26.el6uek.x86_64.rpm: Header V3 RSA/SHA256 signature: NOKEY, key ID ec551f03
- fs: protect write with sb_start/end_write in generic_file_write_iter (Guangyu Sun) [Orabug: 17487738] <<<<<<======================================== You can download the UEK3 kernel from ULN or from public-yum repo. 
http://public-yum.oracle.com/repo/OracleLinux/OL6/UEKR3/latest/x86_64/getPackage/kernel-uek-firmware-3.8.13-26.el6uek.noarch.rpm
http://public-yum.oracle.com/repo/OracleLinux/OL6/UEKR3/latest/x86_64/getPackage/kernel-uek-3.8.13-26.el6uek.x86_64.rpm Hope this helps! ~Siju 
Subscribe to the Oracle Linux el-errata mailing list .
The latest kernel-uek-3.8.13-26.el6uek.x86_64 version fixed the problem.
- Matt