Most efficient way to constantly read, queue and parse multi-size​d RS232 data (multi-thr​eaded)

Question

I've tried tackling this problem a few different ways, and figured it was time to get some others' advice.  My system essentially works, although it looks like a hackjob and not entirely confident.
My RS-232 connection has the following properties/constraints:
-Will be getting unsolicited data at a high data rate.  (ya thats subjective, but assume near constant at whatever baud rate its set at up to 115200)
-Different segment lengths of data receiving
-Two stop bytes (0x10 0x03), while start byte is 0x10.  (Byte stuffing/packing implemented)
-There is a size byte within a packet (3rd one in), however currently relying on stop bytes only.
I have tried the ComCallback within CVI only to find that it is VERY slow at processing events compared to implementing it manually in its own thread.  In addition, it can only trigger 1 stop bit, not 2.  Triggering on size is sometimes okay, but I found that it was possible it would get triggered on only part of the data, the when its called the qlength be larger, and then sometimes I would only read part of a data packet, and half of my segment was still in the queue.  And sometimes, I would get semaphore locks and lots of waiting, and ya it was a mess (hence you will see lots of CMT locks commented out)
I tried implementing a FIFO type queue (copied below), but I have very little experience in doing this, and may not be very efficient in the way I implemented and could definately use some advice in this area.  Also perhaps in how thread safe I have everything.
I thought about a circular buffer, but since data can be different segment lengths, it kind of makes it difficult to cleanly wrap around and read.  I think its still possible, just may require additional checks which I haven't seen done anywhere when google searching. (and made me think a FIFO queue was better).
So if anyone has any good suggestions or examples, that would be great.  Using Labwindows 2010.
//in Main before GUI loaded
programRunning = 1;
CmtScheduleThreadPoolFunction (DEFAULT_THREAD_POOL_HANDLE, ComCallback, NULL, &funcID);
/* Function Used to parse Com data */
static int ComCallback (void *callbackData)
static int bufLen = 0;
int strLen,qLen=0;
unsigned char tempBuf[1024];
int packet_length = 0;
char temp_string[250];
unsigned char *start_ptr;
int i;
int start_offset;
while(programRunning) {
//First lets copy all data from com port to output buffer
if( com_open) qLen = GetInQLen (comPort);
if( qLen <= 0 ) {
ProcessSystemEvents();
} else {
if( qLen > 1024) qLen = 1024-bufLen; //set max length to read
strLen = ComRd (comPort, tempBuf, qLen);
//CmtGetLock(lock);
memcpy(readBuf+bufLen,tempBuf,strLen);
bufLen += strLen;
//CmtReleaseLock(lock);
//Try to read until we hit stop bytes, or until we think we have at least 1 or 2 packets to process
if(tempBuf[strLen-2] != 0x10 && tempBuf[strLen-1] != 0x03 && bufLen < 100 ) )
goto skip;
//ensure start pointer is at beginning command (0x10 0xAA) (in event we just started reading in middle of packet???)
i = 0;
while ( (readBuf[i] != 0x10) && (readBuf[i+1] != 0xAA) && (i < strLen) )
//start_ptr++;
i++;
start_offset = i;
parse_some_more:
start_ptr = readBuf + start_offset;
//lets try to do one packet at a time.
for(i = start_offset; i < bufLen-start_offset-1; i++)
if(readBuf[i] == 0x10)
if(readBuf[i+1] == 0x03)
i += 2;
break;
//trial of unpacking/unstuffing byte buffer
else if( readBuf[i+1] == 0x10 ) //two tens in a row, remove it
memmove( &readBuf[i],&readBuf[i+1],bufLen-i);
bufLen--;
//at this point, we should have a full packet. What if we don't....???
packet_length = i - start_offset;
//CmtGetLock(lock);
ParseResponse(start_ptr,packet_length);
PostPacketToOutputBuffer(start_ptr,packet_length);
if(start_ptr[start_offset] == 0x10 && start_ptr[start_offset+1] == 0xAA )
com_Send_Acknowledge(comPort);
start_offset = packet_length + start_offset;
if(start_offset < bufLen-1)
goto parse_some_more;
//For now, assume everything else we don't care about in buffer, however, probably shouldn't. Should it be bufLen -=start_offset; if so, need to handle partial data better; timeout??
bufLen = 0;
//CmtReleaseLock(lock);
skip:
return 0;
Thanks!

Answer 1

Hi ngay528,
I think there is a great example for you to use that comes with CVI which can be found by clicking Find Examples on the splash screen or Help >> Find Examples in your project. From there, click into Optimizing Applications >> Multithreading. In that folder there is a project that is called BuffNoDataLoss that shows how to create a thread safe queue and setup a producer/consumer type program. In this example the data is a random sine wave but could be adapted to your RS232 data. If you have any questions concerning this example please let me know but this should be a great starting point.
Patrick H | National Instruments | Software Engineer