Handle rounding error

Question

hello,
I understand that rounding errors exist and I need help determining the best way to handle them.  The particular situation I am dealing with relates to finding the equation of a line and comparing the calculated values to a data set.  
The big picture:  I have a data set that looks generally exponential with sharp peaks along the curve.  The exponential part is background noise that I want to eliminate. After trying various methods, I've determined that getting a convex hull is the best method of eliminating the background noise.  The convex hull is found by finding a line through point A and point B of the data and determining if all of the data is above (greater than) the line (similar to a tangent line).  Once a point B is found which satisfies the condition, point B becomes the new point A and a new point B is looked for.  
I compare the data set to the points that create the line.  The X values of the data set and the line are the same integer values.  The Y values of the data set are DBL.  I have attached a VI demonstrating the simple calculations I am doing to find the Y values along the line.  It is simply the (y1-y0)/(x1-x0) to get the slope and using the slope to find the y-intercept (b = y0-m*x0).  I then use the slope and y-intercept to calculate the Y values along the line.  I subtract the line Y values from the data set Y values and check if they are all positive.  If they are, then I can conclude that all the points are above the line and the point B (x1,y1) is kept as part of the convex hull.  
The problem is that rounding error is causing "false negatives."  In other words, the rounding error is causing the actual and calculated (through equation of line) y1 to be different.  Since (x1, y1) is a data set point that defined the line it should be equal to the calculated (x1, y1), but the rounding error is not allowing that.  
Currently, I am adding a very small (1.5E-12) constant to the data set point to account for the error and prevent false negatives (which would be the existence of a negative difference, when it should be 0).  However, this small number is not always sufficient.
How can I find the maximum error, so that I can add the error to the data set point and therefore always prevent false negatives?  
Thanks!
Kristen
Attachments:
Rounding Error Example.vi ‏13 KB

Answer 1

Hi Lynn,
Thanks for your help.
I have attached a text file with example data set.  The range is 0-6000  (look around x=187 and x=300 for max).   There is a middle region (220- 260ish) that is irrelevant, otherwise, there are eight peaks in total, four peaks on each side, which are symmetric from the center.  I am doing the convex hull on each half of the data.   The data set actually begins as an x-ray diffraction image (irrelevant region is the backstop preventing the x-ray from saturating the detector so only the diffraction is captured) of which I take an ROI and sum down the columns.  The result is this data which is pixel location and summed pixel values.  (Yes, I know the x values in the text file are not integers- this is because some of the end points and the irrelevant area is extracted out and the ramp function is used to create new x values even spaced between the new beginning and end of the data set.  This part of the code is inherited and I need to check with the programmer why this method was used) 
Previously, I tried doing a curve fit of the whole data set (background not subtracted) with the background described as h0+h1*exp(-h2*(x-r))+h3*exp(-h4*(x-r)) and the peaks described as a convolution of a gaussian and lorentzian peak: abs(a0)*((1/(1+f^2))*exp(-((x-(r+d0))/w)^2)+(1-(1/(1+f^2)))/(1+((x-(r+d0))/w)^2))
where
r is the theoretical median of the data (point about which the image is horizontally symmetric)
a0 is the amplitude of the peak
d0 is the distance from symmetric center to the center of the peak
w is the full width half max of the peak
f is a factor to determine if the peak is more Gaussian or Lorentzian 
I added the background model with the peak models (subtract d0 from r for the symmetrically equivalent peak) and used the Non-linear Curve Fit VI to fit the equation.  The results were decent, but the convex hull -> subtract background -> peak fit method seemed to be fitting the peaks better (correlation .999 vs .99), which is the important part.
Thoughts?
Kristen 
Attachments:
ConvexHullTest Example Data.txt ‏7 KB