Rounding Error
I have 501.30 typed into a cell (E74). In the next cell (F74) I have the formula: =ROUND(E74*.15, 2)
Why am I getting 75.19 instead of 75.20? If you use a calcutator or show the above formula without the ROUND function you get 75.195. It's not rounding this number up.
This comes up from time to time. It has to do with how decimal numbers are represented in binary digital computers. There is an IEEE standard that Apple follows strictly in Numbers, versus doing some "massaging" of the results to better meet customer expectations. I believe it is the strict adherence to the standard that is the problem you see. Decimal numbers often cannot be represented exactly in binary and it results in small errors. As a test, do the following
B2 = .1
B3= -.3
B4= .2
B5 = SUM(B2:B4)
The result will not be zero
The answer may also change depending on the order of operations
501.3*0.15 -75 - 0.195 is equal to -6.8E-15
501.3*0.15 - 0.195 - 75 is equal to 0
Whether Apple is right or wrong in sticking strictly to the standard is debatable. There are cases in Excel where the "massaging" creates a false zero when the true result is supposed to be a very small number. My personal preference is for the numbers to be massaged to meet my expectations but that's not how it is in Numbers.
Similar Messages
-
hello,
I understand that rounding errors exist and I need help determining the best way to handle them. The particular situation I am dealing with relates to finding the equation of a line and comparing the calculated values to a data set.
The big picture: I have a data set that looks generally exponential with sharp peaks along the curve. The exponential part is background noise that I want to eliminate. After trying various methods, I've determined that getting a convex hull is the best method of eliminating the background noise. The convex hull is found by finding a line through point A and point B of the data and determining if all of the data is above (greater than) the line (similar to a tangent line). Once a point B is found which satisfies the condition, point B becomes the new point A and a new point B is looked for.
I compare the data set to the points that create the line. The X values of the data set and the line are the same integer values. The Y values of the data set are DBL. I have attached a VI demonstrating the simple calculations I am doing to find the Y values along the line. It is simply the (y1-y0)/(x1-x0) to get the slope and using the slope to find the y-intercept (b = y0-m*x0). I then use the slope and y-intercept to calculate the Y values along the line. I subtract the line Y values from the data set Y values and check if they are all positive. If they are, then I can conclude that all the points are above the line and the point B (x1,y1) is kept as part of the convex hull.
The problem is that rounding error is causing "false negatives." In other words, the rounding error is causing the actual and calculated (through equation of line) y1 to be different. Since (x1, y1) is a data set point that defined the line it should be equal to the calculated (x1, y1), but the rounding error is not allowing that.
Currently, I am adding a very small (1.5E-12) constant to the data set point to account for the error and prevent false negatives (which would be the existence of a negative difference, when it should be 0). However, this small number is not always sufficient.
How can I find the maximum error, so that I can add the error to the data set point and therefore always prevent false negatives?
Thanks!
Kristen
Attachments:
Rounding Error Example.vi 13 KBHi Lynn,
Thanks for your help.
I have attached a text file with example data set. The range is 0-6000 (look around x=187 and x=300 for max). There is a middle region (220- 260ish) that is irrelevant, otherwise, there are eight peaks in total, four peaks on each side, which are symmetric from the center. I am doing the convex hull on each half of the data. The data set actually begins as an x-ray diffraction image (irrelevant region is the backstop preventing the x-ray from saturating the detector so only the diffraction is captured) of which I take an ROI and sum down the columns. The result is this data which is pixel location and summed pixel values. (Yes, I know the x values in the text file are not integers- this is because some of the end points and the irrelevant area is extracted out and the ramp function is used to create new x values even spaced between the new beginning and end of the data set. This part of the code is inherited and I need to check with the programmer why this method was used)
Previously, I tried doing a curve fit of the whole data set (background not subtracted) with the background described as h0+h1*exp(-h2*(x-r))+h3*exp(-h4*(x-r)) and the peaks described as a convolution of a gaussian and lorentzian peak: abs(a0)*((1/(1+f^2))*exp(-((x-(r+d0))/w)^2)+(1-(1/(1+f^2)))/(1+((x-(r+d0))/w)^2))
where
r is the theoretical median of the data (point about which the image is horizontally symmetric)
a0 is the amplitude of the peak
d0 is the distance from symmetric center to the center of the peak
w is the full width half max of the peak
f is a factor to determine if the peak is more Gaussian or Lorentzian
I added the background model with the peak models (subtract d0 from r for the symmetrically equivalent peak) and used the Non-linear Curve Fit VI to fit the equation. The results were decent, but the convex hull -> subtract background -> peak fit method seemed to be fitting the peaks better (correlation .999 vs .99), which is the important part.
Thoughts?
Kristen
Attachments:
ConvexHullTest Example Data.txt 7 KB -
Rounding error in BAPI_PO_CREATE1 Function module
Hi experts ,
In my project there are two steps behind PO creation 1-> Authorization of po creation And 2-> PO creation.
In authorization a logic in smart form calculates the amount for which the po is to be created and then goes go 2 nd step.
before creating po gross discount percentage is calculated the same way calculated in step1.
But at the end total amount in smartform and total amount of po created is differing by max of .01 to 0.1.
Unfortunetely the Z logic in the smartforms is caculation the creec value but standard BAPI BAPI_PO_CREATE1 is creating some rounding errors .
Please suggest some thing for this problem.
Thanks in advance
ShoaibmemonHi Md shoaib Memon,
there is another thread with the same issue.
http://scn.sap.com/thread/922418
Solution:
Set the following flag
poitem-no_rounding = 'X'.
poitemx-no_rounding = 'X'.
Best regards,
Ademir -
I've been working on an egyptian fraction program and for some reason I cant seem to figure out a way to fix this rounding error. An egyptian fraction is a fraction that can be expressed as a sum of fractions eg. 3/4 = 1/2 + 1/4. For some reason on certain fractions it rounds up and skips the correct fraction to subtract. When I run 2/7 its supposed to equal 1/4 + 1/28, but gives me 1/4 + 1/29 and decides to round. This is my code for the problem.
public class EgyptianFraction{
private static double epsilon = 1.0e-7;
public static void main(String args[]){
greedySearch(2.0/7.0);
public static void greedySearch(double fraction){
for(int i = 2; fraction > epsilon; i++){
if(fraction - (1.0/i) >= 0){
fraction -= (1.0/i);
//*****Output******
//0.0357142857142857 - 0.03571428571428571 = -1.3877787807814457E-17
}When I print out all of the math involved it says that it gives the output above. They are fairly close but for some reason it makes the 1/28 bigger then the current fraction. The program should subtract 1/28 and then the fraction should be close enough to 0 and end. Is there any way you guys can think of to fix this problem?You have given the error as epsilon, so you can't expect `fraction - 1.0/i >= 0` to give an exact answer.
Given your error is +/- epsilon, this expression should be
fraction - 1.0/i >= epsilon || fraction - 1.0/i >= -epsilon
or just
fraction - 1.0/i >= -epsilon
If you change this you get 1/4 + 1/28 as the answer, -
Tables and Rounding Errors on Board Game Gui
Hello,
So I am in a software development class , and my team and I are developing a software version of a board game that uses numbered tiles placed on a board in a somewhat scrabble-esque way.
Here is a picture of the game board:
[http://img90.imageshack.us/img90/1052/untitledqv3.png|http://img90.imageshack.us/img90/1052/untitledqv3.png]
Currently, a problem that we are working on is that as the tiles get further and further away from the center of the board, they are displayed further and further askew from the board lines. I have another picture to demonstrate what I'm talking about more clearly.
[http://img225.imageshack.us/img225/4605/untitled2nn0.png|http://img225.imageshack.us/img225/4605/untitled2nn0.png]
As the tiles get further away from the center, they are displayed more askew.
We think that this happens because we are using a gridbag layout to add tile objects to, which displays the tiles in a certain spacing and orientation, and then we draw the board ourselves. When we draw the board, we get rounding errors inherent in the use of ints, and the lines become a bit off, with the problem getting worse as it gets further and further away from the center.
Here is the code that we use to initialize the layout and add the tiles to it:
//set the layout
setLayout(new GridLayout(7, 7, 7, 7));
//initialize the array with the references to all the tiles that we are going to add
//to the layout
squares = new SquareView[7][7];
for (int i = 0; i < 7; i++) {
for (int j = 0; j < 7; j++) {
//create the tile, put a reference to it in the array
squares[i][j] = new SquareView(boardModel.getSquare(new Point(j, i)), listener, handler);
//add the tile to the layout
add(squares[i][j]);
}And here is the code that we are using to draw the lines for the board:
GridLayout layout = (GridLayout) getLayout();
//getting the dimensions of the board
int rows = layout.getRows();
int columns = layout.getColumns();
int width = (getWidth() / columns);
int height = (getHeight() / rows);
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, i * width, rows * height);
// Horizontal lines
g2.drawLine(0, i * height, columns * width, i * height);
}I think that our problems come from the innacuracy of ints, but if there is some addition or subtraction trick that we could pull, then let me know.
Also, I was sort of looking into tables, and I was wondering if maybe we could use tables to do this, and have Java do our resizing for us.j.law wrote:
We think that this happens because we are using a gridbag layout to add tile objects to, From the snippets of code, it's looking as though you're using GridLayout. But that's OK as GridLayout should work fine here.
GridLayout layout = (GridLayout) getLayout();
//getting the dimensions of the board
int rows = layout.getRows();
int columns = layout.getColumns();
int width = (getWidth() / columns);
int height = (getHeight() / rows);
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, i * width, rows * height);
// Horizontal lines
g2.drawLine(0, i * height, columns * width, i * height);
}I have no idea of this will improve things, but what about:
GridLayout layout = (GridLayout) getLayout();
int rows = layout.getRows();
int columns = layout.getColumns();
double width = (getWidth() / columns); //** how about using doubles here
double height = (getHeight() / rows); //** how about using doubles here
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, (int)(i * width), (int)(rows * height)); // and casting?
// Horizontal lines
g2.drawLine(0, (int)(i * height), (int)(columns * width), (int)(i * height));
} -
Hi everybody,
how should I/can I handle rounding errors in Oracle.
I'm using the LOG function of Oracle and for SELECT LOG(2,16) FROM DUAL; Oracle returns 3,99999999999999999999999999999999999998 which is incorrect. I cannot use the Round-Function cause my whole mathematical expression (written in Excel Syntax) is:
FLOOR(LOG(x; 2); 1)
So if I use the round function and SELECT ROUND(LOG(2,15)) FROM DUAL; would return 4 which is not correct thinking about my FLOOR(4) => 4, but the result of the mathematical expression is 3!
Any ideas on how to deal with such problems... is there somehow a datatype to use?
Thanks!user10835064 wrote:
Thanks Max - I guess you just wanted to illustrated the problem? cause I don't see a solution there...
Thanks Ravi - but 7 digits... can someone assure that there is no number in the world where LOG(2, x) wont return something like y.999999987389... I mean LOG(2, 15.9999990018681) is 3,99999991 so even a rounding of 7 is not enough :)...
Maybe another question should be asked :)... can you count on this rounding error? I mean if so, than I could add 1 => FLOOR(LOG(2, x + 1)), but if I cannot count on this rounding error, than there could be wrong results if e.g. Oracle returns the correct value!I just showed the parameter usage of round, which you can use for rounding the digits.
While comparision like round(somevalue)>=2 as you said it is not required to round the digit, you can directly do the
somevalue>=2
But when you want to show the o/p to an end user who doesn't need 30+digits of precision, for him you can show using the above round function.
Ravi Kumar -
Rounding error when running prime
hi, i was testing my laptop since i have been having windows failed to start problems randomly. it fails about once every 10 starts on average. its a dv6-6b19wm with i3-2330m, intel 3000 graphics, 16gb ddr3-1333 pc3-10600 centon 2x8 modules. 120 gb patriot pyro se with latest firmware. windows 7 home premium with all available updates, all drivers are updated as well. Bios NOT modded so ram NOT overclocked. my laptop runs at about 1.5 gb of ram used at idle so i ran prime95 blend with custom ram usage of 14.5 gb. which i wanted to add up to almost 100% used. i got an fatal rounding error on worker #4 almost instantly. ram usage was at 15.95gb with prime running. i have clean mem running at the bottom corner of desktop, i also have cpu meter running too. so thats how i knew of my ram usage. was my error caused because I took it to almost 100%? should i leave some unused? Could the ram be failing? it has a lifetime warranty. I have just re ran the test rtying to reproduce the error and now i get a different error code on worker #4 and it wont start now. Ill put screenshot up so you can see it. any insight would be great. thanks.
ok, when i run small fft all workers run normally. and now im only running 13.5gb for a total of 93% of ram usage @ 14.8 gb total and its running now without errors so far. ill keep it running and update any errors that come up. here is a screenshot of that. im thinking it was because i tried to use more than 16gb of ram. but im not an expert. any comments will be of great help.
-
The data type for x and y properties is a double precision floating-point Number, so why is it rounding off values? Rounding is an unnecessary operation, and furthermore it's not saving any memory, because it just requires me to store a more precise value elsewhere, in addition to the lower-bit rounded value.
This rounding causing unnecessary additional work when coding against these values, because I have to round other values as well to make sure there aren't discrepancies when transforming coordinates.
So I was really surprised to discover that if I assign a value to a DisplayObject's x or y coordinates such as 10.3333333, and trace the value, it becomes 10.3.
If I assign 10.666666, it becomes 10.65. Apparently it's rounding everything to the nearest 20th of a unit. So now, I have to override the x and y properties to store the Number-type value, once again as a Number-type, which is not rounded.
Flash's arbitrary rounding of coordinates is causing erratic rounding errors when performing coordinate system transformations using localToGlobal and globalToLocal to find the composite scale of an object on the stage.
For example, suppose an object was laid out to occupy one third of the display, and it's width ends up being 200.3333333. One calculation of my docking framework involves obtaining the orthagonal bounding box of the child by transforming its corner points into stage coordinates using localToGlobal, which accounts for things like scaling and rotation. So despite everything having a scale of 1, and having zero rotation, you'd still end up with a rectangle with a width of 200.3 instead of the expected 200.3333333 in stage coordinates. So it would appear as though the composite scale is slightly smaller than 1, since 200.3 / 200.333333 is 0.99983361081528. But the composite scale is in fact 1, we just don't know that because Flash unexpectedly rounded some coordinates to an arbitrary 1/20 unit.
No game engine in existence does that with its transformation matrices, because it's retarded to round so early, and then allow those rounding errors to accumulate through a display hierarchy via functions like localToGlobal.
This rounding is causing jittering by a pixel or so when animating a drop down panel in my my docking framework, because it's constantly correcting for unexpected anomalies in the scaling factor on each frame. Despite the parent container having a constant fixed width, the child object, once its corner coordinates are passed through localToGlobal, end up reporting rounded widths, which ultimately leads to a series such as the following:
dockedChild.width: 538.3842482614723, parent.width: 558.3412118444024
dockedChild.width: 538.3754595467979, parent.width: 558.3412118444024
dockedChild.width: 538.3666709755926, parent.width: 558.3412118444024
dockedChild.width: 538.3578825478539, parent.width: 558.3412118444024
dockedChild.width: 538.3490942635798, parent.width: 558.3412118444024
dockedChild.width: 538.3903098666023, parent.width: 558.3412118444024
dockedChild.width: 538.3815210529766, parent.width: 558.3412118444024
dockedChild.width: 538.3727323828218, parent.width: 558.3412118444024
dockedChild.width: 538.3639438561353, parent.width: 558.3412118444024
dockedChild.width: 538.3551554729148, parent.width: 558.3412118444024
dockedChild.width: 538.346367233158, parent.width: 558.3412118444024
dockedChild.width: 538.3875826274011, parent.width: 558.3412118444024
dockedChild.width: 538.3787938582956, parent.width: 558.3412118444024
dockedChild.width: 538.37000523266, parent.width: 558.3412118444024
dockedChild.width: 538.3612167504922, parent.width: 558.3412118444024
dockedChild.width: 538.3524284117897, parent.width: 558.3412118444024
dockedChild.width: 538.3436402165502, parent.width: 558.3412118444024
dockedChild.width: 538.384855402015, parent.width: 558.3412118444024
dockedChild.width: 538.3760666774294, parent.width: 558.3412118444024
dockedChild.width: 538.3672780963132, parent.width: 558.3412118444024
dockedChild.width: 538.3584896586638, parent.width: 558.3412118444024
dockedChild.width: 538.349701364479, parent.width: 558.3412118444024
dockedChild.width: 538.3909170139807, parent.width: 558.3412118444024
Is there any way to turn off this rounding to 0.05 units?
To override the x and y values to have greater precision, I must do the following:
public class Control extends MovieClip
public function Control()
super(); //Flash performs timeline/graphics initialization here, which means after this call, the object may have non-zero x and y values
_x = super.x; //acquire them immediately, so if we try to set x or y to zero, the 'if (_x != value)' check does not think it's already positioned at zero and ignore the call
_y = super.y;
private var _x:Number;
private var _y:Number;
override public function get x():Number { return _x; } //return precise value, rather than rounded super.x value
override public function set x( value:Number ):void
if (_x != value) //ensure value is actually changing before performing work
_x = value; //store precise value in private variable
super.x = value; //DisplayObject will round value to nearest 0.05
if (stage != null)
stage.invalidate(); //ensure RENDER event is dispatched to re-render anything that may need to account for a repositioned object
override public function get y():Number { return _y; } //return precise value, rather than rounded super.y value
override public function set y( value:Number ):void
if (_y != value) //ensure value is actually changing before performing work
_y = value; //store precise value in private variable
super.y = value; //DisplayObject will round value to nearest 0.05
if (stage != null)
stage.invalidate(); //ensure RENDER event is dispatched to re-render anything that may need to account for a repositioned object }
Most importantly, you must initialize the _x and _y values to super.x and super.y in the constructor immediately after a call to super(), in order to acquire any non-zero values that the object instance may have been initialized with on the timeline.
I just cannot fathom why they didn't leave the x and y coordinates as-is, instead of rounding them, when it causes so many problems and complications, and requires overriding not only x and y, but functions like localToGlobal/globalToLocal/getRect.
This has been an issue for a while:
flash - AS3 x and y property precision - Stack Overflow specifically: flash - AS3 x and y property precision - Stack Overflow
http://www.actionscript.org/forums/showthread.php3?t=96510
Problems with Sub-pixel Coordinate Movement
In fact, that last link says: "
Running the code:
The motion is still jerky; and
The distance between the two squares diverges
However, the one benefit is that the distance does not diverge by more than 1 pixel."
That's precisely what I saw happening in my own code, as you can see from the series of widths I posted above, which seem to fluctuate randomly between 358.34 and 358.39.Actually, there is a way.
If you simply activate the 3D transformation by setting z to zero, the matrix3D replaces the matrix and concatatedMatrix properties of the DisplayObject's transform object, and suddenly x and y values maintain a precision higher than a twip. It's not quite the double-precision value of the Number type, however, and looks more like a single-precision 32-bit floating point value.
For example, if you run the following code:
var mc:Sprite = new Sprite();
mc.x = 200.0 + (1/3); //assign high precision value to x
trace(mc.x); //traces 200.3 (rounded)
mc.z = 0; //activate 3D matrix
mc.x = 200.0 + (1/3);
trace(mc.x); //traces 200.33333740234374 (still rounded, but accurate to 5 decimal places)
Based on the traced output, it's clear that it is possible to force the DisplayObject to get and set higher precision values for x and y properties, without any modifications to the underlying classes. However, I'm not happy with that solution for 2 reasons. First, it activates stage 3D and introduces graphical glitches and unnecessary bitmap caching. Second, it's still not "Number" precision; it's something less than that.
Instead, I was able to successfully work around the issue by altering the overrides for properties x, y, scaleX, scaleY, and methods localToGlobal, globalToLocal, and getRect to use privately maintained values. I was already using privately maintained width and height values in order to decouple the size from the scaling factors. No need to override getBounds, since it accounts for stroke widths and will be non-exact anyway.
The consistent, high precision values are vital, and they increased the performance of my layout framework, because it's actually able to prevent unnecessary assignments to x, y, width, and height when the values aren't actually changing. #beginrant: Such detection is impossible when Flash internally rounds everything, because if you try to keep something at, for example, 1/3 of the screen, it will always think you're trying to assign a high precision value of 200.33333 over a less precise value of 200.3 as I had previously described. Alternatively, you'd have to pre-round any value you try to assign, which is more work than it's worth. It's sort of terrible that Flash rounds property values as it does, because the value you assign can never be read back the same. That's generally not how numerical properties should work when assigning values of the same data type and precision. #endrant
In particular, two optimizations were made in the getRect override. If the target coordinate system is null or "this", then it simply returns new Rectangle( 0, 0, _width, _height ), and more importantly, if the target coordinate system is "parent" (which is the case 99% of the time) and rotation is zero and scale is 1 (also the case 99% of the time for GUI elements), then it simply returns new Rectangle( _x, _y, _width, _height ), which is the internal, high-precision values for x, y, width, and height (resemblance to AS2 properties is purely coincidental; this is AS3 code). That allows me to skip the following code path 99% of the time, which would otherwise return the orthagonal bounding box of the element at any rotation in any coordinate system:
//These instance variables are used to accelerate calculations, see comments.
//Upper left is always an empty point, and these variables store upper right, lower right, and lower left corner points.
protected var p_UR:Point; //DO NOT ALTER p_UR.y; LEAVE AT ZERO ALWAYS
protected var p_LR:Point; //Lower right corner: x = width, y = height
protected var p_LL:Point; //DO NOT ALTER p_LL.x; LEAVE AT ZERO ALWAYS
//Returns the orthogonal bounding rectangle of the object in the specified target coordinate system
//based on its own internal height and width (actual contentRect may be larger).
//getContentRect function was added to replace the original functionality of this method
override public function getRect( targetCoordinateSpace:DisplayObject ):Rectangle
switch (_scaleMode) //GUIControl allows decoupling of size and scale
case SCALE_NOSYNC_SIZE:
if (targetCoordinateSpace == null || targetCoordinateSpace == this)
return new Rectangle( 0, 0, _width, _height );
//UPDATE: Created this optimization to ensure rounding errors introduced by
//Flash's tendency to round x and y coordinates to twips are not introduced
//by localToGlobal/globalToLocal calls, so they are avoided if possible.
if (targetCoordinateSpace == parent && rotation == 0 && scaleX == 1 && scaleY == 1)
return new Rectangle( _x, _y, _width, _height );
p_UR.x = _width; //note the p_UR.y is always zero
p_LR.x = _width;
p_LR.y = _height;
p_LL.y = _height; //note the p_LL.x is always zero
break;
case SCALE_SYNC_SIZE:
var contentRect:Rectangle = getContentRect( true ); //must use unscaled points when performing local/global transforms, since this object is scaled
p_UR.x = contentRect.right; //note the p_UR.y is always zero
p_LR.x = contentRect.right;
p_LR.y = contentRect.bottom;
p_LL.y = contentRect.bottom; //note the p_LL.x is always zero
break;
return calcOrthogonalBoundingBox(
targetCoordinateSpace.globalToLocal( localToGlobal( EMPTY_POINT ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_UR ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_LR ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_LL ) )
protected function calcOrthogonalBoundingBox( p0:Point, p1:Point, p2:Point, p3:Point ):Rectangle
//Assuming no rotation, points 0 and 3 are most likely to be min_x. This optimization minimizes the likely number of assignments.
//Similar optimizations are in place for max_x, min_y, and max_y, all patterns are ordered 0,1,2,3... starting with the two most likely candidates in the sequence.
var min_x:Number = p3.x;
if (p0.x < min_x) min_x = p0.x;
if (p1.x < min_x) min_x = p1.x;
if (p2.x < min_x) min_x = p2.x;
var max_x:Number = p1.x;
if (p2.x > max_x) max_x = p2.x;
if (p3.x > max_x) max_x = p3.x;
if (p0.x > max_x) max_x = p0.x;
var min_y:Number = p0.y;
if (p1.y < min_y) min_y = p1.y;
if (p2.y < min_y) min_y = p2.y;
if (p3.y < min_y) min_y = p3.y;
var max_y:Number = p2.y;
if (p3.y > max_y) max_y = p3.y;
if (p0.y > max_y) max_y = p0.y;
if (p1.y > max_y) max_y = p1.y;
return new Rectangle( min_x, min_y, max_x - min_x, max_y - min_y );
In this framework, particularly in SCALE_NOSYNC_SIZE mode (the default), the width and height are assigned and internally maintained, independently of the scaleX and scaleY values. The internally maintained values are used for performing layout and drawing operations such as backgrounds and borders. In the extremely rare occasion where the actual content needs to be measured, I just use getBounds, such as in the constructor when initializing the "original" size of the clip's content, if it has timeline content, or possibly bitmap methods for masked objects.
These changes have all increased the performance of my framework by an order of magnitude and have virtually eliminated 3rd layout passes, so I'm happy with it. I still wish the Flash runtime would be updated to simply stop rounding these values. -
Rounding errors when plotting rectangles (simple code snippet included)
hi,
I'm having a problem with rounding errors, and am unsure what the best thing to do is..
If you run the snippet of code below, you'll see some rectangles with black lines between them. The range of values covered in the model behind the plot is contiguous, and its only after scaling the trouble begins..
does anyone have suggestions for what to do here? I need this resize behaviour.. I'm unable to change the interval class :)
thanks,
asjf
import java.awt.Color;
import java.awt.Graphics;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingError extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
public RoundingError() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
double scale= getWidth()/(double)WIDTH;
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256), r.nextInt(256), r.nextInt(256)));
int sx= (int) (i.start * scale);
int sy= getHeight()>>2;
int sw= (int) ((i.end-i.start) * scale);
int sh= getHeight()>>1;
g.fillRect(sx,sy,sw,sh);
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingError());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end;
}import java.awt.Color;
import java.awt.Graphics;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingError extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
boolean firstTime = true;
public RoundingError() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
double scale= getWidth()/(double)WIDTH;
int count = 0;
int errorX = 0;
int totalW = 0;
int lastX = 0;
int[] errors = new int[intervals.size()];
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256),
r.nextInt(256), r.nextInt(256)));
int sx= (int) (i.start * scale);
int sy= getHeight()>>2;
int sw= (int) ((i.end-i.start) * scale);
int sh= getHeight()>>1;
if(firstTime) {
totalW += sw;
if(count > 0) {
int error = sx - lastX;
if(error != 0) {
errors[count] = error;
errorX += error;
lastX = sx + sw;
System.out.printf("count = %2d sx = %3d sy = %3d " +
"sw = %3d dh = %3d totalW = %d%n",
count++, sx, sy, sw, sh, totalW);
g.fillRect(sx,sy,sw,sh);
if(firstTime) {
System.out.printf("errorX = %d%n" +
"lastX - totalW = %d%n",
errorX, lastX - totalW);
for(int i = 0; i < errors.length; i++) {
if(errors[i] != 0) {
System.out.printf("errors[%2d] = %d%n", i, errors);
firstTime = false;
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingError());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end;
This seems to work okay.
import java.awt.Color;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingErrorRx extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
boolean firstTime = true;
public RoundingErrorRx() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
Graphics2D g2 = (Graphics2D)g;
double scale= getWidth()/(double)WIDTH;
int count = 0;
double errorX = 0;
double totalW = 0;
double lastX = 0;
double[] errors = new double[intervals.size()];
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256),
r.nextInt(256), r.nextInt(256)));
double sx= i.start * scale;
double sy= getHeight()>>2;
double sw= (i.end-i.start) * scale;
double sh= getHeight()>>1;
if(firstTime) {
totalW += sw;
if(count > 0) {
double error = sx - lastX;
if(error != 0) {
errors[count] = error;
errorX += error;
lastX = sx + sw;
System.out.printf("count = %2d sx = %5.1f sy = %5.1f " +
"sw = %5.1f dh = %5.1f totalW = %5.1f%n",
count++, sx, sy, sw, sh, totalW);
g2.fill(new java.awt.geom.Rectangle2D.Double(sx,sy,sw,sh));
if(firstTime) {
System.out.printf("errorX = %5.1f lastX - totalW = %5.1f%n",
errorX, lastX - totalW);
for(int i = 0; i < errors.length; i++) {
if(errors[i] != 0) {
System.out.printf("errors[%2d] = %5.1f%n", i, errors);
firstTime = false;
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingErrorRx());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end; -
The Function Math.Round Has errors try Round Number 1.265
round in 1.26, when the correct answer its 1.27Come again... Math.round returns an integer.
Math.round(1.265) will return
1.
Dave -
Head Developer
http://www.blurredistinction.com
Adobe Community Expert
http://www.adobe.com/communities/experts/ -
Rounding error between FI and CO-PA
Dear All,
We are currently in the process of migrating our NZ business on to our SAP application of ECC6. In CO-PA the Operating Concern Currency is set to AUD and in addition Company Code Currency has also been activated.
When a billing document is created for NZ the SD values and FI values correlate, however for the CO-PA document there is a rounding difference of 1 cent when the Company Code Currency amount is checked. Therefore when reconciling CO-PA with FI we come across quite a number of reconciliation differences for each document where the rounding difference has occurred.
I have read OSS message 429517, and according to this note a rounding difference should not be occurring. This is because in determining the company code value in CO-PA the values should be copied from FI.
Any thoughts on this?
Regards.I've solved this via user-exit COPA0002.
-
WITHHOLDING TAX ROUNDING ERROR
Hi All,
I faced one problem ie; at the time of sub contract service charges invoice posting Education Cess amount whereever below one rupee system generate entries geting debited, actuvalli this was worng in accounting.
that to this was happen some of the cases only.
please give the sloution
THANKS IN ADVANCE
Regards & Thanks
KRISHNA KUMARHi,
Good morning and greetings,
Please go through the following SAP OSS note
Note 720332 - 7.5 "Round Tax Amount in Rows" Setting for PL Localization
Note 927977 - Can't enter a document rounding amount without recalculating
Note 662600 - Calculation of the Rounding field in A/R and A/P documents.
Please reward points if found useful
Thanking you
With kindest regards
Ramesh Padmanabhan -
Condition rounding error in pricing of sales order
Hi all,
I have a problem on rounding of condition type MWST in a sales order.
I have the PR00 condition evaluated with 6,50 euro for 1 piece; on this price should be applied condition type MWST with a percentage of 10%.
The result should be 0,65 euro but the system shows (and calculates) 0,64 euro for MWST.
How can I solve this rounding problem?
Thanks in advance for your helpful answers.
AlessandraHi,
Check if you are using VOFM subroutines in your pricing procedure (V/08) or in the condition type (V/06) for rounding rules.
Check too this notes and related notes in them (surely, they will help you)
Note 80183 - Rounding
Note 315792 - Group conditions of the same amount on item
Note 166952 - Rounding when distributing header conditions
Note 517829 - Source code f new dstrbtn rule 'roundng diffrnce comparison'
I hope this helps you
Regards,
Eduardo -
Has anyone ever noticed AE adding empty frames to the end of QuickTime movie output? It's as if the duration was rounded up to the nearest full second by adding a few black frames.
I also noticed if I enter 60 seconds for a movie trailer composition's duration, it is rounded down to 59;28 instead of 59;29, which is technically correct since it's 29.97 for the frame rate, but it is a bit unintuitive. I believe Final Cut interprets 60;00 as 59;29 with the 29.97 frame rate chosen.
Even if I trick AE by using 60;01 for the duration it only shows as 59;29 duration for that session. When I quit, restart, and reopen the project it's down to 59;28 again. I want my frame count to stay the same!If you think it's a bug, please file here:
http://www.adobe.com/go/wish
Personally I don't think it is, but then again I'm lucky enough to not have to worry about the finer points of NTSC most time of the year. ;-) My guess is that this has to do with calculating the distribution of fields, resulting in some time left over every x seconds which is then simply allocated to empty frames at the end of a file.
Mylenium -
Funny modulus result? If this a rounding error?
Hi,
Strangely, I'm getting the following result in the ESTK Javascript console:
492.1 % 13.3
Result: 13.3
Clearly the result should be 0 (since 492.1 is a multiple of 13.3).
What's going on? Is this because of floating-point arithmetic? In any
case, what can I do about it to get an accurate result?
Thanks,
ArielThanks Stephen for your input.
I don't think it says anywhere that the % operator should only be used
with integers? At any rate, it seems to work fine with decimals in most
cases.
Your mod function seems rather strange to me: try a = 9, b = 3:
a/b = 3
3-9 = -6
-6 * 3 = -18
which clearly isn't the right answer for 9 % 3 ?
At any rate, I've meanwhile established that this is a floating-point
arithmetic error. Because, once again in the ESTK console, the following
shows that what looks like 13.3 isn't:
492.1 % 13.3
Result: 13.3
but....
(492.1%13.3)-13.3
Result: -3.5527136788005e-15
So it looks like I'll have to do something like this to the result:
myMod = a % b;
if (Math.abs(myMod - b) < 0.00001) myMod = 0;
... which is the sort of thing that often needs to be done with floating
point arithmetic, if I'm not mistaken.
Thanks,
Ariel
Maybe you are looking for
-
How do I transfer a PDF file from an email to my Adobe Reader app.
HHow do I transfer a PDF file from an email to my Adobe Reader app.
-
Account Dimension not showing all members in the cube in Analysis Services
Hi, In SAP - BPC 5.1 after processing account dimension all the members are created under account dimension in analysis services but the same cannot be found in the data cube. Hence the reports generated through SAP - BPC is not show
-
How do I pragmmatically determine what comm ports my serial instruments are on?
I am building a test stand that incorporates 5 serial instruments; 2 digital loads and 3 power supplies. I can use my PC's device manager to figure out which comm port each device is on and set the VISA Resource Name so everything runs as expected. A
-
How do I properly install a USB windows keyboard on an iMac?
Greetings everyone, I have a new windows keyboard model Haveit Magic Eagle X1 gaming keyboard that has several problems being installed on the OSX system. This is a windows gaming keyboard that, for the most part has installed properly. The two mai
-
CS4 with Quadro FX 1700 512MB (Tested Video Cards?)
After seeing a topic posted here with tested video cards, I went to the link to confirm my card was indeed supported and it is on the list of tested cards. http://kb.adobe.com/selfservice/viewContent.do?externalId=kb405711&sliceId=1 I only installed