Rounding error in pricing
Hi All,
We have one issue for Russia in Pricing
We have one condition type for Base price and the value is 515.30 $ and
Surcharge is 15% o the Base price which willl be 515.30*15/100 which equals to 77.295
The total is Baseprice + Surchage which was calculated as 515.30+77.295 = 592.595 ( but rounded as 592.60) which is showing a difference
If customer orders for 10 parts, then there will be 2 $ variace between the amount calclated and showed in SAP
Can you please let me know why this is displayed with rounding , is this standard behavious and if so, where was it controled.
thanks
santosh
I dont think this will be treated as an issue by the client. It is general that in exports, there will always be some difference in decimals. Considering all these things only, client would have arrived at their selling price. If I am correct, this would be addressed by having a separate G/L account for Gain or Loss in foreign exchange.
Check one such thread where a similar topic was discussed some time back.
[Re: Exports|Re: Exports]
May be you can check with your FI counter part also on this.
thanks
G. Lakshmipathi
Similar Messages
-
Condition rounding error in pricing of sales order
Hi all,
I have a problem on rounding of condition type MWST in a sales order.
I have the PR00 condition evaluated with 6,50 euro for 1 piece; on this price should be applied condition type MWST with a percentage of 10%.
The result should be 0,65 euro but the system shows (and calculates) 0,64 euro for MWST.
How can I solve this rounding problem?
Thanks in advance for your helpful answers.
AlessandraHi,
Check if you are using VOFM subroutines in your pricing procedure (V/08) or in the condition type (V/06) for rounding rules.
Check too this notes and related notes in them (surely, they will help you)
Note 80183 - Rounding
Note 315792 - Group conditions of the same amount on item
Note 166952 - Rounding when distributing header conditions
Note 517829 - Source code f new dstrbtn rule 'roundng diffrnce comparison'
I hope this helps you
Regards,
Eduardo -
Error in pricing when doing sourcing in EBP 4.0
Hello,
We're experiencing random errors when doing sourcing in EBP 4.0. The shopping carts don't seem to have any similarities that could launch the error. There has been few cases when the same error has occured when user is creating a shopping cart.
The problem occures when doing sourcing and the shopping cart is either being held or a PO is created. The error creates dump in ST22, "UNCAUGHT_EXCEPTION". This is somehow related to pricing and we have checked note 673372 and ran report BBP_CND_CUSTOMIZING_CHECK, all OK. IPC is running the same SP level as SRM, SP level 10.
We have already created an OSS about this, but as this is completely random we haven't been able to reproduce the error and no solution has been found.
Here's some data from ST22:
Transaction......... "BBPSOCO01 "
Program............. "SAPLBBP_PDH"
Screen.............. "SAPLBBP_SOCO_UI_ITS 2000"
The termination occurred in the ABAP program "SAPLBBP_PDH" in "BBP_PD_ABORT"
The main program was "SAPLBBP_SOCO_UI_ITS ".
The termination occurred in line 69 of the source code of the (Include)
program "LBBP_PDHU08"
of the source code of program "LBBP_PDHU08" (when calling the editor 690).
000600 * get the messages from the application log
000610 TRY.
000620 CALL FUNCTION 'BBP_PD_LOG_GET_MESSAGES'
000630 TABLES
000640 E_MESSAGES = LT_MESSAGES.
000650 CATCH CX_BBP_PD_ABORT.
000660 REFRESH LT_MESSAGES.
000670 ENDTRY.
000680
> RAISE EXCEPTION TYPE CX_BBP_PD_ABORT EXPORTING
Thanks already in advance!
-JussiHi
Please refer to following SAP Notes:
Note 860935 - pricing customizing exits are missing
Note 673372 - Short dumps UNCAUGHT_EXCEPTION in SRM 3.0, 4.0, and 5.0
Also try to implement the latest IPC patches following the instructions from note 880749, then restart the VMC. It worked for us, after trying many things with no luck.
Alternatively, try this as well. There must be some problem with VMC. Most probably java connection problem. Please execute program RSVMCRT_HEALTH_CHECK in SE38 transaction. You should see green for all 4 messages in the result. Please contact your basis consultant and refer notes 854170 (and 844817)
<u>Other related OSS notes.</u>
Note 1024374 - Unexpected pricing error for pricing irrelevant item
Note 992259 - NullPointerException in Command GetFilteredCsticsAndValues
Note 860935 - pricing customizing exits are missing
Note 673372 - Short dumps UNCAUGHT_EXCEPTION in SRM 3.0, 4.0, and 5.0
Note 854170 - Activating the component "VM Container"
Note 593109 - Short dumps UNCAUGHT_EXCEPTION in SRM 2.0 (EBP 3.5)
Note 880749 - Patching of VMC based AP 7.00 engines
Note 854170 - Activating the component "VM Container"
Note 844816 - Information for Upgrade from IPC 4.0 to SAP AP 7.00
Hope this will help.
Please award points if helpful.
Regards,
-Atul -
Error :Maintain pricing conditions for the material for the excise invoice
Hi
i have maintained excise data for material, after that i have created scheduled agreement with lp and with 0 value private and dummy tax code and delivery schedule is released. Now when i do the goods receipt i am getting error " Maintain pricing conditions for the material for the excise invoice date
Message no. 8I629" can some body can help me.
with regards
Narendra kumarHi,
goto tcode fv11 u have to maintained the condition.
first enter the condition type like jmop -
then enter, next screen u have the select the below comination.
select the plant/vendor/material base u have to maintain the conditions.
Thank's
MVS -
ORA-01403- No data found error on Pricing and availabiltiy form
ORA-01403- No data found error on Pricing and availabiltiy form
whenever user click on item availability in Pricing and Availability form ,this issue appears. Can any one let me know the cause.
Thanks
Payal941250 wrote:
Thanks Mahendra,
I have already read this article and accorind to it our instance is already ATP enabled. But still we are facing the error.
Thanks
PayalPlease see if (OEXPRAVA ORA-01403 Item Availability Not Displaying In Price and Availability Form [ID 290251.1]) helps.
Thanks,
Hussein -
Internal error during pricing in CRM service order
Hello Guys,
Here is an issue when i add line item in service order, it pops up with an error message " Internal error during pricing "at header & line item. Its observed that output tax condition type MWST is not called from TTE. Pricing procedure has picked up perfectly & even TTE setting is maintained which in turn calls Tax Calculation procedure 11000. But then we are unable to trace where system is falling to call MWST condition type. Request you to guide me on the issue.
Regards
Devika.SHi Devika,
please refer to following SAP notes:
559836 FAQ: tips & tricks for CRM pricing
559876 FAQ: Tricks and hints for the error analysis in pricing
559866 FAQ: tips & tricks for the IPC installation
best regards,
Johannes -
hello,
I understand that rounding errors exist and I need help determining the best way to handle them. The particular situation I am dealing with relates to finding the equation of a line and comparing the calculated values to a data set.
The big picture: I have a data set that looks generally exponential with sharp peaks along the curve. The exponential part is background noise that I want to eliminate. After trying various methods, I've determined that getting a convex hull is the best method of eliminating the background noise. The convex hull is found by finding a line through point A and point B of the data and determining if all of the data is above (greater than) the line (similar to a tangent line). Once a point B is found which satisfies the condition, point B becomes the new point A and a new point B is looked for.
I compare the data set to the points that create the line. The X values of the data set and the line are the same integer values. The Y values of the data set are DBL. I have attached a VI demonstrating the simple calculations I am doing to find the Y values along the line. It is simply the (y1-y0)/(x1-x0) to get the slope and using the slope to find the y-intercept (b = y0-m*x0). I then use the slope and y-intercept to calculate the Y values along the line. I subtract the line Y values from the data set Y values and check if they are all positive. If they are, then I can conclude that all the points are above the line and the point B (x1,y1) is kept as part of the convex hull.
The problem is that rounding error is causing "false negatives." In other words, the rounding error is causing the actual and calculated (through equation of line) y1 to be different. Since (x1, y1) is a data set point that defined the line it should be equal to the calculated (x1, y1), but the rounding error is not allowing that.
Currently, I am adding a very small (1.5E-12) constant to the data set point to account for the error and prevent false negatives (which would be the existence of a negative difference, when it should be 0). However, this small number is not always sufficient.
How can I find the maximum error, so that I can add the error to the data set point and therefore always prevent false negatives?
Thanks!
Kristen
Attachments:
Rounding Error Example.vi 13 KBHi Lynn,
Thanks for your help.
I have attached a text file with example data set. The range is 0-6000 (look around x=187 and x=300 for max). There is a middle region (220- 260ish) that is irrelevant, otherwise, there are eight peaks in total, four peaks on each side, which are symmetric from the center. I am doing the convex hull on each half of the data. The data set actually begins as an x-ray diffraction image (irrelevant region is the backstop preventing the x-ray from saturating the detector so only the diffraction is captured) of which I take an ROI and sum down the columns. The result is this data which is pixel location and summed pixel values. (Yes, I know the x values in the text file are not integers- this is because some of the end points and the irrelevant area is extracted out and the ramp function is used to create new x values even spaced between the new beginning and end of the data set. This part of the code is inherited and I need to check with the programmer why this method was used)
Previously, I tried doing a curve fit of the whole data set (background not subtracted) with the background described as h0+h1*exp(-h2*(x-r))+h3*exp(-h4*(x-r)) and the peaks described as a convolution of a gaussian and lorentzian peak: abs(a0)*((1/(1+f^2))*exp(-((x-(r+d0))/w)^2)+(1-(1/(1+f^2)))/(1+((x-(r+d0))/w)^2))
where
r is the theoretical median of the data (point about which the image is horizontally symmetric)
a0 is the amplitude of the peak
d0 is the distance from symmetric center to the center of the peak
w is the full width half max of the peak
f is a factor to determine if the peak is more Gaussian or Lorentzian
I added the background model with the peak models (subtract d0 from r for the symmetrically equivalent peak) and used the Non-linear Curve Fit VI to fit the equation. The results were decent, but the convex hull -> subtract background -> peak fit method seemed to be fitting the peaks better (correlation .999 vs .99), which is the important part.
Thoughts?
Kristen
Attachments:
ConvexHullTest Example Data.txt 7 KB -
Rounding error in BAPI_PO_CREATE1 Function module
Hi experts ,
In my project there are two steps behind PO creation 1-> Authorization of po creation And 2-> PO creation.
In authorization a logic in smart form calculates the amount for which the po is to be created and then goes go 2 nd step.
before creating po gross discount percentage is calculated the same way calculated in step1.
But at the end total amount in smartform and total amount of po created is differing by max of .01 to 0.1.
Unfortunetely the Z logic in the smartforms is caculation the creec value but standard BAPI BAPI_PO_CREATE1 is creating some rounding errors .
Please suggest some thing for this problem.
Thanks in advance
ShoaibmemonHi Md shoaib Memon,
there is another thread with the same issue.
http://scn.sap.com/thread/922418
Solution:
Set the following flag
poitem-no_rounding = 'X'.
poitemx-no_rounding = 'X'.
Best regards,
Ademir -
I've been working on an egyptian fraction program and for some reason I cant seem to figure out a way to fix this rounding error. An egyptian fraction is a fraction that can be expressed as a sum of fractions eg. 3/4 = 1/2 + 1/4. For some reason on certain fractions it rounds up and skips the correct fraction to subtract. When I run 2/7 its supposed to equal 1/4 + 1/28, but gives me 1/4 + 1/29 and decides to round. This is my code for the problem.
public class EgyptianFraction{
private static double epsilon = 1.0e-7;
public static void main(String args[]){
greedySearch(2.0/7.0);
public static void greedySearch(double fraction){
for(int i = 2; fraction > epsilon; i++){
if(fraction - (1.0/i) >= 0){
fraction -= (1.0/i);
//*****Output******
//0.0357142857142857 - 0.03571428571428571 = -1.3877787807814457E-17
}When I print out all of the math involved it says that it gives the output above. They are fairly close but for some reason it makes the 1/28 bigger then the current fraction. The program should subtract 1/28 and then the fraction should be close enough to 0 and end. Is there any way you guys can think of to fix this problem?You have given the error as epsilon, so you can't expect `fraction - 1.0/i >= 0` to give an exact answer.
Given your error is +/- epsilon, this expression should be
fraction - 1.0/i >= epsilon || fraction - 1.0/i >= -epsilon
or just
fraction - 1.0/i >= -epsilon
If you change this you get 1/4 + 1/28 as the answer, -
Tables and Rounding Errors on Board Game Gui
Hello,
So I am in a software development class , and my team and I are developing a software version of a board game that uses numbered tiles placed on a board in a somewhat scrabble-esque way.
Here is a picture of the game board:
[http://img90.imageshack.us/img90/1052/untitledqv3.png|http://img90.imageshack.us/img90/1052/untitledqv3.png]
Currently, a problem that we are working on is that as the tiles get further and further away from the center of the board, they are displayed further and further askew from the board lines. I have another picture to demonstrate what I'm talking about more clearly.
[http://img225.imageshack.us/img225/4605/untitled2nn0.png|http://img225.imageshack.us/img225/4605/untitled2nn0.png]
As the tiles get further away from the center, they are displayed more askew.
We think that this happens because we are using a gridbag layout to add tile objects to, which displays the tiles in a certain spacing and orientation, and then we draw the board ourselves. When we draw the board, we get rounding errors inherent in the use of ints, and the lines become a bit off, with the problem getting worse as it gets further and further away from the center.
Here is the code that we use to initialize the layout and add the tiles to it:
//set the layout
setLayout(new GridLayout(7, 7, 7, 7));
//initialize the array with the references to all the tiles that we are going to add
//to the layout
squares = new SquareView[7][7];
for (int i = 0; i < 7; i++) {
for (int j = 0; j < 7; j++) {
//create the tile, put a reference to it in the array
squares[i][j] = new SquareView(boardModel.getSquare(new Point(j, i)), listener, handler);
//add the tile to the layout
add(squares[i][j]);
}And here is the code that we are using to draw the lines for the board:
GridLayout layout = (GridLayout) getLayout();
//getting the dimensions of the board
int rows = layout.getRows();
int columns = layout.getColumns();
int width = (getWidth() / columns);
int height = (getHeight() / rows);
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, i * width, rows * height);
// Horizontal lines
g2.drawLine(0, i * height, columns * width, i * height);
}I think that our problems come from the innacuracy of ints, but if there is some addition or subtraction trick that we could pull, then let me know.
Also, I was sort of looking into tables, and I was wondering if maybe we could use tables to do this, and have Java do our resizing for us.j.law wrote:
We think that this happens because we are using a gridbag layout to add tile objects to, From the snippets of code, it's looking as though you're using GridLayout. But that's OK as GridLayout should work fine here.
GridLayout layout = (GridLayout) getLayout();
//getting the dimensions of the board
int rows = layout.getRows();
int columns = layout.getColumns();
int width = (getWidth() / columns);
int height = (getHeight() / rows);
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, i * width, rows * height);
// Horizontal lines
g2.drawLine(0, i * height, columns * width, i * height);
}I have no idea of this will improve things, but what about:
GridLayout layout = (GridLayout) getLayout();
int rows = layout.getRows();
int columns = layout.getColumns();
double width = (getWidth() / columns); //** how about using doubles here
double height = (getHeight() / rows); //** how about using doubles here
g2.drawImage(backgroundImage, 0, 0, getWidth(), getHeight(), null);
for (int i = 0; i < 8; i++) {
// Vertical lines
g2.drawLine(i * width, 0, (int)(i * width), (int)(rows * height)); // and casting?
// Horizontal lines
g2.drawLine(0, (int)(i * height), (int)(columns * width), (int)(i * height));
} -
Internal error during pricing in CRM sales order
Hi,
While creating sales order i am getting following error in sales order
Internal error during pricing.
i am not understading what will be the cause of error
currently working on CRM 7.0
we have replicated all pricing condition types from ECC
Thanks & Regards
kishore kumarHello kishore kumar,
Tell basis Team to restart Java Instances.
After that check in Sales Order. If you still get that error then download below objects in a Sequence.
1. DNL_CUST_CNDALL
2. DNL_CUST_CND
3.DNL_CUST_PRC
Regards,
Rajendra -
Hi everybody,
how should I/can I handle rounding errors in Oracle.
I'm using the LOG function of Oracle and for SELECT LOG(2,16) FROM DUAL; Oracle returns 3,99999999999999999999999999999999999998 which is incorrect. I cannot use the Round-Function cause my whole mathematical expression (written in Excel Syntax) is:
FLOOR(LOG(x; 2); 1)
So if I use the round function and SELECT ROUND(LOG(2,15)) FROM DUAL; would return 4 which is not correct thinking about my FLOOR(4) => 4, but the result of the mathematical expression is 3!
Any ideas on how to deal with such problems... is there somehow a datatype to use?
Thanks!user10835064 wrote:
Thanks Max - I guess you just wanted to illustrated the problem? cause I don't see a solution there...
Thanks Ravi - but 7 digits... can someone assure that there is no number in the world where LOG(2, x) wont return something like y.999999987389... I mean LOG(2, 15.9999990018681) is 3,99999991 so even a rounding of 7 is not enough :)...
Maybe another question should be asked :)... can you count on this rounding error? I mean if so, than I could add 1 => FLOOR(LOG(2, x + 1)), but if I cannot count on this rounding error, than there could be wrong results if e.g. Oracle returns the correct value!I just showed the parameter usage of round, which you can use for rounding the digits.
While comparision like round(somevalue)>=2 as you said it is not required to round the digit, you can directly do the
somevalue>=2
But when you want to show the o/p to an end user who doesn't need 30+digits of precision, for him you can show using the above round function.
Ravi Kumar -
Rounding error when running prime
hi, i was testing my laptop since i have been having windows failed to start problems randomly. it fails about once every 10 starts on average. its a dv6-6b19wm with i3-2330m, intel 3000 graphics, 16gb ddr3-1333 pc3-10600 centon 2x8 modules. 120 gb patriot pyro se with latest firmware. windows 7 home premium with all available updates, all drivers are updated as well. Bios NOT modded so ram NOT overclocked. my laptop runs at about 1.5 gb of ram used at idle so i ran prime95 blend with custom ram usage of 14.5 gb. which i wanted to add up to almost 100% used. i got an fatal rounding error on worker #4 almost instantly. ram usage was at 15.95gb with prime running. i have clean mem running at the bottom corner of desktop, i also have cpu meter running too. so thats how i knew of my ram usage. was my error caused because I took it to almost 100%? should i leave some unused? Could the ram be failing? it has a lifetime warranty. I have just re ran the test rtying to reproduce the error and now i get a different error code on worker #4 and it wont start now. Ill put screenshot up so you can see it. any insight would be great. thanks.
ok, when i run small fft all workers run normally. and now im only running 13.5gb for a total of 93% of ram usage @ 14.8 gb total and its running now without errors so far. ill keep it running and update any errors that come up. here is a screenshot of that. im thinking it was because i tried to use more than 16gb of ram. but im not an expert. any comments will be of great help.
-
The data type for x and y properties is a double precision floating-point Number, so why is it rounding off values? Rounding is an unnecessary operation, and furthermore it's not saving any memory, because it just requires me to store a more precise value elsewhere, in addition to the lower-bit rounded value.
This rounding causing unnecessary additional work when coding against these values, because I have to round other values as well to make sure there aren't discrepancies when transforming coordinates.
So I was really surprised to discover that if I assign a value to a DisplayObject's x or y coordinates such as 10.3333333, and trace the value, it becomes 10.3.
If I assign 10.666666, it becomes 10.65. Apparently it's rounding everything to the nearest 20th of a unit. So now, I have to override the x and y properties to store the Number-type value, once again as a Number-type, which is not rounded.
Flash's arbitrary rounding of coordinates is causing erratic rounding errors when performing coordinate system transformations using localToGlobal and globalToLocal to find the composite scale of an object on the stage.
For example, suppose an object was laid out to occupy one third of the display, and it's width ends up being 200.3333333. One calculation of my docking framework involves obtaining the orthagonal bounding box of the child by transforming its corner points into stage coordinates using localToGlobal, which accounts for things like scaling and rotation. So despite everything having a scale of 1, and having zero rotation, you'd still end up with a rectangle with a width of 200.3 instead of the expected 200.3333333 in stage coordinates. So it would appear as though the composite scale is slightly smaller than 1, since 200.3 / 200.333333 is 0.99983361081528. But the composite scale is in fact 1, we just don't know that because Flash unexpectedly rounded some coordinates to an arbitrary 1/20 unit.
No game engine in existence does that with its transformation matrices, because it's retarded to round so early, and then allow those rounding errors to accumulate through a display hierarchy via functions like localToGlobal.
This rounding is causing jittering by a pixel or so when animating a drop down panel in my my docking framework, because it's constantly correcting for unexpected anomalies in the scaling factor on each frame. Despite the parent container having a constant fixed width, the child object, once its corner coordinates are passed through localToGlobal, end up reporting rounded widths, which ultimately leads to a series such as the following:
dockedChild.width: 538.3842482614723, parent.width: 558.3412118444024
dockedChild.width: 538.3754595467979, parent.width: 558.3412118444024
dockedChild.width: 538.3666709755926, parent.width: 558.3412118444024
dockedChild.width: 538.3578825478539, parent.width: 558.3412118444024
dockedChild.width: 538.3490942635798, parent.width: 558.3412118444024
dockedChild.width: 538.3903098666023, parent.width: 558.3412118444024
dockedChild.width: 538.3815210529766, parent.width: 558.3412118444024
dockedChild.width: 538.3727323828218, parent.width: 558.3412118444024
dockedChild.width: 538.3639438561353, parent.width: 558.3412118444024
dockedChild.width: 538.3551554729148, parent.width: 558.3412118444024
dockedChild.width: 538.346367233158, parent.width: 558.3412118444024
dockedChild.width: 538.3875826274011, parent.width: 558.3412118444024
dockedChild.width: 538.3787938582956, parent.width: 558.3412118444024
dockedChild.width: 538.37000523266, parent.width: 558.3412118444024
dockedChild.width: 538.3612167504922, parent.width: 558.3412118444024
dockedChild.width: 538.3524284117897, parent.width: 558.3412118444024
dockedChild.width: 538.3436402165502, parent.width: 558.3412118444024
dockedChild.width: 538.384855402015, parent.width: 558.3412118444024
dockedChild.width: 538.3760666774294, parent.width: 558.3412118444024
dockedChild.width: 538.3672780963132, parent.width: 558.3412118444024
dockedChild.width: 538.3584896586638, parent.width: 558.3412118444024
dockedChild.width: 538.349701364479, parent.width: 558.3412118444024
dockedChild.width: 538.3909170139807, parent.width: 558.3412118444024
Is there any way to turn off this rounding to 0.05 units?
To override the x and y values to have greater precision, I must do the following:
public class Control extends MovieClip
public function Control()
super(); //Flash performs timeline/graphics initialization here, which means after this call, the object may have non-zero x and y values
_x = super.x; //acquire them immediately, so if we try to set x or y to zero, the 'if (_x != value)' check does not think it's already positioned at zero and ignore the call
_y = super.y;
private var _x:Number;
private var _y:Number;
override public function get x():Number { return _x; } //return precise value, rather than rounded super.x value
override public function set x( value:Number ):void
if (_x != value) //ensure value is actually changing before performing work
_x = value; //store precise value in private variable
super.x = value; //DisplayObject will round value to nearest 0.05
if (stage != null)
stage.invalidate(); //ensure RENDER event is dispatched to re-render anything that may need to account for a repositioned object
override public function get y():Number { return _y; } //return precise value, rather than rounded super.y value
override public function set y( value:Number ):void
if (_y != value) //ensure value is actually changing before performing work
_y = value; //store precise value in private variable
super.y = value; //DisplayObject will round value to nearest 0.05
if (stage != null)
stage.invalidate(); //ensure RENDER event is dispatched to re-render anything that may need to account for a repositioned object }
Most importantly, you must initialize the _x and _y values to super.x and super.y in the constructor immediately after a call to super(), in order to acquire any non-zero values that the object instance may have been initialized with on the timeline.
I just cannot fathom why they didn't leave the x and y coordinates as-is, instead of rounding them, when it causes so many problems and complications, and requires overriding not only x and y, but functions like localToGlobal/globalToLocal/getRect.
This has been an issue for a while:
flash - AS3 x and y property precision - Stack Overflow specifically: flash - AS3 x and y property precision - Stack Overflow
http://www.actionscript.org/forums/showthread.php3?t=96510
Problems with Sub-pixel Coordinate Movement
In fact, that last link says: "
Running the code:
The motion is still jerky; and
The distance between the two squares diverges
However, the one benefit is that the distance does not diverge by more than 1 pixel."
That's precisely what I saw happening in my own code, as you can see from the series of widths I posted above, which seem to fluctuate randomly between 358.34 and 358.39.Actually, there is a way.
If you simply activate the 3D transformation by setting z to zero, the matrix3D replaces the matrix and concatatedMatrix properties of the DisplayObject's transform object, and suddenly x and y values maintain a precision higher than a twip. It's not quite the double-precision value of the Number type, however, and looks more like a single-precision 32-bit floating point value.
For example, if you run the following code:
var mc:Sprite = new Sprite();
mc.x = 200.0 + (1/3); //assign high precision value to x
trace(mc.x); //traces 200.3 (rounded)
mc.z = 0; //activate 3D matrix
mc.x = 200.0 + (1/3);
trace(mc.x); //traces 200.33333740234374 (still rounded, but accurate to 5 decimal places)
Based on the traced output, it's clear that it is possible to force the DisplayObject to get and set higher precision values for x and y properties, without any modifications to the underlying classes. However, I'm not happy with that solution for 2 reasons. First, it activates stage 3D and introduces graphical glitches and unnecessary bitmap caching. Second, it's still not "Number" precision; it's something less than that.
Instead, I was able to successfully work around the issue by altering the overrides for properties x, y, scaleX, scaleY, and methods localToGlobal, globalToLocal, and getRect to use privately maintained values. I was already using privately maintained width and height values in order to decouple the size from the scaling factors. No need to override getBounds, since it accounts for stroke widths and will be non-exact anyway.
The consistent, high precision values are vital, and they increased the performance of my layout framework, because it's actually able to prevent unnecessary assignments to x, y, width, and height when the values aren't actually changing. #beginrant: Such detection is impossible when Flash internally rounds everything, because if you try to keep something at, for example, 1/3 of the screen, it will always think you're trying to assign a high precision value of 200.33333 over a less precise value of 200.3 as I had previously described. Alternatively, you'd have to pre-round any value you try to assign, which is more work than it's worth. It's sort of terrible that Flash rounds property values as it does, because the value you assign can never be read back the same. That's generally not how numerical properties should work when assigning values of the same data type and precision. #endrant
In particular, two optimizations were made in the getRect override. If the target coordinate system is null or "this", then it simply returns new Rectangle( 0, 0, _width, _height ), and more importantly, if the target coordinate system is "parent" (which is the case 99% of the time) and rotation is zero and scale is 1 (also the case 99% of the time for GUI elements), then it simply returns new Rectangle( _x, _y, _width, _height ), which is the internal, high-precision values for x, y, width, and height (resemblance to AS2 properties is purely coincidental; this is AS3 code). That allows me to skip the following code path 99% of the time, which would otherwise return the orthagonal bounding box of the element at any rotation in any coordinate system:
//These instance variables are used to accelerate calculations, see comments.
//Upper left is always an empty point, and these variables store upper right, lower right, and lower left corner points.
protected var p_UR:Point; //DO NOT ALTER p_UR.y; LEAVE AT ZERO ALWAYS
protected var p_LR:Point; //Lower right corner: x = width, y = height
protected var p_LL:Point; //DO NOT ALTER p_LL.x; LEAVE AT ZERO ALWAYS
//Returns the orthogonal bounding rectangle of the object in the specified target coordinate system
//based on its own internal height and width (actual contentRect may be larger).
//getContentRect function was added to replace the original functionality of this method
override public function getRect( targetCoordinateSpace:DisplayObject ):Rectangle
switch (_scaleMode) //GUIControl allows decoupling of size and scale
case SCALE_NOSYNC_SIZE:
if (targetCoordinateSpace == null || targetCoordinateSpace == this)
return new Rectangle( 0, 0, _width, _height );
//UPDATE: Created this optimization to ensure rounding errors introduced by
//Flash's tendency to round x and y coordinates to twips are not introduced
//by localToGlobal/globalToLocal calls, so they are avoided if possible.
if (targetCoordinateSpace == parent && rotation == 0 && scaleX == 1 && scaleY == 1)
return new Rectangle( _x, _y, _width, _height );
p_UR.x = _width; //note the p_UR.y is always zero
p_LR.x = _width;
p_LR.y = _height;
p_LL.y = _height; //note the p_LL.x is always zero
break;
case SCALE_SYNC_SIZE:
var contentRect:Rectangle = getContentRect( true ); //must use unscaled points when performing local/global transforms, since this object is scaled
p_UR.x = contentRect.right; //note the p_UR.y is always zero
p_LR.x = contentRect.right;
p_LR.y = contentRect.bottom;
p_LL.y = contentRect.bottom; //note the p_LL.x is always zero
break;
return calcOrthogonalBoundingBox(
targetCoordinateSpace.globalToLocal( localToGlobal( EMPTY_POINT ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_UR ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_LR ) ),
targetCoordinateSpace.globalToLocal( localToGlobal( p_LL ) )
protected function calcOrthogonalBoundingBox( p0:Point, p1:Point, p2:Point, p3:Point ):Rectangle
//Assuming no rotation, points 0 and 3 are most likely to be min_x. This optimization minimizes the likely number of assignments.
//Similar optimizations are in place for max_x, min_y, and max_y, all patterns are ordered 0,1,2,3... starting with the two most likely candidates in the sequence.
var min_x:Number = p3.x;
if (p0.x < min_x) min_x = p0.x;
if (p1.x < min_x) min_x = p1.x;
if (p2.x < min_x) min_x = p2.x;
var max_x:Number = p1.x;
if (p2.x > max_x) max_x = p2.x;
if (p3.x > max_x) max_x = p3.x;
if (p0.x > max_x) max_x = p0.x;
var min_y:Number = p0.y;
if (p1.y < min_y) min_y = p1.y;
if (p2.y < min_y) min_y = p2.y;
if (p3.y < min_y) min_y = p3.y;
var max_y:Number = p2.y;
if (p3.y > max_y) max_y = p3.y;
if (p0.y > max_y) max_y = p0.y;
if (p1.y > max_y) max_y = p1.y;
return new Rectangle( min_x, min_y, max_x - min_x, max_y - min_y );
In this framework, particularly in SCALE_NOSYNC_SIZE mode (the default), the width and height are assigned and internally maintained, independently of the scaleX and scaleY values. The internally maintained values are used for performing layout and drawing operations such as backgrounds and borders. In the extremely rare occasion where the actual content needs to be measured, I just use getBounds, such as in the constructor when initializing the "original" size of the clip's content, if it has timeline content, or possibly bitmap methods for masked objects.
These changes have all increased the performance of my framework by an order of magnitude and have virtually eliminated 3rd layout passes, so I'm happy with it. I still wish the Flash runtime would be updated to simply stop rounding these values. -
Rounding errors when plotting rectangles (simple code snippet included)
hi,
I'm having a problem with rounding errors, and am unsure what the best thing to do is..
If you run the snippet of code below, you'll see some rectangles with black lines between them. The range of values covered in the model behind the plot is contiguous, and its only after scaling the trouble begins..
does anyone have suggestions for what to do here? I need this resize behaviour.. I'm unable to change the interval class :)
thanks,
asjf
import java.awt.Color;
import java.awt.Graphics;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingError extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
public RoundingError() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
double scale= getWidth()/(double)WIDTH;
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256), r.nextInt(256), r.nextInt(256)));
int sx= (int) (i.start * scale);
int sy= getHeight()>>2;
int sw= (int) ((i.end-i.start) * scale);
int sh= getHeight()>>1;
g.fillRect(sx,sy,sw,sh);
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingError());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end;
}import java.awt.Color;
import java.awt.Graphics;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingError extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
boolean firstTime = true;
public RoundingError() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
double scale= getWidth()/(double)WIDTH;
int count = 0;
int errorX = 0;
int totalW = 0;
int lastX = 0;
int[] errors = new int[intervals.size()];
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256),
r.nextInt(256), r.nextInt(256)));
int sx= (int) (i.start * scale);
int sy= getHeight()>>2;
int sw= (int) ((i.end-i.start) * scale);
int sh= getHeight()>>1;
if(firstTime) {
totalW += sw;
if(count > 0) {
int error = sx - lastX;
if(error != 0) {
errors[count] = error;
errorX += error;
lastX = sx + sw;
System.out.printf("count = %2d sx = %3d sy = %3d " +
"sw = %3d dh = %3d totalW = %d%n",
count++, sx, sy, sw, sh, totalW);
g.fillRect(sx,sy,sw,sh);
if(firstTime) {
System.out.printf("errorX = %d%n" +
"lastX - totalW = %d%n",
errorX, lastX - totalW);
for(int i = 0; i < errors.length; i++) {
if(errors[i] != 0) {
System.out.printf("errors[%2d] = %d%n", i, errors);
firstTime = false;
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingError());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end;
This seems to work okay.
import java.awt.Color;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import javax.swing.JFrame;
import javax.swing.JPanel;
@SuppressWarnings("serial")
public class RoundingErrorRx extends JPanel {
List<Interval> intervals;
private static final int WIDTH= 25000000;
private static final int INC= 1000000;
boolean firstTime = true;
public RoundingErrorRx() {
setBackground(Color.BLACK);
intervals= new ArrayList<Interval>();
for(int i=0; i<WIDTH; i+=INC) {
intervals.add(new Interval(i, i+INC));
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
Graphics2D g2 = (Graphics2D)g;
double scale= getWidth()/(double)WIDTH;
int count = 0;
double errorX = 0;
double totalW = 0;
double lastX = 0;
double[] errors = new double[intervals.size()];
for(Interval i : intervals) {
Random r= new Random(System.identityHashCode(i));
g.setColor(new Color(r.nextInt(256),
r.nextInt(256), r.nextInt(256)));
double sx= i.start * scale;
double sy= getHeight()>>2;
double sw= (i.end-i.start) * scale;
double sh= getHeight()>>1;
if(firstTime) {
totalW += sw;
if(count > 0) {
double error = sx - lastX;
if(error != 0) {
errors[count] = error;
errorX += error;
lastX = sx + sw;
System.out.printf("count = %2d sx = %5.1f sy = %5.1f " +
"sw = %5.1f dh = %5.1f totalW = %5.1f%n",
count++, sx, sy, sw, sh, totalW);
g2.fill(new java.awt.geom.Rectangle2D.Double(sx,sy,sw,sh));
if(firstTime) {
System.out.printf("errorX = %5.1f lastX - totalW = %5.1f%n",
errorX, lastX - totalW);
for(int i = 0; i < errors.length; i++) {
if(errors[i] != 0) {
System.out.printf("errors[%2d] = %5.1f%n", i, errors);
firstTime = false;
public static void main(String[] args) {
JFrame frame= new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.getContentPane().add(new RoundingErrorRx());
frame.setSize(800,600);
frame.setVisible(true);
class Interval {
final int start, end;
* @param start inclusive bound
* @param end exclusive bound
Interval(int start, int end) {
this.start= start;
this.end= end;
Maybe you are looking for
-
ITunes keeps attempting to download a particular app when I update other apps via phone store
These errors don't happen all the time, but it happens enough that it's getting rather tiresome to work around it. Both of these problems happen when I am requested to update an app on my phone, or I try to download an app from the store on my phone.
-
Hi there, New to Fireworks and just getting to grips with it. I have what is probably a simple problem however it is still driving me crazy. I am rounding the edges of images using the rounded rectangle tool and cutting and pasting to get the result.
-
SQL Developer Data Modeler Viewer - domain datatypes
I'm using the SQL Developer Data Modeler Viewer (that comes free with SQL Developer) to view a model that was created in Oracle SQL Developer Data Modeler (the chargeable product). The colums have all been defined using domain datatypes. I have an XM
-
Package the book with loan and buy permission
Jim, I am able to package the book with buy permission by acs4 webservices. Now i want to package the book with loan permission also at same time so that book could be available for borrow also instead of buy. Please assite me what addition parameter
-
Grandfathered Alltell users and Employee Discounts. Taken away!
I originally had a Alltel plan with a Blackberry, and switched to Verizon when they bought Alltel out I bought a Droid when they first came out. I was told nothing would change, and for the last 2 years nothing did. I work for Citigroup which had a