(CS5.1) Free Transform and Warp tool causing pixelation and quality drop.

Question

When I free transform and warp a layer to lie flat, sometimes I'll get this harsh pixellation at one end of the layer. It doesn't appear while I'm doing the warp, just suddenly coming up when I hit enter to apply the transformation.
So for example, Ill be transforming this grid, which is originally about 9x this big.
So when I''m running the transformation/warp, it looks like this
Not great, but at least most of the lines are still intact on the left and back (the right side is basically fine looking)
But once I hit enter, I get this
And now like, half the lines I did have are gone, and what's left is super pixelated.
Is there an interpolation setting I could be using? Is it something unavoidable from how distorted I make the layer?

Answer 1

It is a general AA problem with mapping texture on surfaces in acute angles. In 3D programs this is solved by a technique called mipmaping, The 3D renderer calculates the degree of distortions and applies blurring filter where it is needed based on the depth of the 3D space to solve this problem. Unfortunately Photoshop is not a 3D renderer and can't eliminate this. You can try manually to reduce it yourself by applying the same transformation on several differently blurred copies of the layer and then using masks made with manual brushing over the problem areas to reduce the artifact.
Here's an example image of a 3D program solving the problem.
and a quick attempt I made to illustrate the problem using your grid after making it more dense with several multiplied layers. the bottom part is the grid masked with different degree of blurring at the problem areas. As you can see this is not an easy task and the problem may be considered as impossible to be fixed nicely. On top of that  as you can see this approach may require constructing grid patterns on layers so that you can blur the eventual horizontal or vertical directions separately.
Click the image to see it larger without the additional distortion of forum scaling