Using Refine edge with Quick Selection......

Question

I think I'm doing something wrong, because I can do a quick selection, but when I hit the "Refine Edge" button and make a change to the selection, my change doesn't "take"...
I have an image, one layer (background), absolutely normal, boring image (for example, a white swan sitting in the blue water).  I go in with Quick Select and select PART OF
the swan.  I have areas that aren't selected, and areas that are selected and shouldn't be.  So, I need to refine my selection.  I subtract and add bits of selection with the
Quick Select tool, but leave a big piece of the swan's body NOT selected, and a bit of water SELECTED.  So, I need to go in with a tool and refine the edge of my selection...
SO, I hit "Refine Edge" from the tool bar and select "Overlay" for my view mode.  I can now see what's selected, and know what I'd like to change.  And, I have a red overlay
so I have a visual guide to changing things....
Go to the "Refine Radius Tool" and paint over an area I want ADDED to my selection (a piece of swan immediately adjacent to my original selection)...  It runs red, like I'd expect. 
I release the mouse and a second later, the red goes away, but the selection has NOT been changed.  I've tried this directly from the Quick Select tool.  I've double-clicked the
background layer and changed it so it's not background any more - made no difference.  I've created a layer mask on the no-longer background layer (the mask contains my
Quick Selection just fine).  I've then used the Mask panel to specify "Mask Edge" and again tried the "Refine Radius Tool" to add an area to my selection.  Again, it turns red
while selecting, then stops being red when I let go, and the area I selected is NOT added to the selection...
I watched a presentation by a Photoshop person who DEMONSTRATED exactly what I'm doing, and it worked.  I looked at Martin Evening's CS5 book, and as near as I can tell,
I did exactly what he described...  He got a selection, added to the selection, and it worked fine... 
I figure I've got something configured wrong, or something else goofed up.  What am I missing here? 

Answer 1

It's not my intention (intent?) to beat up on the "Refine Edge".  I figured when I couldn't get the same pristine results as all the demos that I was doing something wrong, and kept watching more people doing this 'cause I figured I was doing something dumb...
Mostly, I've been trying this on different images over the last couple weeks just to see if I could figure out how to get it to work.
From the responses in here I suspect I'm doing it correctly, but the images I'm trying it on just don't lend themselves to perfect edges without some additional work.....  The Quick Select and a bit of edge cleanup is still faster than the "good old days", so its definitely a step forward. 
As far as the swans, yes, no matter what I did, I was getting a lot of extraneous selections and wonky edges and stuff.  One of these days I'll try pulling an eagle or a great egret out of a clear sky and see how that goes.  I suspect it'll work a lot better... 
I only fiddled with the rodeo image 'cause I figured it was "like" hair.  But my background is pretty busy.  With a little additional work I DID get a very, very good selection, and was able to soften the background, and do a gradient so the ground under the horse is sharp, gradually blurring as it moves to the grandstands... 
Thanks for all the replies...