Problem with JPanel's mouse listener!

Question

I am developing a Windows Explorer-like program. I have an JPanel and added JLabels to that panel to reprensent the folders. I think, I kind of have to add mouse listener to JPanel to interact with mouse clicks, gaining and losing focus of JLabels etc. instead of adding every JLabel to mouse listener. So when I added to JPanel a mouse listener, it didn't work how I had expected. When I clicked on labels mouse click events didn't fire but when I clicked somewhere else in the panel it did fire. Is this a bug or am I using mouse listener incorrectly??
Thank for advance :)
Here is my JPanel's mouse listener ->
public void mouseClicked(MouseEvent e) {
int x = e.getX();
int y = e.getY();
System.out.println("Mouse Clicked");
Component c = this.getComponentAt(x,y);
if(c instanceof JLabel){
JLabel label = (JLabel) c;
label.setBackground(Color.LIGHT_GRAY);
}

Answer 1

My main problem is as in windows explorer (if CTRL or SHIFT not pressed) there is only one selected folder (in my case JLabel) and transfering "selection" to one label to another might need lots of extra code.. So I thought using JPanel's mouse listener can overcome this handling problem. But if you are saying, this is the way it has to be done, then so be it :D :D