OS X Firewall isn't firewalling

Question

Mac mini running OS X Server 10.8. I have an application that creates a web server instance on port 8000—I want this to be visible to localhost but not to outside computers.
Turned on OS X Firewall in System preferences, turned off "allow signed apps to receive connections", but the service is still live on port 8000 to outside computers. What am I missing here? Would prefer avoiding ipfw and pf if I can.

Answer 1

This is a comment on why you might, or might not, want to use the built-in Application Firewall.
The firewall blocks incoming network traffic, regardless of origin, on a per-application basis. By default it's off, and when turned on, it allows applications digitally signed by Apple, and only those applications, to listen on the network. It does not block outgoing traffic, nor can it distinguish between different sources of incoming traffic, nor does it filter traffic by content.
No matter how it's configured, the firewall is not, as some imagine, a malware filter. If that's what you expect it to do, forget it. All it will do is bombard you with pointless alerts.
Suppose you enable file sharing, and you allow guest access to certain folders. That means you want people on your local network, but not outsiders, to be able to access those shared folders without having to enter a password. In the default configuration, the firewall will allow that. The router prevents outsiders from accessing the shares, whether the application firewall is on or off. But if your computer is portable and you connect it to an untrusted network such as a public hotspot, the firewall will still allow access to anyone, which is not what you want. It does not protect you in this scenario.
Now suppose you unknowingly install a trojan that steals your data and uploads it to a remote server. The firewall, no matter how it's configured, will not block that outgoing traffic. It does nothing to protect you from that threat.
A more likely scenario: The web browser or the router is compromised by an attacker. The attack redirects all web traffic to a bogus server. The firewall does nothing to protect you from this threat.
Another scenario: You're running a public web server. Your router forwards TCP connection requests on port 80 to your Mac, and the connections are accepted by the built-in web server, which is signed by Apple. The application firewall, still configured as above, allows this to happen. A different attack tries to hijack port 80 and replace the built-in web server. The good news here is that the firewall does protect you; it blocks incoming connections to the malicious server and alerts you. But the bad news is that you've been rooted. The attacker who can do all this can just as easily turn off the firewall, in which case it doesn't protect you after all.
Now suppose you're running a Minecraft server on the local network. It listens on a high-numbered port. You, as administrator, have reconfigured the firewall to pass this traffic. An attacker is able to log in to a standard account on the server. He figures out how to crash Minecraft, or he just waits for you to quit it, and then he binds his own, malicious, Minecraft server to the same port. The firewall blocks his server, and because he's not an administrator, he can't do anything about it. In this scenario, the security is genuine.
Here is a more realistic scenario in which you might have reason to enable the firewall. Your MacBook has sharing services enabled. You want those services to be available to others on a home or office network. When you're on those networks, the firewall should be off. When you move to an untrusted network, you can either turn off all the services, or enable the firewall with a non-default configuration to block them. Blocking is easier: one click instead of several.