Ethernet interface priority explanation

Question

I'm looking for an explanation behind the solution to a problem I recently faced. My configuration (this is on the latest Xserve w/ 10.5.6):
Ethernet 1
- NAT'd, static IP
The Xserve was not providing NAT services to the local subnet. A linksys router with another static IP was acting as a gateway to the local subnet.
Ethernet 2
- public, static IP sitting behind a T1 router
The problem I had was that initially I had configured Ethernet 1 as the "primary interface" so that the server "looked in" rather than out by default. Consequently, I was unable to ping the external IP on Ethernet 2 from my home server (via and SSH connection). (At one point, I thought I was able to ping the external IP from another device sitting behind my T1 router, but I'm not sure this was ever the case). And yes, I had the firewalls configured to let pings through.
The only solution (and I was able to see this real time by watching my home server attempt to ping the Xserve's external IP) was to make Ethernet 2 the "primary interface". At this point, ping replies started coming back. I have no problem with this solution--I would just like to understand the reasoning behind it.
My theory is that the Xserve was trying to send replies out the local NAT'd interface (since it was primary) even though the initial ping message came from the secondary interface.
Background: my overall goal was to use multiple Xserves to distribute remotely accessible network services filtered through the firewall while still maintaining internal NAT'd interfaces so that the Xserves could be administered via the private, local subnet. DNS is setup so that the NAT'd subnet is a subdomain of the external domain.
Could somebody provide a little insight into this phenomenon so that more of us may better understand how things work?

Answer 1

... that ping reply still should have made it back to the source albeit routed through internal network first and out through the NAT gateway. Is this not the case?
No. Absolutely not.
Say your machine has two interfaces, a real-world IP 234.56.78.9 and a NAT'd address 192.168.1.2.
The NAT interface has a router address set to route through 192.168.1.1. The router has a public interface address of 65.43.2.1
Now, any outgoing traffic flowing through the NAT device gets NAT'd to the 65.43.2.1 address.
So here's the scenario. Remote user pings 234.56.78.9
Mac hears the ping and sends a reply via it's 192.168.1.2 to its default router.
Now, depending on the router make/model one of two things happen. Either:
1) the router is smart enough to realize that the ping reply is in response to a ping request it never saw and drops the packet (it won't send a reply packet to a query that never came in). or
2) the router NATs the traffic to 65.43.2.1 and sends the reply.
In the case of 2, the remote user gets a ping reply from a completely different address than it sent out - it pinged 234.56.78.9 and got a reply from 65.43.2.1
It quite rightly ignores the ping reply as being bogus. It doesn't know your internal network and doesn't expect to get a ping reply from a completely different address than it pinged in the first place.