Can connect via VPN, but can't access AFP server on same Xserve

Question

Hi:
I've set up our XServe with MacOS X Server 10.5.2 to do AFP and VPN (L2TP only; PPTP is disabled). The XServe is a standalone server, not connected to any other direstory server.
I can connect to the XServe's AFP server from my Mac over our wired and wireless network. The AFP server shows up in the sidebar of Finder windows. So far, so good.
I am able to successfully connect to our network via the VPN with Mac OS X 10.5.2 client (on two different machines) using L2TP through our network's firewall (on a Netopia T1 router; UDF ports 500 and 4500 and IP Protocol 50 and 51 are open) using a shared secret.
But I cannot connect to the XServe itself to use Server Admin or AFP (using afp://server.company.com or afp://xxx.xxx.xxx.xxx via the Go > Connect to Server command).
The error I get while connecting to the 10.5.2 AFP server is Some data in apf://server.mycompany.com could not be read or written (Error Code -36 ). I saw this error associated with a SMB problem in 10.4.x, but SMB is not running.
Other iChat users in my office also do not automatically show up in the Bonjour list when I connect to the network. Other computers on our network do not appear in the sidebar of a Finder window. (I'm told these are to be expected, as Bonjour isn't supported (in the "local area Bonjour" over a WAN link - it's purely a multicast feature on the network in the office, and won't be routed across the VPN link. True?)
Now, here's the odd part. There is a second server (v10.4.11) on our network running AFP. I can connect to it (using afp://server.company.com via the Go > Connect to Server command) and mount its various sharepoints via the VPN.
The only thing I see in the VPN log that seems amiss is this (but I have no idea what it means):
Tue Mar 11 23:09:27 2008 : Unsupported protocol 0x8057 received
--Both the 10.5.2 and the 10.4.11 servers have DNS properly configured (though our ISP; we're not running our own DNS).
--Both servers and the client have public IP addresses and have the same subnet mask. Network Utility confirms this while connected to the VPN.
--NAT is not running. The ISP is responding with public IPs for the servers.
--The firewall for the 10.5.2 server is not running (but will be once I get this all working).
--The IP address range for the VPN server doesn't overlap our DHCP pool (which also currently uses public IP addresses).
--Any user can access any service.
--No network routing definitions have been set up.
--In essence, I've followed the steps on Pages 141-142 of the Network Services Admin Guide.
One other note: After I connect, the Network Preferences > VPN > Advanced > TCP/IP window shows the IP address for the client just fine (assigned from the VPN pool), but lists the router as having the IP address of the XServe (rather than the router on the network). Is that normal?
I'm hoping I don't need to have the XServe run DNS as an internal LAN DNS server.... And I'm not sure why I would have to if I can already successfully connect to the 10.4.11 AFP server .
What simple step am I missing?
TIA,
mm

Answer 1

"I am able to successfully connect to our network via the VPN with Mac OS X 10.5.2 client (on two different machines) using L2TP through our network's firewall (on a Netopia T1 router; UDF ports 500 and 4500 and IP Protocol 50 and 51 are open) using a shared secret."
I suspect you mean UDP ports and you might need UDP port 1701 open too.
You only need IP protocol 50 (ESP), protocol 51 (AH) isn't used. And ESP is only used when client and server isn't behind NAT (when NAT is used only the UDP ports are used).
"Unsupported protocol 0x8057 received"
This is usually seen when you can't get GRE through but since you don't use PPTP I can't be sure why this is registered in the logs. Sometimes when connecting using PPTP you have to disconnect and then reconnect for everything to work - you might try this for L2TP too.
But if you already can reach services on any LAN nodes through the VPN I wouldn't bother with it.
As you have a firewall in front of the server you need a second alias IP on the server that you can use to get at the services running on the server through the VPN. The firewall blocks all ports protocols not opened - that's why you can't use the server main IP even if the VPN is up.
The netmask is used by all nodes to determine how big your subnet is: what part of the IP number is the network number and what range the node number is in => really: should traffic be directed to a node on the same LAN or sent directly to the gw/router for forwarding.
What you can't do is connect from a NATed network to another NATed network that both are using the same network number. (That's why people should stay away from using the "default" 192.168.0.0/24 and 192.168.1.0/24 networks for VPN server LANs).
Try your settings at http://www.jodies.de/ipcalc to see what I mean.
"...lists the router as having the IP address of the XServe (rather than the router on the network). Is that normal?"
Yes. The VPN server is the VPN gw/router.
"The firewall for the 10.5.2 server is not running (but will be once I get this all working)."
If you already have a firewall in front of your servers that is a bit redundant.
"--No network routing definitions have been set up."
"I'm hoping I don't need to have the XServe run DNS as an internal LAN DNS server"
You need routing definitions if you want to setup a split tunnel VPN or all traffic is routed through the VPN when connected. The VPN becomes the default gw.
Without ipforwarding ON in the server you can only reach nodes on the server LAN - not Internet.
DNS is needed for your servers forward and reverse names/IPs for advanced services but doesn't need to run in any of your own servers.
If you decide to do a split tunnel VPN config (adding public and private routing definitions) a reachable DNS IP for VPN clients (in VPN config on server) is needed for VPN clients or they can't use names to find anything. To reach this DNS IP if public/not on your server LAN, you need your server to forward IP DNS lookups and have a routing definition for it.
A split tunnel VPN only send traffic for your server LAN through the VPN and all other traffic directly to the local gw/router (Internet).