Cannot Connect to VPN

Question

I'm just about ready to ragequit for the day. I've been pouring through dozens of support pages, youtube videos, tutorials. The lack of true documentation on problems like this has me considering a start-up business that specifically deals with these frustrations. Clearly I could make millions!
I will detail everything about this problem as best as I can, to avoid confusion later with questions:
Here's what I have for hardware:
1) A Public IP Address. We'll just call it X.X.X.X.
2) A D-Link DI-604 router (yes they DO support VPN services, with a router address of 192.168.1.254.
     This router is running Firmware Version 3.53, the last firmware released for it on Wed, 18 Apr 2007 (YES I AM AWARE THE ROUTER IS OLD, DEFLECTING THIS ISSUE BY TELLING ME TO GET A NEWER ROUTER WITHOUT FIRST READING THROUGH EVERYTHING BELOW IS NOT A HELPFUL CONTRIBUTION TO THE PROBLEM, D-LINK HAS CONFIRMED THIS ROUTER SUPPORTS VPN PASSTHROUGHS).
3) A Mac Mini Server running 10.6.8, router address of 192.168.1.10.
Here are the ports that I've allowed through the router, pointed directly at 192.168.1.10 (aka my Server):
UDP Port 500
UDP Port 1701
UDP Port 4500
TCP & UDP Port 1723
Here is how I have the VPN Service configured on my Server:
L2TP is Enabled.
Starting IP address range of 192.168.1.180
Ending IP Address range of 192.168.1.189
PPP Authentication: Directory Service with Authentication set to MS-CHAPv2
IPSec Authentication is set to Shared Secret, let's just say the secret is "derp" without quotes.
PPTP is Disabled.
Client Information:
DNS Servers point to my router: 192.168.1.254
Search Domains is empty.
Network Routing Definition is empty.
Logging:
Verbose logging is enabled.
VPN Service is: Running.
Server User Information
Access to VPN Services:
Allow only users and groups below:
(I have users dedicated to this, but for the sake of this topic let's just say one of them is "misterderp" without quotes)
The Hardware I'm Using to Connect to the VPN Server:
I have a Macbook Pro running 10.6.8, another laptop running Windows XP Professional Service Pack 3, and another laptop running Windows 7 Home Premium 64-bit Service Pack 1. All 3 laptops acquire an IP Address via DHCP from the Router (192.168.1.254). Below is what happens when I try to set up a VPN connection on all 3 machines:
Computer #1: MacBook Pro, running 10.6.8
Settings: (this is in System Preferences > Network, by the way):
New VPN Connection
Server Address: X.X.X.X. (this is our Public IP Address)
Account Name: misterderp (this is the account who has access granted to use VPN)
Authentication Settings > User Authentication:
Password: (password given to misterderp from server)
Authentication Settings > Machine Authentication:
Shared Secret: derp (as specified in the L2TP tab of the VPN Service on the Server)
At this point I will try to connect. I receive the following error message:
=========
VPN Connection
The L2TP-VPN server did not respond. Try reconnecting. If the problem persists, verify your settings and contact your Administrator.
=========
Computer #2: Laptop, running Windows XP Professional Service Pack 3
Settings: (this is in Control Panel > Network Connections, by the way):
Add a New Connection
VPN Server Selection: X.X.X.X. (this is our Public IP Address)
Smart Card
Do not use my Smart Card
New VPN Connection Properties
General Tab:
Host Name: X.X.X.X.
Security Tab:
Security Options:
     Advanced Custom Settings
          Data Encryption: Require encryption (disconnect if server declines)
          Allow These Protocols: Microsoft CHAP Version 2 (MS-CHAP v2)
     IPSec Settings
          Use Preshared key for authentication: derp (as specified in the L2TP tab of the VPN Service on the Server)
At this point I will try to connect. I am using the Account Name misterderp, and the password given to this account from the server. I receive the following error message:
=========
Error 800: Unable to establish the VPN connection. The VPN server may be unreachable, or security parameters may not be configured properly for this connection.
=========
Computer #3: Laptop, running Windows 7 Home Premium x64 Service Pack 1
Settings: (this is in Control Panel > Network and Internet > Network and Sharing Center, by the way):
Set Up a Connection or Network:
     Connect to a workplace
     Use my Internet Connection
Internet Address: X.X.X.X. (this is our Public IP Address)
Type your username and password:
     User name: misterderp (specified on the Server to have VPN access)
     Password: password given to the misterderp account
VPN Connection Properties:
Security Tab:
Type of VPN: Layer 2 Tunneling Protocol with IPsec (L2TP/IPSec)
     Advanced Settings: Use preshared key for authentication: derp (as specified in the L2TP tab of the VPN Service on the Server)
Data Encryption: Require encryption (disconnect if server declines)
Allow these protocols: Microsoft CHAP Version 2 (MS-CHAP v2)
At this point I will try to connect. The window hangs at "Connecting to X.X.X.X. using "WAN Miniport (L2TP)"". After about 30 seconds, I receive the following error message:
=========
Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during intiial negotiations with the remote computer.
=========
So there you have it, 3 sources of unintelligible frustration.
You're probably wondering, HEY, WHAT ABOUT THE LOG ON YOUR SERVER ADMIN PAGE?
I've been looking at the log, and there's a whole lot of nothing. The only thing I have is this:
#Start-Date: 2012-02-17 14:01:46 CST
#Fields: date time s-comment
2012-02-17 14:01:46 CSTLoading plugin /System/Library/Extensions/L2TP.ppp
2012-02-17 14:01:46 CSTListening for connections. . .
So the Server's not getting ANYTHING, let alone spit out errors.
Now you might be wondering, ALRIGHT, WHAT ABOUT VPN-ING WITHIN YOUR OWN NETWORK, THAT PROBABLY WORKS RIGHT?
Yes it does. Without any question, my MacBook Pro will connect to the VPN Service so long as I'm connecting DIRECTLY to the Server through its local IP address, and not trying to reach it through a public IP address that's forwarding the requests through the ports I've assigned.
At this point I am at a complete loss. I believe I have done everything correctly, but it would appear that my router isn't playing nice with VPN requests. If there is/are any other ports I should be turning on to point to my server, I would like to know what ones those are.
If there are any tweaks or additional settings I should know about for the Windows computers (especially Windows 7), I would like to know what those are.
If at the end of this post that you've just read and know with irrefutable proof or a reasonably educated decision that this router magically will not serve my VPN needs AT ALL, I would like to know a reasonably-priced alternative, preferably something that is not an Extreme Base Station, Time Capsule, or other product because my ISP hates Apple-based routers for a reason even they do not understand
If at the end of htis post that you've just read and know with irrefutable proof or a reasonably educated decision that I would be better off attempting this with PPTP on this D-Link Router, and if you know how to set the correct settings on Server Admin, forward the correct ports on the router I have, I would like to know that
Thank you for reading this wall of text, anyone willing to help me with this is an amazing person

Answer 1

Hi Esther,
After 3 months, I was finally able to revisit this issue. Here are the results of my nmap TCP test using your code:
Gerchak$ nmap -T5 XX.XX.XXX.XX
Starting Nmap 6.00 ( http://nmap.org ) at 2012-05-22 17:50 CDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 1.68 seconds
And here are the results of my UDP test using your code:
Gerchak$ sudo nmap -sU -T5 -p 500,1701,4500,9999 XX.XX.XXX.XX
Starting Nmap 6.00 ( http://nmap.org ) at 2012-05-22 17:51 CDT
Nmap scan report for xxx-bb-xxx-3-ws-6.xxx.xxxxxxxxxxxx.net (XX.XX.XXX.XX)
Host is up (0.096s latency).
PORT     STATE         SERVICE
500/udp  open          isakmp
1701/udp open|filtered L2TP
4500/udp open|filtered nat-t-ike
9999/udp open|filtered distinct
Obviously there's something wrong since the TCP scan registered a major problem, so I redid the scan per nmap's recommendations:
Gerchak$ nmap -Pn XX.XX.XXX.XX
Starting Nmap 6.00 ( http://nmap.org ) at 2012-05-22 17:55 CDT
Nmap scan report for xxx-bb-xxx-3-ws-6.xxx.xxxxxxxxxxxx.net (XX.XX.XXX.XX)
Host is up (0.14s latency).
Not shown: 990 filtered ports
PORT     STATE  SERVICE
22/tcp   closed ssh
427/tcp  closed svrloc
500/tcp  closed isakmp
548/tcp  open   afp
1723/tcp closed pptp
5002/tcp closed rfe
5003/tcp open   fm
5004/tcp closed avt-profile-1
5222/tcp open   xmpp-client
8080/tcp open   http-proxy
So, where should I go from here? 1723 is closed off yet my router says it's open. I'm just about ready to throw my hands up in the air and just purchase a different router.