I don't get the Mail service to work - tried everything!

Question

Hey there,
I am frustrated. It's day 4 that I try to get the mail service on my machine running. Maybe someone has sme hints?
The setup:
- Mac Mini mid 2010
- OS X Mavericks and Server 3.0.1
- static IP (no Firewall or anything else, absolute in the wild)
- registered domain-name
- DNS A-Record: server.mydomain.org -> static IP
- DNS MX-Record: domain, *.domain -> server.mydomain.org; server.mydomain.org -> static IP of server
Server Configuration:
- Server.app shows everything is fine.
- ssh enabled for one user (works as a charme)
- Service-Data is on another partition than Mavericks
- Certificate: self-signed intermediate OD-Certificate (might be problematic?)
- users: one local user and one network-user (services-only)
- groups: workgroup, local network-group (added by the server.app itself)
Services:
- Websites: configured with 12 virtual-hosts some ssl (intermediate certificate) some on port 80, php enabled (All websites work perfect, no problems at all!)
- OpenDirectory: 1 Master on server.mydomain.org
- Mail:
-- Service-Bubble is green, says available under server.mydomain.org
-- Preferences: Mail available for mydomain.org (no other virtual domains)
-- authentification: hooked everything...(any method)
-- no relais
-- no MB-restriction for users
-- Filter: Virus and Junk
I can telnet on port 110 - feedback: dovecot ready!
I can telnet on port 995 - feedback: works... escape characeter is ' ^'.
I can telnet on port 143 - feedback: dovecot ready!
I can telnet on port 993 - feedback: works... escape characeter is ' ^'.
telnet on port 25 - unable to connect, operation timed out
telnet on port 465 - unable to connect, operation timed out
telnet on port 587 - -feedback works: 220 server.pscience.de ESMTP Postfix
Seems like having problem on the smtp standard ports... but why?
When I now try to connect a client with the server (adding an OS X server Account in the System Preferences) it recognizes the server and lists its services as expected. But when Iopen Mail, the new account shows the litlle lightning and says it can't connect.
By the way: Notes, Calendar, Reminders work fine!
Has anyone an Idea what it could be?
Do I need some additional DNS-stuff?
If you need some further informations, please let me know.

Answer 1

If TCP port 25 isn't open, then SMTP services won't work.  That's the port that all SMTP servers use to communicate amongst themselves.    (Sure, you can change the port your SMTP server is listening on, but you'd also have to change the port processing on all of the other SMTP servers around — and if you did manage to implement that change everywhere, the local network folks would likely just block the new port for the same reasons they blocked the old port.)
If you have a public static IP address and have correct forward and reverse translations — host domain to IP address, and IP address to host domain name — then you don't need and very likely don't want to run your own DNS server. 
More to help you learn how these pieces fit together and not something I'd expect you would need given you have a public static IP and public DNS here is a detailed DNS server configuration article for OS X Server; enable Show All Records and it applies to Server.app on 10.7 and later.  If you have questions after reading that, I can certainly answer them and also update the article to try to reduce the confusion or answer the question.
Your host didn't have valid reverse DNS, so you'll want to get that cleared up, and you'll want to get the port blocks cleared, or work with the local network folks to set up a relay through one of the existing mail servers.