A unix-like security question?

Question

Something happened on my Mac the other day, that kind of scared me. First, a little bit about my home setup.
I have a small smtp mailserver running on my desktop Mac for family members - a big whoppin' five accounts. Three of the five of us live at home, two don't and remotely access email via SSL-enabled imap and SSL-enabled smtp. There is a laptop computer at home that accesses the imap and smtp servers on ports 143 and a non-standard smtp port. Traditionally, it has been used at home only, so I don't require SSL because it runs inside a WPA-protected wireless channel. Its mailserver info, set up in Mail.app, uses {computerName}.local as its mailservers. So, no access from outside my local WLAN. I also get my foreign mail, virus-scanned and spam-assassinated, from an MX agent that downloads that traffic to me on that same non-standard smtp port. For what it's worth, outbound smtp from the home mailserver is via port 25.
On the desktop Mac, I also have ssh running, but on a non-standard port, and in sshd_config, I specify protocol 2 only, root login disabled, no password/no PAM authentication, only DSA public key authentication. In NetInfo Manger, I keep the root account disabled.
On the Mac, in System Prefs' Sharing firewall, I have the non-standard smtp port, imap, imaps, non-standard port ssh, ard and vnc (so I can run CotVNC from the laptop at home), and afp (also for the laptop at home) open for uinvited traffic. Also ntp (probably don't need that since I'm not running a time server), and dns (for reasons discussed below). On my DSL router, I only have the non-standard smtp and ssh ports, and the imapS ports open. (When outside my home WLAN, on a foreign network, I port-forward VNC and afp over ssh if I want to do one of those things)
So anyways, for the benefit of the laptop, I enabled DNS on the desktop, so that I could change the laptop's Mail.app's accounts' preferences to point to the same imaps and smtp server using my external WAN host name, whether it was inside or outside my home LAN (inside the home LAN, the laptop couldn't resolve my external domain name, and outside the home LAN, {computerName}.local was not routable). But by enabling DNS, I could reference my external host name to my 192.168.x.x IP address, and the laptop would find the server inside the home LAN, as well as find it outside the home LAN (by virtue of services like DynDNS and NoIP DNSs resolving it to my ISP-assigned dynamic external WAN IP). For what its worth, yes, the laptop's mail preferences enabled SSL for both smtp and imaps, so SSL would be used even inside the WPA-protected channel, just as my users that don't live at home have SSL enabled as they network .
Now for the scary part: the other day, while at home and with the laptop affiliated to my home wireless (WPA-protected) LAN, I ssh'ed into my desktop computer. Either the ssh connection or the desktop computer was running dirt slow. For some reason, I decided to do a tcpdump, and I saw all kinds of traffic going out to hosts all over the world.
After the fact, I think it was just my DNS talking to the sixteen or so root servers, although none of the tcpdump entries used names like "a.root-servers.net" -- there was stuff with an army.mil, a nasa.gov, etc. I think I remember seeing something with a "umd.edu" in it, which there is a commented entry in /var/named/named.ca that has that has a "umd.edu" in it, so that's why I am thinking that my DNS was just gabbing with a bunch of root servers. Not sure why it was gabbing with them since I can't think of any reason why it would have been trying to do name resolutions or anything. At the time, seeing all these packets being initiated by my computer and being sent worldwide freaked me out.
But what really freaked me out is when I control-C'ed the tcpdump and did a "users" to see who or what might be generating them and saw my username and ... root! Repeating the "users" command a few times more, and it still showed "root" as one of the active users. I immediately ran to the computer and pulled the DSL plug out of the wall, and tried to figure out what was going on. I've got HenWen running, and didn't see anything outside of the usual unicast ARP warnings. After thinking that it might be DNS itself, I disabled DNS just to see what sort of traffic I would see in a tcpdump. Just local subnet broadcasts and arp requests. I have not re-enabled DNS yet.
And the story gets better: a day or two later, I glance at my System Preferences firewall settings, and the firewall was OFF! Fortunately, the DSL modem's firewall was still on, only allowing uninvited inbound imaps, smtp, and ssh traffic. I don't remember ever turning off the desktop's firewall, and no one else uses that computer -- they all hop on the laptop, plus they don't know the admin password anyways. So that was a little freaky, too, but, I'll assume for now that I must have inadvertently turned it off when I was doing something, and never turned it back on.
My immediate question is, if you have DNS turned on, would it ever do anything as root, and hence, show up as an active user in response to a "users" command? And not that there were any (/var/cron/tabs) cron jobs scheduled to be active at that particular time, but if there were a /var/cron/tabs/root job actively running, would root then show up as an active user in response to a "users" command?
Signed,
Scared!

Answer 1

Hi J.V.
   First, I have to say that yours is an impressive setup. If you're not a sysadmin, you certainly could be. Also, you have a knowledge of much of this that surpasses mine so I may be of no help. However, I do use the "who" command to see if anyone has broken in and I've never seen the root user listed.
   There are doubtless more processes running as root on a typical system than those running as the user logged into the GUI. However, none of those root processes are the result of a login. I believe that the "users" and "who" commands only report users that are logged in. I don't see the root user with the "who" command even if I create a root shell with sudo. Although I don't know this for a fact, I don't believe that it should be possible to see the root user with the "who" command if the root account is disabled in NetInfo. By the way, I recommend the "who -u" command to the "users" command as it provides quite a bit more information. When I login to my machine via SSH, the domain name of the remote host is included in the output of the "who" command.
   There was a situation on Panther where the root user could be listed in NetInfo Manager as disabled when it was actually enabled. I don't believe that is possible in Tiger but you can check with the command,
nicl . -read /users/root
If the password is only a single asterisk or ideally the authentication_authority string contains ";DisabledUser;", the root user should really be disabled.
   I can see that you're quite knowledgeable about networking and comfortable with tools that examine packets. However, there are methods of intrusion detections that aren't directly network related. They may be of use in your situation.
   The simplest is the /var/log/secure.log. Acquisition of root privileges via sudo does show up in this log but there may be enough information about the circumstances to determine which uses of root privileges are normal.
   A more complex method is process accounting. This records every command executed on the system. It provides information similar to the "who" command but doesn't provide the arguments that were used in executing the command nor any process IDs. If you actually do discover unusual activity real time, a full dump of process information with the "ps" command can provide a useful complement to the information recorded by process accounting. You can turn on process account simply by creating a /var/account/acct file and executing:
sudo accton /private/var/account/acct
You can read the result with the command:
sudo lastcomm
I should warn you that process accounting shouldn't be left on without developing a log rotation mechanism for the above file as it can grow large rather quickly.
   The mechanism for doing for system what Snort does for the network is Security Auditing. This system was developed by Sun and distributed by Apple for OS X in their Common Criteria Tools. To understand the the output of auditing and to customize the configuration requires at least as much study as mastering snort. It can also output a lot of information. However, like Snort, it is the ultimate at what it does.
   There is a minor rootkit for Mac OS X named Opener. Unlike a "real" rootkit it is easy to detect if you know what you're looking for. In reported versions, there is a StartupItem in /Library/StartupItems named "opener". I would check that directory for any unusual StartupItem.
Gary
~~~~
   Adam was but human--this explains it all. He did not want
   the apple for the apple's sake, he wanted it only because
   it was forbidden. The mistake was in not forbidding the
   serpent; then he would have eaten the serpent.
         -- Mark Twain, "Pudd'nhead Wilson's Calendar"