Explanation of Leopard NetBIOS (Windows Sharing) Wackyness With Solutions!

Question

Kick back and grab some coffee. This could be a while. I agree, this fly's in the face of the 'It Just Works' we're all used to, but if you want to get things working, understanding the information below is critical.
I dazed off writing this up, so if I don't make sense someplace, please comment and I'll clarify.
Summary
Leopard does some 'Interesting' things with how it handles announcing and learning about Windows services that may make it feel different than previous versions. Understanding how it does things can help in resolving your Windows related networking issues.
Please do not reply to this post unless it has helped to solve your issue, you have something useful to add (such as where I might be incorrect), or you are posting the output of diagnostics commands. Any other posts have already been made in 1,000 other threads and will only cause clutter.
This post will first define many of the terms used, describe the basics of simple Windows networking with NetBIOS/SMB, explain how Leopard makes things a big goofy, and provide step by step solutions to these issues. Troubleshooting steps are also listed for cases in which things do not work.
Definitions
NetBIOS/NBT (NetBIOS over TCP) - This is a legacy windows protocol that allows for systems to share information about their presence on the network and what they have to offer for resources.
SMB/CIFS - This standards for Server Message Block / Common Internet File System and is just a newer way of doing the same old stuff.
Browse List - This is a list of systems and services advertised over the network that describes who you are, what you are, and what you have to give.
Workgroup - This is a logical grouping of systems into browse lists.
Master Browser - This is a single system elected on the network to be the official holder of the browse list. If enough systems are on the network, there may be backups and backup of backups. Your systems place in this election is determined by your 'os level' and other information.
Domain Master - This very well may be the same system as the Master Browser, but maintains information regarding workgroups other than its own.
WINS - This stands for Windows Internet Naming Service and is a client/server protocol used in larger Windows network environments to share workgroup information (browse lists) between separate locations.
DNS - This stands for Domain Name Services and is used to resolve names (such as 'bob.apple.com') to an IP address (such as 123.123.123.1). It can also maintain 'Special' generic records used to locate special types of systems in your network.
Unicast - A 'unicast' is a packet sent directly from one node on a network to another.
Broadcast - A 'broadcast' is a packet that reaches all nodes of a local area network.
Multicast - A 'multicast' is a packet that goes only to systems that subscribe to a specific multicast address. In a single segment local area network, this typically hits the same number of hosts as a broadcast. However, in larger networks, it can be more restricted. Multicasts are be sent from network segment to network segment, but only if properly configured.
Shell - This is the text mode command interpreter that is available within OS X. You can enter this mode by searching 'terminal' in spotlight. Many of the graphical configurations in OS X really modify text files that you can also view and modify through the shell.
Privileged Access - OS X is a multi-user operating system and as such has separates roles and functions from one user to another. Some commands require you to grant yourself additional rights than you normally have. This is normally seen when you install software and get prompted for your username and password. In the shell, you will only be prompted for your password. Privileged access is typically achieved by starting a command with 'sudo' or 'super-user do'.
_Windows Networking_
This description is no where near complete nor do I guarantee that it is 100% accurate. Most of this is from experience and I will provide links to additional information where it isn't. Advanced topics will not be covered - only what I feel is the minimum knowledge to understand what is happening, how to know if something is going wrong, and how you might fix it.
In a small network, such as a home or small office environment, there is a need to share files, printers, and other services between systems on a network. It's why networks were installed in the first place. In order to do this, four main things are required (I'm sure you can come up with more).
1. Something to Share (which we'll just call the 'Share')
2. Users that have permissions to access the Share
3. Something to announce to the rest of the world that you have something to share
4. A method to connect to the share
If you're reading this because Leopard broke your Windows networking, you're already familiar with #1 and #2 and somewhat annoyed at #3. You may have even jumped to #4 and gained access to your resources by using Option-K in the finder and pointing at the device with smb://x.x.x.x/, although smb://hostname/ may not be working.
This post will assume that you have already setup a windows share and know the username/password on the Windows machine that you will be connecting to.
So that leaves 'Something to announce to the rest of the world that you have something to share' and 'a method to connect to the share'.
That Something is known as the NetBIOS browser service.
In basic IP networking, three primary things allow two systems to talk to each other.
The first is the source IP address. This is the network address of the device that wants to connect to another device or the device that is announcing its services to the world.
The second is the destination IP address. This is the network address of the device or devices that you want to connect (or get information) to. The destination may be a unicast (single host), multicast (subscribed hosts), or broadcast (all hosts).
And finally, the third is the Port number. This is a number between 0 and 65535 that lets the computer know what application, such as Safari, iTunes, etc., that is to receive the data in the packets that were sent.
To share things with each other using Windows protocols, we're primarily concerned about ports udp/137 and tcp/445. Other ports may be used depending on what version of windows you are connecting to and what you are trying to do with it.
Possible destinations are the broadcast address (such as 192.168.15.255) or a unicast address of a system that has something to share (such as 192.168.15.100).
We'll go over how to know what your system is talking to in the diagnostics section of this post.
When a system boots up on the network, it will announce itself and what workgroup it is part of with a broadcast to udp port 137. All systems will see this broadcast and add this information to their local browse cache.
As long as there is already a master browser on the network, things will be fairly quiet at this point as systems that have been running for more than 12 minutes will only broadcast this information every 12 minutes! This means that if you just joined the network, it my be 12 minutes before you see another system.
You, on the other hand, being a system that just turned on, will send your information:
1) When you first come on the network
2) At 4 minutes
3) At 8 minutes
4) At 12 minutes ... and then every 12 minutes after.
This is to make sure that if packets are lost (broadcasts are udp and NOT guaranteed to be delivered) that you'll be seen within this 12 minute interval.
This can be a problem if you're trying to figure out why something doesn't work because you have to wait for 12 minutes to see if your change actually had any effect on solving your problem. If you don't wait this full 12 minutes, you very well may have missed out on seeing the other systems broadcast.
_Finding Systems on the Network_
Most of us are already familiar with a protocol called DNS. It's how you probably got to this website. You give the computer a friendly name, such as discussions.apple.com, and your computer hands it off to its configured DNS server. That DNS server then looks to see if it has the IP address, and if not, queries a root domain server to find out who does. The root domain server sends you off to apple's DNS servers who finally give you an 'authoritative' response. Your configured DNS server then saves that information for its configured or permitted time to live (ttl) and gives you the IP address for the name you just asked for.
You can also do this in reverse (hand DNS an IP address and make it give you the hostname). This is the topic of a number of other threads causing 30, 60, 90 (or some other multiple of 30) second hangs when you try to bring up a web page or before a webpage finishes loading. You can find the details of this issue and solutions in other threads (or post here and I'll write something up on that too).
As you probably guessed, there are other ways to turn names into IP addresses as well and NetBIOS has one as well.
If you recall, when your PC booted up, it broadcast its name, workgroup, and IP address out for everyone to see. The master browser recorded this information in its browse list and therefore has all the information necessary to turn names of systems on the network to IP addresses.
So let's say you want to connect to 'meatwad', which is a local windows system. You type in 'smb://meatwad/' and wait for the system to show up.
In the background, a broadcast just went out on the wire on port 137 asking for meatwad. The master browser responds back saying that 'meatwad' is at 192.168.15.100. This information is given to the smbclient and up pops the remote system.
This isn't working for you? You say that if you do 'smb://192.168.15.100/' that things work?
Well then, there's only a few things that could be wrong.
1) 'Meatwad' never registered with the master browser.
2) Your system doesn't know who the master browser is.
3) There isn't a master browser!
Without going into too many details, if you follow the steps listed in the solutions section below, your resolution should work.
_Leopard Wackeyness_
Leopard changed a few things that make it not quite so obvious that you are sharing files or services with Windows. In addition, it's even more confusing in that even if you don't have anything to share, in order to see other peoples' shares (without connecting to them directly), you need to enable file sharing yourself - and just not share anything.
On top of this, the new application based firewall can be confusing to people.
In addition, the behavior of Leopard when it comes to the master browser role and what workgroup you are in can also cause you headaches.
Here are some examples:
1) You enable File Sharing, click Options, and know for certain that SMB is checked. Yet after 20 minutes, you still don't see anyone and can't find the master browser.
Why? Because unless you have the firewall set to 'Allow All Incoming Connections' or 'Set Access for Specific Services and Applications', you are tossing every one of those broadcasts we talked about that are so critical (and only come in every 12 minutes!) out the window.
How do you know you're doing this?
In the firewall tab, click on Advanced and 'Open Log'. Do you see that "Nov 30 08:40:12 Err Firewall[49]: Deny nmblookup data in from 192.168.15.100:137 uid = 0 proto=17"? Yeah - that's just the packet you were looking for. And guess what, you won't see it again for a while (unless you go asking for it).
2) You had everything working, but you disabled file sharing because you were going to roam off network and didn't want to go blabbing about your system on the wire. You come back home and now things don't work.
Could be a few things going on here. One that is simple and one that is just plain wacky.
When you uncheck File Sharing and re-Check it, guess what? Leopard has just decided it doesn't want to talk to Windows anymore. You need to go back into options and re-check the SMB box EVEN IF ALL YOU WANT TO DO IS SEE OTHER PEOPLE.
Why? Without this box checked, your computer isn't running the necessary services to hear the broadcasts on port 137 that you just made sure in the firewall could get through. And if your computer gets a packet for a socket (port) that it doesn't have an application bound to, guess what, it gets thrown away too.
You can see this if you do a 'sudo tcpdump -nvvXi en0 -s 1500 broadcast or icmp'. Note: Change en0 to en1 if you're using Airport.
Here's a similar example where 192.168.15.109 is the Windows system asking 192.168.15.53, an OS X Leopard system, for info about its shares. This failed.
bash-3.2# tcpdump -ni en1 host 192.168.15.109
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes
23:02:49.458342 ARP, Request who-has 192.168.15.53 tell 192.168.15.109, length 46
23:02:49.458475 ARP, Reply 192.168.15.53 is-at 00:1c:b3:7c:a3:18, length 28
23:02:49.459573 IP 192.168.15.109.137 > 192.168.15.53.137: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:02:49.459656 IP 192.168.15.53 > 192.168.15.109: ICMP 192.168.15.53 udp port 137 unreachable, length 36
23:02:50.969783 IP 192.168.15.109.137 > 192.168.15.53.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
23:02:50.969874 IP 192.168.15.53 > 192.168.15.109: ICMP 192.168.15.53 udp port 137 unreachable, length 36
23:02:52.482892 IP 192.168.15.109.137 > 192.168.15.53.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
23:02:52.483023 IP 192.168.15.53 > 192.168.15.109: ICMP 192.168.15.53 udp port 137 unreachable, length 36
When you see the next broadcast come in on port 137, you immediately see an ICMP packet go back out from your box saying that the port is unreachable. That is unless you have turned on stealth mode in the firewall. Then it'll just go in the bit bucket with the rest of the garbage.
Here is what it should look like. A simple query from .109 and a simple response from .53.
bash-3.2# tcpdump -ni en1 host 192.168.15.109
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes
23:05:47.226351 IP 192.168.15.109.137 > 192.168.15.53.137: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:05:47.226667 IP 192.168.15.53.137 > 192.168.15.109.137: NBT UDP PACKET(137): QUERY; POSITIVE; RESPONSE; UNICAST
3) You could SWEAR that your system was in workgroup 'MSHOME' and you set it up that way, but you're not being seen there.
This one is interesting. It kinda makes sense, but not really.
In the network settings for your network adapter, under Advanced, you'll notice a tab marked WINS. WINS is really a bad choice of words - it should probably just say 'Windows Networking' since you're not really running wins at home normally, but this is where OS X stores information needed to talk to Windows systems.
The first box (NetBiOS Name) is the name of your system. This should match what you have elsewhere in sharing. The second box is your workgroup and by default is left blank. The default workgroup is WORKGROUP. Why? Because that's what Microsoft uses by default. Usually. There was a period of time in which Windows started using MSHOME. How do you know? Go to your windows system and check. If it's MSHOME, type MSHOME here. If it's WORKGROUP, trust me on this one, FILL IN WORKGROUP. DO NOT leave it BLANK. Also make sure that when you built your windows computer 5 years ago that you didn't put in 'FREE_BIRD' or something you just don't remember anymore.
When you hit 'ok' and apply this change, OS X changes a critical system file in /var/run/smb.conf. Specifically, it sets the workgroup value in the global settings and restarts the name service (nmbd).
Why is this important to know? Because if you switch between a Wired and Wireless network, it will do this again and again each time you connect to one or the other.
So if you set things to 'MSHOME' on Airport and then go plug it into your Ethernet network, it WILL reset you back to WORKGROUP if you didn't also put in MSHOME under the wired interface. When you unplug from the wired network and re-connect to airport, it will put it back to MSHOME again. However, if you turn on Wireless while you're on the Wired network, it will keep the Wired setting. Why? I guess because 'Ethernet' is on top (at least on my system). So what happens if you then unplug the wired network? You guessed it, it goes to whatever setting is on the interface left up.
So the lesson here is to make sure that the WINS tab is configured the same in each of your interface types AND that you don't leave the workgroup field blank. This ensures that you don't toggle between, 'MYHOMENET' and 'WORKGROUP' when you go in and out of your office.
Solutions
How do I go about making sure everything is right with all of these conflicting and wacky things going on?
Here's a step through that should make sure everything is up to snuff and stays that way.
1) Go to system Preferences. Select Security. Select Firewall. Set 'Set Access for specific services and applications'.
2) Click 'Show All' and go back to System Preferences and select Network.
3) Select Airport. Select Advanced. Select WINS. Set your workgroup to 'WORKGROUP' unless you know it's something else. Delete any entries out of WINS unless you KNOW you need it. Select OK. Select Apply.
4) Select Ethernet. Select Advanced. Select WINS. Set your workgroup to 'WORKGROUP' unless you know it's something else. Delete any entries out of WINS unless you KNOW you need it. Select OK. Select Apply.
5) Click 'Show All' and go back to System Preferences and select 'Sharing'
6) Check 'File Sharing'. Select Options. Check 'Add Files and Folders using SMB' and any other methods you want to use. Only check the box next to your users if you want to share their data. Select 'Done'.
7) Wait. Sooner or later, you should start seeing hosts showing up in your Finder. Either on the sidebar or within 'Network'.
You don't? Proceed to Troubleshooting.
Troubleshooting
OS X provides a number of utilities in the shell to see what's going on with network services and NetBIOS specifically. I'll briefly go over each with examples of how you might use them.
0) We're going to assume you already checked to make sure that your Windows firewall was correctly configured as well as your Mac's. So if you're using the default windows firewall, McAfee, ZoneAlarm, etc. - you got to be sure that it's allowing this file sharing stuff in and out.
1) netstat -anp udp
This command will list all ports that your system is using (-a) without translating the IP addresses to names (-n) and only for protocol udp (-p udp). You can also use -p tcp or remove -p xxx altogether.
What you are looking for here are entries for the services that watch for and manage the NetBIOS broadcasts. From earlier in this post, you'll recall that is udp port 137.
You'll see:
*.137
192.168.15.53.137
showing that your host is on the lookout for those packets (your firewall was setup correctly, right?)
2) nmblookup -M -- -
This command will send a query out on the network looking for the master browser on your network. You should get a response back such as:
Err:~ eb$ nmblookup -M -- -
querying _MSBROWSE_ on 192.168.15.255
192.168.15.109 _MSBROWSE_<01>
You don't? You SURE about this whole firewall thing? And your windows machine is on and sharing?
3) nmblookup hostname
Obviously replace 'hostname' with the name of the system you're trying to connect to. This will tell you if you can properly resolve that systems name.
Example:
Err:~ eb$ nmblookup mastershake
querying mastershake on 192.168.15.255
192.168.15.109 mastershake<00>
4) nmblookup -S hostname
Same deal applies here - replace hostname with the name of the system you're trying to connect to. This command will list out all of the services and the name of the workgroup that system is part of. It's the same as what you configured in Network prefs for all your interfaces, right?
Err:~ eb$ nmblookup -S mastershake
querying mastershake on 192.168.15.255
192.168.15.109 mastershake<00>
Looking up status of 192.168.15.109
MASTERSHAKE <00> - B <ACTIVE>
WORKGROUP <00> - <GROUP> B <ACTIVE>
MASTERSHAKE <20> - B <ACTIVE>
WORKGROUP <1e> - <GROUP> B <ACTIVE>
WORKGROUP <1d> - B <ACTIVE>
.._MSBROWSE_. <01> - <GROUP> B <ACTIVE>
MAC Address = 00-1B-B9-52-65-2B
5) tcpdump -nvvXi en0 -s 1500
This command is really one of your best friends. At the end of the day, if everything is configured right and not working, see what is hitting the wire. OS X can't do anything if you're not seeing packets.
Use control-c to get out of this. It may look like garbage, but if you spend some time with it, you'll learn pretty quickly how to read it.
6) Are all of your systems DHCP or did you static them? Make sure that the netmasks are the same. Remember, broadcasts don't cross network boundaries. So if one system has an ip address of 192.168.15.2/255.255.255.0 and the other has an address of 192.168.15.3/255.255.255.252, technically, these are not on the same network and will not be able to see broadcasts from each other.
7) So all of the above seems to be great, and you still don't see things in your finder.
Can you connect manually with option-k by BOTH IP address (smb://192.168.15.109) and hostname (smb://mastershake)?
If so, the issue is with browsing only. If you can do both of the above (IP AND NetBIOS name) and you still cannot see entries in Network, there may be a real bug. I've noticed that Vista machines (with Network Discovery enabled) don't always show up correctly, while XP systems show up every time.
_Other Options_
For those that are familiar with Windows Networking, there have been some great comments regarding some methods to speed this up.
1) Setup your own WINS server. You'll find this in one of the threads. Basically, setup smb.conf to allow it to act as a WINS server and then setup those wins entries I said to leave blank to point to it in each of your clients. Since WINS is client/server, it's much easier to figure out what's happening when it doesn't work.
2) Increase your os value or let yourself be the domain master in smb.conf. You'll find this in the threads as well. Keep in mind, it could still take up to 12 minutes (or more - up to an hour really) to see everything on the network.
Message was edited by: mreckhof

Answer 1

Let's make sure we're using the same definition of 'Browse'. Are you able to see them (before you have ever connected to them with option-k smb://x.x.x.x/) in your network list dynamically?
Or are you only talking about being able to connect to them with option-k?
The reason why this is necessary is that the nmbd daemon is not running unless you check the File Sharing box and enable SMB.
Example:
- File Sharing Disabled - Note that nmbd is not running and nothing is listening on port 137.
Err:~ eb$ ps -aef | grep -i nmb
501 5778 5743 0 0:00.00 ttys000 0:00.00 grep -i nmb
Err:~ eb$ netstat -anp udp | grep 137
Err:~ eb$
- File Sharing Enabled and SMB Checked - Note we now have the nmbd process and a listener.
Err:~ eb$ ps -aef | grep -i nmb
0 5789 1 0 0:00.01 ?? 0:00.03 /usr/sbin/nmbd -F
501 5791 5743 0 0:00.00 ttys000 0:00.00 grep -i nmb
Err:~ eb$ netstat -anp udp | grep 137
udp4 0 0 192.168.15.53.137 .
udp4 0 0 *.137 .
If you don't have anything listening for the broadcasts, you're not going to be able to browse and see them in your Network list (unless they're also advertising by some other protocol such as Bonjour).
That is not to say that you can't connect to them - you certainly can by smb://x.x.x.x/ (ip) or smb://DNS_NAME/ (not NetBIOS name).
If you can add more detail if you're seeing something else, like netstat output it would be great.