SSH from work machine to home machine is very slow to connect

Question

If I try to SSH to my home computer (using its domain name or IP) from any computer on my office's network I get very long delays (using the -vv flag to debug). But if I SSH to my home machine from a computer outside of my office's network my home machine responds fine, no delays. And I can SSH from my office's network to any other remote machine no problem. It's only my home machine that is experiencing delays. I should add that I can AFP/VNC connect from my office to my home without issues either, this ONLY affects SSH connections.
Here's the output from the initial SSH attempt from my office to my home machine (I've made the info anonymous):
OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to domain.com [XX.XX.XXX.XXX] port 22.
debug1: Connection established.
debug1: identity file /Users/username/.ssh/identity type -1
debug1: identity file /Users/username/.ssh/id_rsa type -1
debug1: identity file /Users/username/.ssh/id_dsa type -1
Here is waits about 60 seconds before the next output.
In researching this it seems like it could be DNS issue? Any help would be appreciated. It's driving me crazy.

Answer 1

Well, I spent long time with Apple Phone support, they made me delete the library, move some iTunes file from Library folder on my MBP but that didn't help.
I also did reset my Apple TV, and did all the settings again. No change.
I read somewhere on this forum itself I think to remove "Share my Library on local network" option from iTunes preferences, under sharing. As per the suggestion, this is an old option, and when you have home sharing on, you don't need this to be checked. More over it causes sharing to be slow. So after I unchecked it, I ATV3 and my MBP on for about an hour, and when I came back, it was all just perfect!