Mavericks Server Issues

Question

We have a server that was upgraded to Mavericks and OS X Server 3 and that's when the issues started. Folders becoming inaccessible, permission issues, file getting corrupted. We thought it was because it was an older Mac Mini, which was fine running Snow Leopard server, but the issues started after Mavericks. It was assumed part of the problem was the upgrade as opposed to a clean install. As this is a production machine, there was no time to wipe and reinstall. We also worried it was possible the hardware was going after running 24/7 for 3.5 years.
To be safe, we replaced it will a brand new Mac mini running a clean version of Mavericks server. It was fine for a few dys then the issues started creeping in. Permissions issues again, inaccessible folders, Windows shares getting kicked off and finally the log file filling up with errors (4,000+) have pushed us to the breaking point. We cannot get a full days work done on the server without having to repair something or reboot.
The error that comes up in the log and console is: Kernel: proc: table is full. It fills up the log.
DNS is not running and neither is open directory as this server is hosting a small network of users (less than 10) for file sharing of graphics files only. Some of the errors occur once a windows machine connects.
Any suggestions as to what could be causing these issues would be greatly appreciate as we are stumped. It ran fine under Snow Leopard it seems it's more Mavericks specific. The next step is to revert to Snow Leopardand pray that work.
Thanks in advance.
Chris

Answer 1

This procedure is a diagnostic test. It makes no changes to your data. If you have more than one user account, you must be logged in as an administrator to carry out these instructions.
Please triple-click anywhere in the line below on this page to select it:
sudo dtrace -n 'syscall::fork*:entry{printf("%s %d",execname,pid);}' | head | pbcopy
Copy the selected text to the Clipboard by pressing the key combination command-C.
Launch the built-in Terminal application in any of the following ways:
☞ Enter the first few letters of its name into a Spotlight search. Select it in the results (it should be at the top.)
☞ In the Finder, select Go ▹ Utilities from the menu bar, or press the key combination shift-command-U. The application is in the folder that opens.
☞ Open LaunchPad. Click Utilities, then Terminal in the icon grid.
Paste into the Terminal window by pressing the key combination command-V. I've tested these instructions only with the Safari web browser. If you use another browser, you may have to press the return key after pasting. You'll be prompted for your login password. Nothing will be displayed when you type it. If you don’t have a login password, you’ll need to set one before you can run the command. You may get a one-time warning to be careful. Confirm. You don't need to post the warning.
If you see a message that your username "is not in the sudoers file," then you're not logged in as an administrator. Log in as one and start over.
Wait for a new line ending in a dollar sign ($) to appear below what you entered.
The output of the command will be automatically copied to the Clipboard. If the command produced no output, the Clipboard will be empty. Paste into a reply to this message.
The Terminal window doesn't show the output. Please don't copy anything from there.