When to scale up and/or scale out?

Question

When do you guys scale up/out, and how do you justify it?
We have a small farm with a number of new projects we expect to have in production this year. I've been tasked with assessing our current environment, providing recommendations, and performing any scale out Sharepoint configuration.
I’m reviewing Microsoft’s capacity management and performance testing documentation. I want requests for architecture changes to be as quantifiable as possible, but I'm finding them incredibly frustrating.
I've checked our current server performance. SQL server's RAM is maxed 90% of the time which, I'm assured by my network guys, is normal for SQL boxes. There are occasional CPU spikes but nothing major.
I also took a look at the CA Admin Report Library for Crawl Rates, Query Latency, etc, but need something to compare against.
Any guidance/recommendations greatly appreciated.
Thanks,
Scott

Answer 1

Fist, let's look at your immediate concern.
High memory use on SQL is a normal sign of a healthy SQL box. i.e. you want SQL to store commonly accessed data in memory so it's not pulling from disk. SQL will try to use as much memory as it can to achieve this goal.
High memory use on SQL is a normal sign of an unhealthy SQL box when you're seeing lots of page swaps to disk. If the server is constantly caching data from memory to disk this suggests the server needs more memory.
Microsoft has a good document for troubleshooting performance. Check out Monitoring and maintaining SharePoint Server 2010. 
The trick with the documentation about performance is every environment is different. What may be unacceptable performance in a farm that serves thousands or users may be completely tolerable and reasonable for one that serves a small organization or department.
You're not going to find specific guidance from Microsoft along the lines of "if you see CPU at ABC then add a CPU. If you see CPU at XYZ then add a server" because the decision is more nuanced than simply looking at a value.
There are lots of great guides that others in the community have published that can provide some real-world context to how you can monitor performance for scale.
Jason Warren
@jaspnwarren
jasonwarren.ca
habaneroconsulting.com/Insights