Advice on how to build a business case for partitioning

Question

Hi Everyone,
I'm the architect of a business intelligence/data warehouse team. We've grown to the point where I believe we need database partitioning. I believe we are at about 1.5 TB of data. Our largest tables are in the 100 million row counts with relatively wide record lengths (~100 columns). Our just sort of large tables are in the 10 million to 50 million range, however, they have very long row lengths (~300 columns). Based on our future source system plans, our data will become even more granular and there is still a need to maintain our historical data.
Those aren't the only reasons, but are some of the larger driving issues that are making most members of our team think it's time we purchased database partitioning. Our team is pretty unanimous for partitioning, many of our source system partners also consider it a "no-brainer", however, we're getting push back from our CIO saying that 1.5 TB and 100 million records isn't that much data as compared to other "big data" companies and that it doesn't require partitioning.
Does anyone have a advice on how to build an business case for purchasing partitioning? Is there a good way to demonstrate cost-vs-benefit value?
Thanks for your feedback.
-=Joe

Answer 1

I'm going to be contrarian and start from the end of your post.
>
Does anyone have a advice on how to build an business case for purchasing partitioning? Is there a good way to demonstrate cost-vs-benefit value?
>
Yes - forget about cost-vs-benefit. That's the CIOs job not yours. You're an architect. Stick to architecture. Design and implement the architecture needed to meet the business requirements of your projects. If you are given requirements that can't be met with your current infrastructure then that is when you should propose alternative infrastructure.
>
We've grown to the point where I believe we need database partitioning. I believe we are at about 1.5 TB of data. Our largest tables are in the 100 million row counts with relatively wide record lengths (~100 columns). Our just sort of large tables are in the 10 million to 50 million range, however, they have very long row lengths (~300 columns). Based on our future source system plans, our data will become even more granular and there is still a need to maintain our historical data.
Those aren't the only reasons, but are some of the larger driving issues that are making most members of our team think it's time we purchased database partitioning.
>
NONSENSE! That's what ALL of the above statements represent - UTTER AND ABSOLUTE NONSENSE.
Who cares what you believe? Beliefs don't carry weight, facts do. You do list some facts: 1.5 TB of data, 100 million row counts, wide record lengths, large sorts, long rows, data will get more granular, need to maintain historical data.
Ho Hum! So what? How do ANY of those facts indicate any sort of issue or problem, existing or potential, at all? They don't. You haven't provided information that suggests that ANY of those facts illustrate what Justin referred to as 'pain points'.
Unless those facts prevent you from doing your job of providing an architecture that meets the business requirements you don't have a leg to stand on.
My first reaction to your request would be similar to your cio's. Implicit in your cio's response are some questions you need to ask and answer
1. What is it that you do now that you can't do with the current architecture?
2. What is it that you need to do for an upcoming requirement that you can't do with the current architecture?
3. What alternatives are available to modify the current architecture to do what can't be done in items #1 and #2 above?
That #3 is where my skepticism lies. You seem to have already concluded that partitioning is the answer. Why? What problems do you think it will solve? Why not RAC? Why not a separate data warehouse system? Why not split that historical data off into an OLAP system? Maybe that can reduce the size of the OLTP system to where you won't have whatever problems you are having.
Partitioning is just about addressing some performance issues. A lot of times partitioning is used more for management purposes such as loading/offloading data without impacting those huge tables you are talking about.
If my hot queries use a primary key value to retrieve data it isn't going to matter much if the tables are partitioned or not.
That is what sets the context for what Justin is telling you. You and your team need to document the constraints (and SLAs) your system currently has, how close you are to hitting those constraints, your options for avoiding/evading those constraints using the current infrastructure and the options for avoiding/evading those constraints using a modified architecture.
It is up to the CIO to do the cost-benefit analysis of working within the limits imposed by the current infrastructure versus modifiying the architecture.