Table partitioning best practices

Just looking for some ideas.
We have a large information warehouse table that we are looking to partition on 'service_period' id. We have been requested to partition on every month of every year which right now will create approximately 70 partitions. The other problem is that this is a rolling or dynamic partition meaning we will have a 'new' partition vale with each new month. I understand in 11g there is a rolling partition functionality but we are not there yet.
So right now we are looking for a best practice for this scenario. We are thinking of possibly creating a partition on each year and indexing on the service period within each partition, maybe hash partitioning on the service period id (although does not seem to group the service periods distinct within each partition), somehow creating the partition dynamically via pl/sql (creating the table with a basic partition and then running an alter table on the data creating the proper number of partitions within a list partition.
I am also wondering if there is a point of too many partitions on a table. I am thinking 70 may be a little extreme but not sure. We are going to do some performance testing but would be nice to hear from the community. We have 5,000,000 over approx 70 partitions giving us approx 70,000 records per partition. The other option would be to create the partition based on year and then apply an index over top on the service period to reduce the number of partitions.
Thanks in advance,
Bruce Carson

This is not a lot of data, so the effort of partitioning may not be worth the benefit you receive. 70 partitions is not unreasonable. Do you have performance problems ? Do the majority of your queries reference service_period ? Do you have a lot of full table scans ?
Partitioning strategies depend on the queries you plan to run, your data distributions, and your purge / archival strategy.
Think about whether you should pre-create partitions for years in advance. Think about whether you should put every partition in a seperate tablespace for easy purging and archival. Think about what indexing you will use (can you use local indexes, or do you need global ?) Think about what data changes are happening ? Are they all on the newest data ? Can you compress older partitions with PCT_FREE 0 to improve performance ?

Similar Messages

Azure table design best practice

What's the best practice for designing tables in Azure Tables to optimize query performance?

Hi Raj,
When we design the azure table, we need to consider the scalability of the azure table.
and selecting the PartitionKey is very more important to scalability.
Basically, we have two options which have their advantages and disadvantages:
One Option: having a single partition by having the same value for PartitionKey for all entities to
Second Option: having a unique value for PartitionKey for every entity
More information about how to get the most out of windows azure tables ,please refer to the link below:
http://blogs.msdn.com/b/windowsazurestorage/archive/2010/11/06/how-to-get-most-out-of-windows-azure-tables.aspx
There is also a detailed article which explain how to design a scalable partitioning strategy for Windows Azure Storage,please refer to the link below:
http://msdn.microsoft.com/en-us/library/hh508997.aspx
Best Regards,
Kevin Shen

Varying table columns, best practices

I've been wondering about this for quite sometime now. JTable is very complex, but it has a lot of funcationality that hints at reusable models. The separation of TableModel and ColumnModel seems to hint at being able to reuse a TableModel that stores some sort of objects, and apply different ColumnModels to view the data in different ways. Which is really cool.
However, who is in charge of managing the columns? The default implementation is usually good enough. But, it doesn't do anything special to the columns like: assign renderers, or editors. Should the column model be in charge of this? But, then you have to swap full on column models out when you want to change the look of the table. What if you just want to vary the renderer on a column, or remove one column. Would you build a whole new ColumnModel for this?
Should the JTable be in charge of setting himself up in these matters? But that seems to impose the view's representation on the model. What if you change views in some way that affects your model's structure.
Should there be some external controller in charge of this?
Sometimes you don't plan for these things at it hurts you when you need to reuse models, but maybe modify them in some way. What are your best practices?
charlie

The practice you described is what I'm doing right now, and I feel that it is cumbersome for reuse.The practice you described is what I'm doing right now, and I feel that it is cumbersome for reuse.
What I'm wondering is that if anyone has come up with a very elgant way to organize their class' responsibilities between who populates the column model. I know I can subclass and fill it in the subclass, but it seems that I might NOT need to subclass, use the default, and have another class ( maybe the JTable or a controller ), that populates the ColumnModel.
Then I can get better reuse between TableModel's and ColumnModels.
charlie

External Table Authorization Best practices

Hi,
I am working on OBIEE External table Authorization. I am able to successfully implement for one Project (catalog). The field for Authorization table (AuthTable) are
Windows_ID Employeeid Name EmpEMail GroupName Process_ID Process_Name Portal_Path
Here as per requirement a user should see data for a few process. So, I put a column for Process_ID and subsequently I created a INIT block in repository where query are like
Select 'PROCESS_ID',AuthTable. Process_id
From AuthTable
WHERE upper(AuthTable.AD_ID) = upper(':USER')
Then for User Groups I applied FILTERs for all the tables E.G for every Logical Table I applied Filter
Dim_Process."Process ID" = VALUEOF(NQ_SESSION."PROCESS_ID")
I checked data and every thing is correct. But My question is:
We have many projects/catalog for which Filter Criteria will be different so shall we insert a new column for each criteria in SAME AuthTable or there is any other and better way to maintain it. Because if we maintain one table for all the projects/catalog it will be very messy I would prefer to keep different tables for different projects/catalog as there data marts are different.
But Problem is for all other session variables we may use different INIT BLOCKS and hence different tables BUT for PORTALPATH there should be only one INIT BLOCK so only for PORTALPATH sake we need to keep every thing in same table ?
Tell me if I am wrong some where in my understanding or there is a better way to do it.
Regards
Saurabh

Hi,
Pls refer to this link. Kumar explained it very clearly
http://obieeblog.wordpress.com/category/obiee/obiee-security/
Pls award points, if helpful
Regards,
Sarat Nallapati

VC table use, best practices?

Hi,
I'm updating a table in the back end with an RFC. I would like to send only the rows I've modified or added on the VC client to the RFC and not the whole table. Is this possible?

Hey Joel,
Make a condition(say check box) on changing the table row (user need to select what are all the rows he is modifying).
while sending the values to the RFC, give the guard condition as valid only when check box is selected.
Regards,
Pradeep

What's the best practice to partition an existing table?

We are using Oracle 11g EE 11.2.0.3.0. We want to partition an existing table. I found some information from this link [http://www.oracle-base.com/articles/misc/partitioning-an-existing-table.php], but I am not sure whether that's the best way. Could anyone share the best practice of partitioning an existing table?
Thanks.
Jun

>
We want to partition a few related tables. A couple of tables have about 30 million rows, and a couple of other tables have about 10 million rows. The data for those tables is for ~2000 accounts, each of them is identified by a column named "account_id". We want to partition the tables by using the "account_id" as the partition key to gain better performance for queries, so we would probably use hash partition. Please advise.
>
1. What evidence do you have that your performance will be any better if you partition those tables?
2. What evidence do you have that there is anything wrong with the performance of your current queries?
3. Have you reviewed the actual execution plans for your most important queries to see if there is even any room for improvement?
If your query uses account_id as a predicate but only pulls a few records from a 'master' table for that key value your performance may actually be worse if you partition that table. That is because using an index range scan Oracle can easily find the ROWIDs of the rows that need to be selected and then easily retrieve them regardless of their physical location.
If you have queries that do NOT use account_id but currently use indexes the performance of those queries may also be worse if you partition that table. Those indexes will be GLOBAL.
There are two major implications when you use hash partitioning:
1. Any query that does NOT include the partitioning key (account_id for your case) will, of necessity, have to use a full table scan if an appropriate index is not available.
2. You will get NO management benefits such as being able to add/remove partitions that include old data.
Before you decide to partition a table you should conduct tests and examine the execution plans for your important queries.

Best practice for replicating Partitioned table

Hi SQL Gurus,
Requesting your help on the design consideration for replicating a partitioned table.
1. 4 Partitioned tables (1 master table with foreign key constraints to 3 tables) partitioned based on monthly YYYYMM
2. 1 table has a XML column in it
3. Monthly switch partition to remove old data, since it is having foreign key constraint; disable until the switch is complete
4. 1 month partitioned data is 60 GB
having said the above, wanted to create a copy of the same tables to a different servers.
I can think of
1. Transactional replication, but then worried about the XML column,snapshot size and the alter switch will make the same thing
on the subscriber or row by row delete.
2. Logshipping with standby with every 15 minutes, but then it will be for the entire database; because I have other partitioned monthly table which is of 250 GB worth.
3. Thinking about replicating the Partitioned table as Non Partitioned, in that case how the alter switch will work. Is it possible to ignore delete when setting up the replication.
3. SSIS or Stored procedure method of moving data on a daily basis.
4. Backup and restore on a daily basis, but this will not work when the source partition is removed.
Ganesh

Plz refer to
http://msdn.microsoft.com/en-us/library/cc280940.aspx

Best Practice for Designing Database Tables?

Hi,
I work at a company for tracking devices (GPS Devices). Our SQL Server database is designed to have a table for each device we sell, currently there is 2500 tables in our database and they all have the same columns they only differ in table name. Each device
sends about 4K records per day.
currently each table hold from 10K records to 300K records
What is the best practice to design a database in this situation?
When accessing database from a C# application, which is better to use, direct SQL commands or views?
a detailed description about what is best to do in such scenario would be great.
Thanks in advance.
Edit:
Tables columns are:
[MessageID]
,[MessageUnit]
,[MessageLong]
,[MessageLat]
,[MessageSpeed]
,[MessageTime]
,[MessageDate]
,[MessageHeading]
,[MessageSatNumber]
,[MessageInput]
,[MessageCreationDate]
,[MessageInput2]
,[MessageInput3]
,[MessageIO]

Hello Louis, thank you so much for your informative post. I'll describe in detail what situations I came through my 9 months of work in the company (working as a software engineer, but I am planning to take over database maintenance since no one is maintaining
it right now and I cannot do anything else in the code to make it faster)
At every end of the month our clients generate report for the previous month for all their cars, some clients have 100+ cars, and some have few. This is when real issue start, they are calling their data from our server through internet while having 2000
unit sending data to our server, they keep on getting read time out since SQL Server gives priority to insert and hold all select commands. I solved it temporary in the code using "Read Uncommitted" once I initialize a connection through C#.
The other issue is generating reports for a month or two takes lots of time when selecting 100+ units. Thats what I want to solve, the problem is the one who wrote the C# app used hard coded SQL Statements
AND
the company is refusing to upgrade from SQL Server 2003 and Windows Server 2003.
Now talking about reports, there are summary reports, stops reports, zone reports ..etc most of them depend usually on at least MessageTime, MessageDate, MessageSpeed, MessageIO and MessageSatNumber.
So from your post I conclude that for now I need to set snapshots so that select statements don't get kicked out in favor for insert commands, but does SQL Server automatically select from the snapshots or do I have to tell it to do so?
Other than proper indexing what else I need? Tom
Phillips suggested Table partitioning but I don't think it is needed in my case since our database size is 78GB
When I run code analysis on the app, Visual Studio tells me I better use stored procedures, views than using hard coded Select Statements, what difference will this bring me when talking about performance?
Thanks in advance.

Logical level in Fact tables - best practice

Hi all,
I am currently working on a complex OBIEE project/solution where I am going straight to the production tables, so the fact (and dimension) tables are pretty complex since I am using more sources in the logical tables to increase performance. Anyway, what I am many times struggling with is the Logical Levels (in Content tab) where the level of each dimension is to be set. In a star schema (one-to-many) this is pretty straight forward and easy to set up, but when the Business Model (and physical model) gets more complex I sometimes struggle with the aggregates - to get them work/appear with different dimensions. (Using the menu "More" - "Get levels" does not allways give the best solution......far from). I have some combinations of left- and right outer join as well, making it even more complicated for the BI server.
For instance - I have about 10-12 different dimensions - should all of them allways be connected to each fact table? Either on Detail or Total level. I can see the use of the logical levels when using aggregate fact tables (on quarter, month etc.), but is it better just to skip the logical level setup when no aggregate tables are used? Sometimes it seems like that is the easiest approach...
Does anyone have a best practice concerning this issue? I have googled for this but I haven't found anything good yet. Any ideas/articles are highly appreciated.

Hi User,
For instance - I have about 10-12 different dimensions - should all of them always be connected to each fact table? Either on Detail or Total level.It not necessary to connect to all dimensions completely based on the report that you are creating ,but as a best practice we should maintain all at Detail level only,when you are mentioning any join conditions in physical layer
for example for the sales table if u want to report at ProductDimension.ProductnameLevel then u should use detail level else total level(at Product,employee level)
Get Levels. (Available only for fact tables) Changes aggregation content. If joins do not exist between fact table sources and dimension table sources (for example, if the same physical table is in both sources), the aggregation content determined by the administration tool will not include the aggregation content of this dimension.
Source admin guide(get level definition)
thanks,
Saichand.v

Temp Tables - Best Practice

Hello,
I have a customer who uses temp tables all over their application.
This customer is a novice and the app has its roots in VB6. We are converting it to .net
I would really like to know the best practice for using temp tables.
I have seen code like this in the app.
CR2.Database.Tables.Item(1).Location = "tempdb.dbo.[##Scott_xwPaySheetDtlForN]"
That seems to work, though i do not know why the full tempdb.dbo.[## is required.
However, when i use this in the new report I am doing I get runtime errors.
i also tried this
CR2.Database.Tables.Item(1).Location = "##Scott_xwPaySheetDtlForN"
I did not get errors, but I was returned data i did not expect.
Before i delve into different ways to do this, i could use some help with a good pattern to use.
thanks

Hi Scott,
Are you using the RDC still? It's not clear but looks like it.
We had an API that could piggy back the HDBC handle in the RDC ( craxdrt.dll ) but that API is no longer available in .NET. Also, the RDC is not supported in .NET since .NET uses the framework and RDC is COM.
Work around is to copy the temp data into a data set and then set location to the data set. There is no way that I know of to get to the tempdb from .NET. Reason being is there is no CR API to set the owner of the table to the user, MS SQL Server locks the tempdb to that user has exclusinve rights on it.
Thank you
Don

Best Practice Table Creation for Multiple Customers, Weekly/Monthly Sales Data in Multiple Fields

We have an homegrown Access database originally designed in 2000 that now has an SQL back-end. The database has not yet been converted to a higher format such as Access 2007 since at least 2 users are still on Access 2003. It is fine if suggestions
will only work with Access 2007 or higher.
I'm trying to determine if our database is the best place to do this or if we should look at another solution. We have thousands of products each with a single identifier. There are customers who provide us regular sales reporting for what was
sold in a given time period -- weekly, monthly, quarterly, yearly time periods being most important. This reporting may or may not include all of our product identifiers. The reporting is typically based on calendar-defined timing although we have
some customers who have their own calendars which may not align to a calendar month or calendar year so recording the time period can be helpful.
Each customer's sales report can contain anything from 1,000-20,000 rows of products for each report. Each customer report is different and they typically have between 4-30 columns of data for each product; headers are consistently named. The
product identifiers included may vary by customer and even within each report for a customer; the data in the product identifier row changes each week. Headers include a wide variety of data such as overall on hand, overall on order, unsellable on hand,
returns, on hand information for each location or customer grouping, sell-through units information for each location or customer grouping for that given time period, sell-through dollars information for each location or customer grouping for that given time
period, sell-through units information for each location or customer grouping for a cumulative time period (same thing for dollars), warehouse on hands, warehouse on orders, the customer's unique categorization of our product in their system, the customer's
current status code for that product, and so on.
Currently all of this data is stored in a multitude of Excel spreadsheets (by customer, division and time period). Due to overall volume of information and number of Excel sheets, cross-referencing can take considerable time. Is it possible to
set-up tables for our largest customers so I can create queries and pivot tables to more quickly look at sales-related information by category, by specific product(s), by partner, by specific products or categories across partners, by specific products or
categories across specific weeks/months/years, etc. We do have a separate product table so only the product identifier or a junction table may be needed to pull in additional information from the product table with queries. We do need to maintain
the sales reporting information indefinitely.
I welcome any suggestions, best practice or resources (books, web, etc).
Many thanks!

Currently all of this data is stored in a multitude of Excel spreadsheets (by customer, division and time period). Due to overall volume of information and number of Excel sheets, cross-referencing can take considerable time. Is it possible to
set-up tables .....
I assume you want to migrate to SQL Server.
Your best course of action is to hire a professional database designer for a short period like a month.
Once you have the database, you need to hire a professional DBA to move your current data from Access & Excel into the new SQL Server database.
Finally you have to hire an SSRS professional to design reports for your company.
It is also beneficial if the above professionals train your staff while building the new RDBMS.
Certain senior SQL Server professionals may be able to do all 3 functions in one person: db design, database administration/ETL & business intelligence development (reports).
Kalman Toth Database & OLAP Architect
SELECT Video Tutorials 4 Hours
New Book / Kindle: Exam 70-461 Bootcamp: Querying Microsoft SQL Server 2012

Best Practice to fetch SQL Server data and Insert into Oracle Tables

Hello,
I want to read sqlserver data everry half an hour and write into oracle tables ( in two different databases). What is the best practice for doing this?
We do not have any database dblinks from oracle to sqlserver and vice versa.
Any help is highly appreciable?
Thanks

Well, that's easy:
use a TimerTask to do the following every half an hour:
- open a connection to sql server
- open two connections to the oracle databases
- for each row you read from the sql server, do the inserts into the oracle databases
- commit
- close all connections

Best practice for a same query against 2 different tables

Hello all,
I want to extract info about tablespaces storage, both permanent and temporary. For that I use 2 different cursors that do exactly the same query but against a different table (dba_data_files and dba_temp_files).
CURSOR permanentTBSStorageInfo (tablespaceName VARCHAR2) IS
SELECT file_name, bytes, autoextensible, maxbytes, increment_by
FROM dba_data_files
WHERE tablespace_name = tablespaceName;
CURSOR temporaryTBSStorageInfo (tablespaceName VARCHAR2) IS
SELECT file_name, bytes, autoextensible, maxbytes, increment_by
FROM dba_temp_files
WHERE tablespace_name = tablespaceName;
First I'm bothered that I have to use 2 cursors to execute the same query against 2 different tables. Is there no another way around?
Then I fetch the results of this cursors in 2 different loops because I didn't find a way to dynamically call the cursors. I am looking for best practice here, knowing that I will do the same parsing against the results of the 2 cursors.
Thank you,

Hi
Check whether the below query is helpful or not
select      fs.tablespace_name "Tablespace",
     fs.tempspace "Temp MB",
     df.totalspace "Total MB"
     from
     (select
     tablespace_name,
     round(sum(bytes) / 1048576) TotalSpace
     from
     dba_data_files
     group by
     tablespace_name
     ) df,
     (select
     tablespace_name,
     round(sum(bytes) / 1048576) tempSpace
     from
     dba_temp_files
     group by
     tablespace_name
     ) fs
     where
     df.tablespace_name = fs.tablespace_name;
Thanks

Best practice for migrating data tables- please comment.

I have 5 new tables seeded with data that need to be promoted from a development to a production environment.
Instead of the DBAs just using a tool to migrate the data they are insistent that I save and provide scripts for every single commit, in proper order, necessary to both build the table and insert the data from ground zero.
I am very unaccustomed to this kind of environment and it seems much riskier for me to try and rebuild the objects from scratch when I already have a perfect, tested, ready model.
They also require extensive documentation where every step is recorded in a document and use that for the deployment.
I believe their rationale is they don't want to rely on backups but instead want to rely on a document that specifies each step to recreate.
Please comment on your view of this practice. Thanks!

>
Please comment on your view of this practice. Thanks!
>
Sounds like the DBAs are using best practices to get the job done. Congratulations to them!
>
I have 5 new tables seeded with data that need to be promoted from a development to a production environment.
Instead of the DBAs just using a tool to migrate the data they are insistent that I save and provide scripts for every single commit, in proper order, necessary to both build the table and insert the data from ground zero.
>
The process you describe is what I would expect, and require, in any well-run environment.
>
I am very unaccustomed to this kind of environment and it seems much riskier for me to try and rebuild the objects from scratch when I already have a perfect, tested, ready model.
>
Nobody cares if if is riskier for you. The production environment is sacred. Any and all risk to it must be reduced to a minimum at all cost. In my opinion a DBA should NEVER move ANYTHING from a development environment directly to a production environment. NEVER.
Development environments are sandboxes. They are often not backed up. You or anyone else could easily modify tables or data with no controls in place. Anything done in a DEV environment is assumed to be incomplete, unsecure, disposable and unvetted.
If you are doing development and don't have scripts to rebuild your objects from scratch then you are doing it wrong. You should ALWAYS have your own backup copies of DDL in case anything happens (and it does) to the development environment. By 'have your own' I mean there should be copies in a version control system or central repository where your teammates can get their hands on them if you are not available.
As for data - I agree with what others have said. Further - ALL data in a dev environment is assumed to be dev data and not production data. In all environments I have worked in ALL production data must be validated and approved by the business. That means every piece of data in lookup tables, fact tables, dimension tables, etc. Only computed data, such as might be in a data warehouse system generated by an ETL process might be exempt; but the process that creates that data is not exempt - that process and ultimately the data - must be signed off on by the business.
And the business generally has no access to, or control of, a development environment. That means using a TEST or QA environment for the business users to test and validate.
>
They also require extensive documentation where every step is recorded in a document and use that for the deployment.
I believe their rationale is they don't want to rely on backups but instead want to rely on a document that specifies each step to recreate.
>
Absolutely! That's how professional deployments are performed. Deployment documents are prepared and submitted for sign off by each of the affected groups. Those groups can include security, dba, business user, IT and even legal. The deployment documents always include recovery steps so that is something goes wrong or the deployment can't procede there is a documented procedure of how to restore the system to a valid working state.
The deployments themselves that I participate in have representatives from the each of those groups in the room or on a conference call as each step of the deployment is performed. Your 5 tables may be used by stored procedures, views or other code that has to be deployed as part of the same process. Each step of the deployment has to be performed in the correct order. If something goes wrong the responsible party is responsible for assisting in the retry or recovery of their component.
It is absolutely vital to have a known, secure, repeatable process for deployments. There are no shortcuts. I agree, for a simple 5 new table and small amount of data scenario it may seem like overkill.
But, despite what you say it simply cannot be that easy for one simple reason. Adding 5 tables with data to a production system has no business impact or utility at all unless there is some code, process or application somewhere that accesses those tables and data. Your post didn't mention the part about what changes are being made to actually USE what you are adding.

Best Practice : how 2 fetch tables, views, ... names and schema

hi,
I am looking for the best practice about getting the catalog of a database.
I have seen that I can make some select in system-tables(or views) such as DBA_TABLES and DBA_VIEWS, or DBA_CATALOG, but is that the best way to grab thses informations ?
(I ask this question because It seems a strange way to me to get the table names using a simple select, but getting column info using a specialized function, OCIDescribeAny(). this does not look like a coherent API...)
thanks for your advice
cd

in the same idea, why use OCIDescribeAny instead of doing an appropriate select in DBA_TAB_COLUMNS ?
cd

Table partitioning best practices

Similar Messages

Maybe you are looking for