Table count comparison in a SSIS package

In our production environment, I have a SSIS package to import from OLTP SQL Server database to Data Warehouse (in SQL Server) and from there another package imports from Data warehouse to a Tabular SSAS database.
For health check reason, I would like to develop a SSIS package to compare table counts between production OLTP, Data warehouse and Tabular databases. I know how to do it for SQL Server databases but have no idea how to calculate table counts of Tabular tables
and save the results to a SQL Server table to compare the counts.
Has any body done this? Any clue?
Thanks

one option is to query the a ssas DMV in a DataFlow...
SELECT *
FROM $SYSTEM.DBSCHEMA_TABLES
WHERE TABLE_CATALOG = 'AdventureWorks Tabular Model SQL 2012' AND
TABLE_TYPE = 'TABLE' AND
TABLE_SCHEMA = 'Model'
...and capture the RowCount to a variable.
BI Developer and lover of data (Blog |
Twitter)

Similar Messages

How to change color of rows count dispalyed while debugging ssis packages

Hello,
in visual 2013 ssdt version I have row count displayed in white color on pale background: bit.ly/1oWP1Cm
I'd like to have it in black. I know it should be changeable in Options -> Fonts and Colors, but what option should I change from myriads available?
Thank you in advance.

My colleague had the same issue in that we couldn't see the numbers. We couldn't figure out why we couldn't see the numbers and the design screen was white. It looked like the numbers were displaying in white over the connections lines. So
we changed the color scheme to light and the numbers appeared in black. WHen we switched the color scheme back to dark, the canvas turned dark and the text was in white.
So there must be a bug when selecting the color scheme on the first run of Visual Studio 2013 in that it doesn't set the correct canvas color for the SSIS designer. Luckily toggling the color scheme resolves the issue.
Christian Jones Software Developer, former Network/Sys Admin

SSIS Package compiled successful, in SQL Server Integration Service package executed sucessful, But fail to run in MS SQL Job Scheduler

Hi Everyone,
I having a problem to transfer data from MS SQL 2005 to IBMAS400. Previously my SSIS was running perfectly but there is some changes I need to be done in order for the system to work well. Considers my changes are minimal & just for upgrades (but I did
include DELETE statements to truncate AS400 table before I insert fresh data from MS SQL table to the same AS400 table), so I compile my SSIS package & it run successfully & I passed it into MS SQL Integrated Service as 1 of the packages & manually
executed the package & the result is the same, that mean it was successful again but when I try to run it in a MS SQL Job Scheduler, the job failed with these message shown below as extracted from the job View history.
Date today
Log Job History (MSSQLToAS400)
Step ID 1
Server MSSQLServer
Job Name MSSQLToAS400
Step Name pumptoAS400
Duration 00:00:36
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: MSSQLServer\SYSTEM. ... 9.00.4035.00 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: today time
Error: on today time
Code: 0xC0202009 Source: SSISMSSQLToAS400 Connection manager "SourceToDestinationOLEDB"
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred.
Error code: 0x80004005. An OLE DB record is available.
Source: "IBMDA400 Session"
Hresult: 0x80004005
Description: "CWBSY0002 - Password for user AS400ADMIN on system AS400SYSTEM is not correct ". End Error
Error: today
Code: 0xC020801C
Source: Data Flow Task OLE DB Destination [5160]
Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "DestinationClearData" failed with error code 0xC0202009. There may be error messages posted before
this with more information on why the AcquireConnection method ca... The package execution fa... The step failed.
So I hope somebody can shed some hints or tips for me to overcome time problem of mine. Thanks for your help in advance. As I had scoured thoroughout MSDN forums & found none solution for my problem yet.
PS: In the SQL Integrated Services when I deployed the package I set the security of the packages to Rely on server...
Hope this will help.

Hi Ironmaidenroxz,
From the message “Executed as user: MSSQLServer\SYSTEM”, we can see that the SQL Server Agent job ran under the Local System account. However, a Local System account doesn’t have the network rights natively, therefore, the job failed to communicate with
the remote IBMAS400 server.
To address this issue, you need to create a proxy account for SQL Server Agent to run the job. When creating the credentials for the proxy account, you can use the Windows domain account under which you executed the package manually.
References:
How to: Create a Credential
How to: Create a Proxy
Regards,
Mike Yin
TechNet Community Support

Error executing SSIS packages from Agent

Hi All,
     I designed SSIS package that loads data from Oracle table to SQl server table along with loading data into Diension and fact tables using SSIS packages only. First I created all SSIS packages that loads data into Staging tables from
Oracle table in next level loading data into Dimension table and final level is loading data into Fact table. At lasts I created One master package there I'm calling all packages (Execute package task) differently.
Configuration type: Parent package configuration for All child packages. XML config for parent package.
When I execute SSIS package from BIDS data is loading up to Fact table. When I execute SSIS packages through Sql server agent ( Using File System Deployment)When I execute SSIS package from Sql server agent it is throwing an error message like
Message
Executed as user: . Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 32-bit Copyright (C) Microsoft Corporation. All rights reserved.    Started: 10:45:10 PM Error: 2013-10-17 22:45:12.60
Code: 0xC00220E6     Source: Master Data      Description: There is no project to reference. End Error Error: 2013-10-17 22:45:12.61     Code: 0xC0024107     Source:
Master Data      Description: There were errors during task validation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 10:45:10 PM Finished: 10:45:13 PM Elapsed:
2.59 seconds. The package execution failed. The step failed.
I'm uploaded config file also while at the time of scheduling job.
Regards,
Praveen
Regards, Praveen

Hi Praveen,
The issue may occur if the SQL Server Agent Service Account or Proxy account doesn’t have sufficient permissions. Please create a SQL Server Agent Proxy account that has the permissions to execute the package and perform update operations on the SQL Server
fact tables.
Reference:
http://technet.microsoft.com/en-us/library/ms175834.aspx
Regards,
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

SSIS package to check if source table have rows

Hi,
I want to create an ssis package that moves data from source table to destination table. What I am doing right now is truncating the destination table and then populating destination table based on source table.
I want to chnage this process to check if the source table have any rows in that. if the source table have 0 rows I do not want to truncate the destination table and stop the process right away.
Can someone please let me know how to achive that.

What is your Source and Destination. Do they exist on same server, if so you could use Vaibhav's query to check if there are rows in the Source table and then truncate the destination table with one query. Otherwise, after your Source, use a Row Count
transformation and check if the value <> 0 then connect the data flow path to the Destination and in the destination, you can have your TRUNCATE statement. That way, it'll executed only if the RowCount is <> 0.
Narsimha

SSIS package flow based on a scheduling configuration table.

I have a table called Tbl_Scheduler.
This scheduler table contains
scheduler id ,scheduler description,as shown below.
1 ,Runs between 10 th of current month and 20th of current month
2 ,Runs at last day of the month
3 ,Runs at first day of the month
Scheduler Id will come from first task(it is given in a configuration table),
second task should check this scheduler id on scheduler table and
verify whether the current run date is falling under the criteria,which is given as scheduler description.
If it is not,need to stop the package flow.
SSIS package is scheduled from SQL agent job in every 10 mins.Could you please let us know,how I can design the same.
Also please feel free to advise to change the table design.

Stopping running packages is a bad idea in general. You must specifically design a package for being stopped any time which isn't always possible, often not at all.
And running a package repeatedly in short intervals is not a SOP either.
To trigger a SSIS package run, the easiest would be via sp_start_job <name> once you have your package set up to run in Agent.
Arthur My Blog

How do I join two tables in the same database and load the result into a destination table in a SSIS package

Hi,
I have a query that joins two tables in the same database, the result needs to be loaded in a destination DB table. How do I do this in SSIS package?
thank you !
Thank You Warmest Fanny Pied

Please take a look at these links related to your query.
http://stackoverflow.com/questions/5145637/querying-data-by-joining-two-tables-in-two-database-on-different-servers
http://stackoverflow.com/questions/7037228/joining-two-tables-together-in-one-database

Copy table from Ms SQL Server 2005 to Oracle with SSIS package

Hi all
I am witting an SSIS package to copy 2 tables from Ms SQL Server 2005 to an Oracle 10g database.
Problem: I can only see the Microsoft OLDB Provider for Oracle and it is too slow. How can I use the Oracle ODBC in aMs SQL Server 2005 or building a .net SSIS package.
thanks

Yes I am able to see the driver on the Data Source (ODBC), I also created a connection to the Oracle database. The problem in Ms SQL Server 2005 is that the ODBC connection I created is not listed. None of the lo local connections are listed, only Microsoft OLEDB connections

SSIS Package : Needs SQL Tables get refreshed every hour

Hello,
I have created a SSIS Package which includes,
Step 1 : Using Execute SQL Task to truncate all the tables
Step 2 : Data Flow Tasks to extract data from SharePoint lists and fill the SQL Tables for SSRS Reporting from that.
Requirement is to refresh SQL Tables every hour. I can do this using SQL Server Agent Job to run SSIS Package.
But, I am not sure is this the feasible way of do this, as we are truncating all the tables. There are around 10-15 tables. Database size is also not large.
Is there any in SSIS to impose only changes done in the SharePoint list to SQL tables rather than truncating them and creating new ones?
Thank you,
Mittal.

@Visakh,
Yes I am getting Created,CreatedBy, Modified, ModifiedBy audit fields data from all the SharePoint List.
So, can we just trust these fields and update tables accordingly ?
And how can I do this using SSIS ? Is there any particular control I can use to compare the data and time and update the SQL Tables?
Thank you,
Mittal.
Yes. Provided they have proper data thiat should work fine.
Yes you can use it inside SSIS
You can have a logic like below
1. In execute sql task capture max date for the source table
2. Use data flow task with source query having filter based on date field
ie like
WHERE Modified > ?
And map parameter to maxdate
3. add destination step to populate data
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Join the SQL Server Msdb.dbo.SysJobSteps to the SSIS Package table dbo.sysssispackages

I have the following query which returns jobs and jobsteps - for those items which have SSIS Packages attached, I would like to include a link to the results of this query below - to the actual SSIS Package Id - Can someone tell me what the column is that
references the two ?
dbo.sysssispackages= dbo.SysJobSteps
I could parse the "command" label - but would prefer something along the lines of job_id unique identifier.
SELECT
Job.job_id,
Job.name,
Job.enabled,
Job.description,
Job.date_created,
Job.Date_modified,
Step.Step_Id,
Step.step_name,
Step.subsystem,
Step.command,
Step.server,
Step.database_name,
step.last_run_duration,
Step.Last_run_date
FROM Msdb.dbo.SysJobs JOB
INNER JOIN Msdb.dbo.SysJobSteps STEP ON STEP.Job_Id = JOB.Job_Id
John

Parse the command column to get the SSIS package file name may be your only option here.

SSIS Package developed in SQL Server 2008 R2 Using BIDS 2008 is not working in SQL Server 2012

I am working in a Product based company.
We have given a Pre-requisites to our customers like SQL Server version should be 2008 or later.
I have developed and deployed my SSIS package in SQL Server 2008 R2 with BIDS 2008.
I have used "Script Component" in my package.
We will take only DTSX package file to the customer places and run using SQL Agent Job daily.
Who are all(Customer Environments) using SQL 2008 or R2 my package is running fine.
Who are all(Customer Environments) using SQL 2012, I am facing the below issue
The component metadata for “Script Component, clsid {874F7595-FB5F-4OFF-9BAF-FBFF825OE3EF}” could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
Still the same package is running fine for some of the customers having SQL 2012.
I am getting the above error only for few customers.
Can someone please guide me how to proceed on this. Is there any solution to go without upgrading the package to SQL 2012. Because we need to maintain only one package for all customers

Hi Katherine,
I am just using the script component only for generating Identity column purpose. I will get the maximum count from the table and using this count value as Input (JurisCount in my below code) for script component and increment the counter for new records.
Finally I am mapping the Script component output value (JurisKey in my below code) to Key column of the table. That's all I am doing with Script Component.
Here is my Simple .net code
/* Microsoft SQL Server Integration Services Script Component
* Write scripts using Microsoft Visual C# 2008.
* ScriptMain is the entry point class of the script.*/
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
public override void PreExecute()
base.PreExecute();
Add your code here for preprocessing or remove if not needed
public override void PostExecute()
base.PostExecute();
Add your code here for postprocessing or remove if not needed
You can set read/write variables here, for example:
Variables.MyIntVar = 100
int counter = 0;
public override void Input0_ProcessInputRow(Input0Buffer Row)
if (counter == 0)
counter = Convert.ToInt32(Row.JurisCount);
Row.JurisKey = counter;
counter = counter + 1;
PAVAN MSBI

Ssis 2012 parallel execution of ssis packages using catalog database hangs

i have a simple ssis package in 2012 where I am executing several data flow tasks in parallel using sequence containers, however very often the sql job just runs forever although all the tables have been loaded or I cant see whats going on, the job neither
fails nor succeeds, just goes on executing,is there an issue with executing several data flow tasks in parallel, why does the job continue to run forver, how to troubleshoot and fix the error, please guide, are there any issues using the catalog db execution,
is that the reason this behavior is showing?

multiple execute sql tasks within sequence containers running to execute the ssis packages deployed in ssisdb in sunchronous mode using the catalog database execution model
all of the ssis packages that have the data flow taks retrieve data from source to 3 different tables.
some of them push data into the same table.
please let me know if you need more information.
the job just runs forever, looks like all the data flow tasks ran and it is still running, because I don't see an increase in the row count of the tables, I may be wrong, if I check the execution status, 1 task does not show as succeeded, I don't know why
thanks a lot for ur help, ur help is much appreciated nik

Issue with Dataflow Task in SSIS Package

Hello Everyone,
I have recently created an SSIS package to Load data from Flat file to SQL Server Table, it was working fine. Now I have bought a new laptop and copied the projects into this new system and changed all the connection managers(Flat files) accordingly.
Problem is, when I previously ran the package(With File1) in old laptop and check the distinct count on a table it was perfect, but when I ran the same package with same File on New Laptop, the Distinct count gives me an issue with particular record.
Eg: Distinct Count with old laptop Run , 10,20,100
Distinct Count with old New Run , 10,20,100 , 100
There is a Space being generated for a particular record, when I run the package on new laptop with same File as old one.
Any suggestions on this, I could not exactly figured out the Issue.
Please Help.
Thanks
Thanks, Please Help People When they need..!!! Mark as answered if your problem is solved.

You need to tell the responders here how exactly to reproduce your issue and provide the file.
I do not know how the "distinct count" works.
Seems to me it appears to be a database issue not SSIS.
Arthur
MyBlog
Twitter
When I perform the above query on old laptop, this gives me a result set of
10
20
100
When I perform the Same query on New laptop, this gives me a result set of
10
20
100
'100 ' (No Quotes please see the spaces generated)
Thanks, Please Help People When they need..!!! Mark as answered if your problem is solved.
Can you give us an idea of tasks you're having after the file source?
Particulary see steps where you are doing some data manipulations.
Please Mark This As Answer if it solved your issue
Please Mark This As Helpful if it helps to solve your issue
Visakh
My MSDN Page
My Personal Blog
My Facebook Page
I am using simple Flatfile source and oledb destination in here. this is causing issue, all the other tasks after this are loading perfectly except this.
Thanks, Please Help People When they need..!!! Mark as answered if your problem is solved.

Migrating SQL Server SSIS package to Oracle

Dear All,
We are using Microsoft SQL server 2005 in our application, we have two separate Server instances where one instance synchronize data from other server table using SQL Server SSIS package. What can be the counterpart of this functionality in Oracle. I have looked into materialized views but in my case the target table will have more columns than the base table, and the query used to get data from base table will extract some values from one of the BLOB type column of base table and then populate target table multiple columns.
Kindly suggest me any solution.
Best Regards
Irfan

You can start with Oracle doc,
Oracle® Database PL/SQL User's Guide and Reference
http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14261/toc.htm
I don't think there's side by side comparison reference between them. Some syntax is very different.

Unicode and non-unicode string data types Issue with 2008 SSIS Package

Hi All,
I am converting a 2005 SSIS Package to 2008. I have a task which has SQL Server as the source and Oracle as the destination. I copy the data from a SQL server view with a field nvarchar(10) to a field of a oracle table varchar(10). The package executes fine
on my local when i use the data transformation task to convert to DT_STR. But when I deploy the dtsx file on the server and try to run from an SQL Job Agent it gives me the unicode and non-unicode string data types error for the field. I have checked the registry
settings and its the same in my local and the server. Tried both the data conversion task and Derived Column task but with no luck. Pls suggest me what changes are required in my package to run it from the SQL Agent Job.
Thanks.

What is Unicode and non Unicode data formats
Unicode :
A Unicode character takes more bytes to store the data in the database. As we all know, many global industries wants to increase their business worldwide and grow at the same time, they would want to widen their business by providing
services to the customers worldwide by supporting different languages like Chinese, Japanese, Korean and Arabic. Many websites these days are supporting international languages to do their business and to attract more and more customers and that makes life
easier for both the parties.
To store the customer data into the database the database must support a mechanism to store the international characters, storing these characters is not easy, and many database vendors have to revised their strategies and come
up with new mechanisms to support or to store these international characters in the database. Some of the big vendors like Oracle, Microsoft, IBM and other database vendors started providing the international character support so that the data can be stored
and retrieved accordingly to avoid any hiccups while doing business with the international customers.
The difference in storing character data between Unicode and non-Unicode depends on whether non-Unicode data is stored by using double-byte character sets. All non-East Asian languages and the Thai language store non-Unicode characters
in single bytes. Therefore, storing these languages as Unicode uses two times the space that is used specifying a non-Unicode code page. On the other hand, the non-Unicode code pages of many other Asian languages specify character storage in double-byte character
sets (DBCS). Therefore, for these languages, there is almost no difference in storage between non-Unicode and Unicode.
Encoding Formats:
Some of the common encoding formats for Unicode are UCS-2, UTF-8, UTF-16, UTF-32 have been made available by database vendors to their customers. For SQL Server 7.0 and higher versions Microsoft uses the encoding format UCS-2 to store the UTF-8 data. Under
this mechanism, all Unicode characters are stored by using 2 bytes.
Unicode data can be encoded in many different ways. UCS-2 and UTF-8 are two common ways to store bit patterns that represent Unicode characters. Microsoft Windows NT, SQL Server, Java, COM, and the SQL Server ODBC driver and OLEDB
provider all internally represent Unicode data as UCS-2.
The options for using SQL Server 7.0 or SQL Server 2000 as a backend server for an application that sends and receives Unicode data that is encoded as UTF-8 include:
For example, if your business is using a website supporting ASP pages, then this is what happens:
If your application uses Active Server Pages (ASP) and you are using Internet Information Server (IIS) 5.0 and Microsoft Windows 2000, you can add "<% Session.Codepage=65001 %>" to your server-side ASP script.
This instructs IIS to convert all dynamically generated strings (example: Response.Write) from UCS-2 to UTF-8 automatically before sending them to the client.
If you do not want to enable sessions, you can alternatively use the server-side directive "<%@ CodePage=65001 %>".
Any UTF-8 data sent from the client to the server via GET or POST is also converted to UCS-2 automatically. The Session.Codepage property is the recommended method to handle UTF-8 data within a web application. This Codepage
setting is not available on IIS 4.0 and Windows NT 4.0.
Sorting and other operations :
The effect of Unicode data on performance is complicated by a variety of factors that include the following:
1. The difference between Unicode sorting rules and non-Unicode sorting rules
2. The difference between sorting double-byte and single-byte characters
3. Code page conversion between client and server
Performing operations like >, <, ORDER BY are resource intensive and will be difficult to get correct results if the codepage conversion between client and server is not available.
Sorting lots of Unicode data can be slower than non-Unicode data, because the data is stored in double bytes. On the other hand, sorting Asian characters in Unicode is faster than sorting Asian DBCS data in a specific code page,
because DBCS data is actually a mixture of single-byte and double-byte widths, while Unicode characters are fixed-width.
Non-Unicode :
Non Unicode is exactly opposite to Unicode. Using non Unicode it is easy to store languages like ‘English’ but not other Asian languages that need more bits to store correctly otherwise truncation will occur.
Now, let’s see some of the advantages of not storing the data in Unicode format:
1. It takes less space to store the data in the database hence we will save lot of hard disk space.
2. Moving of database files from one server to other takes less time.
3. Backup and restore of the database makes huge impact and it is good for DBA’s that it takes less time
Non-Unicode vs. Unicode Data Types: Comparison Chart
The primary difference between unicode and non-Unicode data types is the ability of Unicode to easily handle the storage of foreign language characters which also requires more storage space.
Non-Unicode
Unicode
(char, varchar, text)
(nchar, nvarchar, ntext)
Stores data in fixed or variable length
Same as non-Unicode
char: data is padded with blanks to fill the field size. For example, if a char(10) field contains 5 characters the system will pad it with 5 blanks
nchar: same as char
varchar: stores actual value and does not pad with blanks
nvarchar: same as varchar
requires 1 byte of storage
requires 2 bytes of storage
char and varchar: can store up to 8000 characters
nchar and nvarchar: can store up to 4000 characters
Best suited for US English: "One problem with data types that use 1 byte to encode each character is that the data type can only represent 256 different characters. This forces multiple
encoding specifications (or code pages) for different alphabets such as European alphabets, which are relatively small. It is also impossible to handle systems such as the Japanese Kanji or Korean Hangul alphabets that have thousands of characters."<sup>1</sup>
Best suited for systems that need to support at least one foreign language: "The Unicode specification defines a single encoding scheme for most characters widely used in businesses around the world.
All computers consistently translate the bit patterns in Unicode data into characters using the single Unicode specification. This ensures that the same bit pattern is always converted to the same character on all computers. Data can be freely transferred
from one database or computer to another without concern that the receiving system will translate the bit patterns into characters incorrectly.
https://irfansworld.wordpress.com/2011/01/25/what-is-unicode-and-non-unicode-data-formats/
Thanks Shiven:) If Answer is Helpful, Please Vote

Table count comparison in a SSIS package

Similar Messages

Maybe you are looking for