Run Multiple SSIS Packages in parallel using SQL job

Hi ,
We have a File Watcher process to determine the load of some files in a particular location. If files are arrived then another package has to be kick-started.
There are around 10 such File Watcher Processes to look for 10 different categories of files. All 10 File Watcher have to be started at the same time.
Now these can be automated by creating 10 different SQL jobs which is a safer option. But if a different category file arrives then another job has to be created. Somehow I feel this is not the right approach.
Another option is to create one more package and execute all the 10 file watcher packages as 10 different execute packages in parallel.
But incase if they don’t execute in parallel, i.e., if any of the package waits for some resources, then it does not satisfy our functional requirement . I have to be 100% sure that all 10 are getting executed in parallel.
NOTE: There are 8 logical processors in this server.
(SELECT cpu_count FROM sys.dm_os_sys_info
i.e., 10 tasks can run in parallel, but somehow I got a doubt that only 2 are running exactly in parallel and other tasks are waiting. So I just don’t want to try this option.
Can someone please help me in giving the better way to automate these 10 file watcher process in a single job.
Thanks in advance,
Raksha
Raksha

Hi Jim,
For Each File Type there are separate packages which needs to be run.
For example package-A, processes FileType-A and package-B processes FileType-B. All these are independent processes which run in parrallel as of now.
The current requirement is to have File Watcher process for each of these packages. So now FileWatcher-A polls for FileType-A and if any of the filetype-A is found it will kick start package-A. In the same way there is FileWatcher-B, which looks for FileType-B
and starts Package-B when any of FileType-B is found. There are 10 such File Watcher processes.
These File Watcher Processes are independent and needs to start daily at 7 AM and run for 3 hrs.
Please let me know if is possible to run multiple packages in parallel using SQL job.
NOTE: Some how I find it as a risk, to run these packages in parallel using execute package task and call that master package in job. I feel only 2 packages are running in parallel and other packages are waiting for resources.
Thanks,
Raksha
Raksha

Similar Messages

Execute the SSIS packages (2008 version) using SQL job 2008

We need to execute the SSIS packages (2008 version) using SQL job Please help with details information how to execute it.
1.Protection level of the SSIS packages.
2.Configuration File (Curently using XML file)
3.Please provide details how to create Credential (Windows User etc) Do we need to create windows cred explicitly to run the Job etc..
4.Please provide details for creating Proxy.
5.How can we configure the xml file since the xml is reading from local box but when the package is moved from developement box to production box
then the local drive might not be mapped with Production box .
6.Roles that need to be tagged for the Proxy/Credential.
7.Which is the best and safest way to execute ssis packages either msdb deployment or filesystem deployment and other necessary details.

Hi,
If you want to Execute your SSIS package using SQL job then best solution is to first deploy your Package
either using msdb deployment or filesystem deployment.The simplest approach to deployment is probably to deploy to the file system.
So, here is the deployment steps;
There are four steps in the package deployment process:
    The first optional step is optional and involves creating package
configurations that update properties of package elements at run time. The configurations are automatically included when you deploy the packages.
    The second step is to build the Integration Services project to create a package deployment utility. The deployment utility for the project contains the packages that you want to deploy
    The third step is to copy the deployment folder that was created when you built the Integration Services project to the target computer.
    The fourth step is to run, on the target computer, the Package Installation Wizard to install the packages to the file system or to an instance of SQL Server.
you can use below link also;
http://bharath-msbi.blogspot.in/2012/04/step-by-step-ssis-package-deployment.html
After that you need to Schedule package using SQL Agent then Have a look at the following link. It explains how to create an SQL job that can execute an SSIS package on a scheduled basis.
http://www.mssqltips.com/sqlservertutorial/220/scheduling-ssis-packages-with-sql-server-agent/
Deployment in msdb deployment or filesystem deployment has there own merits and demerits.
For security point of view msdb deployment is good.
it provides Benefits from database security, roles and Agent interaction
Protection Level;
In SQL Server Data Tools (SSDT), open the Integration Services project that contains the package.
Open the package in the SSIS designer.
If the Properties window does not show the properties of the package, click the design surface.
In the Properties window, in the Security group, select the appropriate value for the
ProtectionLevel property.
If you select a protection level that requires a password, enter the password as the value of the
PackagePassword property.
On the File menu, select Save Selected Items to save the modified package.
SSIS Provide different Protection levels;
    DontSaveSensitive
    EncryptSensitiveWithUserKey
    EncryptSensitiveWithPassword
    EncryptAllWithPassword
    EncryptAllWithUserKey
    ServerStorage
For better undertsnading you can use below link;
http://www.mssqltips.com/sqlservertip/2091/securing-your-ssis-packages-using-package-protection-level/
Thanks

Running multiple SSIS packages using SQL Server Agent question.

I have a multitude of SSIS packages I want to run using SQL Server Agent. What would the best practice be for running these jobs using SQL Server Agent? One job per package or running all pakages from one job? If you have an answer can
you explain the technical reasoning behind your answer? Thanks in advance.
Stan Benner

Hi, maybe a bit more analysis will give a better answer
Do all the packages have to run in sequence? (if yes, single job better)
Can the list of packages to be executed be grouped by dependency (ex package 1,2 and 5 must run in sequence and can be executed by one job, while package 3,4 are not dependent on package 1,2 and 5 can be run by a separate job).
Can any jobs be run in parallel?
How often will the package execution sequence change?
How will you deploy your packages and job? (the more jobs to create the more install script needed and upgrade scenarios become messy).
My personal preference:
I create ONE ssis package which is executed by ONE sql agent job. lets call this 'PackageExecutionWrapper.dtsx'
PackageExectionWrapper then contains multiple 'Execute Package' tasks for the packages you want to execute.
In the package you can apply any package execution rules - which packages have to run after the other, which packages can run concurrently, which packages should only run if previous succeeded.
If you need to change the sequence, simple, just update the PackageExecutionWrapper package.

Error coming after using an ssis package to create an sql job

Hi
I have created an SSIS package and try to create a job on the same when i run the job it if failing an dgiving me the below error
Executed as user: xxxxx\FESGSQLA. Microsoft (R) SQL Server Execute Package Utility Version 10.50.4000.0 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights reserved.    Started: 12:33:03 AM Error: 2014-03-19
00:33:03.12     Code: 0xC0011007     Source: {DD3B52A8-850A-4328-94DC-CF51B4376BCB}      Description: Unable to load the package as XML because of package does not have a valid XML format. A
specific XML parser error will be posted. End Error Error: 2014-03-19 00:33:03.12     Code: 0xC0011002     Source: {DD3B52A8-850A-4328-94DC-CF51B4376BCB}      Description: Failed
to open package file "D:\Users\xxxxxx\Desktop\I11_Load_SAP_Recon_Data.dtsx" due to error 0x80070003 "The system cannot find the path specified.". This happens when loading a package and the file cannot be opened or loaded correctly
into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. End Error Could not load package "D:\Users\H139933\Desktop\I11_Load_SAP_Recon_Data.dtsx"
because of error 0xC0011002. Description: Failed to open package file "D:\Users\xxxxxx\Desktop\I11_Load_SAP_Recon_Data.dtsx" due to error 0x80070003 "The system cannot find the path specified.". This happens when loading a package
and the file cannot be opened or loaded correctly into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. Source: {DD3B52A8-850A-4328-94DC-CF51B4376BCB}
Started: 12:33:03 AM Finished: 12:33:03 AM Elapsed: 0.046 seconds. The package could not be found. The step failed.
Please throw some light on the issue and stpes to resolve.
Thanks
Geeth If you feel that the answer which i gave you is Helpful please select it as Answer/helpful.

The package was created using a newer SSIS version than you have in the target (server).
Arthur My Blog

Error "To run a SSIS package outside of SQL Server data tools you must install task name used in package of Integration service or highter.

Hello Team,
I am trying to execute a SSIS package from web page. When i try to do that i am getting following error.
"To run a SSIS package outside of SQL Server data tools you must install <task name used in package> of Integration service or highter."
In my machine Integration Services are installed and its service is also in running state.
Please help me on this.
Thanks,
Ramesh
Thanks, Ramesh Arige

The SSIS package developed using SSIS 2008 Server R2 and Integrations Services 10.0 is exists in my machine. Is this wrong configuration, please help me on this.
I am using the below code copied from CodeProject
Thank you so much for responding.
Ramesh
Thanks, Ramesh Arige
Which way are you using from the provided blog? Using 1) C# Code or 2) C# and Stored Procedure?
Cheers,
Vaibhav Chaudhari
MCP, MCTS, MCSA (SQL Server 2012)

Run the SSIS Package using sql server Agent--Memory utilization Problem

Hi,
I have Created one SSIS Package it was working fine in BIDS but in SQL Server Agent it takes all the server memory and server gets very slow now.
in that ssis package i am using merge join,sorting tasks.Can you please suggest me how to handle in the memory usage for this..pls refer the below screen

Hi aravindhrm,
You can enable BufferSizeTuning logging event on the Data Flow Task to get how many rows in buffer. Or you can use Performance Counter to obtain more details about the buffers.
http://microsoft-ssis.blogspot.com/2013/04/performance-best-practice-more-rows-per.html
http://www.sql-server-performance.com/2009/ssis-an-inside-view-part-3/
Then, you can tweak the DefaultBufferMaxRows and DefaultBufferMaxSize size settings.
Besides, you can also go through the blog mentioned by Visakh and try suggestions such as increase OS pagefile size.
Regards,
Mike Yin
TechNet Community Support

Getting an error running an SSIS package from SQL server agent

This is the first time I'm running an SSIS package in SQL server agent.
I'm running a SQL server agent job and getting this error message:
Am I using the correct options? (options list below)
Message
Executed as user: SITEX\stxdatasqlexec. Microsoft (R) SQL Server Execute Package Utility Version 10.50.1600.1 for 32-bit Copyright (C) Microsoft Corporation 2010. All rights reserved.    Started: 1:42:00 PM Error: 2014-04-17
13:42:01.01     Code: 0xC0011007     Source: {B72BF421-785A-44EC-9570-D2A40B046CA7}      Description: Unable to load the package as XML because of package does not have a valid XML format. A
specific XML parser error will be posted. End Error Error: 2014-04-17 13:42:01.02     Code: 0xC0011002     Source: {B72BF421-785A-44EC-9570-D2A40B046CA7}      Description: Failed
to open package file "C:\Users\halexandrovich\Documents\Nbr_Rolls_base.dtsx" due to error 0x80070003 "The system cannot find the path specified.". This happens when loading a package and the file cannot be opened or loaded correctly
into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. End Error Could not load package "C:\Users\halexandrovich\Documents\Nbr_Rolls_base.dtsx"
because of error 0xC0011002. Description: Failed to open package file "C:\Users\halexandrovich\Documents\Nbr_Rolls_base.dtsx" due to error 0x80070003 "The system cannot find the path specified.". This happens when loading a
package and the file cannot be opened or loaded correctly into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. Source:
{B72BF421-785A-44EC-9570-D2A40B046CA7} Started: 1:42:00 PM Finished: 1:42:01 PM Elapsed: 0.047 seconds. The package could not be found. The step failed.
Here are the options I'm using:
package source - file system
data source destination = my SQL server db
execution option - use 32 bit runtime
command line entry:
- /FILE "C:\Users\halexandrovich\Documents\Nbr_Rolls_base.dtsx" /CONNECTION DestinationConnectionOLEDB;"\"Data Source=esxdspr-sql01.sitex.com;Initial Catalog=SuesTbls;Provider=SQLNCLI10;Integrated Security=SSPI;Auto Translate=false;\""
/CONNECTION SourceConnectionFlatFile;"\\sitex.com\dfs\Production\SFTPData\LPSEFTProcess\FromAADFS\Rpt_Nbr_Rollsperbatch_20140331_20_0.txt"
/X86 /CHECKPOINTING OFF /REPORTING E
I checked the filename so it must be the XML format it's complaining about.

Thank you, pituach.
I tried running the package using just my c drive without partition, C:\Nbr_Rolls.dtsx, but still getting the same
error. I do not see any option on this site to post my package. I know it's not the path that's the issue. Our support desk already fixed that because I was getting a different message before.
Message
Executed as user: SITEX\stxdatasqlexec. Microsoft (R) SQL Server Execute Package Utility Version 10.50.1600.1 for 32-bit Copyright (C) Microsoft Corporation 2010. All rights reserved.    Started: 11:23:00 AM Error: 2014-04-18
11:23:00.44     Code: 0xC0011007     Source: {2BEE62EB-DD08-4867-9F5C-660167964650}      Description: Unable to load the package as XML because of package does not have a valid XML format. A
specific XML parser error will be posted. End Error Error: 2014-04-18 11:23:00.45     Code: 0xC0011002     Source: {2BEE62EB-DD08-4867-9F5C-660167964650}      Description: Failed
to open package file "C:\Nbr_Rolls.dtsx" due to error 0x800C0006 "The system cannot locate the object specified.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of
either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. End Error Could not load package "C:\Nbr_Rolls.dtsx" because of error 0xC0011002. Description: Failed
to open package file "C:\Nbr_Rolls.dtsx" due to error 0x800C0006 "The system cannot locate the object specified.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of
either providing an incorrect file name was specified when calling LoadPackage or the XML file was specified and has an incorrect format. Source: {2BEE62EB-DD08-4867-9F5C-660167964650} Started: 11:23:00 AM Finished: 11:23:00 AM
Elapsed: 0.125 seconds. The package could not be found. The step failed.

Running a SSIS package with SQL Job and Linked Server

I have a SSIS 2008 package. In one of the Script task I am calling a stored procedure which is using Openquery using linked server. I deployed this package with protection level as "EncryptWithPassword" and gave a password to the package.
Created a SQL job and edited its command line to include the password. If I login to SQL Server Mgmt Studio with Windows Authentication and run the job manually it runs fine. But when I schedule it then I get an error that "The Communication link to Linked
server failed".
Please help

Hi Vivek.B,
The issue should occur because the SQL Server Agent Service Account or SQL Agent Proxy account under which the job step runs doesn’t have sufficient permissions on the linked server.
If the job owner is the sysadmin fixed server role, the job can be run under the SQL Server Agent Service Account or a proxy account, then please make sure the SQL Agent Service Account or the proxy account has corresponding login on the linked server. If
the job owner is not a sysadmin fixed server role, the job must run under a proxy account. In this case, make sure the proxy account has a corresponding login on the linked server.
Reference:
http://blogs.msdn.com/b/dataaccesstechnologies/archive/2009/10/13/who-owns-my-job-and-who-runs-it.aspx
Regards,
Mike Yin
TechNet Community Support

SSIS: To run a SSIS package outside of SQL Server Data Tools you must install SCR - DP1 Connections of Integration Services or higher

We have SSIS installed on a machine that is not part of a cluster but it is accessible by the cluster.
Our job we have running on the cluster is failing with the following error:
Error: 2014-01-31 09:14:37.52 Code: 0xC000F427 Source: SCR - DP1 Connections
Description: To run a SSIS package outside of SQL Server Data Tools you must install SCR - DP1 Connections of Integration Services or higher. End Error
Any advice or information on how to resolve this would be great.
Many Thanks.

Hi NessaBella,
Integration Services service is not a cluster-aware service, and does not support failover from one cluster node to another. Therefore, in a clustered environment, Integration Services should be installed and started as a stand-alone service on each node
in the cluster.
Based on the error message, it seems that SSIS is not installed on the cluster node on which the job was running. Although SSIS is installed on a machine that is not part of the cluster and can be accessed by each cluster node, the SSIS service installed
on a remote server cannot be used a cluster node. So, please install the shared feature SQL Server Integration Services on each cluster node. Besides, if certain packages need to run in 32-bit mode in certain jobs and the SQL Server installed is 64-bit version,
you also need to install BIDS/SSDT on the cluster node to get the 32-bit runtime of SSIS.
References:
Integration Services (SSIS) in a Cluster
Loading and Running a Remote Package Programmatically
Regards,
Mike Yin
TechNet Community Support

Please help with this issue "To Run a SSIS Package outside of Server Data Tools You must install Drived Column of Integration sercvies of higher" From C# app

i have searched all over the google and here on MSDN(I have read all the threads related to my problem ) , i am new to SISS and doing my clg project ,
I have this package which loads data from dim tables to a fact table , my Package runs without any problem in BIDS (2012)
I have installed SQl Server integration services on SQL and service is also running in services.msc,
the problem comes when i execute my package using my C# application i am using MSDN Method to execute a SSIS package programmatically given here.
"http://msdn.microsoft.com/en-us/library/ms136090.aspx"
For loading my FactTable i have used a Derived columns and some lookups ,
Now if i don't use derived columns and looks ups and just use source and destination component then my package runs fine from my C# application but if i use derived columns or looks ups etc it gives the following error
"Error in Microsoft.SqlServer.Dts.Runtime.TaskHost/SSIS.Pipeline: To Run a SSIS package outside of SQL Server Data Tools you must Install Derived Column of Integration services of higher"
I am searching and trying to resolve this issue from previous 2 days but without any luck please help or guide me what is solution to this problem ,,i will be very thankful
I have also found that integration services that are running on my system have version 10.0
but the BIDS have version Version 11.0.3402.0
IS this causing problem ? if yes then what i have to do ?

Hi BlaxButt,
The package is developed in SSDT installed by SQL Server 2012, so it is a SSIS 2012 package. However, the Integration Services you have installed is SQL Server 2008 R2 version. To run the package outside SSDT, you need SSIS 2012 installed. The reason why
the package runs fine with only Source and Destination components is that such a simple package can be executed by the DTExec utility installed by SQL Server 2012 Data base Engine or Client Tools (SQL Server Import and Export Wizard). To run a package that
uses other tasks/components outside SSDT/BIDS, the SSIS runtime is also required except the DTExec utility. To obtain the SSIS 2012 runtime, we have to install SSIS 2012 on the server where the package runs.
Reference:
http://stackoverflow.com/questions/19989099/getting-error-running-ssis-package-on-non-ssis-server
Regards,
Mike Yin
TechNet Community Support

How to Run 2012 SSIS packages on 2008R2 Version

HI Friends,
I have scheduled a job to run 2012 SSIs packages on 2008R2. I am getting this error can you please Give any ideas how can I run them without converting!
ERROR Is :
Package migration from version 6 to version 3 failed with error 0xC001700A "The version number in the package is not valid. The version number cannot be greater than current version number."

Hello,
There is backward compatibility in SSIS but no forward compatibility, meaning, when 2008 R2 is developed they were aware of all versions less than 2008 R2 and included execution scenarios , this is backward compatibility
while 2008 R2 is being developed , 2012 is not yet developed. so we cannot execute a 2012 package on 2008 R2 Server where vice versa is possible
Correct.
The "backwards compatibility" is implemented by SSIS upgrading older versions of packages to the current version of SSIS when executing. If the upgrade fails, you're out of luck.
MCSE SQL Server 2012 - Please mark posts as answered where appropriate.

How to Load Multiple Files in Oracle Database using Sql Loader

Hi All,
I want to import multiple files in my DB using Sql*Loader. Please tell me the Syntax, how can I import multiple files using one Control File?
Thanks & Regards,
Imran

Hi,
You might get a good response to your post in the forum dedicated to data movement , including SQL*Loader . You can find it here Export/Import/SQL Loader & External Tables
Regards,

How to store Multiple Ec\xcels into Table Using sql loder

Plese guide me to store Multiple Ec\xcels into Table Using sql loder

I am not clear with your question. Do you want to load multipel Excel Files into a table. If so you can follow this link.
http://www.orafaq.com/wiki/SQL*Loader_FAQ#Can_one_load_data_from_multiple_files.2F_into_multiple_tables_at_once.3F
Thanks,
Karthick.

Can we run multiple sapinst programs in parallel?

We are planning to upgrade our EP, XI and BW simultaneously. These SIDs all reside on the same server and same OS. Is it possible to run multiple sapinst programs in parallel? I know it may require specifying separate ports for the 2nd and 3rd instance of sapinst. For example: sapinst 1 gets default ports 21212, 21213. sapinst 2 will take 21214, 21215 and sapinst 3 takes 21216 and 21217.
Has anyone attempted this sort of a thing before? Appreciate your thoughts.

Never done that before, maybe it works .. however it could run into problems when the sapinst are trying to access the same paths / files.
Regards,
Siddhesh

To configure SSIS packages out of these SQL Server Standard, Web and Workgroup editions which is preferred?

Hi All,
To configure SSIS packages out of these SQL Server Standard, Web and Workgroup editions which is preferred?
Grateful to your time and support. Regards, Shiva

Hi Shiva,
SSIS designer and runtime, basic transforms, and data profiling tools are only supported by Standard and higher editions. Besides, some advanced SSIS adapters such as High performance Oracle/Teradata destination and Dimension processing destination adapter
are only supported in Enterprise edition. So, if you need to develop/modify SSIS packages on this server, Standard or even Enterprise editor is recommended.
Reference:
http://msdn.microsoft.com/en-us/library/cc645993.aspx#SSIS
Regards,
Mike Yin
TechNet Community Support

Run Multiple SSIS Packages in parallel using SQL job

Similar Messages

Maybe you are looking for