100 packages for 100 seperate files in SSIS 2012---------

Hi,
I am new to this Microsoft Forum.
I have worked for last 3 years in MSBI(mainly SSIS and SSAS). Recently I have joined a new company and been assigned to new project. Here we have to first build Operational Data Stores from 50 source files (which is built-100 tables). then from this we need
to pull data in Data Warehousing and later Cube.
My question is: My architect telling me to create 100 separate packages for each source file from source to staging (one to one source to staging mapping) and connect it through master package. then when we pull data from staging to Dev. there would
be 30 destination tables so 30 packages!!
I have not designed any ETL in this way.
the files can be segregated as per daily, monthly, weekly and quarterly, so I would have created daily, monthly and so on package. In each package I would have included source to staging and staging to development phase.
His points are, testing team can work separately on each package, multiple developers can work on separate files at the same time. can anyone tell me how feasible the design is.....advantage/disadvantage... But Would it be good to handle
100 odd packages for one database where I can achieve this by creating 6 packages.
thanks, Rims

Hi Rims,
Personally, I agree with your architect's opinion from the perspective of "multi-developer" as well as "unit test your SSIS packages".
If your main concern of creating dedicated SSIS packages is time consuming, I would recommend you to have a look at BIML / Mist. You can quickly get the idea from this YouTube video: https://www.youtube.com/watch?v=RKi2zIoFUAg
Basically, BIML / Mist can help you to build a SSIS code generator to loop through your database objects and generate SSIS packages on the fly. If you take this approach, it would be faster to create 100 SSIS packages than 6 customized SSIS packages.
Regards,
George Qiao

Similar Messages

How to create a dynamic SSIS package for multiple flat file destinations

Hi,
I have to create a ssis package which has single data flow task and inside that I have 23 source (sql- select * from - statements)- destination (flat files, 23 distinct) connection.
Now for each product I have to create separate SSIS package (i.e. if prod=abc then these read select * from abc_tables and 23 abc_ txt files)
I want to do it dynamically, means only single package and inside that variables will take select * values for each source-dest connection (so i believe 23 variables) and same for destination flat files.
Let me know. :)
ANK HIT - if reply helps, please mark it as ANSWER or helpful post

Sorry It seems you're contradicting yourself. you say I know my source and dest structure and the you're
asking all I want is to have a dynamic structure
what does that mean?
and reading your next sentence
I want to run a package for 5 products, instead of creating 5 ssis packages with 23 source- dest connection, I would
like to have one with only 23 source- dest connections
What I feel is what you're looking for is to have a looping structure to loop through each of the 5
products.
In that case what you could do is this
1. Create a object variable in SSIS
2. Use a Execute SQL Task to populate the variable with all available products (I think you'll have a master table for that). Set ResultSet property to Full ResultSet and then in ResultSet tab map Object variable to 0 th index
3. use a ForEachLoop container with ADO .NET recordset enumerator and map to object variable. Create a variable of datatype same as that of Product identifier field to get individual values out
4. Inside loop create your data flow task with 23 source destination connection. In the query part use a parameter for product field and map it to the variable containing product value to get only data for the product.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Scripting Language Options for the Script Task Options SSIS 2012

Can python or perl be used as a script language in the drag and drop script task? I believe only VB.Net and C# are, but I see under SSDT Tools -> Options there are some choices for Python Debugging? I'd like to use python as my SSIS scripting language
in some cases. If it can not be used as the script task language, what are the Python options under Tools -> Options for?
Any experts out there who can illuminate?
Thanks all

Hello,
In SSDT in a SSIS Script task you can only use Visual Basic.NET and C#, no other programming / scripting languages.
SSDST is a plugin for Visual Studio and in VS you will find several options which are irrelevant for SSIS projects, like this Phyton debugging option.
Olaf Helper
[ Blog] [ Xing] [ MVP]

Ssis 2012 configuration package wizard is hanging

Hi,
I am trying to edit the dtsconfig package xml file in SSIS 2012 VS envoirnemnt and the wizard is hanging up for minutes until I need to close VS.
Has anyone experienced this problem and what can be done to solve other than editing the XML file directly.

Hi danisroyi2 and Mike Yin Pactera
I am also experiencing this issue.
Edit an existing package configuration and click next.
It hangs on the loading properties form until you eventually (I have waited for over half an hour) have
to kill the process in task manager.
thanks

How to migrate to SSIS 2012 from Integration Services dtsx file?

I am struck at migrating the dtsx file to SSIS 2012. How to migrate that dtsx file to SSIS 2012.I made it to migrate from SQL2000 to SQL 2005 dtsx. Now I want to migrate or upgrade it to SSIS 2012.How to achieve it.Help me please.
sammy

Hi sammy,
Based on your description, you want to upgrade SQL Server 2005 packages to SQL Server 2012. If there are some individual packages, I suggest that we can directly add the SQL Server 2005 package to an existing project in SQL Server Data Tools (SSDT). For
more details, please see:
Right click on the SSIS Packages folder in the Integration Services Project in SSDT and Select Add Existing Package from the context menu.
In the Add Copy of Existing Package dialog box select Package location as File System from the dropdown.
Click on the ellipsis button next to the Package path textbox.
Locate the SSIS package (.dtsx file) which you want to add from the file system.
Click OK in the Add Copy of Existing Package dialog box to finish adding and upgrading the package.
If those packages are located in a project, we can open a SQL Server 2005 Integration Services (SSIS) project file in SSDT, and then use the SSIS Package Upgrade Wizard to upgrade multiple packages in the project.
If there are any questions, please feel free to ask.
Thanks,
Katherine Xiong
Katherine Xiong
TechNet Community Support

SSIS 2012 - Parent/Child Package "Object" Type Variables

How do I pass Variables of Object type from Parent to Child packages?
I am using the SSIS 2012 project deployment model.
Thanks

You can do it as outlined in
http://sqlblog.com/blogs/andy_leonard/archive/2010/01/25/ssis-snack-passing-parent-starttime-to-the-child-package.aspx
But
When you need an object it is typically a recordset, then just pass the results, or execute the query in a package.
Arthur My Blog

Creation of one single config file for 100 packages

Hello Everyone,
I need to create a single config file for multiple packages. Is it possible to do it?
Could you please let me know how can it be achieved?
Regards,
Chinni

Its possible
Depending on whether all packages contain same configuration items and names you can use any of below approaches
1. If all package configuration items and names are same, you can point all packages to same config file and it will use values from file for all of them
2.If packages have different set of conigurations, then one method is to use parent package (wrapper package) to call and execute all of them. Then define all configurations as variable values in parent and add a config file in it. Pass only required configurations
to each of child packages by using parent package configuration method.
http://www.sqlis.com/sqlis/post/Using-Parent-Package-Variables-in-Package-Configurations.aspx
The parent package has to call child packages using execute package task
Then change any of values in your config file for parent package and change gets reflected to child package property values.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Error 100 File contains erroneous data. Normally for user data files.

Hello. We have a LabVIEW program that reads in test steps and data in order to execute a test sequence. Recently we had to apply Retina and Gold Disk security patches in accordance with DOD security policies. Now we are getting the following error:
Error 100
LabVIEW: File contains erroneous data. Normally for user data files.
We have not changed the code or files that the program is reading in. My guess would be it is some sort of permission issue. However, we have given the user modify permission to the entire C drive, and still get this error. Does anyone have any ideas on what could be causing it? Thanks!

Do you have any backup copies of the files, by any chance? Is it possible the files were modified somehow (perhaps something extra was added when the new security measures were implemented)?
How is the file being accessed? Is it occurring on the local machine, or are the files accessed from a remote location?
Caleb Harris
National Instruments | Mechanical Engineer | http://www.ni.com/support

Doc2text process using 100% cpu for way too long

I've run into this issue before, and have never received a satisfactory resolution.
When we publish certain PDF documents via collab to the KD, the doc2text process on our portal servers (also our API servers) spin up to 100% CPU for about 90 seconds. This causes the publish from collab to fail, so we resort to directly uploading the file to the KD via content upload.
1) Why is doc2text running on our portal servers in the first place? Is this a result of the portal server or api service? If i move the API service off to another machine, will that remove this CPU hog from our front end servers?
2) Has anyone seen this before, or found a way around it? Is there a newer version of doc2text in 10gR3 that isn't so terrible?
I'm preparing to move to 1-cpu virtuals in our new environment so any process that takes up 100% of the CPU is going to kill performance.

Doc2text is native search client functionality used by portal, automation, ws, collab, publisher, etc. The portal components use it when indexing content to search server. It is installed seperately with each portal component. If you have multiple portal components installed on the same box, the ones that use Doc2text will share it from the common folder where it is located. Doc2text uses different libraries for the purpose of raw text extraction and processing, the processed information being sent in a proprietary format to the search server.
Doc2text is a processor heavy component as it needs to execute intensives tasks necessary to parse and extract raw data from such file formats as Microsoft Office and Adobe Acrobat. Some files may be more complex to extract data from and take longer for this task to complete.
The best way to increase performance would be to install the various portal components onto their own machines. For example not installing portal and automation on seperate servers. This way when you run a crawler job that imports files, the running of doc2text will not use cpu cycles on the portal web server.

Why can I only create 4 forms when I have purchased the entire package at 100+ dollars?

Why can I only create 4 forms when I have purchased the entire package at 100+ dollars?

FormsCentral can be used from anywhere, but to purchase a subscription you need a credit card with the mailing address in one of the countries supported for purcahse:
http://forums.adobe.com/docs/DOC-1375
The partner can purchase if they have access to a card in one of those countries. Alternatively they do not need to purchase a subscription to edit forms that you've created, they do in order to create multiple forms of their own, but by your sharing forms with them at the "Co-author" level they have access to the editing. Here are the instructions to share a form:
http://forums.adobe.com/docs/DOC-2462
Thanks,
Josh

Package Configuration Files Not loading SSIS 2012

I've migrated an SSIS 2008 project to SSIS 2012 that uses xml configuration files. When I execute the first package through the IDE, the values in the config file are not populating variables as they did in SSIS 2008. I've enabled Package Configurations
and added the config files and not sure what I could be doing wrong. Could anyone offer any suggestions.
Thanks

Someone else migrated it for me before I had vs2012 installed. So I decided I would try migrating it myself. Now the config files are loading. When I migrated it there was an option for ignore config files that was checked. I unchecked it. Could that have
been the problem?
I just have one more issue now...Whenever I open the solution, it loads the config files, but two passwords are encrypted. For some reason it tries to login to the database when I load the package. This didn't occur in SSIS 2008. Do you know of any way to
disable this? I get about 2 failures before I'm locked out of the db.
Thanks

Audio Bin - Select unused - Delete - Hangs on 100% CPU for long time

Hi
I am running Logic Studio 9.0.2 on OS X 10.5.8 on a Mac Pro. Logic was upgraded from version 7.
In Audio Bin and "Select Unused" and then "Delete", the application might hang on 100% CPU for hours on large projects.
Anyone with an idea of what the problem might be? This was not a problem i Logic Pro 7.
Thanks!
GD

I have this same problem, though I'm on 10.5.7. I was hoping maybe updating the OS would help but apparently not. I also get very slow file-selection windows sometimes, like when I go to bounce a file. Navigating to a specific folder is painfully slow. Would like to hear if any Snow Leopard users are experiencing this.
Bull

Webcam Blackmail asking for $100 otherwise will po...

URGENT: Webcam Blackmail asking for $100 otherwise will post to porn sites and youtube
Hi,
the user with username: fooled me into thinking it is a girl, and now has acquired a personal video of me. He/She is now threatening me that they will post the video to porn sites and youtube, if I dont pay them $100. They ask me to send it through Western Union, the person name is[removed for privacy], in Philippines. Also the persons email address is [removed for privacy]
This person is a criminal. What should I do, should I try to meet the demands hoping the file goes away?
Thank you

report them to Skype so they can do something with regard to their account, then also report them to your local police.
IF YOU FOUND OUR POST USEFUL THEN PLEASE GIVE "KUDOS". IF IT HELPED TO FIX YOUR ISSUE PLEASE MARK IT AS A "SOLUTION" TO HELP OTHERS. THANKS!
ALTERNATIVE SKYPE DOWNLOAD LINKS | HOW TO RECORD SKYPE VIDEO CALLS | HOW TO HANDLE SUSPICIOS CALLS AND MESSAGES
SEE MORE TIPS, TRICKS, TUTORIALS AND UPDATES in
| skypefordummies.blogspot.com |

Can SSIS 2012 execute packages built on 2008 R2 BI for 2008 R2 SQL Server?

We have a query regarding SSIS instalattion on application server. As SQL Server 2008 R2 is getting out of life, DBAs are suggesting to install SSIS 2012. Now our concern is
that our SQL Server DB Server is on 2008 R2, and developers are building packages using SSIS 2008.
Considering we deploy file packages, and backward compatibility, theoretically it shall work. There are 3 things here -
1. {2008 R2 SSIS Package} ------ Deployed On -------> 2. {Server with 2012 SSIS installation} ------- Connects to -------> 3. {SQL Server 2008 R2 DB Instance}
Any recommendation on this. Would it work? Anything which could cause issue?
As we have to pay for license, do not want be in a situation where we pay for SSIS 2012 installation on server (#2 above) and then realize that there is some issue
Regards, Amit Gupta

Hi Amit,
Just as Visakh and hsbal said, there will be no issues as SSIS 2012 will work with standard previous versions of SQL Server packages. But some custom SSIS Tasks can always throw error after upgrading packages from SQL Server 2008 R2 to SQL Server 2012.
About this, please remember to do the following things:
Change your custom tasks and components to target the .NET 4.0 Framework in each of the project properties, in the Application Tab, under Target framework.
Update the assembly references in each of your projects to point to the appropriate SqlServer dlls.
Build and copy your project dlls to the appropriate directory, depending on what kind of components you are updating, i.e. Program Files (x86)\Microsoft SQL Server\110\DTS\PiplelineComponents
Register your components in the GAC.
Close Visual Studio and then reopen; open your SSIS packages and refresh the SSIS Toolbox, by right clicking in the tool box and selected Refresh Toolbox.
Besides, if some of your script task throw errors to you, we can open the script task in VSTA, insert Breakpoint to debug the scripts, then find the root cause. For more details, please see:
Debug a Script by Setting Breakpoints in a Script Task and Script Component
For more detailed information regarding to the license issue, please call 1-800-426-9400, Monday through Friday, 6:00 A.M. to 6:00 P.M. (Pacific Time) to speak directly to a Microsoft licensing specialist. For international customers, please use the Guide
to Worldwide Microsoft Licensing Sites to find contact information in your locations.
You can also visit the following site for more information and support on licensing issues:
http://www.microsoft.com/licensing/mla/default.aspx
Thanks,
Katherine Xiong
If you have any feedback on our support, please click
here.
Katherine Xiong
TechNet Community Support

When saving a file in [.jpg] format, i am asked to choose a quality from 1 to 100. what changes in the file created based on the quality parameter chosen?

when saving a file in [.jpg] format, i am asked to choose a quality from 1 to 100. what changes in the file created based on the quality parameter chosen?. i would like to know what changes, so in the future i can set my camera to a setting that will give me the highest quality to begin with,allowing me to make crops and still preserve the quality.
thank you
dovid

It's the level of compression. Lower number, more aggressive compression, more visual artifacts.
Aside from that you should never use jpeg as a working format. The compression is destructive and cumulative, and the file deteriorates every time you resave it.
Use TIFF or PSD, and if you need jpeg save out a copy as a single final step.

100 packages for 100 seperate files in SSIS 2012---------

Similar Messages

Maybe you are looking for