Parsing PROMPT values to custom SSIS package

Hi,
I need to extract data from our customers ERP system. I have created a SSIS package doing this. I'm using two SSIS variables 'STARTDATE' and 'ENDDATE' to specify the range in transactions I want to extract. I have tested the SSIS package with the variables pre-filled (in SSIS) from SAP BPC 7.5 and the package is working as expected.
Now i want to change the pre-filled values with values from my PROMPTs in my BPC package. I have created two PROMPTs called BEGIN and END. I have tried to parse the values in different ways, but havn't found the solution. Can anyone guide me in the right direction?
Here the script on my BPC package:
DEBUG(ON)
GLOBAL(STARTDATE,%BEGIN%)
GLOBAL(ENDDATE,%END%)
PROMPT(TEXT,%BEGIN%,"Enter start date:",,"")
PROMPT(TEXT,%END%,"Enter end date:",,"")
INFO(%TEMPFILE%,%TEMPPATH%%RANDOMFILE%)
TASK(CONVERT data,INPUTFILE,%FILE%)
TASK(CONVERT data,OUTPUTFILE,%TEMPFILE%)
TASK(CONVERT data,CONVERSIONFILE,%TRANSFORMATION%)
TASK(CONVERT data,STRAPPSET,%APPSET%)
TASK(CONVERT data,STRAPP,%APP%)
TASK(CONVERT data,STRUSERNAME,%USER%)
TASK(LOAD AND PROCESS,APPSET,%APPSET%)
TASK(LOAD AND PROCESS,APP,%APP%)
TASK(LOAD AND PROCESS,USER,%USER%)
TASK(LOAD AND PROCESS,DATATRANSFERMODE,4)
TASK(LOAD AND PROCESS,CLEARDATA,%CLEARDATA%)
TASK(LOAD AND PROCESS,FILE,%TEMPFILE%)
TASK(LOAD AND PROCESS,RUNTHELOGIC,%RUNLOGIC%)
TASK(LOAD AND PROCESS,CHECKLCK,%CHECKLCK%)
Regards,
Lars

This is confusing, I can't understand why you are getting this error, it seems to me that BPC is trying to look for your stored procedure in your APPSET DB (Source Server) . If your connection object, which is defined in the package is pointing to the correct destination server, then it should work, unless you have modified te connection using Dynamic Script (which it appears not to be the case)
You can try to create the stored procedure in the source system, but I suspect you will get the error message "Object not found, etc... ", becuase your source server doesnt know about your database in your query.
You can try add the following dynamic script, which implicitly tells BPC to override the connection object with the settings you have provided.
DEBUG(ON)
INFO(%DBName%,YourSourcedBName)
INFO(%DSTSrv%,YourDestinationServerName/IP)
PROMPT(TEXT,%BEGINDATE%,"Enter start date:",,"")
PROMPT(TEXT,%ENDDATE%,"Enter end date:",,"")
CONNECTION(SQLconn,ConnectionString,Data Source=%DSTSrv%;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;)
CONNECTION(SQLconn,InitialCatalog,%DBName%)
CONNECTION(SQLconn,ServerName,%DSTSrv%)
OLEDBCONN(Source - LEDGERTRANS,SqlCommand,"EXEC dbo.AXdata '%BEGINDATE%', '%ENDDATE%'")
OLEDBCONN(Source - LEDGERTRANS,OpenRowSet,"EXEC dbo.AXdata '%BEGINDATE%', '%ENDDATE%'")
Make sure you amend the following
YourSourcedBName - Your Destination Server Database Name (The name of the database where you are extracting data from)
DSTSrv - The destination server ne or IP Address
SQLConn - This is the name of your OLE DB connection object defined in the package. This should be the same connection object in which your OLEDB Source is using..
And try to fully qualify your SQL code, meaning try to put the full path, for example
EXEC DBName.dbo.AXdata '%BEGINDATE%', '%ENDDATE%'"
I hope this helps and resolves your error..
Kind Regards
Daniel

Similar Messages

  • How to find a certain usage of a specific value inside an SSIS package

    Hi,
    In T-SQL, there are many tools to search for instances of a specific value in all the objects such as tables and SPs.
    The only way I can think of in SSIS, is to search for a value in the XML of the DTSX, but it is not that user friendly.
    Are there any existing tools or other ways for a value search that can assist ?
    Thanks,
    Dani

    Since a package is just a plain text XML file thus  you are able to perform a text search in the file.
    http://sqlblog.com/blogs/jamie_thomson/archive/2012/06/12/using-find-replace-with-regular-expressions-inside-a-ssis-package.aspx
    Best Regards,Uri Dimant SQL Server MVP,
    http://sqlblog.com/blogs/uri_dimant/
    MS SQL optimization: MS SQL Development and Optimization
    MS SQL Consulting:
    Large scale of database and data cleansing
    Remote DBA Services:
    Improves MS SQL Database Performance
    SQL Server Integration Services:
    Business Intelligence

  • Child package cannot read param value from parent ssis package

    Hi all,
    I think I must be crazy !! i have a big problem in my ssis project .
    i have more than 40 package one parent and 39 childs 
    i use configuration method to get variable values from parent to child packages and it work well .. but when i copy my ssis project
    to the production server i face the following problem .. childs package didn't read the variables values from the parent package 
    i revised the configuration many times i can't find any missing !!
    any help please ...
    Thanks,

    Do not copy the packages from the projet to the production server, but you must generate the manifest installation including your configuration, and deploy them.
    Note : not forget to include the configuration.
    To learn more about how to install the package :
    Lesson 1:
    http://msdn.microsoft.com/en-us/library/ms365332(v=sql.105).aspx
    Lesson 2 :
    http://msdn.microsoft.com/en-us/library/ms365329(v=sql.105).aspx
    Lesson 3 :
    http://msdn.microsoft.com/en-us/library/ms365321(v=sql.105).aspx

  • Call Function from SSIS package

    Hello.  I have a field pulled from an Access database that I need to manipulate before processing.  I need to strip out illegal characters.  What's the best way to use a function to process the data in the table.  I was hoping to do this
    within the package if possible.  
    From within Access, a query passed values to a function defined within a code module and returned the new value.  The SSIS package needs to be able to do this as well.

    Hi,
    You can can remove the illegal characters in the package. On your Data Flow Task drag a script component in the script component input columns only check the field you will process.
    private string NewCharacter(string character)
    string replacestr= System.Text.RegularExpressions.Regex.Replace(character, "[^a-zA-Z0-9.]+", " ");
    return replacestr.Trim();
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    if(Row.FieldName_IsNull == false)
    {//Assign the new value
    Row.FieldName = NewCharacter(Row.FieldName);

  • SSIS Package, Use Datepicker control in VB script task

    Dim dateTimePicker1 As New DateTimePicker()
    Controls.AddRange(New Control() {dateTimePicker1})
    MessageBox.Show(dateTimePicker1.Value.ToString())
    dateTimePicker1.Value = DateTime.Now.AddDays(1)
    MessageBox.Show(dateTimePicker1.Value.ToString())
    In an SSIS package, 2008r2....   I trying to add a messagebox to a script task that contains a datepicker control. 
    I will pass the date to a variable then stored procedure.  The code above almost works except I get "Name Controls not declared".  I am using
    System.Windows.Forms. 
    Anyone know how to correct?
    Thanks,
    MIke

    Nope
    You cant have forms etc added to script task inside SSIS and make it interactive
    If your intention is to pass a value from outside you need to make use of configurations or use command line arguments with utilities like dtexec.
    http://www.bidn.com/blogs/DevinKnight/ssis/1655/passing-values-into-an-ssis-package-at-runtime-from-outside-the-package
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • How to parse crystal report query prompt value via url parameters

    HI All,
    I am creating a employee detail report in crystal report. The recordset is huge and i have created a parameter prompts in the query and thinking to parse the prompt value via url parameters. Not sure why i still could not make it works.
    for example, in my query panel i have created a prompt called "pwwid"
    and this is how i parse the prompt value via open document url.
    OpenDocument/opendoc/openDocument.jsp?sIDType=CUID&iDocID=AV8U6HlIq4FBl_MhcBwHqGI&lsSpwwid=12345678
    I read the sap documentation, this is how it parse the prompt value via url parameter. but it is not working for me and i am getting prompt to fill in the wwid whenever i run the reports.
    any idea why i am not getting this works?
    Regards,
    KeatAun

    Could you try:
    OpenDocument/opendoc/openDocument.jsp?sIDType=CUID&iDocID=AV8U6HlIq4FBl_MhcBwHqGI&lsSpspwwid=12345678
    -Abhilash

  • Error trying to run SSIS Package via SQL Server Agent: DTExec: Could not set \Package.Variables[User::VarObjectDataSet].Properties[Value] value to System.Object

    Situation:
    SSIS Package designed in SQL Server 2012 - SQL Server Data Tools
    Windows 7 - 64 bit.
    The package (32 bit) extracts data from a SQL Server db to an Excel Output file, via an OLE DB connection.
    It uses 3 package variables:
    *) SQLCommand (String) to specify the SQL Statement to be executed by the package
    Property path: \Package.Variables[User::ExcelOutputFile].Properties[Value]
    Value: f:\Output Data.xls
    *) EXCELOutputFIle (String) to specify path and filename of the Excel output file
    Property path: \Package.Variables[User::SQLCommand].Properties[Value]
    Value: select * from CartOrder
    *) VarObjectDataSet (Object) to hold the data returned by SQL Server)
    Property path: \Package.Variables[User::VarObjectDataSet].Properties[Value]
    Value: System.Object
    It consists out of 2 components:
    *) Execute SQL Task: executes the SQL Statement passed on via a package variable. The resultng rows are stored in the package variable VarObjectDataSet
    *) Script Task: creates the physical output file and iterates VarObjectDataSet to populate the Excel file.
    Outcome and issue:The package runs perfectly fine both in SQL Server Data Tools itself and in DTEXECUI.
    However, whenever I run it via SQL Server Agent (with 32 bit runtime option set), it returns the errror message below.
    This package contains 3 package variables but the error stating that a package variable can not be set, pops up for the VarObjectDataSet only.  This makes me wonder if it is uberhaupt possible to set the value of a package variable
    of type Object.
    Can anybody help me on this please ?
    Message
    Executed as user: NT Service\SQLSERVERAGENT. Microsoft (R) SQL Server Execute Package Utility  Version 11.0.2100.60 for 32-bit  Copyright (C) Microsoft Corporation. All rights reserved.    Started:  6:40:20 PM  DTExec: Could
    not set \Package.Variables[User::VarObjectDataSet].Properties[Value] value to System.Object.  Started:  6:40:20 PM  Finished: 6:40:21 PM  Elapsed:  0.281 seconds.  The package execution failed.  The step failed.
    Thank you very much in advance
    Jurgen

    Hi Visakh,
    thank you for your reply.
    So, judging by your reply, not all package variables used inside a package need to be set a value for when run in DTEXEC ?
    I already tried that but my package ended up in error (something to do with "... invocation ...." and that error is anything but clearly documented. Judging by the error message itself, it looks like it could be just about anything. that is why I asked my
    first question about the object type package variable.
    Now, I will remove it from the 'set values' list and try another go cracking the unclear error-message " ... invocation ...". Does an error message about " ... invocation ..." ring any bells, now that we are talking about it here ?
    Thx in advance
    Jurgen
    Yes exactly
    You need to set values only forthem which needs to be controlled from outside the package
    Any variable which gets its value through expression set inside package or through a query inside execute sql task/script task can be ignored from DTExec
    Ok I've seen the invocation error mostly inside script task. This may be because some error inside script written in script task. If it appeared after you removed the variable then it may because some reference of variable existing within script task.
    Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

  • Running SSIS Package from SQL Job - with Set Values, a string containing";"

    Can anybody help with this problem please?
    I’m using “set value” in SQL agent to pass in an email address to my SSIS package, if I put in a value of:
    [email protected]
    It works fine, but if I attempt to input multiple email addresses it fails
    [email protected];[email protected]
    It doesn’t appear to like the “;”.
    Any ideas anyone???
    BoroFC

    Hi Jamie, many thanks for the response.
    If I look at my command line table when Email = [email protected], it looks like:
    /SET "\package.Variables[EmailAddress].Value";"[email protected]" /REPORTING E
    If I change the value to “[email protected];[email protected]”, it looks like:
    /SET "\package.Variables[EmailAddress].Value";"\""" [email protected];"" [email protected] ""\"" /REPORTING E
    But the weird thing is, if I close the job properties window and re-open it again, I lose this line from my set values screen all together!!!
    Help ???
    BoroFC

  • SSIS Package developed in SQL Server 2008 R2 Using BIDS 2008 is not working in SQL Server 2012

    I am working in a Product based company.
    We have given a Pre-requisites to our customers like SQL Server version should be 2008 or later.
    I have developed and deployed my SSIS package in SQL Server 2008 R2 with BIDS 2008.
    I have used "Script Component" in my package.
    We will take only DTSX package file to the customer places and run using SQL Agent Job daily. 
    Who are all(Customer Environments) using SQL 2008 or R2 my package is running fine.
    Who are all(Customer Environments) using SQL 2012, I am facing the below issue 
    The component metadata for “Script Component, clsid {874F7595-FB5F-4OFF-9BAF-FBFF825OE3EF}” could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
    Still the same package is running fine for some of the customers having SQL 2012.
    I am getting the above error only for few customers.
    Can someone please guide me how to proceed on this. Is there any solution to go without upgrading the package to SQL 2012. Because we need to maintain only one package for all customers

    Hi Katherine,
    I am just using the script component only for generating Identity column purpose. I will get the maximum count from the table and using this count value as Input (JurisCount in my below code) for script component and increment the counter for new records.
    Finally I am mapping the Script component output value (JurisKey in my below code) to Key column of the table. That's all I am doing with Script Component.
    Here is my Simple .net code
    /* Microsoft SQL Server Integration Services Script Component
    *  Write scripts using Microsoft Visual C# 2008.
    *  ScriptMain is the entry point class of the script.*/
    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
    using Microsoft.SqlServer.Dts.Runtime.Wrapper;
    [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
    public class ScriptMain : UserComponent
        public override void PreExecute()
            base.PreExecute();
              Add your code here for preprocessing or remove if not needed
        public override void PostExecute()
            base.PostExecute();
              Add your code here for postprocessing or remove if not needed
              You can set read/write variables here, for example:
              Variables.MyIntVar = 100
        int counter = 0;
        public override void Input0_ProcessInputRow(Input0Buffer Row)
            if (counter == 0)
                counter = Convert.ToInt32(Row.JurisCount);
            Row.JurisKey = counter;
            counter = counter + 1;
    PAVAN MSBI

  • How to insert CLEAR SSIS Package in another SSIS Package

    Hi All,
    We have a requirement to run CLEAR SSIS Package in another SSIS Package whish extracts data from Oracle database system. Needed to run CLEAR SSIS Package if data exists in BPC application else run extract to Oracle system. Basically continue with SSIS Package and extract to Oracle system.
    In Outlooksoft 4.2 we didn't have any issue, since all evModify is going to be in DTS Package and we didn't have any issues in placing a condition connection to CLEAR Package. Basically we were running CLEAR Package inside anther DTS Package twice successfully.
    When we tried to create same functionality in SAP BPC 5.1, the code from EVMODIFY not doing correctly in executing the CLEAR inside another SSIS Package.
    If any body knows how to create  SSIS Package with a combination of 2 or more SSIS Packages by taking above scenario, please let me know.
    The basic requirement is, how do we include a SSIS Package in another SSIS Package and use EVMODIFY to run this meta package in SAP BPC 5.1 environment.
    Any information would be appreciated.
    Regards,
    Sreekanth.

    Yes, I have the job , and tried running this, Looks like it started executing.  Will update in next 5 mins if it runs fine.
    Looking at the command it generated, it looks like it is setting few more parameters like LoggingLevel, Synchronized , callerInfo etc.. I believe these should be optional right?
    Also now I have to set two Parameters out of 10 parameters the project takes, If I add additional /Par Project.starttime , will the package pick the values passed from this prompt?
    /ISSERVER "\"\SSISDB\FFF\PPP\Xpackage2.dtsx\"" /SERVER "\".\"" /Par "\"$ServerOption::LOGGING_LEVEL(Int16)\"";1 /Par "\"$ServerOption::SYNCHRONIZED(Boolean)\"";True /CALLERINFO SQLAGENT /REPORTING E

  • Auditing a Custom Import package

    We are having issues with BPC data auditing on a custom import package.  We are using the convert and dump load tasks just like in the standard import package.  The difference is that we have those surrounded by other SSIS tasks to perform other pieces of the import process.  The specific issue is that when we look at the detailed audit data, we are not seeing all of the records from our custom import.  We usually see that the import package took place, but we also usually see only a small fraction (if any) of the actual records that were posted (and which we can see in BPC) in the audit.
    My first question is, does the "Data Manager Import" option under the audit settings work with custom import packages using the convert and dump load tasks, or is it exclusive to the standard import package?  Secondly, if it does work with custom import packages, what might be some causes and/or resolutions to this issue?

    Just an update on the two issues that we are seeing...
    First, I have tested the standard Import and the Append into Fact Table packages, and both are audited correctly and completely.  This is important because our import package utilizes both of these styles of import.  One set of data is transactional, so it is loaded similar to the Append into Fact Table package.  The other set is a merge style import, so it is very similar to the standard import.  I see all records in the AuditDataFincance table in the Extra database for both of these packages.  This would lead me to believe that we are seeing some interference either between these two dump load tasks in the same package or some other SSIS package.  It is important to note that these dump load tasks are wrapped in foreach loops to pickup all files in a directory.  Looking at the package, my guess would be that either having two dump load tasks or having said tasks wrapped in a foreach loop would probably be the culprit.  My next step will be to strip the package down to test just these pieces.  Does anyone know if either of these would cause an issue with auditing?
    Secondly, I stated that we are not seeing the temp audit files in the STAGED or LOADED folders, which is making it more difficult to debug this issue.  After testing several packages, I noticed that the standard import and clear packages audit correctly, but they never create files in the STAGED or LOADED folders.  However, the Append into Fact Table package DOES create these files.  What I noticed in the AuditHdrFinance table in the Extra database was that the StagingCode values for these packages were different before they were loaded.  A value of 0 signifies that the package has already been loaded.  Append into Fact Table had a value of 1, which appears to signify that an audit file is waiting to be loaded in the STAGED directory.  The others had a value of 2, which I am guessing signifies that they are currently being staged in memory or some other location.  Does anyone have any information on what the different StagingCode values signify and why different packages have different codes?

  • SSIS Package execution error from powershell

    Hi I have written a powershell script to exec SSIS packages. I am passing variable values at runtime by reading it from a text file.
    But it throws an error.
    Here is my script.
    #Read Data from Parameter File
    $variable=Get-Content D:\Batch\Bin\ETLParameters.txt
    $variable.GetType()
     $dest_conn_str=$variable[1]
     $mail_cc=$variable[2]
     $mail_from=$variable[3]
     $mail_str=$variable[4]
     $mail_to=$variable[5]
     $pkg_path=$variable[6]
     $ps_exe_path=$variable[7]
     $ps_script_cmd=$variable[8]
     $ps_script_path=$variable[9]
     $src_conn_str=$variable[10]
     $src_sys_id=$variable[11]
    #Executing SSIS packages from command prompt
    $cmd = 'dtexec /f "' + $pkg_path + 'pkg_ntas_all.dtsx' + '"' + ' /SET \Package.Variables[User::src_conn_str].Properties[Value];"' + $src_conn_str + '"' + ' /SET \Package.Variables[User::dest_conn_str].Properties[Value];"'
    + $dest_conn_str + '"' + ' /SET \Package.Variables[User::src_sys_id].Properties[Value];"' + $src_sys_id + '"' + ' /SET \Package.Variables[User::pkg_file_path].Properties[Value];"' + $pkg_path + '"' +
    ' /SET \Package.Variables[User::mail_str].Properties[Value];"' + $mail_str + '"' + ' /SET \Package.Variables[User::mail_from].Properties[Value];"' + $mail_from + '"' + ' /SET \Package.Variables[User::mail_to].Properties[Value];"'
    + $mail_to + '"'+ ' /SET \Package.Variables[User::mail_cc].Properties[Value];"' + $mail_cc + '"' + ' /SET \Package.Variables[User::ps_exe_path].Properties[Value];"' + $ps_exe_path + '"' + ' /SET
    \Package.Variables[User::ps_script_path].Properties[Value];"' + $ps_script_path + '"' + ' /SET \Package.Variables[User::ps_script_cmd].Properties[Value];"' + $ps_script_cmd + '"'  
    $cmd
    DECLARE @returncode int
    EXEC @returncode = xp_cmdshell @cmd
    Types of error that I get -
    If I run the $cmd directly. Which is the concatenated string of dtexec command with parameters. It doesn't do anything, not even an error and the pwoershell execution just completes.
    If I use exec EXEC @returncode = xp_cmdshell @cmd    ----- Then it throws me the error that the exec is not recognized name of cmdlet.
    Please help as I a noob in any kind of scripting.

    I did but it just printed the entire string on the command line and assigned all the values.
    PS C:\Users\dev_setup> . D:\Batch\Bin\runETL.ps1
    IsPublic IsSerial Name                                     BaseType
    True     True     Object[]                                
    System.Array
    dtexec /f "D:\ETL\test\test\test_ETL\pkg_all.dtsx" /SET "\Package.
    Variables[User::src_conn_str].Properties[Value]";"Data Source=SOURCE1;Initial Catalog=tagtest;Provider=SAOLEDB.11;Integrated Security=SSPI;
    " /SET "\Package.Variables[User::dest_conn_str].Properties[Value]";"destinationDb;User ID=FP_DEV;Password=**********;Initial Catalog=xyz;Provider=SQLOLEDB.1;Persist Security Info=True;Auto Translate=False;Packet Size=10240;" /SET " \Package.Variables[User::src_sys_id].Properties[Value]
    ";"100200" /SET "\Package.Variables[User::pkg_file_path].Properties[Value]";"D:\ETL\test\test\test_ETL\" /SET "\Package.Variables[User::mail_str].Properties[Value]";"SmtpServer=mailhost.test.com;UseWindowsAuthentication=False;EnableSsl=False;" /SET "\Package.Variables[User::mail_from].Properties[Value]
    ";"[email protected]" /SET "\Package.Variables[User::mail_to].Properties[Value]";"[email protected]" /SET "\Package.Variables[User::mail_cc].Properties[Value]";"[email protected]" /SET "\Package.Variables[User::ps_exe_path].Properties[Value]";"C:\Windows\System32\WindowsPower
    Shell\v1.0\PowerShell.exe" /SET "\Package.Variables[User::ps_script_path].Properties[Value]";"D:\ETL\Powershell_Script\" /SET "\Package.Variables[User::ps_script_cmd].Properties[Value]";"create_historical_reject_file.ps1 -input_variable"
    PS C:\Users\svc_sp_fp_dev_setup>
    But when I just run it without passing the parameters it runs. What might be the issue.

  • SSIS package configurations

    Hi Experts,
    I am trying to configure my SSIS packages.
    I have a master package to call child packages for Dimernsions and Facts.
    Below are my two main approaches for configuration
    1. Configuring all packages to read a single XML file for database connections and File paths.
        In this case if someone want to change the location of config file then I need to change all my package.  Is there any other issue with this approach?
    2. Configuring via parent package variable. Here I will use the values from the parent package.
       But after I did this , if we open the package the connection managers will throw error when we migrate to other server as the hard coding of the connection manager not matches with the actual database.  If we run the master package it will
    run successfully as it get the correct value from the master package.
    Here Is there any way avoid this poping of error?
    Please suggest what would be the best way ?
    Regards
    Mukejee

    Hi Mukejee,
    The error occurs because the variables passed from the parent child cannot get their runtime value in the child package at design-time. To avoid this issue, we can set the For your first approach, to avoid updating the Package Configurations after moving
    to a new server, you can create environment variables on the each server, and export the relevant package configurations to the environment variables.
    In reality, we can set the DelayValidation property of the Connection Manager as well as the Data Flow Tasks in the source package so that they don’t prompt for the error messages after being opened in the BIDS/SSDT on a new server.
    Regards,
    Mike Yin
    TechNet Community Support

  • Unicode and non-unicode string data types Issue with 2008 SSIS Package

    Hi All,
    I am converting a 2005 SSIS Package to 2008. I have a task which has SQL Server as the source and Oracle as the destination. I copy the data from a SQL server view with a field nvarchar(10) to a field of a oracle table varchar(10). The package executes fine
    on my local when i use the data transformation task to convert to DT_STR. But when I deploy the dtsx file on the server and try to run from an SQL Job Agent it gives me the unicode and non-unicode string data types error for the field. I have checked the registry
    settings and its the same in my local and the server. Tried both the data conversion task and Derived Column task but with no luck. Pls suggest me what changes are required in my package to run it from the SQL Agent Job.
    Thanks.

    What is Unicode and non Unicode data formats
    Unicode : 
    A Unicode character takes more bytes to store the data in the database. As we all know, many global industries wants to increase their business worldwide and grow at the same time, they would want to widen their business by providing
    services to the customers worldwide by supporting different languages like Chinese, Japanese, Korean and Arabic. Many websites these days are supporting international languages to do their business and to attract more and more customers and that makes life
    easier for both the parties.
    To store the customer data into the database the database must support a mechanism to store the international characters, storing these characters is not easy, and many database vendors have to revised their strategies and come
    up with new mechanisms to support or to store these international characters in the database. Some of the big vendors like Oracle, Microsoft, IBM and other database vendors started providing the international character support so that the data can be stored
    and retrieved accordingly to avoid any hiccups while doing business with the international customers.
    The difference in storing character data between Unicode and non-Unicode depends on whether non-Unicode data is stored by using double-byte character sets. All non-East Asian languages and the Thai language store non-Unicode characters
    in single bytes. Therefore, storing these languages as Unicode uses two times the space that is used specifying a non-Unicode code page. On the other hand, the non-Unicode code pages of many other Asian languages specify character storage in double-byte character
    sets (DBCS). Therefore, for these languages, there is almost no difference in storage between non-Unicode and Unicode.
    Encoding Formats: 
    Some of the common encoding formats for Unicode are UCS-2, UTF-8, UTF-16, UTF-32 have been made available by database vendors to their customers. For SQL Server 7.0 and higher versions Microsoft uses the encoding format UCS-2 to store the UTF-8 data. Under
    this mechanism, all Unicode characters are stored by using 2 bytes.
    Unicode data can be encoded in many different ways. UCS-2 and UTF-8 are two common ways to store bit patterns that represent Unicode characters. Microsoft Windows NT, SQL Server, Java, COM, and the SQL Server ODBC driver and OLEDB
    provider all internally represent Unicode data as UCS-2.
    The options for using SQL Server 7.0 or SQL Server 2000 as a backend server for an application that sends and receives Unicode data that is encoded as UTF-8 include:
    For example, if your business is using a website supporting ASP pages, then this is what happens:
    If your application uses Active Server Pages (ASP) and you are using Internet Information Server (IIS) 5.0 and Microsoft Windows 2000, you can add "<% Session.Codepage=65001 %>" to your server-side ASP script.
    This instructs IIS to convert all dynamically generated strings (example: Response.Write) from UCS-2 to UTF-8 automatically before sending them to the client.
    If you do not want to enable sessions, you can alternatively use the server-side directive "<%@ CodePage=65001 %>".
    Any UTF-8 data sent from the client to the server via GET or POST is also converted to UCS-2 automatically. The Session.Codepage property is the recommended method to handle UTF-8 data within a web application. This Codepage
    setting is not available on IIS 4.0 and Windows NT 4.0.
    Sorting and other operations :
    The effect of Unicode data on performance is complicated by a variety of factors that include the following:
    1. The difference between Unicode sorting rules and non-Unicode sorting rules 
    2. The difference between sorting double-byte and single-byte characters 
    3. Code page conversion between client and server
    Performing operations like >, <, ORDER BY are resource intensive and will be difficult to get correct results if the codepage conversion between client and server is not available.
    Sorting lots of Unicode data can be slower than non-Unicode data, because the data is stored in double bytes. On the other hand, sorting Asian characters in Unicode is faster than sorting Asian DBCS data in a specific code page,
    because DBCS data is actually a mixture of single-byte and double-byte widths, while Unicode characters are fixed-width.
    Non-Unicode :
    Non Unicode is exactly opposite to Unicode. Using non Unicode it is easy to store languages like ‘English’ but not other Asian languages that need more bits to store correctly otherwise truncation will occur.
    Now, let’s see some of the advantages of not storing the data in Unicode format:
    1. It takes less space to store the data in the database hence we will save lot of hard disk space. 
    2. Moving of database files from one server to other takes less time. 
    3. Backup and restore of the database makes huge impact and it is good for DBA’s that it takes less time
    Non-Unicode vs. Unicode Data Types: Comparison Chart
    The primary difference between unicode and non-Unicode data types is the ability of Unicode to easily handle the storage of foreign language characters which also requires more storage space.
    Non-Unicode
    Unicode
    (char, varchar, text)
    (nchar, nvarchar, ntext)
    Stores data in fixed or variable length
    Same as non-Unicode
    char: data is padded with blanks to fill the field size. For example, if a char(10) field contains 5 characters the system will pad it with 5 blanks
    nchar: same as char
    varchar: stores actual value and does not pad with blanks
    nvarchar: same as varchar
    requires 1 byte of storage
    requires 2 bytes of storage
    char and varchar: can store up to 8000 characters
    nchar and nvarchar: can store up to 4000 characters
    Best suited for US English: "One problem with data types that use 1 byte to encode each character is that the data type can only represent 256 different characters. This forces multiple
    encoding specifications (or code pages) for different alphabets such as European alphabets, which are relatively small. It is also impossible to handle systems such as the Japanese Kanji or Korean Hangul alphabets that have thousands of characters."<sup>1</sup>
    Best suited for systems that need to support at least one foreign language: "The Unicode specification defines a single encoding scheme for most characters widely used in businesses around the world.
    All computers consistently translate the bit patterns in Unicode data into characters using the single Unicode specification. This ensures that the same bit pattern is always converted to the same character on all computers. Data can be freely transferred
    from one database or computer to another without concern that the receiving system will translate the bit patterns into characters incorrectly.
    https://irfansworld.wordpress.com/2011/01/25/what-is-unicode-and-non-unicode-data-formats/
    Thanks Shiven:) If Answer is Helpful, Please Vote

  • SSIS 2012 Scheduled SSIS package fail to start – Execution timed out

    Hi
    I am currently running into an issue that seem to have a live Connect issue raised for it. The following is the link to the connect issue https://connect.microsoft.com/SQLServer/feedback/details/783291/ssis-package-fails-to-start-application-lock-timeout-in-ssisdb-catalog-create-execution
    There is also a blog post that explains this issue in more detail, however the solution
    it provides does not work all the time and I am very reluctant to create custom stored procedures in the SSISDB. http://speaksql.wordpress.com/2013/06/27/ssis-2012-fail-to-start-execution-timed-out/
    Just to give some more background. When scheduling 10 or more, in my case I scheduled 14, SQL Server Agent Jobs all executing a package in the SSIS Catalog and all kicking off on exactly the same time about 10% to 20% of these jobs fail stating one of the
    following error messages:
    The operation failed because the execution timed out.  Source: .Net SqlClient Data Provider  Started:  12:20:01  Finished: 12:20:07  Elapsed:  5.797 seconds.  The package execution failed.  The step failed.
    Description: Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  Source: .Net SqlClient Data Provider  Started:  14:06:48  Finished: 14:06:53
     Elapsed:  4.719 seconds.  The package execution failed.  The step failed.
    This timeout takes place both when using TSQL to start the package in the Job or when the step in the Job is set to start the package as an Integration Services Package.
    Steps to recreate this:
    1. Create one SSIS package that simply executes for around a minute. For example you can use an Execute SQL Task and simply execute the following TSQL "WAITFOR DELAY '00:01:00'"
    2. Create 14 SQL Server Agent Jobs each executing the same package and schedule all of them to run at the same time.
    3. Watch the execution of the jobs take place and note the failures. (This does not always happen and you might get one run in which everything executes without a problem. The next run might have 6 or 7 of the jobs fail.)
    You can also create 14 different SSIS packages, one for each job, and the result is the same.
    I am running on SQL 2012 SP1 CU4. Windows Server 2012 R2 patched to the lastest patch level.
    This issue does not happen when the packages are deployed to the SSIS Package Store the way that it was done in SQL2008 and before. It only occurs when the SSIS packages are deployed to the SSIS Catalog.
    Any help or feedback on this would be greatly appreciated.
    Kind Regards,
    Christo Olivier

    Hi Tom
    Thank you for the fast response. I went and increased the value by 1024KB instead of the initial
    512KB. Unfortunately this seems to only alleviate the issue but does not stop it from occurring. It
    is happening much less often now but still occurring intermittently.
    Regards,
    Christo

Maybe you are looking for