SQL DM - Data Transformation and Data Movement option ?

SQL DM - Data Transformation and Data Movement option ?
I am using SQL DM 3.0.0.665. I need your thoughts on following.
We find that Erwin introduced Data Transformation and Data Movement functionality to support ETL need. We were able to generate ETL spec using this feature.
Does SQL DM have any plan to introduce such features?
How do we use the current SQL DM to build ETL spec ?
Thanks in helping us out.

Hello,
I am currently experimenting with SQL Data Modeler to produce high level solution designs and ETL specifications.
Have not completed what I am doing but so far have come up with the following:
Current assumption I am working on:
All objects specified within the SQL Data Modeler will export to the Reporting Schema tables set up in an Oracle database. Once the data is within these tables then it will be a simple task to develop a SQL report to extract the data in any format required.
1) There is nothing in the physical (Relational) Model section that supports this
- though I have yet to fully use the Dimensional Modelling functionality which may have the mapping functionality required to specify an ETL
2) We need diagrams of the processes as well as the ETL mapping
- Process modelling is available in the Logical
- Reverse Engineer all Physical objects to become Logical object i.e. one Table to one Entity
- For each Entity set up an Information Structure
(Currently this can only be done in a convoluted method via creating a diagram, creating a Flow and editing the Flow then drilling down)
MESSAGE to SQL Data Modeler Support: Can things be set up so that Information Structures can be set up directly from the Browser, current method is a bit nonsensical
- You are now set up to use the Logical Process Modeling functionality to capture the ETL requirements
- Advise that you reference the training to understand what primitive, composite and transformation processes objects are
- Also, take the time to understand what an external agent object is
- Will assume you know what a Data Store is
Here is the standard I am heading towards that seems feasible, will need to run a proof of concept within the larger team to ensure it works though:
- A Logical is kept that is a one for one with the Physical
(The only reason for this is that there is no process modeling functionality for the Physical objects
MESSAGE to SQL Data Modeler Support: Can you duplicate the Process Modeling for the Logical to be available for the Physical objects too, would be a great help to specify ETL jobs.
- An External Agent is used to represent an external source e.g. Billing application
- A primitive process is used to represent the high Level design
- A composite process is used to specify processes which can be further broken down to ETL jobs
- A transformation process is used to represent an ETL job
Within a Transformation process you can specify the mapping from multiple sources to a target table
There are some negatives to this approach:
- You lose the physical schemas the tables are part of, though a naming convention will get round this
- You need to maintain a logical that is one for one with the physical, this is not a big overhead
However, as I have stated in my message to the SQL Data Modeler support team, would all be resolved if the Process Modeling functionality were also made available within the Physical objects environment.
Please note that we have not as yet adopted the above approach and are still assessing is SQL Data Modeler will meet this requirement to our satisfaction. The critical bit will be if the data exports to the Reporting Schema, if it does then we have plenty of SQL resource that can produce the reports required procided the data can be captured.
Hope that all helps.
Also, hope I have not missed the point of your email.
Kind regards,
Yusef

Similar Messages

SQL developer online data move

Hi,
I am using SQL developer 2.1 to migrate tables from Sybase 12 database to oracle 11g.
I have used online data move option for moving sybase data into oracle tables, but even after data move is completed not all rows have been moved from sybase tables to corresponding oracle tables. Some rows are missing but still there is no error messege being displyed, how to find out what's going wrong.
Is there any log file that is being generated in case of oracle sql developer?
Please help me with this.

Hi Mandark,
This is a weakness of SQL Developers online datamove.
It doesn't have logging and you cannot tweak the data move itself.
It very basic SELECT * FROM -> INSERT INTO ... under the covers.
If some rows are failing to be inserted,but some rows are being inserted, in the same table.
Then my guess is that the data in some rows doesn't "fit" into the target row/column.
And the datatype mapping used to convert the tables/columns was not good enough.
Ex: Sybase column of VARCHAR(10000) mapped to Oracle VARCHAR2(4000).
Some data which is greater than 4000 chars.
Which causes the insert to fail for those rows with >4000 chars. But ok for other rows.
The solution (in this example) is to map Sybase VARCHAR(100000) to Oracle CLOB.
You can specify the data type mapping in the Convert page of the migration wizard.
Another workaround is to use Copy To Oracle.
http://dermotoneill.blogspot.com/2010/11/copy-to-oracle.html
This is a very simple data move feature again. But it works slightly differently than Online Data Move.
It has no logging or tweaking either, but may do the job "online" for you.
You should temporarily turn off any check constraints, triggers, keys on each table you want to move data with using this method.
If all those fail, then the only sure fire way would be to use Offline Data Move.
Regards,
Dermot
SQL Developer Team.

Why cannot find data movement for clone database option on OEM 11gr2

Hi Expert,
I have Oracle 11gr2 11.2.3 normal OEM install to monitor our RAC database. When i go click data movement option, i can't find the clone database option. Any idea why ? Thanks

Hi;
Similar question mention in below links:
Clone Database: Select Destination in OEM 10g Database Control
Also see:
http://www.stanford.edu/dept/itss/docs/oracle/10g/em.101/b12016.pdf << part 6-6
Regard
Helios

Sql Developer Data Modeler 3.0 EA1: Custom transformation script

What is the object model used by:
Tools -> Design Rules -> Transformations
From looking at the examples (and guessing) I have managed to create ID attributes on all my entities, but how do I set its sequence number or set it as the primary key?
entities = model.getEntitySet().toArray();
for (var e = 0; e<entities.length; e++)
entity = entities[e];
att = entity.createAttribute();
att.setName('ID');
- Marc de Oliveira

content of XML files:
1) class described, parent class if there is one
2)<property .../>
describes property - set and get method, data type of property, default value
3) <collection ... />
describes collection of objects belonging to class - 5 collections are defined in Table.xml - columns, indexes, table check constraints, column groups and spatial definitions
- get method - to get collection
- create item method - method of described class used to create item for that collection; so for columns collection table.createColumn() will create column in table
- other methods - add, move, remove if are defined
4)other meta data
definitions in 1) .. 4) are used by Oracle SQL Developer Data Modeler, so it'll be wrong to modify those xml files
5)<roproperty ../>
Data Modeler doesn't parse/process these definitions, we decided to not filter them out however you should not rely that much on them
6)some junk left
For each object you can set your own properties and they will be persisted (also included in compare/merge in versioning) together with other information for object - look at "template table" example in transformation scripts. Here are methods you can use:
void setProperty(String key, String value);
     String getProperty(String key);
     boolean hasProperty(String key);
     boolean hasProperty(String key, String value);
     void removeProperty(String key);
     void clearProperties();
     Iterator getPropertyNames();
Philip

Sql Developer MSSql Migration, Online data move, image to long raw mapping

Hello,
my task is to migrate a MSSql Server Database to Oracle 10.2.0.4 with SQL Developer Version 2.1.1.64
and JTDS JDBC Driver oracle.sqldeveloper.thirdparty.drivers.sqlserver 11.1.1.58.17.
Capture, convert and online data move works except data mapping from MSSql image to Oracle long raw dataype.
I need to map MsSql image to oracle blob, to achieve online data move!
is Sql Developer online migration MsSql image to oracle long raw possible?
I know that long raw is outdated, but i'm dba and not responsible for the software part.
What options should i try? offline data move? can MS Bulk Copy Program export image data and SQL*Loader import into long raw?
Any experience or tipp?
Thanks
Michael

Hi user12132314 ,
I do not see any 'long raw' options on our current SQLServer 2005 data type mapping. (Raw is a separate 2000 byte data type).
The usual route for SQLDeveloper is to go to Blob.
Online - this is simple & automated.
Offline - this actually goes via hex output in bcp to sqlldr, read in as CLOB which is then encoded to blob. (This is automated)
(Online 'Copy to Oracle' option, right click on table, may be of use if advanced features not included in the select such as defaults are not required, it copies over a select * from table, including blobs)
To move you from blob to long raw after migration can be done (easily enough if Oracle database >=10g so the stream does not have to be in smaller 'chunks')
http://asktom.oracle.com/pls/asktom/f?p=100:11:3938270166267830::::P11_QUESTION_ID:702825000306
I can look into this if you want to go down this route.
Note however that long raw has drawbacks and blob is preferred, see
Oracle® Database SQL Language Reference
11g Release 2 (11.2)
Part Number E17118-04
Data Types
LONG Data Type
for example one restriction is:
A table can contain only one LONG column.
-Turloch
SQLDeveloper Team

Data Virtualization options with SQL Server technologies

Please let me know the available options for data virtualization using Microsoft BI Technologies for scenarios where high data volume and complex transformations are involved. I'm looking for a virtualization solution such that there is no persistent DWH
built, no ETL involved and the virtualized layer can be built directly on top of the source databases using which reports can be generated. The main objective is to let the business users generate their reports in real-time/near real-time.
Power Query/ROLAP SSAS/SSAS Tabular with DirectQuery serve as virtualization options but it doesn't appear to be an optimal solution for high data volume/complex transformations. Please share your thoughts if you have any other ideas.

MSBI Dev,
Not sure what you mean by data visualization. If you are talking about an OLAP layer that only contains metadata that sits directly on a OLTP system with large data volumes, I would suggest ROLAP with SSAS. Check out this paper for further details.
Analysis Services ROLAP for SQL Server Data Warehouses
Hope this helps.
Reeves
Denver, CO

Different LOV behavior between SQL query data model and data template

I have noticed different behavior when using parameters linked to list of values (LOV) of type menu with the multiple selection option enabled and a SQL query data model vs a data template. Here's the example because that first sentence was probably really confusing.
SQL Query:
select
plmc.MonthCode, plmc.ModalityDim, plmc.ModalityName,plmc.RegionDim
from
DataOut.dbo.PatientLabMonthlyCross plmc
where
plmc.MonthCode = 200202
and plmc.RegionDim = 1209
and 1 =
case
when coalesce(:modalityDim,null) is null
then 1
else
case
when plmc.ModalityDim in (:modalityDim)
then 1
else 0
end
end
Putting BI Publisher into debug mode, defining a data model of type SQL Query, defining a parameter called :modalityDim linked to a LOV that allows multiple selections, and selecting a couple of values from the LOV the output of the prepared statement is:
[081607_122647956][][STATEMENT] Sql Query : select
plmc.MonthCode,
plmc.ModalityDim,
plmc.ModalityName,
plmc.RegionDim
from
DataOut.dbo.PatientLabMonthlyCross plmc
where
plmc.MonthCode = 200202
and plmc.RegionDim = 1209
and 1 =
case
when coalesce(?,?,null) is null
then 1
else
case
when plmc.ModalityDim in (?,?)
then 1
else 0
end
end
[081607_122647956][][STATEMENT] 1:6
[081607_122647956][][STATEMENT] 2:7
[081607_122647956][][STATEMENT] 3:6
[081607_122647956][][STATEMENT] 4:7
[081607_122654713][][EVENT] Data Generation Completed...
[081607_122654713][][EVENT] Total Data Generation Time 7.0 seconds
Note how the bind variable :modalityDim was changed into two parameters in the prepared statement.
When I use this same SQL Query in a data template the output is:
[081607_012113018][][STATEMENT] Sql Query : select
plmc.MonthCode,
plmc.ModalityDim,
plmc.ModalityName,
plmc.RegionDim
from
DataOut.dbo.PatientLabMonthlyCross plmc
where
plmc.MonthCode = 200202
and plmc.RegionDim = 1209
and 1 =
case
when coalesce(?,null) is null
then 1
else
case
when plmc.ModalityDim in (?)
then 1
else 0
end
end
[081607_012113018][][STATEMENT] 1:'6','7'
[081607_012113018][][STATEMENT] 2:'6','7'
[081607_012113574][][EXCEPTION] java.sql.SQLException: Syntax error converting the nvarchar value ''6','7'' to a column of data type int.
Note the exception because it is trying to convert the multiple parameter values.
Am I doing something completely wrong here? I really need to use a data template because I will need to link a couple of queries together from different database vendors.
-mark

This is for 10.1.3.4 - because in 11g every SQL query is automatially part of a data model.
In 10g SQL query is for simple unrelated SQL queries.
If you need to use advance features such as:
a) multiple SQL queries that are joined in master-detail relation ships
b) before/after report triggers
Then you will need to use the data template, which is an XML description
of the queries, links, and PL/SQL calls.
hope that helps,
Klaus

SQL LOADER , EXTERNAL TABLE and ODBS DATA SOURCE

hello
Can any body help loading data from dbase file .dbt to an oracle 10g table.
I tried last day with SQL LOADER, EXTERNAL table , and ODBC data source.
Why all of these utilities still failing to solve my problem ?
Is there an efficient way to reach this goal ?
Thanks in advance

export the dbase data file to text file,
then you have choice of using either sql loader or external table option to use.
regards

Data Mover login error (SQL 2008R2 as backend DB)

Hi,
This question was asked in this thread -> Continued Discussion - Steps to Upgrade PT8.49 to 8.50 Manually
The difference between problem in that thread and mine is that I have WIN08R2 SQL08R2 Peopletools9.52 and Peoplesoft HRMS 9.0 installed. I was on the steps of creating the databases manually with the sql scripts and that was also preformed.
I also have different ConnectID: people and AccessID: peoplesa with respective passwords. I try login in with the AccessID and I get the below error.
File: SQL Access ManagerSQL error. Stmt #: 2 Error Position: 0 Return: 8077 - Invalid SNAC client version (minimum is 10.00.1763) (SQLSTATE PS077)
Any suggestions?
-Vishal

Hi there,
I upgraded the SNAC client to 1033\x64\sqlncli.msi from the Microsoft and still I get the same Login error in the DataMover.
I checked the registry location --> HKEY_Local_Machine\Software\Microsoft\Microsoft SQL Server Native Client 10.0\Current Version = 10.51.2500.0
The Data Mover login error says that it requires minimum is 10.00.1763 so i guess the one I have should work fine?
Any suggestions?
-Vishal

Use of Trace key in transformer and parser of ODI Data Quality

Hi All,
Can someone explain to me the significance of Trace key in the transformer and parser processes (modules) in the ODI data profiling and quality part.
I have read all the documentation present of the ODI dataquality getting started and the user guide. I was not able to find any answer.
It would be great anyone can answer this question.
Thanks in advance,
Chapanna

Hi Julien,
The Trace key option is present in the "transformer" and "transformer address reconstructor" steps when executing an ODQ Name and Address Quality project.
The purpose of this trace key is to help developer to easily debug and identify records that have incorrectly transformed. A primary key/surrogate key is usually chosen as the trace key to facilitate the debugging process by identifying the exact record.
Thanks,
Chapanna

My imported movie shows up in the event library under 2006 date, month of march, the movie name. I want the movie to show up in the event library under the date I choose and the movie name. How do I do this?

My imported movie shows up in the event library under 2006 date, month of march, the movie name. I want the movie to show up in the event library under the date I choose and the movie name. How do I do this?

rogerhermans wrote:
I did change the name of the event. I cannot change the date of the event in the Modify command. The date and month was selected by the system as I did not name my event to start with. I now want to change the date (from 2006 to 1987) , delete the month and show the event name. I want the event library to post all my events by year (specified by me) and name of event.
you can change the date, so that FCPX categorizes it properly. First delete the clip(s) from FCPX, but not from your computer entirely, make sure you have the originals somewhere.
Open a terminal window and use the following command:
touch -t 198711280800.00 <path/filename of clip>
Where the format is: (use leading zeros if necessary for date & time)
first four digits: year
next four: month and day
next four: time of day (in 24 hour format)
next 2: seconds as a decimal (this is optional, if you leave it off you will get 00 seconds)
The above example will change the file date to Nov. 8, 1987 08:00:00 AM
This will change the created, modified and last access dates of the clip/file.
Then reimport the clip(s) and FCPX should categorize it properly.

How do I "Init without data transfer' using 7.0 transformations and DTP's?

I have a data-mart situation where I have a standard DSO object that sends deltas up to a standard cube. This data flow is created with 7.0 transformation and a delta type DTP and has been running fine.
I now have a new single activated request loaded to the DSO object but I DO NOT want this request to delta update to the cube. In 3.x, I used to go into the init infopackage, manually delete the initialization pointer, and run a 'Initialize Without Data Transfer' to make this work. Deltas could then continue as before.
I do not see a way to do this using 7.0 transformations and DTP's. I want to reset the init from DSO to cube so this request does not get updated. Does anyone have step by step ideas on how this can be accomplished?
Thanks in advance for any help on this issue.

Hi,
delete the data in the cube by using oprtion selective deletion based on request number.
(if delta request is deleted from cube, with next load last delta records will also come)
in this scenario there is no need to delete initiaizations option.
in BI 7.0, initialization load is not there.
see below documentation on DTP in SAP help.
"On the Extraction tab page, specify the parameters:
a. Choose Extraction Mode.
You can choose Delta or Full mode.
In contrast to a delta transfer with an InfoPackage, an explicit initialization of the delta process is not necessary for the delta transfer with a DTP. When the data transfer process is executed in delta mode for the first time, all existing requests are retrieved from the source and the delta status is initialized. "
Link: [http://help.sap.com/saphelp_nw70/helpdata/EN/42/f98e07cc483255e10000000a1553f7/frameset.htm]
Regards
Daya Sagar

Put some data form the databse of SQL Server into SAP and auto general PR?

Dear all,
Can you tell me how I can put some data from the databse of SQL Server into SAP and auto general PR?
Best regards,
Merryzhang
Edited by: merry zhang on Feb 4, 2009 9:10 AM

Hi Zhang,
If i have understood your question properly, I guess you wanted to create Purchase Requistion automatically right?
Then you have lots of option simpilest one would be create a couple of Z table with all the fields that you require for header and Item.
create a Program and use BAPI function module to create the PR by reading the values from those Z tables. after creation delete the contents from the table. Schedule this program in Background periodically like everyday or weekly once however you would like it to be. So once the table has contents in it and when the program runs it will create the PR's (Purchase Requisition).
You can also use a workflow with the Z table. Once the Z table is filled you can trigger a workflow based on the Table change. I'm not sure like how to do it but I nkow that this process can also be used.
Hope this might have thrown some light upon.
Thanks,
Prashanth

ImplementationOpt Design[Opt 31-67] Problem for AXI bus between data mover ip and axi interconnect

I'm using vivado 2014.4 and win7 64bit for my zynq design. Previously, the design is good. I made some revisions. Then I came across the problem.
If the synthesis strategies option is flow_runtime_optimatized(Vivadio synthesis 2014), everything works
If the synthesis strategies option is default options (the synthesis settings are shown in the picture), Synthesis is still fine, but the implementation failed. Some of the errror msg is shown below.
The error message shows there are unconnected pins on axi_interconnect_1. The connections between AXI master from my own IP(it is the axi port from a DataMover IP) and axi_interconnected is shown in the attached picture.
The synthesis schematic is also checked. The connections of the axi_interconnect have some pins unconnected as shown in the picture (interconnect_schematic_synth.PNG). The connections of my IP are good, but it misses some pins (like _arready, _rvalid, _ruser, _bid...). The master AXI port in my port is from the data mover IP. By default, the data mover IP doesn't have these missed pins. The AXI port on my IP declares these missed pins, but actually they are not connected to any inside my IP.
Btw, previously, the project works well in my design. But now it doesn't.
I also check the connections of the axi_interconnect when synthesis strategies option is flow_runtime_optimatized, the schematic show all pins are connected.
Please help. Thx.
Sam
ImplementationOpt Design[Opt 31-67] Problem: A LUT1 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/size_mask_q[3]_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/access_is_incr_q_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I1, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/access_is_incr_q_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/command_ongoing_i_2__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/size_mask_q[1]_i_1__0.
[Opt 31-67] Problem:

Hi Muzaffer
THx for your reply.
I tried implementation with opt_design option -directive Explore, it didn't work.
Aslo I disabled the opt_design and enabled phys_opt_design, it still has the same error in the implementation.
I would try to delete the pins related _AR_(like _arready, etc. ) and _R_ (like _rdata, _rid, etc.) pins in the AXI4 port in my IP. The data mover ip doesn't contains thes pins. I want to see whether it works this way.
Hopefully, the new vivado version will help.

Iphone 5 - "use cellular data for" option only shows Facetime and Passbook

I have an iphone 5 and I am trying to manage my cellular data usage,
When I go into Settings - Cellular and I look at the "use cellular data for" option, there are only 2 apps on the list Facetime and Passbook.
I have seen other posts online that show a long list of apps that you can toggle Cellular data on and off.
How can I see a longer list of apps.

In Settings/General/Restrictions if you tap on Cellular Data Use there should be 2 options: Allow Changes or Don't Allow Changes. If you don't see these options is that where you see "Cellular data is turned off for 'Setting'?
Do you have a corporate MS Exchange email account on your phone? If so your administrator may have added an override on this setting.

SQL DM - Data Transformation and Data Movement option ?

Similar Messages

Maybe you are looking for