Data Warehouse and ETL tools for data verification ?

Data Warehouse and ETL tools for data verification ?
How need to to data verification using ETL tool ? Also how to relate this thing to datawaehouse ?
Thanks in Advance

Hi Shyamal Kumar,
1) BW it self facilitates to do the ETL (Extraction Transformation Loading) steps:
     example:
                 Extraction - from SAP or other data bases
                 Transformation - using transfer rules, Updates rules
                 Loading - Loading into ODS, Cube, master data
2) Typically used ETL tools in the industry are:
     a)   datastage from Ascential (owned by IBM)
     b)   Informatica
     c)   Mercator
Regards, BB

Similar Messages

Low-Cost data migration and ETL tools

Hi, Im building a database for my company. We are a rather small size book company with a lot of references and still growing.
We have a Mysql database here and are trying to find some good tools to use it at its best. Basically we are just starting up the database after dealing with Excel: we had a size problemu2026 So im trying to find a program that will allow us to do two different things: the migration of our data from the old system to the new one and a specialized software to perform ETL (Extract, transform and load) on our database.
About the price of the tools, if we were one year ago, the accounting department would have been pretty relaxed about this. But today, we have some budget restrictions and therefore need a low cost tool. So could you give me some advice on a good data migration and etl tool for a low cost?
Thanks for your help.

Hi,
Some companies have budget problems and are having to reduce their cost of operation.
Have you ever heard of open source tools? They can do the job easily and less expensive than proprietary solutions that cost a lot of money.
You can have a look at a good ETL open source program called Talend Open Studio: it is user-friendly but also has advanced features intended for technical users (java debugger, code injectionu2026). It can perform data migration and ETL as you wrote in your first post.
The website is [here|http://www.talend.com/solutions-data-integration/data-migration.php] to download the open source program. They have a forum and documentation you can read. Tell us what you think about the software.
For an ETL benchmark: [ETL Benchmark|http://blogs.sun.com/aja/entry/talend_s_new_data_processing]

Monitoring and Alerting tools for Business Object Data Integrator

Hi,
We have Business Objects DI installed in AIX 5.3 environment. Please advise any type of monitoring and alerting tools for DI.
Monitor tools to monitor Job servers(like job service and web services running properly) and also to monitor long running extraction jobs and alerting the same. Thanks.
Thanks.
Raj.

Hi Thomas
Though both the tools objectives are for Data integration, the purpose for both tools are different.
SAP BODS/ BOBJ Data Integrator is an ETL tool which involves combining data residing in different sources and providing users with a unified view of the data
SAP PI is a Middleware/EAI tool which have a mechanism/technology that connects different parts of an application or a series of applications
SAP BOBJ Data Integrator
Bulk Data Integration
Extraction Transformation and Loading
Deals of huge volume of data
SAP PI
It is process based integration of application data
It is a middle ware which integrates various project applications into a common programming package that can be operate/access as a single application
SAP NetWeaver PI is SAP’s implementation of Service-oriented Architecture (SOA) middleware and facilitates the integration of business processes that span different departments, organizations, or companies.
SAP PI uses SAP NetWeaver Application Server and message servers where in fail over cases the messages can be resent again.
The below differences will gives you more clarity
Verdict is we can use both the tools for data transfer/migration and for choosing the tool depends upon the business requirement
Please reply back if you need more information.
Regards
Murthy

Diff b/w Data warehouse and Business Warehouse

Hi all,
what is the Diff b/w Data warehouse and Business Warehouse?

hi..
The diferrence between Datawarehousing and Business Warehouse are as follows.
DataWarehousing is the concept and BIW is a tool that uses this concept in Business applicaitons.
DataWarehousing allows you to analyze tons of data (millions and millions of records of data) in a convinent and optimum way, it is called BIW when applied to Business applications like analyzing the sales of a company.
Advantages- Consedering the volume of business data, BIW allows you to make decisions faster, I mean you can analyze data faster. Support for multiple languges easy to use and so on.
Refer this
Re: WHAT IS THE DIFFERENCE BETWEEN BIW & DATAWAREHOUSING
hope it helps...

Please help to get onhand stock report with last purchase and billed date warehouse and item wise

please help to get onhand stock report with last purchase and billed date warehouse and item wise

Hi Rajeesh Ambadi...
Try This
SELECT distinct T0.ITEMCODE , t1.ItemName, T0.ONHAND as 'Total Qty',
T1.LASTPURDAT ,t1.LastPurPrc
FROM OITW T0 INNER JOIN OITM T1 ON T0.ITEMCODE = T1.ITEMCODE
INNER JOIN OITB T2 ON T1.ITMSGRPCOD=T2.ITMSGRPCOD left join ibt1 t3 on t3.itemcode = t0.itemcode and t3.whscode = t0.whscode
WHERE
T0.ONHAND>0
AND T0.WhsCode ='[%0]'
Hope Helpful
Regards
Kennedy

Changing of data class and size category for keyfigures .

Hi,
I am not able to change the data class and size category for keyfigures .
can you please let me know how to enable the changing of data class and size category for keyfigure under Maintain DB storage parameters.

Ok then it sounds like your primary key of 8 fields and secondary index of 3 non-unique fields appear somewhat similar to the database and it wrongly uses the secondary index. Perhaps you can try to declare the table as a SORTED table with index fields as key. I doubt this will do any good, but you can try.
You can try to deactivate the sec. index if it is not being used.
Or by far the best but also debatable, try to pass hints to the SQL parser by %_hints statement. Please refer to SAP note 129385 for details on hints in general and note 772497 for various hints statements for ORACLE DB. This will surely make the DB interface use the primary index and the update would be faster. But with 1 million records, hopefully you are not looking at response time in micro seconds, are you?
rgds,
Prabhu

Backpac: A package state snapshot and restore tool for Arch Linux

backpac:
A package state snapshot and restore tool for Arch Linux with config file save/restore support.
https://aur.archlinux.org/packages.php?ID=52957
https://github.com/altercation/backpac (see readme on the github repository for more information)
Summary & Features
It's a common method of setting up a single system: take some notes about what packages you've installed, what files you've modified.
Backpac creates those notes for you and helps back up important configuration files. Specifically, backpac does the following:
maintains a list of installed groups (based on 80% of group packages being installed)
maintains a list of packages (including official and aur packages, listed separately)
maintains a list of files (manually created)
backs up key config files as detailed in the files list you create
The package, group and files lists along with the snapshot config files allows system state to be easily committed to version control such as git.
Backpac can also use these lists to install packages and files. Essentially, then, backpac takes a snapshot of your system and can recreate that state from the files and lists it archives.
Use Cases
Ongoing system state backup to github
Quick install of new system from existing backpac config
Conform current system to given state in backpac config
Backpac is a very, very lightweight way of saving and restoring system state.
It's not intended for rolling out and maintaining multiple similar systems, it's designed to assist individual users in the maintainance of their own Arch Linux box.
Status
Alpha, release for testing among those interested. Passing all tests right now but will continue to rework and refine. Bug reports needed.
Why?
There are a lot of 'big-iron' solutions to maintaining, backing up and restoring system state. Setting these up for a single system or a handful of personal systems has always seemed like overkill.
There are also some existing pacman list making utilities around, but most of them seem to list either all packages or don't separate the official and aur packages the way I wanted. Some detect group install state, some don't. I wanted all these features in backpac.
Finally, whatever tool I use, I'd like it to be simple (c.f. the Arch Way). Lists that are produced should be human readable, human maintainable and not different from what I'm using in non-automated form. Backpac fulfills these requirements.
Regarding files, I wanted to be able to backup arbitrary system files to a git repository. Tools like etckeeper are interesting but non /etc files in that case aren't backed up (without some link trickery) and there isn't any automatic integration with pacman, so there is no current advantage to using a tool like that. I also like making an explicit list of files to snapshot.
Sample Output
This is the command line report. Additionally, backpac saves this information to the backpac groups, packages and files lists and the files snapshot directory.
$ backpac -Qf
backpac
(-b) Backups ON; Files will be saved in place with backup suffix.
-f Force mode ON; No prompts presented (CAUTION).
(-F) Full Force mode OFF; Prompt displayed before script runs.
(-g) Suppress group check OFF; Groups will be checked for currency.
(-h) Display option and usage summary.
(-p) Default backpac: /home/es/.config/backpac/tau.
-Q Simple Query ON; Report shown; no changes made to system.
(-R) Auto-Remove OFF; Remove/Uninstall action default to NO.
(-S) System update OFF; No system files will be updated.
(-U) backpac config update OFF; backpac files will not be updated.
Sourcing from backpac config directory: /home/es/.config/backpac/tau
Initializing.................Done
GROUPS
============================================================================
/home/es/.config/backpac/tau/groups
GROUPS UP TO DATE: group listed in backpac and >80% local install:
base base-devel xfce4 xorg xorg-apps xorg-drivers xorg-fonts
GROUP PACKAGES; MISSING?: group member packages not installed:
(base: nano)
(xfce4: thunar xfdesktop)
PACKAGES
============================================================================
/home/es/.config/backpac/tau/packages
PACKAGES UP TO DATE: packages listed in backpac also installed on system:
acpi acpid acpitool aif alsa-utils augeas cowsay cpufrequtils curl dialog
firefox gamin git ifplugd iw mesa mesa-demos mutt netcfg openssh rfkill
rsync rxvt-unicode sudo terminus-font vim wpa_actiond wpa_supplicant_gui
xmobar xorg-server-utils xorg-twm xorg-utils xorg-xclock xorg-xinit xterm
yacpi yajl youtube-dl zsh
AUR UP TO DATE: aur packages listed in backpac also installed on system:
flashplugin-beta freetype2-git-infinality git-annex haskell-json
package-query-git packer wpa_auto xmonad-contrib-darcs xmonad-darcs
AUR NOT IN backpac: installed aur packages not listed in backpac config:
yaourt-git
FILES
============================================================================
/home/es/.config/backpac/tau/files
MATCHES ON SYSTEM/CONFIG:
/boot/grub/menu.lst
/etc/acpi/handler.sh
/etc/rc.conf
/etc/rc.local

firecat53 wrote:I think your plan for handling an AUR_HELPER is good. If AUR_HELPER is defined by the user, then either you might need a list of major AUR helpers and their command line switches so you can pick the correct switch for what needs to be done (most use some variation of -S for installing, but not all), or have the user define the correct switch(es) somehow for their chosen AUR helper.
That's a good idea. I'll add that to my AUR refactoring todo.
I also found directory tracking to be a weakness in other dotfile managers that I tried. I think you would definitely have to recursively list out the contents of a tracked directory and deal with each file individually. Wildcard support would be nice...I just haven't personally found a use case for it yet.
I've been thinking that I could just add the directory and scan through it for any non-default attribute files. If those are found then they get automatically added to the files list. That's pretty close to what etckeeper does.
Edit: I just compiled the dev version and removed my comments for already fixed things...sorry!
The master branch should have those fixes as well, but I didn't update the version number in the package build. I'll have to do that.
1. Still apparently didn't handle the escaped space for this item: (the file does exist on my system)
Ok, good to know. This wildcard directory business will require some new code and refactoring so I'll also rework my filenames handling.
2. Suggestion: you should make that awesome README into a man page!
I was working on one (the pkgbuild has a commented out line for the man page) but I had to leave it for later. Definitely want a man page. Once this stabilizes and I'm sure there aren't any big structural changes, I'll convert it to man format.
3. Suggestion: add the word 'dotfile' into your description somewhere on this page, the github page, and in the package description so people looking for dotfile managers will find it. You could also consider modularizing the script into a dotfile manager and the package manager, so people on other distros could take advantage of your dotfile management scheme.
I actually have a different script for dotfile management that doesn't touch packages, but there is definitely overlap with this one. That script isn't released yet, though, and if people find this useful for dotfile management that's great. I'll add that in.
4. Suggestion: since -Q is a read-only operation, why not just make it run with -f automatically to avoid the prompt?
Originally, running backpac without any command line options produced the Query output. I was concerned that since it is a utility that can potentially overwrite system files, it is important to give users a clear statement prior to execution about what will be done. Since the Query output is essentially the same as the Update and System reports in format and content, I wanted to be explicit about the Query being a passive no-change operation. The current command line options aren't set in stone though. If you feel strongly about it being different, let me know.
Long answer to a short question
5. Another suggestion: any thought to providing some sort of 'scrub' function to remove private information from the stored files if desired? This would be cool for publishing public dotfiles to github. Perhaps a credentials file (I did this with python for my own configs). Probably detecting email addresses and passwords without a scrub file would be rather difficult because dotfiles come in so many flavors.
Yes, absolutely. In fact, if you look at the lib/local file (pretty sure it's in both master and dev branches in this state) you'll see some references to a sanitize function. The idea there is that the user will list out bash associative arrays like this:
SANITIZE_WPA_=(
[FILE]='/etc/wpa_supplicant.conf'
[CMD]='sed s/expungepattern/sanitizedoutput/g'
Question: am I missing an obvious option to remove a file from the files.d directory if I delete it from the files list? Or do I have to delete it manually? It might be helpful to add a section to the README on how to update and delete dotfiles from being tracked, and also a more detailed description of what the -b option does (and what is actually created when it's not used).
You are only missing the function I didn't finish. There should be either dummy code or a TODO in the backpac main script referencing garbage collection, which isn't difficult but I just haven't finished it. The idea being another loop of "hey I found these old files in your files.d, mind if I delete them?" It's on my list and I'll try to get it in asap.
And finally, just out of curiosity, why did you choose to actually copy the files instead of symlink like so many other dotfile managers do?
git not following symlinks, hardlinks also out for permissions issues (git wouldn't be able to read the files, change them, etc.)
I definitely would prefer to not make an entire copy of the file, but I haven't come up with a better option. Shout with ideas, though. Also, if there is a way around the link issues I noted above, let me know. I don't see one but that doesn't mean it's not there.
edit: I think a Seattle area Arch meetup would be cool! Perhaps coffee someplace? Bellevue? U-district? Anyone else? BYOPOL (bring your own pimped out laptop)
A general meetup sounds good. I was also thinking it would be fun to do a mini archcon with some demos.

Date type-specific Customizing missing for date type ACTActPln

Hi all.
I'm using a Web Service obtained by copying the CRM 7.0 built-in WS APPTACTCRTRC.
I'm using it to create Appointments using an Adobe Interactive Form.
It goes everything fine (a Date is created in the system, with all the input data properly set) excepted the dates...
The WS uses two TIMESTAMPS, TIMESTAMP_FROM for StartDateTime and TIMESTAMP_TO for EndDateTime, whereas the GUI transaction to create Dates accepts dates and times in separated fields.
Going in Debug I can see that the execution ends with the following error, referred to the involved entity (CRM_APPOINTMENT):
Date type-specific Customizing missing for date type ACTActPln
CRM_APPOINTMENT uses date profile 0000000001 with three possible date types: ORDERACTUAL, ORDERPLANNED and ORDERPROPOSED and I cannot find anywhere date type ACTActPln (just to add it in customizing).
Is anyone able to enlight me?
Thank you in advance!
Alessandro
PS I've years of experience in Java programming but I'm quite new to the SAP world. Maybe the problem is not a problem at all, but it seems to me as such. Even if it has a trivial solution, please let me know. Thx!

Hi,
Well I was about to write that you should enter date in decimal format. but you already did that as mentioned in How to set a date in a Form where the bound date format is a decimal?
In response to following from above thread (copied for others to understand my response)
Hi all.
I'm successfully using the LiveCycle Designer and I've managed to remotely create Appointments using a CRM Web Service (a mere copy of APPTACTCRTRC).
The only problem I'm encountering is that I'm not able to set the dates/times (start and end of the appointment).
Dragging each of the two fields from the WSDL based Data Connection I obtain a Decimal Field limited to 15 digits.
How can I write a date and time in such a field?
I've tried with a retroverse date as YYYYMMDDHHmmss (for example, 20100301124500) but it does not work
(the appointment is indeed created, but with the start date and time fields filled with
the current date/time and the end date and time fields filled with the current date and time plus 10 minutes).
The retroverse date above is only 14 digit long, so, maybe, this is the problem.
But, given the field definition, I'm not able to fill a blank space between the date and the time.
I'm not able to put a dot either.
What should I do?
Any suggestions?
Thank you in advance.
Alessandro
15 digit in UTC time format here is fraction of second so you should enter 201003011245000 (add one more zero).
14th Oct 2010 16:00 should be written as 201010141600000, so just try it and and see if it works, I guess if you do not supply end time then by default it will be starttime + 10 min but this is just guess.
Regards,
Gourav

Can start date be set before valid for date for an equipment?

Can start date be set before valid for date for an equipment?

Hello,
Generally when a asset/equipment is purchased that is the valid from date and when commissioning done and it becomes operational then that date is start up date. So logically start up date should not before valid from date, it could be same.
Can you elaborate why you want start up date before valid from date ?
Thanks
Naveen

Native data warehouse products vs non-native data warehouse products

Hi Experts!
Can any one help me on this topic if you have any idea on native data warehouse products idea.
2)discuss the systems ability to interface with a non-native data-warehouse.
Discuss the architecture in both cases.
Describe or illustrate how data in the data warehouse can be utilized for reporting with the data in the ERP system.
Your help will be Appreciated.
Advance thanks,
vikram.c

Hi Experts!
Can any one help me on this topic if you have any idea on native data warehouse products idea.
2)discuss the systems ability to interface with a non-native data-warehouse.
Discuss the architecture in both cases.
Describe or illustrate how data in the data warehouse can be utilized for reporting with the data in the ERP system.
Your help will be Appreciated.
Advance thanks,
vikram.c

Collection : Popular DVD and Video Tools (For Mac)

Collection : Popular DVD and Video Tools (For Mac)
iPod softwares http://www.oursdownload.com/Mac-DVD-Video-iPod.html
iPhone softwares http://www.oursdownload.com/Mac-DVD-Video-iPhone.html
PSP softwares http://www.oursdownload.com/Mac-DVD-Video-PSP.html
Apple TV softwares http://www.oursdownload.com/Mac-DVD-Video-Apple-tv.html
DVD to Mobile(3GP) softwares http://www.oursdownload.com/Mac-DVD-3GP.html
DVD Video to MP4 softwares http://www.oursdownload.com/Mac-DVD-Video-MP4.html
DVD Converter http://www.oursdownload.com/Mac-DVD-Converter.html
Video Converter http://www.oursdownload.com/Mac-Video-Converter.html
DVD Copy softwares http://www.oursdownload.com/Mac-DVD-Copy.html
DVD ripper softwares http://www.oursdownload.com/Mac-DVD-Ripper.html
DVD to Blackberry http://www.oursdownload.com/Mac-DVD-Blackberry.html
DVD Creator http://www.oursdownload.com/Mac-DVD-Creator.html
DVD Studio Pack http://www.oursdownload.com/Mac-DVD-Studio.html

I addition to running less concurrent processes or installing more RAM as Kappy mentioned.
I also see that you have less than 10% of free space left on your HDD.
Disk Information: ?
ST3500418AS disk0 : (500.11 GB)
S.M.A.R.T. Status: Verified
EFI (disk0s1) <not mounted>: 209.7 MB
Macintosh HD (disk0s2) / [Startup]: 499.25 GB (40.6 GB free)
Recovery HD (disk0s3) <not mounted>: 650 MB
Which is fairly low and forcing it to work on the slowest part of the HDDs platter.

Mapping between oracle data types and ms sql server data types

hello
i need mapping between oracle data types and ms sql server data types
where can i find them ?

read this
http://download.oracle.com/docs/cd/E10405_01/doc/appdev.120/e10379/ss_oracle_compared.htm

OWB Paris, 10k$/CPU for Enterpr. ETL and 15k$/ CPU for Data Quality option

Hi,
I just heard that the licensing model changed completely with OWB 10g R2 (Paris).
As far as I get informed Warehouse Builder is now shipped as part of the database and for each Database (development, test, production...) you have to license the required option.
The 2 main options are the:
-Enterprise ETL Option which costs 10'000$ pro CPU
-Data Quality Option which costs 15'000$ pro CPU
This means that from now on you pay for OWB with the the Enterprise ETL and Data Quality Option as much as 50% of what you already pay for the EE DB license with the partitioning option.
So what's your opinion to this new licensing model?
Regards
Maurice

Hi Maurice. It seems like Oracle is trying to reduce number of licenses with the OLAP option, since it should be possible to have both OLTP and OLAP systems in the very same instance.
In the other hand, Oracle seems to want more money for "advanced ETL options", such as scheduling, metadata management, bla bla bla...
But those "advanced" functions seems to me as something every ETL tool should provide. For free...

Are there any tools for data encryption and decryption ?

Hi,
i am using oracle 9i R2, i want encrypt my data. Are there any tools available in market.
Please let me know the ways to do data encryption and decryption.
Thanks in advance
Prasuna.

970489 wrote:
using DBMS_OBFUSCATION_TOOLKIT.Encrypt /DESEncrypt we can't secure our password...So i am looking for an another alternative.As Blue Shadow said, what are you really trying to achieve?
Encrypting a password is itself not secure. Anything that can be encrypted can be decrypted. That is why Oracle itself DOES NOT encrypt passwords.
Surprised??
Here's what Oracle does with passwords, and what others should be doing if they have to store them.
When the password is created, the presented password - clear text - is concatenated with the username. The resulting character string is then passed through a one-way hashing function. It is that hashed value that is stored. Then when a user presents his credentials to log on to the system, the presented credentials are combined and hashed in the same manner as when the password was created, and the resulting hash value compared to the stored value.

RMAN crosscheck and expire guidelines for data warehouse environment

O/S: Windows Server 2008
DB: Oracle 11gR2
Are there any guidelines to how often one should do a RMAN crosscheck and set an expiration on archivelogs for data warehouse environments?
It would seem once a day would be enough for the crosscheck in a data warehouse environment that gets refreshed nightly. Expiration I would expect no less than 1 week.
Cheers!

I agree with damorgan
refer the below links for best practices.
http://www.oracle.com/technetwork/database/features/availability/311394-132335.pdf
https://blogs.oracle.com/datawarehousing/entry/data_warehouse_in_archivelog_m
Hope this helps,
Regards
http://www.oracleracepxert.com
Understand the Power of Oracle RMAN
http://www.oracleracexpert.com/2011/10/understand-power-of-oracle-rman.html
Duplicating RAC database using RMAN
http://www.oracleracexpert.com/2009/12/duplicate-rac-database-using-rman.html

Data Warehouse and ETL tools for data verification ?

Similar Messages

Maybe you are looking for