ETL Tools Training

Data Warehousing and Cleansing Concepts

What is data warehousing, understanding the extract, transform and load processes, what is data aggregation, data scrubbing and data cleansing and the importance of Informatica PowerCenter ETL

Informatica Installation and Configuration

Configuring the Informatica tool and how to install the Informatica operational administration activities and integration services

Hands-on Exercise: Step-by-step process for the installation of Informatica PowerCenter

Working with Active and Passive Transformatio

Understanding the difference between active and passive transformations and the highlights of each transformation

Working with Expression Transformation

Learning about expression transformation and connected passive transformation to calculate value on a single row

Hands-on Exercise: Calculate value on a single row using connected passive transformation

Working with Sorter, Sequence Generator and Filter Transformation

Different types of transformations like sorter, sequence generator and filter, the characteristics of each and where they are used

Hands-on Exercise: Transform data using the filter technique, use a sequence generator and use a sorter

Working with Joiner Transformation

Working with joiner transformation to bring data from heterogeneous data sources

Hands-on Exercise: Use joiner transformation to bring data from heterogeneous data sources

Working with Ranking and Union Transformation

Understanding the ranking and union transformation, the characteristics and deployment

Hands-on Exercise: Perform ranking and union transformation

Syntax for Rank and Dense Rank

Learn the rank and dense rank functions and the syntax for them

Hands-on Exercise: Perform rank and dense rank functions

Router Transformation

Understanding how router transformation works and its key features

Hands-on Exercise: Perform router transformation

Source Qualifier Transformation and Mappings

Lookup transformation overview and different types of lookup transformations: connected, unconnected, dynamic and static

Hands-on Exercise: Perform lookup transformations: connected, unconnected, dynamic and static

Slowly Changing Dimension in Informatica

What is SCD, processing in xml, learn how to handle a flat file, list and define various transformations, implement ‘for loop’ in PowerCenter, the concepts of pushdown optimization and partitioning, what is constraint-based loading and what is incremental aggregation

Hands-on Exercise: Load data from a flat file, implement ‘for loop’ in PowerCenter, use pushdown optimization and partitioning, do constraint-based data loading and use incremental aggregation technique to aggregate data

Mapplet and Loading to Multiple Designer

Different types of designers: Mapplet and Worklet, target load plan, loading to multiple targets and linking property

Hands-on Exercise: Create a mapplet and a worklet, plan a target load and load multiple targets

Performance Tuning in Informatica

Objectives of performance tuning, defining performance tuning and learning the sequence for tuning

Hands-on Exercise: Do performance tuning by following different techniques

Repository Manager

Managing repository, Repository Manager: the client tool, functionalities of previous versions and important tasks in Repository Manager

Hands-on Exercise: Manage tasks in Repository Manager

Best Practices in Informatica

Understanding and adopting best practices for managing repository

Workflow Informatica

Common tasks in workflow manager, creating dependencies and the scope of workflow monitor

Hands-on Exercise: Create workflow with dependencies of nodes

Parameters and Variables

Define the variable and parameter in Informatica, parameter files and their scope, the parameter of mapping, worklet and session parameters, workflow and service variables and basic development errors

Hands-on Exercise: Define variables and parameters in functions, use the parameter of mapping, use worklet and session parameters and use workflow and service variables

Error Handling and Recovery in Informatica

Session and workflow log, using debuggers, error-handling framework in Informatica and failover and high availability in Informatica

Hands-on Exercise: Debug development errors, read workflow logs and use the error-handling framework

High Availability and Failover in Informatica

Configurations and mechanisms in recovery and checking health of PowerCenter environment

Hands-on Exercise: Configure recovery options and check health of PowerCenter environment

Working with Different Utilities in Informatica

Using commands: infacmd, pmrep and infasetup and processing of a flat file

Hands-on Exercise: Use commands: infacmd, pmrep and infasetup

Flat File Processing (Advanced Transformations)

Fixed length and delimited, expression transformations: sequence numbers and dynamic targeting using transaction control

Hands-on Exercise: Perform expression transformations: sequence numbers and dynamic targeting using transaction control

Dynamic Targeting

Dynamic target with the use of transaction control and indirect loading

Hands-on Exercise: Use of transaction control with dynamic target and indirect loading

Working with Java Transformations

Importance of Java transformations to extend PowerCenter capabilities, transforming data and active and passive mode

Hands-on Exercise: Use Java transformations to extend PowerCenter capabilities

Unconnected Stored Procedure Usage

Understanding the unconnected stored procedure in Informatica and different scenarios of unconnected stored procedure usage

Hands-on Exercise: Use the unconnected stored procedure in Informatica in different scenarios

Advanced Concepts in SCD

Using SQL transformation (active and passive)

Hands-on Exercise: Use SQL transformation (active and passive)

Incremental Data Loading and Aggregation

Understanding incremental loading and aggregation and comparison between them

Hands-on Exercise: Do incremental loading and aggregation

Constraint-based Loading

Working with database constraints using PowerCenter and understanding constraint-based loading and target load order

Hands-on Exercise: Perform constraint-based loading in a given order

XML Transformation and Active Lookup

Various types of XML transformation in Informatica and configuring a lookup as active

Hands-on Exercise: Perform XML transformation and configure a lookup as active

Profiling in PowerCenter

Understanding what data profiling in Informatica is, its significance in validating content and ensuring quality and structure of data as per business requirements

Hands-on Exercise: Create data profiling in Informatica and validate the content

Workflow Creation and Deletion

Understanding workflow as a group of instructions/commands for integration services and learning how to create and delete workflow in Informatica

Hands-on Exercise: Create and delete workflow in Informatica

Database Connection

Understanding the database connection, creating a new database connection in Informatica and understanding various steps involved

Hands-on Exercise: Create a new database connection in Informatica

Relational Database Tables

Working with relational database tables in Informatica, mapping for loading data from flat files to relational database files

Hands-on Exercise: Create mapping for loading data from flat files to relational database files

LinkedIn Connection

Understanding how to deploy PowerCenter for seamless LinkedIn connectivity with Informatica PowerCenter

Hands-on Exercise: Deploy PowerCenter for seamless LinkedIn connectivity with Informatica PowerCenter

Connection with Sources

Connecting Informatica PowerCenter with various data sources like social media channels such as Facebook, Twitter, etc.

Hands-on Exercise: Connect Informatica PowerCenter with various data sources like social media channels such as Facebook, Twitter, etc.

Pushdown Optimization and Partitioning

Pushdown optimization for load-balancing on the server for better performance and various types of partitioning for optimizing performance

Hands-on Exercise: Optimize using pushdown technique for load-balancing on the server for better performance and create various types of partitioning for optimizing performance

Cache Management

Understanding session cache, the importance of cache creation, implementing session cache and calculating cache requirement

Hands-on Exercise: Implement cache creation and work with session cache

What projects I will be working on this Informatica training?

Project 1: Admin Console

Problem Statement:It includes following actions:

• Creation of users
• Building roles
• Forming groups
• Collaboration of users, roles and groups
• Lock handling
• Creating sessions, workflow and worklets

Project 2: Deploying Informatica ETL for Business Intelligence

Industry:General

Problem Statement: Disparate data needs to be converted into insights using Informatica

Topics: In this Informatica project, you have access to all environments like dev, QA, UAT and production. You will first configure all the repositories in various environments. You will receive the requirement from client through source to target mapping sheet. You will extract data from various source systems and fetch it into staging. From staging, it will go to the operational data store; from there, the data will go to the enterprise data warehouse, and from there it will be directly deployed for generating reports and deriving business insights.

Highlights:

• Access data from multiple sources
• Manage current and historic data with SCD
• Import source and target tables

Project 3: Deploying the ETL Transactions on Healthcare Data

Industry:Healthcare

Problem Statement: How to systematically load data within a hospital scenario so that it is easily available

Topics: In this Clinical Research Data Warehouse (CRDW) Informatica project, you will be working on various types of data coming from diverse sources. The warehouse contains remitted claims that are both approved or disapproved for end-user reporting. You will create CRDW load schedules that are on daily, weekly and monthly bases.

Highlights:

• Extracting data from multiple sources
• Cleansing data and putting in right format
• Loading the data into the CRDW

Case Study 1

Project:Banking Products Augmentation

Industry: Banking

Problem Statement: How to improve the profits of a bank by customizing the products and adding new products based on customer needs

Topics:In this Informatica case study, you will construct a multidimensional model for the bank. You will create a set of diagrams depicting the star-join schemas needed to streamline the products as per customer requirements. You will implement slowly changing dimensions, understand the customer–account relationships and create diagram for the description of the hierarchies. You will also recommend heterogeneous products for the customers of the bank.

Highlights:

• Deploy a star-join schema
• Create demographic mini-dimensions
• Informatica aggregator transformations

Case Study 2

Project:Employee Data Integration

Industry:General

Problem Statement:How to load a table with employee data using Informatica

Topics:In this Informatica case study, you will create a design for a common framework that can be used for loading and updating the employee ID and other details lookup for multiple shared tables. Your design will address the regular loading of shared tables. You will also keep a track of when the regular load runs, when the lookup requests run, prioritization of requests if needed and so on.

Highlights:

• Creating multiple shared tables
• Plug-and-play capability of the framework
• Code and framework reusability

What is BI?

Introduction to Business Intelligence, understanding the concept of Data Modeling, Data Cleaning, learning about Data Analysis, Data Representation, Data Transformation.

ETL Overview

Introduction to ETL, the various steps involved Extract, Transform, Load, using a user’s email ID to read a flat file, extracting the User ID from email ID, loading the data into a database table.

Working with Connection Managers

Introduction to Connection Managers – logical representation of a connection, the various types of Connection Managers – Flat file, database, understanding how to load faster with OLE DB, comparing the performance of OLE DB and ADO.net, learning about Bulk Insert, working with Excel Connection Managers and identifying the problems.

Data Transformations

Learning what is Data Transformation, converting data from one format to another, understanding the concepts of Character Map, Data Column and Copy Column Transformation, import and export column transformation, script and OLEDB Command Transformation, understanding row sampling, aggregate and sort transformation, percentage and row sampling.

Advance Data Transformation

Understanding Pivot and UnPivot Transformation, understanding Audit and Row Count Transformation, working with Split and Join Transformation, studying Lookup and Cache Transformation, Integrating with Azure Analysis Services, elastic nature of MSBI to integrate with the Azure cloud service, scale out deployment option for MSBI, working with cloud-borne data sources and query analysis. Scaling out the SSIS package, deploying for tighter windows, working with larger amount of data sources, SQL Server vNext for enhancing SQL Server features, more choice of development languages and data types both on-premise and in the cloud.

Slowly Changing Dimensions

Understanding data that slowly changes over time, learning the process of how new data is written over old data, best practices.Detail explanation of three types of SCDs –Type1, Type2 and Type3, and their differences.

Overview of Fuzzy Look-up Transformation and Lookup and Term Extraction

Understanding how Fuzzy Lookup Transformation varies from Lookup Transformation, the concept of Fuzzy matching,

Concepts of Logging & Configuration

Learning about error rows configuration, package logging, defining package configuration, understanding constraints and event handlers.

What projects I will be working on this SSIS training?

Project 1:  SSIS

Problem Statement: Create a data flow task to extract data from the XLS format and store it into the SQL database, store the subcategory and category-wise sales in a table of the database. Once you get the output, split the dataset into two other tables. Table 1 should contain three columns (Sales < 100,000), Category, and Subcategory. Table 2 should contain (Sales > 100,000), Subcategory, and Category columns. Also, the Sales column should be sorted in both tables. Divide the whole dataset into a ratio of 70:30 percent and store the results in two different tables in the database

Topics: Data Flow, ODBC Set up and Connection Manager, Flat File Connection, Transformation, Import Export Transformation, Split and Join Transformation, Merge and Union All Transformation

Highlights:

• Creating a Data Flow Task
• ODBC Set up and Connection Manager
• Transformations

Case Study: SSIS

Problem Statement: Create the connection of OLDB & load the data in SQL Server from excel; Create transformation where you have to split the people’s age group; How to create constants and events in package; Create a project level and package parameter at the package level; How to extract the data in an Incremental Order;

Topics: Data Flow, ODBC Set up and Connection Manager, Transformation, Split & Join Transformation, Term Extraction and Lookup

Highlights:

• ODBC Set up and Connection Manager
• Transformations

Data Modeling Concepts

Introduction to OBIEE,installation of OBIEE,What are data models and why you need them? The scope, reach and benefits of data modeling,data warehousing,sample OBIEE Report, the business requirement intrinsic in data modeling, various case studies, the data modeling implications and the impact of data modeling on business intelligence.

Business Intelligence Concepts

Introduction to Business Intelligence, the architecture of data flow,OBIEE architecture, stack description of BI technology,BI Server,BI Scheduler,displaying report with data,need for reporting in business, distinction between OLTP and OLAP, the BI platform in BI technology stack, the product and dimension hierarchy, multidimensional and relational analytical processing, types of Reports, multidimensional modelling.

Understanding OLAP

Online Analytical Processing,the OBIEE admin tools, RPD, the important concepts & terminology, significance of OLAP in business intelligence life cycle, understanding various data schemas like star,designing with Star Schema,creation of physical layer & simple RPD, enterprise information model, snow flake and constellation, aggregate and calculated measures.

Oracle Business Intelligence Suite

Introduction to Oracle Business Intelligence Enterprise Edition, overview of the OBIEE product, the Architecture of OBIEE, key features and components,creating a simple report, business model, hierarchy, presentation and mapping.

Oracle BI Repository

Understanding what is Oracle Business Intelligence Repository, installation of OBIEE on Windows system, directory structure installation, services,, analytics and interactive reporting, dashboard creation, multiple report creation, formula editing, column properties, altering

BI Repository Business Model

Understanding how to build a Business Model and Mapping Layer in BI Repository, creating the Presentation Layer,formatting of data, conditional formatting, saving the report, creating and sharing folder. Topics – Data format, Conditional format, Removing filters,Like,Advanced ,Save the report , Shared folder and my folder, Creating new folder

Business Model Continued

Working with the Enterprise Manager, testing and validating the Repository, cache disabling,dashboard prompt, filtering, editing of dashboard with action link.Water fall model.

Working with Repository

Working with the Repository, creating Test Report, adding calculations, deploying OBIEE analysis, coming up with landing page UI and its features,repository variables, session and presentation variables.

BI Presentation Catalog

Learning about the Oracle BI Presentation Catalog, accessing and managing objects, Report archiving and exporting, data grouping and limiting in analyses, data formatting, conditional formatting, master detail report, report creation with multiple subject areas, data mashup, visual analyzer, performance tile, BI functionality, waterfall model, graphs, pivot table, Pie chart, KPI watchList.

Dashboard Creation

The OBIEE dashboard setup, basics of dashboard and dashboard pages, deploying Dashboard Builder for building Dashboards, editing, sharing, and saving Dashboard analysis,cache creation & clearing, ODBC functions in OBIEE, Logical Table Source, summary & detail report.

OBIEE Security & Management

Securing the Oracle Business Intelligence Suite with Enterprise Manager, creating alerts, managing grouping and maintenance,administrating, the various types of security in OBIEE, object, task and folder level security, Report scheduling.

What projects I will be working on this OBIEE training?

Project :  Report formatting using OBIEE

Industry : General

Problem Statement : How to find the revenue generated for a business

Topics : This is an Oracle Business Intelligence project that is associated with creating complex dashboards and performing formatting of the report. You will gain hands-on experience in filtering and sorting of the report data depending on the business requirements. This project will also help you understand how to convert the data into graphs for easy visualization and analysis. As part of the project you will gain experience in calculating the subtotal and grand total in a business scenario while finding the revenue generated.

Highlights :

• Filtering and sorting of Reports
• Deploying visualization & analysis techniques
• Designing an OBIEE dashboard.

Getting started with Talend

Working of Talend,Introduction to Talend Open Studio and its Usability,What is Meta Data?

Jobs

Creating a new Job,Concept and creation of Delimited file,Using Meta Data and its Significance,What is propagation?,Data integration schema,Creating Jobs using t-filter row and string filter,Input delimation file creation

Overview of Schema and Aggregation

Job design and its features,What is a T map?,Data Aggregation,Introduction to triplicate and its Working,Significance and working of tlog,T map and its properties.

Connectivity with Data Source

Extracting data from the source,Source and Target in Database (MySQL),Creating a connection, Importing Schema or Metadata

Getting started with Routines/Functions

Calling and using Functions,What are Routines?,Use of XML file in Talend,Working of Format data functions,What is type casting?

Data Transformation

Defining Context variable,Learning Parameterization in ETL,Writing an example using trow generator,Define and Implement Sorting,What is Aggregator?,Using t flow for publishing data,Running Job in a loop.

Connectivity with Hadoop

Learn to start Trish Server,Connectivity of ETL tool connect with Hadoop,Define ETL method,Implementation of Hive,Data Import into Hive with an example,An example of Partitioning in hive,Reason behind no customer table overwriting?,Component of ETL,Hive vs. Pig,Data Loading using demo customer,ETL Tool,Parallel Data Execution.

Introduction to Hadoop and its Ecosystem, Map Reduce and HDFS

Big Data, Factors constituting Big Data,Hadoop and Hadoop Ecosystem,Map Reduce -Concepts of Map, Reduce, Ordering, Concurrency, Shuffle, Reducing, Concurrency ,Hadoop Distributed File System (HDFS) Concepts and its Importance,Deep Dive in Map Reduce – Execution Framework, Partitioner Combiner, Data Types, Key pairs,HDFS Deep Dive – Architecture, Data Replication, Name Node, Data Node, Data Flow, Parallel Copying with DISTCP, Hadoop Archives

Hands on Exercises

Installing Hadoop in Pseudo Distributed Mode, Understanding Important configuration files, their Properties and Demon Threads,Accessing HDFS from Command Line

Map Reduce – Basic Exercises,Understanding Hadoop Eco-system,Introduction to Sqoop, use cases and Installation,Introduction to Hive, use cases and Installation,Introduction to Pig, use cases and Installation,Introduction to Oozie, use cases and Installation,Introduction to Flume, use cases and Installation,Introduction to Yarn

Mini Project – Importing Mysql Data using Sqoop and Querying it using Hive

Deep Dive in Map Reduce

How to develop Map Reduce Application, writing unit test,Best Practices for developing and writing, Debugging Map Reduce applications,Joining Data sets in Map Reduce

Hive

A. Introduction to Hive

What Is Hive?,Hive Schema and Data Storage,Comparing Hive to Traditional Databases,Hive vs. Pig,Hive Use Cases,Interacting with Hive

B. Relational Data Analysis with Hive

Hive Databases and Tables,Basic HiveQL Syntax,Data Types ,Joining Data Sets,Common Built-in Functions,Hands-On Exercise: Running Hive Queries on the Shell, Scripts, and Hue

C. Hive Data Management

Hive Data Formats,Creating Databases and Hive-Managed Tables,Loading Data into Hive,Altering Databases and Tables,Self-Managed Tables,Simplifying Queries with Views,Storing Query Results,Controlling Access to Data,Hands-On Exercise: Data Management with Hive

D. Hive Optimization

Understanding Query Performance,Partitioning,Bucketing,Indexing Data

E. Extending Hive

Topics : User-Defined Functions

F. Hands on Exercises – Playing with huge data and Querying extensively.

G. User defined Functions, Optimizing Queries, Tips and Tricks for performance tuning

Pig

A. Introduction to Pig

What Is Pig?,Pig’s Features,Pig Use Cases,Interacting with Pig

B. Basic Data Analysis with Pig

Pig Latin Syntax, Loading Data,Simple Data Types,Field Definitions,Data Output,Viewing the Schema,Filtering and Sorting Data,Commonly-Used Functions,Hands-On
Exercise: Using Pig for ETL Processing

C. Processing Complex Data with Pig

Complex/Nested Data Types,Grouping,Iterating Grouped Data,Hands-On Exercise: Analyzing Data with Pig

D.  Multi-Data set Operations with Pig

Techniques for Combining Data Sets,Joining Data Sets in Pig,Set Operations,Splitting Data Sets,Hands-On Exercise

E. Extending Pig

Macros and Imports,UDFs,Using Other Languages to Process Data with Pig,Hands-On Exercise: Extending Pig with Streaming and UDFs

F. Pig Jobs

Impala

A. Introduction to Impala

What is Impala?,How Impala Differs from Hive and Pig,How Impala Differs from Relational Databases,Limitations and Future Directions Using the Impala Shell

B. Choosing the best (Hive, Pig, Impala)

Major Project – Putting it all together and Connecting Dots

Putting it all together and Connecting Dots,Working with Large data sets, Steps involved in analyzing large data

ETL Connectivity with Hadoop Ecosystem

How ETL tools work in big data Industry,Connecting to HDFS from ETL tool and moving data from Local system to HDFS,Moving Data from DBMS to HDFS,Working with Hive with ETL Tool,Creating Map Reduce job in ETL tool,End to End ETL PoC showing Hadoop integration with ETL tool.

Job and Certification Support

Major Project, Hadoop Development, cloudera Certification Tips and Guidance and Mock Interview Preparation, Practical Development Tips and Techniques, certification preparation

What projects I will be working on this Talend For Hadoop training?

Project Work

1. Project – Jobs

Problem Statement – It describes that how to create a job using metadata. For this it includes following actions:

Create XML File,Create Delimited File,Create Excel File,Create Database Connection

2. Hadoop Projects

A. Project – Working with Map Reduce, Hive, Sqoop

Problem Statement – It describes that how to import mysql data using sqoop and querying it using hive and also describes that how to run the word count mapreduce job.

B. Project – Connecting Pentaho with Hadoop Eco-system

Problem Statement – It includes:

Quick Overview of ETL and BI,Configuring Pentaho to work with Hadoop Distribution,Loading data into Hadoop cluster,Transforming data into Hadoop cluster
Extracting data from Hadoop Cluster