Wednesday, January 13, 2010

IBM WebSphere Information Analyzer

IBM WebSphere Information Analyzer evaluates the content and structure of your data for consistency and quality.IBM WebSphere Information Analyzer helps you to assess the quality of your data by identifying inconsistencies, redundancies, and anomalies in your data at the column, table, and cross-table levels. IBM WebSphere Information Analyzer also makes inferences about the best choices regarding data structure.Inferences help you to learn more about the optimal structure of your data and what you can do to improve the quality of your data. In addition, IBM WebSphere Information Analyzer provides a mechanism called Baseline Analysis to help you assess whether a data quality procedure that you implemented has resulted in an improvement in data quality by comparing a prior version of analysis results with the current analysis results for a given data source.
The main functions provided by IBM WebSphere Information Analyzer are:
 Column Analysis
 Primary Key Analysis
 Foreign Key Analysis
 Cross Domain Analysis
 Publish Analysis Results
 Baseline Analysis
 Reports



Meta Data/Domain Integrity
•Column Analysis
•Completeness
•Consistency
•Pattern Consistency
•Translation table creation

Structural Integrity
•Table Analysis
•Key Analysis

Entity Integrity
•Duplicate Analysis
•Targeted Data Accuracy

Relational Integrity
•Cross-Table Analysis
•Redundancy Analysis

DQA process. Those steps are as follows:

Prepare the data for assessment
Select the data sources to be investigated and analyzed.
2. Conduct data discovery
The DA and SME perform the investigation and analyses using tools such as
IBM WebSphere Information Analyzer and IBM WebSphere AuditStage. This
discovery involves checking metadata integrity, structural integrity, entity
integrity, relational integrity, and domain integrity.
3. Document data quality issues and decisions
After all information about data quality is known, you can make and
implement the appropriate data alignment and cleansing decisions.




Sunday, January 10, 2010

Balanced Hardware Configuration for Data Warehouse

Balanced Hardware Configuration
A properly sized and balanced hardware configuration is required to maximize data
warehouse performance. The following sections discuss important considerations in
achieving this balance:
How Many CPUs and What Clock Speed Do I Need?
■ How Much Memory Do I Need?
■ How Many Disks Do I Need?
■ How Do I Determine Sufficient I/O Bandwidth?


How Many CPUs and What Clock Speed Do I Need?
Central processing units (CPUs) provide the calculation capabilities in a data Warehouse. You must have sufficient CPU power to perform the data warehouse Operations. Parallel operations are more CPU-intensive than the equivalent serial
operations.

Use the estimated highest throughput as a guideline for the number of CPUs you
need. As a rough estimate, use the following formula:

< No. of CPUs > = < maximum throughput in MB/secon> / 200 MB

This formula assumes that a CPU can sustain up to about 200 MB per second. For
example, if a system requires a maximum throughput of 1200 MB per second, then the
system needs = 1200/200 = 6 CPUs. A configuration with 1
server with 6 CPUs can service this system. A 2-node clustered system could be
configured with 3 CPUs in both nodes.
----------------------------------------------------------------------------------
How Much Memory Do I Need?

Memory in a data warehouse is particularly important for processing
memory-intensive operations such as large sorts. Access to the data cache is less
important in a data warehouse because most of the queries access vast amounts of
data. Data warehouses do not have the same memory requirements as mission-critical
OLTP applications.

The number of CPUs provides a good guideline for the amount of memory you need.
Use the following simplified formula to derive the amount of memory you need from
the CPUs that you select:

< amount of memory in GB > = 2 * < No. Of CPUs >

For example, a system with 6 CPUs needs 2 * 6 = 12 GB of memory. Most
standard servers fulfill this requirement.
----------------------------------------------------------------------------------
How Many Disks Do I Need?

A common mistake in data warehouse environments is to size the storage based on the
maximum capacity needed. Sizing that is based exclusively on storage requirements
will likely create a throughput bottleneck.

Use the maximum throughput you require to find out how many disk arrays you
need. Use the storage provider's specifications to find out how much throughput a
disk array can sustain. Note that storage providers measure in Gb per second, and
your initial throughput estimate is based on MB per second. An average disk
controller has a maximum throughput of 2 Gb per second, which translates to a
sustainable throughput of about (70% * 2 Gbit/s) /8 = 180 MB/s.
Use the following formula to determine the number of disk arrays you need:

< No. of disk controllers > = < throughput in MB/secon > / < individual controller throughput in MB/secon >

For example, our system with 1200 MB per second throughput requires at least 1200 /
180 = 7 disk arrays.
-------------------------------------------------------------------
How Do I Determine Sufficient I/O Bandwidth?

The end-to-end I/O system consists of more components than just the CPUs and disks.
A well-balanced I/O system must provide approximately the same bandwidth across
all components in the I/O system. These components include:
■ Host Bus Adapters (HBAs), the connectors between the server and the storage.
■ Switches, in between the servers and a Storage Area Network (SAN) or Network
Attached Storage (NAS).
■ Ethernet adapters for network connectivity (GigE NIC or Infiniband). In a Oracle
RAC environment, you need an additional private port for the interconnect
between the nodes that you should not include when sizing the system for I/O
throughput. The interconnect must be sized separately, taking into account factors
such as internode parallel execution.
■ Wires that connect the individual components.
Each of the components has to be able to provide sufficient I/O bandwidth to ensure a
well-balanced I/O system. The initial throughput you estimated and the hardware
specifications from the vendors are the basis to determine the quantities of the
individual components you need. Use the conversion in the following table to
translate the vendors' maximum throughput numbers in bits into sustainable
throughput numbers in bytes.

Friday, January 8, 2010

What is a Trigger?

What is a Trigger?

A trigger is a pl/sql block structure which is fired when a DML statements like Insert, Delete, Update is executed on a database table. A trigger is triggered automatically when an associated DML statement is executed.

Syntax of Triggers

The Syntax for creating a trigger is:
CREATE [OR REPLACE ] TRIGGER trigger_name
{BEFORE | AFTER | INSTEAD OF }
{INSERT [OR] | UPDATE [OR] | DELETE}
[OF col_name]
ON table_name
[REFERENCING OLD AS o NEW AS n]
[FOR EACH ROW]
WHEN (condition)
BEGIN --- sql statements
END;

•CREATE [OR REPLACE ] TRIGGER trigger_name - This clause creates a trigger with the given name or overwrites an existing trigger with the same name.
•{BEFORE | AFTER | INSTEAD OF } - This clause indicates at what time should the trigger get fired. i.e for example: before or after updating a table. INSTEAD OF is used to create a trigger on a view. before and after cannot be used to create a trigger on a view.
•{INSERT [OR] | UPDATE [OR] | DELETE} - This clause determines the triggering event. More than one triggering events can be used together separated by OR keyword. The trigger gets fired at all the specified triggering event.
•[OF col_name] - This clause is used with update triggers. This clause is used when you want to trigger an event only when a specific column is updated.
•CREATE [OR REPLACE ] TRIGGER trigger_name - This clause creates a trigger with the given name or overwrites an existing trigger with the same name.
•[ON table_name] - This clause identifies the name of the table or view to which the trigger is associated.
•[REFERENCING OLD AS o NEW AS n] - This clause is used to reference the old and new values of the data being changed. By default, you reference the values as :old.column_name or :new.column_name. The reference names can also be changed from old (or new) to any other user-defined name. You cannot reference old values when inserting a record, or new values when deleting a record, because they do not exist.
•[FOR EACH ROW] - This clause is used to determine whether a trigger must fire when each row gets affected ( i.e. a Row Level Trigger) or just once when the entire sql statement is executed(i.e.statement level Trigger).
•WHEN (condition) - This clause is valid only for row level triggers. The trigger is fired only for rows that satisfy the condition specified.

Exception Handling

Exception Handling

In this section we will discuss about the following,
1) What is Exception Handling.
2) Structure of Exception Handling.
3) Types of Exception Handling.

1) What is Exception Handling?
PL/SQL provides a feature to handle the Exceptions which occur in a PL/SQL Block known as exception Handling. Using Exception Handling we can test the code and avoid it from exiting abruptly. When an exception occurs a messages which explains its cause is recieved.
PL/SQL Exception message consists of three parts.
1) Type of Exception
2) An Error Code
3) A message
By Handling the exceptions we can ensure a PL/SQL block does not exit abruptly.

2) Structure of Exception Handling.
The General Syntax for coding the exception section

DECLARE
Declaration section
BEGIN
Exception section
EXCEPTION
WHEN ex_name1 THEN
-Error handling statements
WHEN ex_name2 THEN
-Error handling statements
WHEN Others THEN
-Error handling statements
END;

General PL/SQL statments can be used in the Exception Block.

When an exception is raised, Oracle searches for an appropriate exception handler in the exception section. For example in the above example, if the error raised is 'ex_name1 ', then the error is handled according to the statements under it. Since, it is not possible to determine all the possible runtime errors during testing fo the code, the 'WHEN Others' exception is used to manage the exceptions that are not explicitly handled. Only one exception can be raised in a Block and the control does not return to the Execution Section after the error is handled.

If there are nested PL/SQL blocks like this.

DELCARE
Declaration section
BEGIN
DECLARE
Declaration section
BEGIN
Execution section
EXCEPTION
Exception section
END;
EXCEPTION
Exception section
END;

In the above case, if the exception is raised in the inner block it should be handled in the exception block of the inner PL/SQL block else the control moves to the Exception block of the next upper PL/SQL Block. If none of the blocks handle the exception the program ends abruptly with an error.

3) Types of Exception.
There are 3 types of Exceptions.
a) Named System Exceptions
b) Unnamed System Exceptions
c) User-defined Exceptions

a) Named System Exceptions
System exceptions are automatically raised by Oracle, when a program violates a RDBMS rule. There are some system exceptions which are raised frequently, so they are pre-defined and given a name in Oracle which are known as Named System Exceptions.

For example: NO_DATA_FOUND and ZERO_DIVIDE are called Named System exceptions.

Named system exceptions are:
1) Not Declared explicitly,
2) Raised implicitly when a predefined Oracle error occurs,
3) caught by referencing the standard name within an exception-handling routine.



For Example: Suppose a NO_DATA_FOUND exception is raised in a proc, we can write a code to handle the exception as given below.

BEGIN
Execution section
EXCEPTION
WHEN NO_DATA_FOUND THEN
dbms_output.put_line ('A SELECT...INTO did not return any row.');
END;

b) Unnamed System Exceptions
Those system exception for which oracle does not provide a name is known as unamed system exception. These exception do not occur frequently. These Exceptions have a code and an associated message.

There are two ways to handle unnamed sysyem exceptions:
1. By using the WHEN OTHERS exception handler, or
2. By associating the exception code to a name and using it as a named exception.

We can assign a name to unnamed system exceptions using a Pragma called EXCEPTION_INIT.
EXCEPTION_INIT will associate a predefined Oracle error number to a programmer_defined exception name.

Steps to be followed to use unnamed system exceptions are
• They are raised implicitly.
• If they are not handled in WHEN Others they must be handled explicity.
• To handle the exception explicity, they must be declared using Pragma EXCEPTION_INIT as given above and handled referecing the user-defined exception name in the exception section.

The general syntax to declare unnamed system exception using EXCEPTION_INIT is:

DECLARE
exception_name EXCEPTION;
PRAGMA
EXCEPTION_INIT (exception_name, Err_code);
BEGIN
Execution sectionEXCEPTION
WHEN exception_name THEN
handle the exception
END;

For Example: Lets consider the product table and order_items table from sql joins.

Here product_id is a primary key in product table and a foreign key in order_items table.
If we try to delete a product_id from the product table when it has child records in order_id table an exception will be thrown with oracle code number -2292.
We can provide a name to this exception and handle it in the exception section as given below.

DECLARE
Child_rec_exception EXCEPTION;
PRAGMA
EXCEPTION_INIT (Child_rec_exception, -2292);
BEGIN
Delete FROM product where product_id= 104;
EXCEPTION
WHEN Child_rec_exception
THEN Dbms_output.put_line('Child records are present for this product_id.');
END;
/

c) User-defined Exceptions
Apart from sytem exceptions we can explicity define exceptions based on business rules. These are known as user-defined exceptions.

Steps to be followed to use user-defined exceptions:
• They should be explicitly declared in the declaration section.
• They should be explicitly raised in the Execution Section.
• They should be handled by referencing the user-defined exception name in the exception section.

For Example: Lets consider the product table and order_items table from sql joins to explain user-defined exception.
Lets create a business rule that if the total no of units of any particular product sold is more than 20, then it is a huge quantity and a special discount should be provided.

DECLARE
huge_quantity EXCEPTION;
CURSOR product_quantity is
SELECT p.product_name as name, sum(o.total_units) as units
FROM order_tems o, product p
WHERE o.product_id = p.product_id;
quantity order_tems.total_units%type;
up_limit CONSTANT order_tems.total_units%type := 20;
message VARCHAR2(50);
BEGIN
FOR product_rec in product_quantity LOOP
quantity := product_rec.units;
IF quantity > up_limit THEN
message := 'The number of units of product ' || product_rec.name || ' is more than 20. Special discounts should be provided. Rest of the records are skipped. '
RAISE huge_quantity;
ELSIF quantity < up_limit THEN
v_message:= 'The number of unit is below the discount limit.';
END IF;
dbms_output.put_line (message);
END LOOP;
EXCEPTION
WHEN huge_quantity THEN
dbms_output.put_line (message);
END;
/

RAISE_APPLICATION_ERROR ( )
RAISE_APPLICATION_ERROR is a built-in procedure in oracle which is used to display the user-defined error messages along with the error number whose range is in between -20000 and -20999.

Whenever a message is displayed using RAISE_APPLICATION_ERROR, all previous transactions which are not committed within the PL/SQL Block are rolled back automatically (i.e. change due to INSERT, UPDATE, or DELETE statements).

RAISE_APPLICATION_ERROR raises an exception but does not handle it.

RAISE_APPLICATION_ERROR is used for the following reasons,
a) to create a unique id for an user-defined exception.
b) to make the user-defined exception look like an Oracle error.

The General Syntax to use this procedure is:

RAISE_APPLICATION_ERROR (error_number, error_message);
• The Error number must be between -20000 and -20999
• The Error_message is the message you want to display when the error occurs.

Steps to be folowed to use RAISE_APPLICATION_ERROR procedure:
1. Declare a user-defined exception in the declaration section.
2. Raise the user-defined exception based on a specific business rule in the execution section.
3. Finally, catch the exception and link the exception to a user-defined error number in RAISE_APPLICATION_ERROR.

Using the above example we can display a error message using RAISE_APPLICATION_ERROR.

DECLARE
huge_quantity EXCEPTION;
CURSOR product_quantity is
SELECT p.product_name as name, sum(o.total_units) as units
FROM order_tems o, product p
WHERE o.product_id = p.product_id;
quantity order_tems.total_units%type;
up_limit CONSTANT order_tems.total_units%type := 20;
message VARCHAR2(50);
BEGIN
FOR product_rec in product_quantity LOOP
quantity := product_rec.units;
IF quantity > up_limit THEN
RAISE huge_quantity;
ELSIF quantity < up_limit THEN
v_message:= 'The number of unit is below the discount limit.';
END IF;
Dbms_output.put_line (message);
END LOOP;
EXCEPTION
WHEN huge_quantity THEN
raise_application_error(-2100, 'The number of unit is above the discount limit.'); END; /

Parameters in Procedure and Functions

Parameters in Procedure and Functions

How to pass parameters to Procedures and Functions in PL/SQL ?
In PL/SQL, we can pass parameters to procedures and functions in three ways.

1) IN type parameter: These types of parameters are used to send values to stored procedures.
2) OUT type parameter: These types of parameters are used to get values from stored procedures. This is similar to a return type in functions.
3) IN OUT parameter: These types of parameters are used to send values and get values from stored procedures.

NOTE: If a parameter is not explicitly defined a parameter type, then by default it is an IN type parameter.


1) IN parameter:
This is similar to passing parameters in programming languages. We can pass values to the stored procedure through these parameters or variables. This type of parameter is a read only parameter. We can assign the value of IN type parameter to a variable or use it in a query, but we cannot change its value inside the procedure.

The General syntax to pass a IN parameter is

CREATE [OR REPLACE] PROCEDURE procedure_name (
param_name1 IN datatype, param_name12 IN datatype ... )

•param_name1, param_name2... are unique parameter names.
•datatype - defines the datatype of the variable.
•IN - is optional, by default it is a IN type parameter.


2) OUT Parameter:
The OUT parameters are used to send the OUTPUT from a procedure or a function. This is a write-only parameter i.e, we cannot pass values to OUT paramters while executing the stored procedure, but we can assign values to OUT parameter inside the stored procedure and the calling program can recieve this output value.

The General syntax to create an OUT parameter is

CREATE [OR REPLACE] PROCEDURE proc2 (param_name OUT datatype)

The parameter should be explicity declared as OUT parameter.

3) IN OUT Parameter:

The IN OUT parameter allows us to pass values into a procedure and get output values from the procedure. This parameter is used if the value of the IN parameter can be changed in the calling program.

By using IN OUT parameter we can pass values into a parameter and return a value to the calling program using the same parameter. But this is possible only if the value passed to the procedure and output value have a same datatype. This parameter is used if the value of the parameter will be changed in the procedure.

The General syntax to create an IN OUT parameter is

CREATE [OR REPLACE] PROCEDURE proc3 (param_name IN OUT datatype)


The below examples show how to create stored procedures using the above three types of parameters.

Example1:

Using IN and OUT parameter:

Let’s create a procedure which gets the name of the employee when the employee id is passed.

1> CREATE OR REPLACE PROCEDURE emp_name (id IN NUMBER, emp_name OUT NUMBER)
2> IS
3> BEGIN
4> SELECT first_name INTO emp_name
5> FROM emp_tbl WHERE empID = id;
6> END;
7> /

We can call the procedure ‘emp_name’ in this way from a PL/SQL Block.

1> DECLARE
2> empName varchar(20);
3> CURSOR id_cur SELECT id FROM emp_ids;
4> BEGIN
5> FOR emp_rec in id_cur
6> LOOP
7> emp_name(emp_rec.id, empName);
8> dbms_output.putline('The employee ' || empName || ' has id ' || emp-rec.id);
9> END LOOP;
10> END;
11> /

In the above PL/SQL Block
In line no 3; we are creating a cursor ‘id_cur’ which contains the employee id.
In line no 7; we are calling the procedure ‘emp_name’, we are passing the ‘id’ as IN parameter and ‘empName’ as OUT parameter.
In line no 8; we are displaying the id and the employee name which we got from the procedure ‘emp_name’.

Example 2:

Using IN OUT parameter in procedures:

1> CREATE OR REPLACE PROCEDURE emp_salary_increase
2> (emp_id IN emptbl.empID%type, salary_inc IN OUT emptbl.salary%type)
3> IS
4> tmp_sal number;
5> BEGIN
6> SELECT salary
7> INTO tmp_sal
8> FROM emp_tbl
9> WHERE empID = emp_id;
10> IF tmp_sal between 10000 and 20000 THEN
11> salary_inout := tmp_sal * 1.2;
12> ELSIF tmp_sal between 20000 and 30000 THEN
13> salary_inout := tmp_sal * 1.3;
14> ELSIF tmp_sal > 30000 THEN
15> salary_inout := tmp_sal * 1.4;
16> END IF;
17> END;
18> /

The below PL/SQL block shows how to execute the above 'emp_salary_increase' procedure.

1> DECLARE
2> CURSOR updated_sal is
3> SELECT empID,salary
4> FROM emp_tbl;
5> pre_sal number;
6> BEGIN
7> FOR emp_rec IN updated_sal LOOP
8> pre_sal := emp_rec.salary;
9> emp_salary_increase(emp_rec.empID, emp_rec.salary);
10> dbms_output.put_line('The salary of ' || emp_rec.empID ||
11> ' increased from '|| pre_sal || ' to '||emp_rec.salary);
12> END LOOP;
13> END;
14> /

PL/SQL Functions

PL/SQL Functions

What is a Function in PL/SQL?
A function is a named PL/SQL Block which is similar to a procedure. The major difference between a procedure and a function is, a function must always return a value, but a procedure may or may not return a value.

The General Syntax to create a function is:

CREATE [OR REPLACE] FUNCTION function_name [parameters]
RETURN return_datatype;
IS
Declaration_section
BEGIN
Execution_section
Return return_variable;
EXCEPTION
exception section
Return return_variable;
END;

1) Return Type: The header section defines the return type of the function. The return datatype can be any of the oracle datatype like varchar, number etc.
2) The execution and exception section both should return a value which is of the datatype defined in the header section.

For example, let’s create a frunction called ''employer_details_func' similar to the one created in stored proc

1> CREATE OR REPLACE FUNCTION employer_details_func
2> RETURN VARCHAR(20);
3> IS
5> emp_name VARCHAR(20);
6> BEGIN
7> SELECT first_name INTO emp_name
8> FROM emp_tbl WHERE empID = '100';
9> RETURN emp_name;
10> END;
11> /

In the example we are retrieving the ‘first_name’ of employee with empID 100 to variable ‘emp_name’.
The return type of the function is VARCHAR which is declared in line no 2.
The function returns the 'emp_name' which is of type VARCHAR as the return value in line no 9.

How to execute a PL/SQL Function?
A function can be executed in the following ways.

1) Since a function returns a value we can assign it to a variable.

employee_name := employer_details_func;If ‘employee_name’ is of datatype varchar we can store the name of the employee by assigning the return type of the function to it.

2) As a part of a SELECT statement

SELECT employer_details_func FROM dual;3) In a PL/SQL Statements like,

dbms_output.put_line(employer_details_func);This line displays the value returned by the function.

Stored Procedures

Stored Procedures

What is a Stored Procedure?
A stored procedure or in simple a proc is a named PL/SQL block which performs one or more specific task. This is similar to a procedure in other programming languages. A procedure has a header and a body. The header consists of the name of the procedure and the parameters or variables passed to the procedure. The body consists or declaration section, execution section and exception section similar to a general PL/SQL Block. A procedure is similar to an anonymous PL/SQL Block but it is named for repeated usage.

We can pass parameters to procedures in three ways.
1) IN-parameters
2) OUT-parameters
3) IN OUT-parameters

A procedure may or may not return any value.

General Syntax to create a procedure is:

CREATE [OR REPLACE] PROCEDURE proc_name [list of parameters]

IS
Declaration section
BEGIN
Execution section
EXCEPTION
Exception section
END;

IS - marks the beginning of the body of the procedure and is similar to DECLARE in anonymous PL/SQL Blocks. The code between IS and BEGIN forms the Declaration section.


The syntax within the brackets [ ] indicate they are optional. By using CREATE OR REPLACE together the procedure is created if no other procedure with the same name exists or the existing procedure is replaced with the current code.

The below example creates a procedure ‘employer_details’ which gives the details of the employee.

1> CREATE OR REPLACE PROCEDURE employer_details
2> IS
3> CURSOR emp_cur IS
4> SELECT first_name, last_name, salary FROM emp_tbl;
5> emp_rec emp_cur%rowtype;
6> BEGIN
7> FOR emp_rec in sales_cur
8> LOOP
9> dbms_output.put_line(emp_cur.first_name || ' ' ||emp_cur.last_name
10> || ' ' ||emp_cur.salary);
11> END LOOP;
12>END;
13> /
How to execute a Stored Procedure?
There are two ways to execute a procedure.

1) From the SQL prompt.


EXECUTE [or EXEC] procedure_name; 2) Within another procedure – simply use the procedure name.


procedure_name;NOTE: In the examples given above, we are using backward slash ‘/’ at the end of the program. This indicates the oracle engine that the PL/SQL program has ended and it can begin processing the statements.