Data Warehousing and

Data Mining
Data Warehouse Design
Logical Design

Requirement analysis
Requirement specification
Conceptual design
Logical design
Physical design

 is not a design --- used just to describe

the business
should be a business model -- and not
data design model
should identify real world business
objects (e.g. Customer, Order, Sale,
Policy, etc)
the relationships between these objects

 Translating the agreed business

requirements into system deliverables
within the scope defined by the
conceptual design results in Logical and
then Physical design
Logical Design
Is more conceptual and abstract than physical
Looks at the logical relationships among the
objects
Defines the types of information that are
needed

 Entity-Relationship model
Identify the things of importance (Entities)
The properties of these things (Attributes)
Show how two or more entities are related
(Relationship)

The entities, attributes and

relationships existing in a ER model are
translated into a star model.

 arrange data into a series of logical

relationships called entities and attributes
An entity
represents a chunk of information
often maps to a table

An attribute
a component of an entity that helps define the uniqueness of the
entity
often maps to a column.

A unique identifier is
what is added to tables so that it is possible to differentiate
between the same item when it appears in different places
usually a primary key.

record the associations between objects and

facts

 In dimensional modeling, instead of

seeking to discover atomic units of
information (such as entities and
attributes) and all of the relationships
between them, we
identify which information belongs to a central
fact table and which information belongs to its
associated dimension tables
identify business subjects or fields of data,
define relationships between business subjects,
and name the attributes for each subject.

 often start with a conceptual schema

and then generates relational structures
should result in
a set of entities and attributes
corresponding to fact tables and dimension
tables
a model of operational data from your
source into subject-oriented information in
the target data warehouse schema.

 mapping the conceptual model

structures to the logical model ones
taking into account implementation
issues, which are not considered in
the conceptual schema

 Identify all applicable entities (Conceptual

Model doesn't express all the details)
Attributize (either fully or mostly) data
entities (with business nomenclature)
Assign datatype domains (e.g. text, date,
numeric) vs. datatypes (varchar, integer)
Resolve M:M relationships (e.g. with an
associative entity, record versioning, etc)

 Formalize keys (primary, alternate,

foreign)
carry out resolution of subtypes
Perform abstraction (e.g. abstracting
conceptual entities such as Customer,
Prospect, Supplier, etc. into a
generalized entity such as Party) as
part of the normalization process (so
that data can be stored once)

 a type of database that is optimized

for data warehouse and online
analytical processing (OLAP)
applications.
is designed to make the best use of
storing and utilizing data

 is created using input from existing

relational databases.
implies the ability to rapidly process
the data in the database so that
answers can be generated quickly

 can receive data from a variety of

relational databases and structure the
information into categories and sections
that can be accessed in a number of
different ways
can obtain data more easily, more quickly
and more succinctly
uses the idea of a data cube to represent
the dimensions of data available to a user

Store data in dimensions

Multiple dimensions, aka cubes (also

called hypercube), allow users to
analyze any view of data

Can consolidate data much faster than

relational database

 "sales" could be viewed in the

dimensions of product model,
geography, time, or some additional
dimension. In this case, "sales" is known
as the measure attribute of the data
cube and the other dimensions are seen
as feature attributes. More hierarchies
and levels can be defined within a
dimension (for example, state and city
levels within a regional hierarchy).

Product Dimension
ProductKey

Time Dimension
TimeKey

Sales Fact Table

Store Dimension
StoreKey

TimeKey (FK)
ProductKey (FK)
StoreKey (FK)
Sale Amount

SALES Fact Table

Time
Key

Product
Key

Stores
Key

Sales
Amount

Rs.1000

Rs.1200

Rs.1500

Product

3
2
1
1
1
2

Store

Time

Product

3
2
1
1

Rs.1000

1
2

Store

Time

Product

3
2
1
1
1
2

Store

Rs.1200

Time

Relational
Database
structured for keyword
searches and building a
query by specifying fields
and perimeters, using SQL

Multi Dimensional
Database
a user simply poses the
question in everyday
verbiage. The user is
helped by the several
online help tools associated
with software programs
such as word processing
and spreadsheet
applications, as well as
several of the more popular
search engines currently in
use

 SQL has several aggregate operators:

SUM(), MIN(), MAX(), COUNT(), AVG()
Some systems extend this with many others:

Stat functions, financial functions

i.e. RANK(), N_TILE(), RATIO_TO_TOTAL()

The basic idea is:

Combine all values in a column
into a single scalar value

AVG(Temp)
Weather;

AVG()

13

SELECT
FROM

17
...

Syntax

 GROUP BY allows aggregates over table sub-groups

Result is a new table
Syntax
SELECT
Time, Altitude, AVG(Temp)
FROM
Weather
GROUP BY Time, Altitude;
Time

Latitude

Longitude

Altitude
(m)

Temp

07/9/5:1500

20

24

Time

Altitude
(m)

AVG(Temp)

07/9/5:1500

20

22

07/9/5:1500

20

23

07/9/5:1500

100

17

07/9/5:1500

100

17

07/9/9:1500

50

19

07/9/9:1500

50

20

07/9/9:1500

50

21

 Users want Histograms

MAX(Temp)

Suppose:
Day(): time day
Nation(): latitude & longitude name of the country
SELECT
day, nation, MAX(Temp)
FROM
Weather
GROUP BY Day(Time) AS day,
Nation(Latitude, Longitude) AS nation;

day, nation

 The following is not a STANDARD SQL

query!!
SELECT
FROM
GROUP BY

day, nation, MAX(Temp)

Weather
Day(Time) AS day,
Nation(Latitude, Longitude) AS nation;

In standard SQL:
SELECT
FROM

day, nation, MAX(Temp)

(SELECT Day(Time) AS day,
Nation(Latitude,Longitude) AS nation,
FROM Weather) AS foo
GROUP BY day, nation;
A Nested Query

 Users want Roll-Up Reports

Attributes: Model, Year, Color, and, Sales
Chevy Sales Roll Up by Model by Year by Color:

Keyword ALL

{Black, White}

{1994, 1995}

Problems with GROUP BY - Roll-Up

Reports

To build the Chevy Sales Roll Up

Unioned GROUP BYs

Too many
GROUP BYs and UNIONs!!

 Users want Cross-Tabulations

Chevy Sales Cross-Tab

By adding the following clause

 Problems with
GROUP BY
GROUP BY cannot directly
construct:
Histograms
Roll-Up Reports
Cross-Tabs

CUBE Operator
Generalize GROUP BY and RollUp and Cross-Tabs!!

CUBE

Think of ALL as a token representing the set

{red, white, blue}

{1990, 1991, 1992}

{Chevy, Ford}

 Sample syntax:
Model, Make, Year, SUM(Sales)
SELECT
FROM
Sales
WHERE
Model IN {Chevy, Ford}
AND
Year BETWEEN 1990 AND 1994
GROUP BY CUBE Model, Make, Year
HAVING
SUM(Sales) > 0;
Note: GROUP BY operator repeats aggregate list

in select list
in group by list

 Allows functional aggregations

(e.g., Sales by quarter):
SELECT
FROM
WHERE
GROUP BY
quarter;

Store, quarter, SUM(Sales)

Sales
nation=Korea AND Year=1994
ROLLUP Store, Quarter(Date) AS

ROLLUP Operator
A Subset of CUBE Operator
Return Sales Roll Up by Store by Quarter in 1994.

 An Example of ..

By Year

Ch
ev

Fo
rd
y

3D Data Cube

90
9
1
91
19

9
19

2
93
9
1
Re

te
hi
W

Sum
By
Co
lo r

By Make

ue
l
B

&

Ye
ar

By

By Color

ke
a
M

&

r
lo
o
C

 A multi-dimensional structure containing

data points that represent unique
combinations of several classifications
A flexible way of storing and
disseminating data

 is a data structure that allows fast

analysis of data
overcomes a limitation of relational
databases
can be thought of as extensions to the
two-dimensional array of a spreadsheet
consists of numeric facts (also called
measures) which are categorized by
dimensions.

 The cube metadata is typically created

from a star schema or snowflake
schema of tables in a relational
database. Measures are derived from
the records in the fact table and
dimensions are derived from the
dimension tables.

Year
Country 2000

2001

2002

2003

AAA

123 456 124 567 125 678 126 789

BBB

987 654 988 654 989 654 999 654

CCC

35 789

36 789

37 789

38 789

 Many recent statistical data management

models and systems are based on cubes
Users can select just those data that are
of interest
Cubes can easily be expanded, e.g. for
extra years, countries, or other
categories
At least in theory, cubes can have an
infinite number of dimensions