Está en la página 1de 51

Ingeniamos el futuro

CAMPUS OF
INTERNATIONAL
EXCELLENCE
Energy-efficient data centers:
Exploiting knowledge about
application and resources
Jos M. Moya <jm.moya@upm.es>
Integrated Systems Laboratory
Jos M.Moya | Madrid (Spain), July 27, 2012 1
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Data centers
Jos M.Moya | Madrid (Spain), July 27, 2012 2
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Jos M.Moya | Madrid (Spain), July 27, 2012 3
13


Figure 1: Worldwide electricity use for data centers (2000, 2005, and 2010)


Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Power distribution
Jos M.Moya | Madrid (Spain), July 27, 2012 4
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Power distribution (Tier 4)
Jos M.Moya | Madrid (Spain), July 27, 2012 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Contents
Motivation
Our approach
Scheduling and resource
management
Virtual machine
optimizations
Centralized management
of low-power modes
Processor design
Conclusions


Jos M.Moya | Madrid (Spain), July 27, 2012 6
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Motivation
Jos M.Moya | Madrid (Spain), July 27, 2012 7
Energy consumption of data centers
1.3% of worldwide energy production in 2010
USA: 80 mill MWh/year in 2011 = 1,5 x NYC
1 data center = 25 000 houses
More than 43 Million Tons of CO
2
emissions per
year (2% worldwide)
More water consumption than many industries
(paper, automotive, petrol, wood, or plastic)
Jonathan Koomey. 2011. Growth in Data center electricity use 2005 to 2010
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Motivation
Jos M.Moya | Madrid (Spain), July 27, 2012 8
It is expected for total data
center electricity use to
exceed 400 GWh/year by
2015.
The required energy for
cooling will continue to be at
least as important as the
energy required for the
computation.
Energy optimization of future
data centers will require a
global and multi-disciplinary
approach.
0
5000
10000
15000
20000
25000
30000
35000
2000 2005 2010
W
o
r
l
d

s
e
r
v
e
r

i
n
s
t
a
l
l
e
d

b
a
s
e

(
t
h
o
u
s
a
n
d
s
)

High-end servers
Mid-range servers
Volume servers
0
50
100
150
200
250
300
2000 2005 2010
E
l
e
c
t
r
i
c
i
t
y

u
s
e


(
b
i
l
l
i
o
n

k
W
h
/
y
e
a
r
)

Infrastructure
Communications
Storage
High-end servers
Mid-range servers
Volume servers
5,75 Million new servers per year
10% unused servers (CO
2
emissions
similar to 6,5 million cars)
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Temperature-dependent
reliability problems
Jos M.Moya | Madrid (Spain), July 27, 2012 9
Time-dependent
dielectric-
breakdown (TDDB)
Electromigration (EM)
Stress
migration (SM)
Thermal
cycling (TC)




Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Cooling a data center
Jos M.Moya | Madrid (Spain), July 27, 2012 10
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Virtualization


- 27%
Energy Star server
conformance


= 6.500
Better capacity
planning

2.500
Server improvements
Jos M.Moya | Madrid (Spain), July 27, 2012 11
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Cooling improvements
Improvements in air flow management and
wider temperature ranges
Jos M.Moya | Madrid (Spain), July 27, 2012 12
Energy savings
up to 25%
25.000
Return of investment
in only 2 years

Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
AC DC
20% reduction of power losses in the
conversion process
47 million dollars savings of real-state costs
Up to 97% efficiency, energy saving enough to
power an iPad during 70 million years

Infrastructure improvements
Jos M.Moya | Madrid (Spain), July 27, 2012 13
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE




Best practices
Jos M.Moya | Madrid (Spain), July 27, 2012 14
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
And
what about IT people?
Jos M.Moya | Madrid (Spain), July 27, 2012 15
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
PUE
Power Usage Effectiveness
State of the Art: PUE 1,2
The important part is IT energy consumption
Current work in energy efficient data centers is
focused in decreasing PUE
Decreasing P
IT
does not decrease PUE, but it is seen in
the electricity bill
But how can we reduce P
IT
?
Jos M.Moya | Madrid (Spain), July 27, 2012 16
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Potential energy savings
by abstraction level
Jos M.Moya | Madrid (Spain), July 27, 2012 17
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Our approach
Global strategy to allow the use of multiple
information sources to coordinate decisions in order
to reduce the total energy consumption
Use of knowledge about the energy demand
characteristics of the applications, and
characteristics of computing and cooling resources
to implement proactive optimization techniques
Jos M.Moya | Madrid (Spain), July 27, 2012 18
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Holistic approach
Jos M.Moya | Madrid (Spain), July 27, 2012 19
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
1. Room-level resource
management
Jos M.Moya | Madrid (Spain), July 27, 2012 20
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Leveraging heterogeneity
Use heterogeneity to minimize energy
consumption from a static/dynamic point of view
Static: Finding the best data center set-up, given a
number of heterogeneous machines
Dynamic: Optimization of task allocation in the
Resource Manager
We show that the best solution implies an
heterogeneous data center
Most data centers are heterogeneous (several
generations of computers)
21
CCGrid 2012
Jos M.Moya | Madrid (Spain), July 27, 2012
M. Zapater, J.M. Moya, J.L. Ayala. Leveraging Heterogeneity for
Energy Minimization in Data Centers, CCGrid 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Current scenario
22
WORKLOAD
Scheduler
Resource
Manager
Execution
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Potential improvements
with best practices
Jos M.Moya | Madrid (Spain), July 27, 2012 23
To appear in Computer Networks, Special Issue on Virtualized Data Centers. All rights reserved.
Cold
Aisle
coldair fromfloor
Racks Racks
Hot Aisle Hot Aisle
heat
recirculation
Fig. 2. Demonstration of heat recirculation: heated air in the hot aisle
loops around the equipment to enter the air inlets.
0
200
400
600
800
1000
1200
1400
0 20 40 60 80 100
P
o
w
e
r

(
K
W
)
job size relative to data center capacity (%)
Total power (computing and cooling) for various scheduling approaches
savings by minimizing computing power
savings by minimizing the recirculations effect
savings by turning off idle machines
unaddressed heat recirculation cost
basic (unavoidable) cost
max computing power, worst thermal placement
min computing power, worst thermal placemenit
optimal computing+cooling
optimal computing+cooling, shut off idles
optimal computing+cooling, shut off idles, no recirculation
Fig. 3. Data center operation cost (in kilowatts) for various savings
modes. Savings are based on heat recirculation data obtained by
FloVENT simulation of the ASU Fulton HPCI data center.
to cost savings, someresearch has included economical models to computing schedules [18]. Most of the schedulers
implement therst t policy in job placement, mainly for reasons of low overhead.
In datacenters, themajority of energy inefficiency isattributed to idlerunning serversand to heat recirculation [2]
(Figure 2). Solutions such as low-voltage ICs [10], ensemble-level power management [11] and energy-efficient de-
sign of data centers [12] which try to address these causes of inefficiency by reducing the inherent heat generation.
Power control schemes, such as Freon-EC (an extension to theFreon power-awaremanagement softwarewhich adds
power control [4, 5]), or using energy-proportional systems [16] can help in addressing the energy inefficiency due
to idle servers. To address heat recirculation, energy-efficient thermal-awarespatial scheduling algorithms havebeen
proposed, such asMinHR and XInt [13]. Spatial scheduling canavoid or even prevent excessiveheat conditions, while
it can greatly reduce thecooling costs, which account for alargeportion (about onethird) of adatacenter sutility.
Todemonstratethemagnitudeof savingsachieved by thermal-awareplacement, and thethermal-awareplacement s
complementary relation to power control schemes, we provide Figure 3, which was produced using XInt and numer-
ical results from previous spatial scheduling work on thermal-aware placement [3]: the top two lines in the gure
represent the most energy-consuming schedules, and they were produced using a variant of XInt that maximizes the
thermal inefficiency of thedata center; thethird line is theXInt curveas obtained in theprevious work [3]; thefourth
line represents the combined use of XInt and turning off idle servers, and it was obtained by removing the power
consumption of all un-assigned servers fromtheXInt line; thebottom linerepresents thepower consumption without
heat recirculation. The gure shows that explicit power control and thermal-aware job placement are mutually com-
plementary, with theformer showing themost savings at low datacenter utilization rates and thelatter at moderate to
high (but not full) datacenter utilization rates. Thegurealso showsthat power-awareyet heat-obli vious approaches
that minimize only thecomputing power (second linefromthetop) do not saveasmuch asthermal-awareapproaches
do.
Theaforementioned thermal-awarejob scheduling algorithmstry tooptimizethespatial scheduling (i.e. placement)
6
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Cooling-aware scheduling and
resource allocation
Jos M.Moya | Madrid (Spain), July 27, 2012 24
To appear in Computer Networks, Special Issue on Virtualized Data Centers. All rights reserved.
0
50
100
150
200
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (a) 40 jobs, 25014 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.197 jobs/hr 0.197 jobs/hr 0.172 jobs/hr 0.197 jobs/hr 0.163 jobs/hr
18.41 hr 18.41 hr 20.75 hr 18.41 hr 51.75 hr
3.4 ms 6.9 ms 213 ms 23 min 40 min
0% 6.2% 8.6% 8.7% 10.2%
cooling energy
computing energy
(a)
0
5
10
15
20
25
30
35
40
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (a) 40 jobs, 25014 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.197 jobs/hr 0.197 jobs/hr 0.172 jobs/hr 0.197 jobs/hr 0.163 jobs/hr
18.41 hr 18.41 hr 20.75 hr 18.41 hr 38.02 hr
3.4 ms 6.9 ms 213 ms 23 min 43 min
0% 11.8% 54.7% 21.8% 60.5%
cooling energy
computing energy
(b)
0
50
100
150
200
250
300
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (b) 120 jobs, 16039 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.580 jobs/hr 0.580 jobs/hr 0.349 jobs/hr 0.580 jobs/hr 0.254 jobs/hr
8.98 hr 8.98 hr 12.17 hr 8.98 hr 48.49 hr
170 ms 186 ms 397 ms 40.8 min 88.6 min
0% 1.7% 4.1% 3.6% 4.7%
cooling energy
computing energy
(c)
0
5
10
15
20
25
30
35
40
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (b) 120 jobs, 16039 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.580 jobs/hr 0.580 jobs/hr 0.349 jobs/hr 0.580 jobs/hr 0.427 jobs/hr
8.98 hr 8.98 hr 12.17 hr 8.98 hr 17.75 hr
171 ms 186 ms 397 ms 42 min 100 min
0% 4.0% 14.6% 14.2% 15.1%
cooling energy
computing energy
(d)
0
50
100
150
200
250
300
350
400
450
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (c) 174 jobs, 45817 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.892 jobs/hr 0.892 jobs/hr 0.861 jobs/hr 0.892 jobs/hr 0.561 jobs/hr
9.99 hr 9.99 hr 13.39 hr 9.99 hr 65.38 hr
173 ms 196 ms 346 ms 20 min 142 min
0% 2.5% 5.9% 9.4% 12.5%
cooling energy
computing energy
(e)
0
20
40
60
80
100
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (c) 174 jobs, 45817 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.892 jobs/hr 0.892 jobs/hr 0.861 jobs/hr 0.892 jobs/hr 0.590 jobs/hr
9.99 hr 9.99 hr 13.39 hr 9.99 hr 61.49 hr
173 ms 191 ms 346 ms 21 min 147 min
0.0% 7.5% 17.3% 25.7% 41.4%
cooling energy
computing energy
(f)
Fig. 8. Energy comparison of the simulated schemes for the three scenarios. The plots correspond in respective positions to the plots of Figure 7.
policy used in the data center, which enables job execution as soon as they arrive if the queue is empty and the data
center is lightly loaded. In the idle-on case (Figure 8a), the total energy consumption using SCINT, EDF-LRH,
FCFS-Backll-FF, FCFS-Backll-LRH and XInt placement is 131.3GJ, 133.9GJ, 146.2GJ, 139.3GJ and 139.3GJ,
respectively. However, in the idle-off case, the energy consumption reduces to 10.3,GJ, 11.8GJ, 26.1GJ, 24.4GJ,
and 24.6GJ, respecti vely. Notice that the savings exceed 80% for any approach. The savings is achieved by: (i) the
22
To appear in Computer Networks, Special Issue on Virtualized Data Centers. All rights reserved.
0
50
100
150
200
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (a) 40 jobs, 25014 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.197 jobs/hr 0.197 jobs/hr 0.172 jobs/hr 0.197 jobs/hr 0.163 jobs/hr
18.41 hr 18.41 hr 20.75 hr 18.41 hr 51.75 hr
3.4 ms 6.9 ms 213 ms 23 min 40 min
0% 6.2% 8.6% 8.7% 10.2%
cooling energy
computing energy
(a)
0
5
10
15
20
25
30
35
40
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (a) 40 jobs, 25014 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.197 jobs/hr 0.197 jobs/hr 0.172 jobs/hr 0.197 jobs/hr 0.163 jobs/hr
18.41 hr 18.41 hr 20.75 hr 18.41 hr 38.02 hr
3.4 ms 6.9 ms 213 ms 23 min 43 min
0% 11.8% 54.7% 21.8% 60.5%
cooling energy
computing energy
(b)
0
50
100
150
200
250
300
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (b) 120 jobs, 16039 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.580 jobs/hr 0.580 jobs/hr 0.349 jobs/hr 0.580 jobs/hr 0.254 jobs/hr
8.98 hr 8.98 hr 12.17 hr 8.98 hr 48.49 hr
170 ms 186 ms 397 ms 40.8 min 88.6 min
0% 1.7% 4.1% 3.6% 4.7%
cooling energy
computing energy
(c)
0
5
10
15
20
25
30
35
40
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (b) 120 jobs, 16039 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.580 jobs/hr 0.580 jobs/hr 0.349 jobs/hr 0.580 jobs/hr 0.427 jobs/hr
8.98 hr 8.98 hr 12.17 hr 8.98 hr 17.75 hr
171 ms 186 ms 397 ms 42 min 100 min
0% 4.0% 14.6% 14.2% 15.1%
cooling energy
computing energy
(d)
0
50
100
150
200
250
300
350
400
450
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (c) 174 jobs, 45817 core-hours, idle servers on
Throughput
Turnaround time
Alg. runtime
Energy savings
0.892 jobs/hr 0.892 jobs/hr 0.861 jobs/hr 0.892 jobs/hr 0.561 jobs/hr
9.99 hr 9.99 hr 13.39 hr 9.99 hr 65.38 hr
173 ms 196 ms 346 ms 20 min 142 min
0% 2.5% 5.9% 9.4% 12.5%
cooling energy
computing energy
(e)
0
20
40
60
80
100
FCFS-FF FCFS-LRH EDF-LRH FCFS-Xint SCINT
e
n
e
r
g
y

c
o
n
s
u
m
e
d

(
G
J
)
Energy consumption, Scenario (c) 174 jobs, 45817 core-hours, idle servers off
Throughput
Turnaround time
Alg. runtime
Energy savings
0.892 jobs/hr 0.892 jobs/hr 0.861 jobs/hr 0.892 jobs/hr 0.590 jobs/hr
9.99 hr 9.99 hr 13.39 hr 9.99 hr 61.49 hr
173 ms 191 ms 346 ms 21 min 147 min
0.0% 7.5% 17.3% 25.7% 41.4%
cooling energy
computing energy
(f)
Fig. 8. Energy comparison of the simulated schemes for the three scenarios. The plots correspond in respective positions to the plots of Figure 7.
policy used in the data center, which enables job execution as soon as they arrive if the queue is empty and the data
center is lightly loaded. In the idle-on case (Figure 8a), the total energy consumption using SCINT, EDF-LRH,
FCFS-Backll-FF, FCFS-Backll-LRH and XInt placement is 131.3GJ, 133.9GJ, 146.2GJ, 139.3GJ and 139.3GJ,
respectively. However, in the idle-off case, the energy consumption reduces to 10.3,GJ, 11.8GJ, 26.1GJ, 24.4GJ,
and 24.6GJ, respecti vely. Notice that the savings exceed 80% for any approach. The savings is achieved by: (i) the
22
iMPACT Lab (Arizona State U)
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
25
LSI-UPM
WORKLOAD
Resource
Manager
(SLURM)
Execution
Profiling and
Classification
Energy
Optimization
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
Workload:
12 tasks from SPEC CPU INT 2006
Random workload composed by 2000 tasks, divided into
job sets
Random job set arrival time
Servers:
26
Scenario
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
27
Energy profiling
WORKLOAD
Resource
Manager
(SLURM)
Execution
Profiling and
Classification
Energy
Optimization
Energy profiling
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Workload characterization
Jos M.Moya | Madrid (Spain), July 27, 2012 28
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
29
Optimization
WORKLOAD
Resource
Manager
(SLURM)
Execution
Profiling and
Classification
Energy
Optimization
Energy Minimization:
Minimization subjected to constraints
MILP problem (solved with CPLEX)
Static and Dynamic

Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
30
Static optimization
Definition of optimal data center
Given a pool of 100 servers of each kind
1 job set from workload
The optimizer chooses the best selection of servers
Constraints of cost and space
Best solution:
40 Sparc
27 AMD

Savings:
5 a 22% energy
30% time
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
31
Dynamic optimization
Optimal workload allocation
Complete workload (2000 tasks)
Good enough resource allocation in terms of energy (not
the best)
Run-time evaluation and optimization


Energy savings
ranging from 24%
to 47%
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Application-aware scheduling and
resource allocation
First proof-of-concept regarding the use of
heterogeneity to save energy
Automatic solution
Automatic processor selection offers notable energy
savings
Easy implementation in real scenarios
SLURM Resource Manager
Realistic workloads and servers
32
Conclusions
Jos M.Moya | Madrid (Spain), July 27, 2012
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
2. Server-level resource
management
Jos M.Moya | Madrid (Spain), July 27, 2012 33
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Scheduling and resource allocation
policies in MPSoCs
A. Coskun , T. Rosing , K. Whisnant and K. Gross "Static and dynamic temperature-
aware scheduling for multiprocessor SoCs", IEEE Trans. Very Large Scale Integr. Syst.,
vol. 16, no. 9, pp.1127 -1140 2008
Jos M.Moya | Madrid (Spain), July 27, 2012 34
1136 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS, VOL. 16, NO. 9, SEPTEMBER 2008
TABLE VI
SUMMARY OF EXPERIMENTAL RESULTS
Fig. 3. Distribution of thermal hot spots, with DPM (ILP).
A. Static Scheduling Techniques
We next provide an extensive comparison of the ILP based
techniques. We refer to our static approach as Min-Th&Sp.
As discussed in Section III, we implemented the ILP for min-
imizing thermal hot spots (Min-Th), energy balancing (Bal-
En), and energy minimization (Min-En) to compare against
our approach. To the best of our knowledge, this is the rst time
in the literature static MPSoC scheduling techniques are com-
pared extensively to evaluate their thermal behavior.
We rst show average results over all the benchmarks. Fig. 3
demonstrates the percentage of time spent at certain temperature
intervals for the case with DPM. The gure shows that Min-
Th&Sp achieves a higher reduction of hot spots in comparison
to the other energy and temperature-based ILPs. The reason for
this is that, avoiding clustering of workload in neighbor cores
reduces the heating on the die, resulting in lower temperatures.
Fig. 4 shows the distribution of spatial gradients for the av-
erage case with DPM. In this plot, we can observe howMin-Th
increases the percentage of high differentials while reducing
Fig. 4. Distribution of spatial gradients, with DPM (ILP).
hot spots. While Min-Th reduces the high spatial differentials
above 15 C, we observe a substantial increase in the spatial
gradients above 10 C. In contrast, our method achieves lower
and more balanced temperature distribution in the die.
In Fig. 5, we showhow the magnitudes of thermal cycles vary
with the scheduling method. We demonstrate the average per-
centage of time the cores experience temporal variations of cer-
tain magnitudes. As can be observed in Fig. 5, Min-Th&Sp
reduces the thermal cycles of magnitude 20 C and higher sig-
nicantly. The temporal uctuations above 15 C are reduced
in comparison to other static techniques, except for Min-En.
The cycles above 15 C (total) occur 17.3% and 19.2% of the
time for Min-Th&Sp and Min-En, respectively. Our formu-
lation targets reducing the frequency of highest magnitude of
hot spots and temperature variations, therefore, such slight in-
creases with respect to Min-En are possible.
In the plots discussed before and also in Table VI, we ob-
serve that the Min-Th&Sp technique successfully reduces hot
spots as well as the spatial and temporal uctuations. Power
1136 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS, VOL. 16, NO. 9, SEPTEMBER 2008
TABLE VI
SUMMARY OF EXPERIMENTAL RESULTS
Fig. 3. Distribution of thermal hot spots, with DPM (ILP).
A. Static Scheduling Techniques
We next provide an extensive comparison of the ILP based
techniques. We refer to our static approach as Min-Th&Sp.
As discussed in Section III, we implemented the ILP for min-
imizing thermal hot spots (Min-Th), energy balancing (Bal-
En), and energy minimization (Min-En) to compare against
our approach. To the best of our knowledge, this is the rst time
in the literature static MPSoC scheduling techniques are com-
pared extensively to evaluate their thermal behavior.
We rst show average results over all the benchmarks. Fig. 3
demonstrates the percentage of time spent at certain temperature
intervals for the case with DPM. The gure shows that Min-
Th&Sp achieves a higher reduction of hot spots in comparison
to the other energy and temperature-based ILPs. The reason for
this is that, avoiding clustering of workload in neighbor cores
reduces the heating on the die, resulting in lower temperatures.
Fig. 4 shows the distribution of spatial gradients for the av-
erage case with DPM. In this plot, we can observe howMin-Th
increases the percentage of high differentials while reducing
Fig. 4. Distribution of spatial gradients, with DPM (ILP).
hot spots. While Min-Th reduces the high spatial differentials
above 15 C, we observe a substantial increase in the spatial
gradients above 10 C. In contrast, our method achieves lower
and more balanced temperature distribution in the die.
In Fig. 5, we showhow the magnitudes of thermal cycles vary
with the scheduling method. We demonstrate the average per-
centage of time the cores experience temporal variations of cer-
tain magnitudes. As can be observed in Fig. 5, Min-Th&Sp
reduces the thermal cycles of magnitude 20 C and higher sig-
nicantly. The temporal uctuations above 15 C are reduced
in comparison to other static techniques, except for Min-En.
The cycles above 15 C (total) occur 17.3% and 19.2% of the
time for Min-Th&Sp and Min-En, respectively. Our formu-
lation targets reducing the frequency of highest magnitude of
hot spots and temperature variations, therefore, such slight in-
creases with respect to Min-En are possible.
In the plots discussed before and also in Table VI, we ob-
serve that the Min-Th&Sp technique successfully reduces hot
spots as well as the spatial and temporal uctuations. Power
UCSD System Energy Efficiency Lab
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Scheduling and resource allocation
policies in MPSoCs
Energy characterization of applications allows
to define proactive scheduling and resource
allocation policies, minimizing hotspots
Hotspot reduction allows to raise cooling
temperature

+1
o
C means around 7% cooling energy savings
Jos M.Moya | Madrid (Spain), July 27, 2012 35
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
3. Application-aware and
resource-aware virtual machine
Jos M.Moya | Madrid (Spain), July 27, 2012 36
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
JIT compilation in
virtual machines
Virtual machines compile
(JIT compilation) the
applications into native
code for performance
reasons
The optimizer is general-
purpose and focused in
performance
optimization
Jos M.Moya | Madrid (Spain), July 27, 2012 37
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Back-end
JIT compilation for
energy minimization
Application-aware compiler
Energy characterization of applications and
transformations
Application-dependent optimizer
Global view of the data center workload
Energy optimizer
Currently, compilers for high-end processors oriented
to performance optimization
Jos M.Moya | Madrid (Spain), July 27, 2012 38
Front-end
Optimizer
Code generator
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Energy saving potential for
the compiler (MPSoCs)
T. Simunic, G. de Micheli, L. Benini, and M. Hans. Source code optimization and
profiling of energy consumption in embedded systems, International Symposium on
System Synthesis, pages 193 199, Sept. 2000
77% energy reduction in MP3 decoder

FEI, Y., RAVI, S., RAGHUNATHAN, A., AND JHA, N. K. 2004. Energy-optimizing source
code transformations for OS-driven embedded software. In Proceedings of the
International Conference VLSI Design. 261266.
Up to 37,9% (mean 23,8%) energy savings in
multiprocess applications running on Linux

Jos M.Moya | Madrid (Spain), July 27, 2012 39
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
4. Global automatic
management of low-power
modes
Jos M.Moya | Madrid (Spain), July 27, 2012 40
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
DVFS Dynamic Voltage
and Frequency Scaling
As supply voltage decreases, power decreases
quadratically
But delay increases (performance decreases)
only linearly
The maximum frequency also decreases
linearly
Currently, low-power modes, if used, are
activated by inactivity of the server operating
system
Jos M.Moya | Madrid (Spain), July 27, 2012 41
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Room-level DVFS
To minimize energy consumption, changes
between modes should be minimized
There exist optimal algorithms for a known
task set (YDS)
Workload knowledge allows to globally
schedule low-power modes without any
impact in performance
Jos M.Moya | Madrid (Spain), July 27, 2012 42
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Parallelism to save energy
Jos M.Moya | Madrid (Spain), July 27, 2012 43
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
5. Temperature-aware floorplanning of
MPSoCs and many-cores
Jos M.Moya | Madrid (Spain), July 27, 2012 44
Chip Server Rack Room Multi-
room
Sched & alloc 2 1
app
OS/middleware
Compiler/VM 3
architecture 4 4
technology 5
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Temperature-aware
floorplanning
Jos M.Moya | Madrid (Spain), July 27, 2012 45
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Potential energy savings
with floorplanning
Up to 21
o
C reduction of maximum temperature
Mean: -12
o
C in maximum temperature
Better results in the most critical examples
Jos M.Moya | Madrid (Spain), July 27, 2012 46
Temperature Reductions
Average MaxTemp reduction: 12
o
C
Larger temperature reductions for benchmarks
with higher maximum temperature
For many benchmarks, temperature reducions are
larger than 20
o
C
Maximum Temperature
0
20
40
60
80
100
120
140
a
m
m
p
a
p
p
l
u
a
p
s
i
a
r
t
b
z
i
p
2
c
r
a
f
t
y
e
o
n
e
q
u
a
k
e
f
a
c
e
r
e
c
f
m
a
3
d
g
a
p
g
c
c
g
z
i
p
l
u
c
a
s
m
c
f
m
e
s
a
m
g
r
i
d
p
a
r
s
e
r
p
e
r
l
b
m
k
s
w
i
m
t
w
o
l
f
v
o
r
t
e
x
v
p
r
w
u
p
w
i
s
e
a
v
g
original modified
Y. Han, I. Koren, and C. A. Moritz. Temperature Aware Floorplanning. In Proc. of the
Second Workshop on Temperature-Aware Computer Systems, June 2005
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Temperature-aware
floorplanning in 3D chips
Jos M.Moya | Madrid (Spain), July 27, 2012 47
3D chips are getting interest due to:
Scalability: reduces 2D equivalent
area
Performance: shorter wire length
Reliability: less wiring

Drawback:
Huge increment of hotspots
compared with 2D equivalent designs
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Temperature-aware
floorplanning in 3D chips
Jos M.Moya | Madrid (Spain), July 27, 2012 48
Up to 30
o
C reduction per layer in a 3D chip
with 4 layers and 48 cores
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
There is still much more
to be done
Smart Grids
Consume energy when everybody else does not
Decrease energy consumption when everybody
else is consuming
Reducing the electricity bill
Variable electricity rates
Reactive power coefficient
Peak energy demand
Jos M.Moya | Madrid (Spain), July 27, 2012 49
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Conclusions
Reducing PUE is not the same as reducing energy
consumption
IT energy consumption dominates in state-of-the-art data
centers
Application and resources knowledge can be effectively
used to define proactive policies to reduce the total energy
consumption
At different levels
In different scopes
Taking into account cooling and computation at the same time
Proper management of the knowledge of the data center
thermal behavior can reduce reliability issues
Reducing energy consumption is not the same as reducing
the electricity bill
Jos M.Moya | Madrid (Spain), July 27, 2012 50
Ingeniamos el futuro
CAMPUS OF
INTERNATIONAL
EXCELLENCE
Contact
Jos M.Moya | Madrid (Spain), July 27, 2012 51
Jos M. Moya
+34 607 082 892
jm.moya@upm.es

ETSI de Telecomunicacin, B104
Avenida Complutense, 30
Madrid 28040, Spain
Gracias:

También podría gustarte