BLUE GENE / L SUPER COMPUTER

CHAPTER 1
INTRODUCTION

Blue Gene is a supercomputer project designed to produce several super

computers, designed to reach operating speeds in the PFLOPS (Peta floating point
operations per second)range and currently its speed of nearly 500 TFLOPS (Tera
floating point operations per second).This is a jointed funded research partnership
between IBM and LLNL (Lawrence Livermore National laboratory) as a part of
United states department of energy. This project was awarded the national medal of
technology and innovation by US president Barack Obama on September
18,2009 .It is a massively parallel supercomputer using thousands of embedded power
PC processors supporting a large memory space with standard compilers and message
passing environment In the term blue Gene, ’Blue’ means the corporate colour of
IBM and ‘ Gene’ means the intended use of Blue Gene clusters- computational
biology especially protein folding. Blue gene is an IBM Research Project dedicated to
exploring the frontiers in supercomputer in computer Architectures, in the software
required to program and control massively parallel systems and in the use of
computation to advance our understanding of important biological process such as
protein folding. Blue Gene systems occupy the #4 and #5 positions in the TOP500
supercomputing list announced in November 2008.

1
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 2

HISTORY

On September 29, 2004, IBM announced that a Blue Gene/L prototype at IBM
Rochester led by Drew Flaada, had overtaken NEC's Earth Simulator as the fastest
computer in the world, with a speed of 36.01 TFLOPS on the Linpack benchmark,
beating Earth Simulator's 35.86 TFLOPS. This was achieved with an 8-cabinet
system, with each cabinet holding 1,024 compute nodes. Upon doubling this
configuration, the machine reached a speed of 70.72 TFLOPS by November.

On March 24, 2005, the US Department of Energy announced that the Blue
Gene/L installation at LLNL broke its current world speed record, reaching 135.5
TFLOPS. This feat was possible because of doubling the number of cabinets to 32.

On the June 2005 Top500 list, Blue Gene/L installations across several sites
world-wide took 5 out of the 10 top positions, and 16 out of the top 64.

On October 27, 2005, LLNL and IBM announced that Blue Gene/L had once
again broken its current world speed record, reaching 280.6 TFLOPS, upon reaching
its final configuration of 65,536 "Compute Nodes" (i.e., 216 nodes) and an additional
1024 "IO nodes" in 64 air-cooled cabinets.

On June 22, 2006, NNSA and IBM announced that Blue Gene/L had broken
its current world speed record for real world applications, sustaining 207.3 TFLOPS.

Blue Gene/L is also the first supercomputer ever to run over 100 TFLOPS
sustained on a real world application, namely a three-dimensional molecular
dynamics code (ddcMD), simulating solidification (nucleation and growth processes)
of molten metal under high pressure and temperature conditions. This won the 2005
Gordon Bell Prize.

2
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 3
GOALS

The major goals of Blue Gene/L supercomputers are massively parallel

computing and computational biology- protein folding. Parallel computing is form
of computation in which many calculations are carried out simultaneously operating
on the principle that large problems can be divided into smaller one ,which are then
solved concurrently There are several different forms of parallel computing ;bit level,
instruction level ,data and task parallelism. Protein folding is the physical process by
which a polypeptide folds into its characteristics and functional 3D structure from a
random coil which necessary for the proper working of gene.Each protein exists as an
unfolded polypeptide or random coil when translated from a sequence of mRNA to a
linear chain of amino acids. This polypeptide lacks any developed three-dimensional
structure (the left hand side of the neighboring figure). Amino acids interact with each
other to produce a well-defined three-dimensional structure, the folded protein (the
right hand side of the figure), known as the native state. The resulting three-
dimensional structure is determined by the amino acid sequence.For many proteins
the correct three-dimensional structure is essential to function. Failure to fold into the
intended shape usually produces inactive proteins with different properties including
toxic prions. Several neurodegenerative and other diseases are believed to result from
the accumulation of misfolded (incorrectly folded) proteins. Many allergies are caused
by the folding of the proteins, for the immune system does not produce antibodies for
certain protein structures.

3
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

FIG 3.1 Protein after and before folding

CHAPTER 4

MAJOR FEATURES

The Blue Gene/L supercomputer is unique in the following aspects:

1. Trading the speed of processors for lower power consumption.

2. Dual processors per node

3. System-on-a-chip design

4. A large number of nodes (scalable in increments of 1024 up to at least 65,536)

5. Three-dimensional torus interconnect with auxiliary networks for global communications,

I/O, and management

6. Lightweight OS per node for minimum system overhead

4
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 5

SYSTEM OVERVIEW

5.1 ARCHITECTURE

Blue Gene/L is a scalable system in which the maximum number of

compute nodes assigned to a single parallel job is 2^16 = 65,536. Blue Gene/L is
configured as a 64 x32x32 three-dimensional torus of compute nodes. The BG/L
node ASIC includes two standard 700Megahertz PowerPC440
processing cores, each with a PowerPC 440 FP2 core, an enhanced
“Double” 64-bit Floating-Point Unit. The 440 is a standard 32-bit
microprocessor core from IBM’s microelectronics division. Since the
440 CPU core does not implement the necessary hardware to
provide SMP support, the two cores are not L1 cache coherent. A
lockbox is provided to allow coherent processor-to-processor
communication. Each core has a small 2 KB L2 cache which is
controlled by a data pre-fetch engine, a fast SRAM array for
communication between the two cores, an L3 cache directory and 4
MB of associated L3 cache made from embedded DRAM, an
integrated external DDR memory controller, a gigabit Ethernet
adapter, a JTAG interface as well as all the network link cut-through
buffers and control. The L2 and L3 are coherent between the two
cores. In normal operating mode, one CPU/FPU pair is used for
computation while the other is used for messaging.

5
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

The nodes are interconnected through five networks:

a 3D torus network for point-to point messaging between compute
nodes, a global combining/broadcast tree for collective operations
such as MPI.MPI is a language-independent communications protocol used to
program parallel computers. Both point-to-point and collective communication are
supported. MPI "is a message-passing application programmer interface, together
with protocol and semantic specifications for how its features must behave in any
implementation. MPI's goals are high performance, scalability, and portability. MPI
remains the dominant model used in high-performance computing today. _All reduce
over the entire application, a global barrier and interrupt network, a Gigabit Ethernet
to JTAG network for machine control, and another Gigabit Ethernet network for
connection to other systems, such as hosts and file systems.

FIG 5.1.1 Logical layout of Blue Gene / L

Double data rate memory is a type of computer memory built off

integrated circuit technology. DDR memory achieves just under double the data rate

6
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

of single data rate memory, which it superseded by using a double pumping data
transfer technology. This technology doubles the data transfer rate without increasing
the clock speed of the memory.Double data rate memory is a type of computer
memory built off integrated circuit technology. DDR memory achieves just under
double the data rate of single data rate memory, which it superseded by using a double
pumping data transfer technology. This technology doubles the data transfer rate
without increasing the clock speed of the memory.DDR memory has a much higher
data transfer rate than single data rate memory. This higher data bandwidth allows for
a computer system that is more responsive and capable of processing data with lower
latencies. This double data rate also does not increase the clock speed. This ability to
operate at standard clock speeds allows for lower operating temperatures and lower
required voltages for operation. DDR memory acts as a volatile memory type within
computer systems for data manipulation during run-time.However, there are no
hardware impediments to fully utilizing the second processing element for algorithms
that have simple message passing requirements such as those with a large compute to
communication ratio. The PowerPC 440 FP2 core consists of a primary side and a
secondary side, each of which is essentially a complete floating-point unit. Each side
has its own 64-bit by 32 element register file, a double-precision computational data
path and a double-precision storage access data path. A single common interface to
the host PPC 1`440 processor is shared between the sides.The primary side is capable
of executing standard PowerPC floating-point instructions, and acts as an off-the-shelf
PPC 440 FPU [K01]. An enhanced set of instructions include those that are executed
solely on the secondary side, and those that are simultaneously executed on both
sides. While this enhanced set includes SIMD operations, it goes well beyond the
capabilities of traditional SIMD architectures. Here, a single instruction can initiate a
different yet related operation on different data, in each of the two sides. These
operations are performed in lockstep with each other. We have termed these type of
instructions SIMOMD for Single Instruction Multiple Operation Multiple Data. While
Very Long Instruction Word (VLIW) processors can provide similar capability, we
are able to provide it using a short (32 bit) instruction word, avoiding the complexity
and required high bandwidth of long instruction words.) on each side, four floating-
point operations can begin each cycle. To help sustain these operations, a dual

7
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

operand memory access can be initiated in parallel each cycle. The core supports
single element load and store instructions such that any element, in either the primary
or secondary register file, can be individually accessed. This feature is very useful
when data structures in code (and hence in memory) do not pair operands as they are
in the register files..

FIG 5.1.2 Architecture of Blue Gene /L super com

8
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

5.2. PACKAGING

The current design for BG/L system packaging is shown in

Figure The design calls for 2 nodes per compute card, 16 compute cards
per node board, 16 node boards per 512-node midplane of approximate
size 17”x 24”x 34,” and two midplanes in a 1024-node rack.

FIG 5.2.1 BlueGene/L packaging

The BG/L system is a cost/performance design, focused on

fault tolerance, high density, low power and thus achieving low
acquisition and runtime cost. The hardware cost is dominated by the
ASIC and DRAM devices themselves. To manage circuit card costs,
the interconnect was developed from the outside in. After
identifying a system package based on standard 19” racks modified
for high power transverse air cooling, we arranged these racks to

9
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

minimize the longest cable as this is the bandwidth limiter in rack to

rack communication. To reduce connector failure all DRAMsand DC-
DC power supplies are directly soldered, cable connectors have
screwed lockdowns, and all connectors are very reliable pin and
socket multi-contact interfaces.

FIG 5.2.2 System arrangements

The BlueGene/L power and cooling network is an example of a

cost/performance fault tolerant design. The system is air cooled,
designed to operate in standard raised floor machine rooms

10
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

FIG 5.2.3 Blue Gene / L cabinets

Here thirty 100mm diameter high speed, DC “smart fans”
arranged on pluggable fans cards cool a midplane. Fan speed is
monitored and adjusted with feedback from thermal sensors to
maintain constant chip temperature in the face of coherent system-
wide power demands. If a fan slows or stops, the others increase
rotation to maintain ASIC junction temperature and an error
condition is reported to the control host. A damaged fan card can be
replaced with the system running. Fan power is supplied by the
same 208V AC->48V DC N+1 redundant supplies that power the
rack electronics.

11
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

5.3. SOFTWARE SUPPORT

Scalable system software that supports efficient execution of

parallel applications is an integral part of the Blue Gene/L
architecture. A Blue Gene/L application is organized as a collection
of compute processes, each executing on its own compute node
from a partition of the system. I/O nodes provide additional services
for the executing application. The IO nodes , which run the linux
Operating System, provide communication with world through an
Ethernet networks.Finally, a separate and private Ethernet network provides
access to any node for configuration, booting and diagnostics. To allow multiple
programs to run coccurrently , a Blue gene /L system can be
partitioned in to electronically isolated sets of nodes The no: of
nodes in a partition must be a positive integer power of 2 and must
contain at least 2^5=32 nodes. The program is then run on all the
nodes with in the partition and no other program may access
nodes with in the partition while it is in use .Up on completion, the
partition nodes are released for future programs to use.

12
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 6

BLUE GENE PROJECTS

There are four Blue Gene Projects in development: Blue Gene/L,Blue

Gene/C,Blue Gene/P and Blue Gene/Q. The first computer in the Blue Gene
Series,Blue Gene/L, developed through a partnership with Lawrence Livermore
National Laboratory.Originally it had a theoretical peak performance of 360 TFLOPS
sustained .After an upgrade in 2007 the performance increased to 478 TFLOPS
sustained and 596TFLOPS peak. The size Blue Gene/L supercomputer is 2500 square
feets and draws low power (~1 MW).

The term Blue Gene/L sometimes refers to the computer installed at LLNL;
and sometimes refers to the architecture.

Blue Gene/C (now renamed to Cyclops64) is a sister-project to Blue Gene/L.

It is a massively parallel, supercomputer-on-a-chip cellular architecture

Cyclops64 is part of the Blue Gene effort, to produce the next several
generations of supercomputers. The projects were started in response to the
announced construction of the Earth Simulator.

Cyclops64 is a cooperative project between the United States Department of

Energy (which is partially funding the project), the U.S. Department of Defense in

13
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

particular), and academia.This project aims to create the first ‘supercomputer on a

chip’.Each 64bit Cyclops64 Chip (processor) will run at 500 Megahertz and contain
80 processors.The theoritical peak performance of a Cyclops64 is 80GegaFLOPS.

On june 26 ,2007,the second generation of the Blue Gene/Lsupercomputer,Blue

Gene/p developed by IBM. Designed to run continuously at 1 PFLOPS
(petaFLOPS), it can be configured to reach speeds in excess of 3 PFLOPS.
Furthermore, it is at least seven times more energy efficient than any other
supercomputer, accomplished by using many small, low-power chips connected
through five specialized networks. Four 850 MHz PowerPC 450 processors are
integrated on each Blue Gene/P chip. The 1-PFLOPS Blue Gene/P configuration is a
294,912-processor, 72-rack system harnessed to a high-speed, optical network. Blue
Gene/P can be scaled to an 884,736-processor, 216-rack cluster to achieve 3-PFLOPS
performance. A standard Blue Gene/P configuration will house 4,096 processors per
rack.

The last known supercomputer design in the Blue Gene series, Blue Gene/Q
is aimed to reach 20 Petaflops in the 2011 time frame. It will continue to expand
and enhance the Blue Gene/L and /P architectures with higher freq uency at much
improved performance per watt.

It will consist of 98,304 compute nodes comprising 1.6 million processor

cores and 1.6 PB memory in 96 racks covering an area of about 3000 square feet,
drawing 6 megawatts of power.

A Blue Gene/Q system called Mira will be installed at Argonne National

Laboratory early in 2012. It will consist of approximately 50,000 compute nodes
(16 cores per node), 70 PB of disk (470 GB/s I/O bandwidth) and will be water-
cooled.

14
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 7
CHALLENGES

The main challenge of deploying this dual-core mode of operation is that the
L1 caches in each core are not hardware coherent. This forces a software-based
approach to cache coherence and guides our design of a programming model for dual-
core mode. Technology challenges of the future promise to result in a very different
design point for supercomputing. The forces and possible directions that these forces
will push computer architectures will be discussed. The impact on applications will
also be addressed

15
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 8

FUTURE

With Blue Gene/L, IBM introduces a new and extremely powerful high-end
computer architecture for capability computing. In the near future, a leadership
system with a performance of nearly 596 TFlops including 65.536 specially
manufactured PowerPC processors will be installed at the Lawrence Livermore
National Laboratory. What makes Blue Gene/L so attractive are its scalability and its
low requirements for power, cooling and floor space. On the other hand, its enormous
number of processors will certainly pose special problems to programmers. The final
generation of Blue Gene / L supercomputer ,Blue Gene / Q will be established with
1.6 millions of processors and occupy space of 3500 square feet in the 2011 time
frame.

16
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

CHAPTER 9

APPLICATIONS

A wide variety of scientific applications, including many from

DOE’s NNSA laboratories, have been used to assist in the design of
BlueGene/L’s hardware and software. BlueGene/L’s unique features
are especially appealing for ASCI-scale scientific applications To
carry out the scientific research into the mechanisms behind protein
folding announced in December 1999, development of a molecular
simulation application kernel targeted for massively parallel
architectures is underway. For additional information about the
science applications. One of the motivations for the use of massive
computational power in the study of protein folding and dynamics is
to obtain a microscopic view of the thermodynamics and kinetics of
the folding process.The major application areas are military
logistics ,air traffic control, genome processing. Nanotechnology,

17
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

chemical reactors, rational drug design, study about

Biosphere/Geosphere.

CHAPTER 10

CONCLUSION

Blue Gene/L Supercomputer is a revolutionary, low-cost machine that delivers

extraordinary computing power for scientific simulations and programmatic work
located in the TerascaleSimulation Facility at Lawrence Livermore National
Laboratory. . The BlueGene/L targets a machine with 65,536 nodes, with a peak
performance of 360 trillion floating-point operations per second (360 TFLOP/s). The
clock will be 700 MHz, it comes out of embedded systems. BlueGene/L is optimized
to run molecular dynamics applications at extreme speeds to address materials aging
issues confronting the Stockpile Stewardship Program. BlueGene/L is also used to
explore the potential of system-on-a-chip technologies to achieve extreme speed while
minimizing floor space and electrical power consumption.

18
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

REFERENCE

1. Wikipedia.org
2. IBM website
– (www.03.ibm.com/servers/deepcomputing/bluegene.html)
3. www.supercomp.org/sc2002/paperpdfs/pap.pap207.pdf
4. IBM Journal of Research and Development, Vol. 49, No. 2-3.
<http://www.research.ibm.com/journal/rd49-23.html>
– “Overview of the Blue Gene/L system architecture”
– “Packaging the Blue Gene/L supercomputer”
– “Blue Gene/L compute chip: Memory and Ethernet subsystems”
– “Blue Gene/L torus interconnection network”
– “Blue Gene/L programming and operating environment”

19
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

“Design and implementation of message-passing services for the Blue Gene/L

supercomputer

CONTENTS

CHAPTER 1
INTRODUCTION
CHAPTER 2
HISTORY
CHAPTER 3
GOALS
CHAPTER 4
MAJOR FEATURES
CHAPTER 5
SYSTEM OVERVIEW
5.1 ARCHITECTURE

20
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

5.2 PACKAGING
5.3 SOFTWARE SUPPORT
CHAPTER 6
MAJOR BLUE GENE PROJECTS
CHAPTER 7
CHALLENGES
CHAPTER 8
FUTURE
CHAPTER 9
APPLICATION
CHAPTER 10
CONCLUSION
REFERENCE

ABSTRACT

This paper gives an overview of the Blue Gene/L Supercomputer.

This is a jointly funded research partnership between IBM and the Lawrence
Livermore National Laboratory as part of the United States Department of
Energy ASCI Advanced Architecture Research Program. Application
performance and scaling studies have recently been initiated with partners at a
number of academic and government institutions, including the San Diego
Supercomputer Center and the California Institute of Technology. This
massively parallel system of 65,536 nodes is based on a new architecture that
exploits system-on-a-chip technology to deliver target peak processing power
of 360 teraFLOPS (trillion floating-point operations per second). The machine

21
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

is scheduled to be operational in the 2004-2005 time frame, at

price/performance and power consumption/performance targets unobtainable
with conventional architectures.

BLUE GENE / L SUPER COMPUTER

Submitted in partial fulfillment of the award of the degree of
Bachelor of Technology in
Electronics and Communication Engineering of
Cochin University of Science and Technology
MANU ARAVIND

22
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

AUGUST 2010
Department of Electronics and Communication Engineering
COLLEGE OF ENGINEERING, ADOOR

College of Engineering,
Adoor

23
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

DEPARTMENT OF ELECTRONICS AND COMMUNICATION

ENGINEERING

Certificate
Certified that this is a bonafide record of the project entitled

BLUE GENE / L SUPER COMPUTER

done by

MANU ARAVIND

During the year 2010 in partial fulfillment of the requirements for the award of the
degree of bachelor of technology in elect and communication engineering of Cochin
University of Science and Technology, Kerala

ANOOP S RAJI. A
Co-Ordinator Head of the Department

ACKNOWLEDGEMENT

With prayers to GOD for His grace and blessing, for without His unforeseen
guidance, this project would remain only in dreams.

I express my sincere gratitude to the Principal Prof. Jyothi John and head of
department (electronics and communication) Mrs. Raji .A for providing in ambiance
for carrying out the work of my seminar.

24
Dept Of Electronics &Communication, University College
of Engineering Kariavattom
 BLUE GENE / L SUPER COMPUTER

I am profoundly indebted to my guides Mrs

Mr Anoop. S and Ms. Aiswarya (lecturers in electronics and communication) for
their incessant guidance, constructive criticism and encouragement to strive for
excellence.

Last, but not the least, I extend my deepest gratitude to my parents and friends
without whose support this project wouldn’t have become a reality.

25
Dept Of Electronics &Communication, University College
of Engineering Kariavattom