OPENSTACK FOR VSPECIALISTS

MODULE 2: OPENSTACK SERVICE FUNDAMENTALS

EMC CONFIDENTIAL—INTERNAL USE ONLY 1

MODULE 2: SERVICE FUNDAMENTALS

Upon completion of this module, you should be able

to:
• Explain key relationships between OpenStack
projects
• Understand common service design patterns and
components

EMC CONFIDENTIAL—INTERNAL USE ONLY 2

WHAT REALLY HAPPENED?

• Cloud is a term invoked to hide complexity

• Under the UI, requesting a VM invoked a series of services
EMC CONFIDENTIAL—INTERNAL USE ONLY 3
CORE PROJECT RELATIONSHIPS - GUI
Provides
Horizon UI for Heat
Dashboard Orchestration
Provides
UI for

Provides
Neutron UI for Provides
Networkin Provides UI for
UI for Provides Provides
g UI for UI for

Nova Glance Swift

Compute Images Objects

Provides
UI for
Cinder
Volumes

Keystone Ceilometer
Identity Metering

EMC CONFIDENTIAL—INTERNAL USE ONLY 4

CORE PROJECT RELATIONSHIPS - AUTH

Horizon Heat
Dashboard Orchestration

Neutron
Networkin Provides
Provides
g auth for auth for

Nova Glance Swift

Compute Images Objects

Provides Provides
Cinder Provides auth for auth for Provides
Volumes auth for auth for

Provides
auth for
Keystone Ceilometer
Identity Provides Metering
auth for

EMC CONFIDENTIAL—INTERNAL USE ONLY 5

CORE PROJECT RELATIONSHIPS – VM’s

Horizon Heat
Dashboard Orchestration

Neutron
Networkin Provides
g networking for

Stores Stores
Nova Images in Disk files in
Glance Swift
Compute Images Objects

Provides
Volumes for
Cinder
Volumes

Keystone Ceilometer
Identity Metering

EMC CONFIDENTIAL—INTERNAL USE ONLY 6

PROJECT RELATIONSHIPS - METERING

Horizon Heat
Dashboard Orchestration

Neutron
Networkin
g

Nova Glance Swift

Compute Images Objects

Polls data
from
Cinder Polls data
Polls data
Volumes from
from
Polls data
from

Keystone Ceilometer
Identity Polls data Metering
from

EMC CONFIDENTIAL—INTERNAL USE ONLY 7

RELATIONSHIPS - ORCHESTRATION

Heat Horizon
Orchestration Dashboard
Orchestrates

Orchestrates
Neutron Orchestrates
Networkin Orchestrates Orchestrates
g Orchestrates

Nova Glance Swift

Compute Images Objects

Cinder
Volumes Orchestrates

Keystone Ceilometer
Identity Metering

EMC CONFIDENTIAL—INTERNAL USE ONLY 8

SERVICE TOOLKIT FOR CLOUDS
• Mix of services adopted for a cloud deployment depends on use cases

Use case Required Projects Optional Projects

Web accessible Swift Keystone

objects
Basic IaaS Nova, Swift*, Cinder, Horizon, Neutron, Heat,
Glance, Keystone Ironic

Container- Magnum, Heat, Nova, Horizon

Glance, Neutron, Cinder,
focused IaaS Swift*, Keystone
NFV undercloud Nova, Glance, Neutron, Horizon, Heat
Cinder, Swift*, Keystone

EMC CONFIDENTIAL—INTERNAL USE ONLY 9

EXAMPLE: OPTIONS FOR COMPUTE
• Services for physical servers, VM’s, or containers
Physical VM1 VM2 C1 C2
App App App Bin Bin
App sLib App sLib
s s
Bins/Libs Bins/Libs Bins/Libs
Docker Image Docker Image
Container Container
Guest OS Guest OS
Operating System Virtual Machine Virtual Machine Control Groups/Namespaces

Hypervisor Docker Engine

GPU CPU RAM Operating System Operating System

GPU CPU RAM GPU CPU RAM

Flash Storage Flash Storage Flash Storage

Bare-Metal Virtual Machines Containers

Performance Efficiency Scalability

Ironic Nova Magnum

OpenStack Projects
EMC CONFIDENTIAL—INTERNAL USE ONLY 10
OPENSTACK SERVICE FUNDAMENTALS
• Most ‘core’ OpenStack projects are intended to deliver a particular
service (Compute, Networking, Storage, Containers, Identity,
Orchestration, …)
• Most OpenStack services are composed of subsidiary services, in a
pervasive service oriented architecture
• Currently, the project is implemented in 100% Python, and
leverages a range of Python projects and technologies
– Paste, WebOb, Eventlet, Kombu, SQLAlchemy, …

• OpenStack Common Libraries (Oslo) project provides Python

libraries for projects to standardize shared code
– Includes messaging, DB access, API middleware, configuration,
logging, i18n, object versioning, …
– Project introduced to manage and pay down technical debt; adopted
in phases by existing projects, and include technologies that intend to
be more general Python libraries (no oslo- namespace used there)
EMC CONFIDENTIAL—INTERNAL USE ONLY 11
‘GENERIC’ SERVICE ARCHITECTURE
Client
s
• Fight Club Rule: there is no
HTTP Control Node Resource Node generic service architecture
across all OpenStack projects
API • All services have stateless,
Local
Sub RESTful API for client
Resources
requests, including from
WSGI Service
other services
A Local
protocol
SQL • RDBMS used for persisting
Driver
Sub entity state
Service Service
SQL
B Agent • AMQP messaging used for
DB communications and calls
between sub-services
AMQP
AMQP • Sub-services include
schedulers, controllers, and
AMQP Message Bus remote agents
EMC CONFIDENTIAL—INTERNAL USE ONLY 12
SERVICE PLUGIN MODEL
Client
s
• A number of primary
HTTP Control Node Resource Pool OpenStack services are
plugin-driven
API • In this pattern, the service
Sub defines a public API and
exercises control of specific
WSGI Service
back-end resources via
A
plugin(s)
SQL
Sub Resource • Some services, like Cinder,
Service Plugin support simultaneous use of
SQL Manager
B Local multiple back-end plugins
DB protocol
• Plugins have provided means
AMQP for vendors to integrate their
products and compete with
defaults
AMQP Message Bus
EMC CONFIDENTIAL—INTERNAL USE ONLY 13
STATELESS RESTFUL API’S
• The public interface for each OpenStack service is a stateless, RESTful
API
– Accessed at a well known endpoint via set of URIs
– Using standard HTTP verbs GET /v1/0b234fa80b694234ba333e26f0da725/volumes/detail?all_tenants=1 HTTP/1.1
– Returned data typically in JSON format (or XML)

• API’s are implemented as a Python Paste Deployment pipeline of WSGI

middleware (more services!)
– Keystone token validation is common WSGI middleware deployment
– Some services provide other middleware options for special features, such as API rate
limiting in Swift
– Users can also develop and deploy custom WSGI

• WSGI is the Python Web Service Gateway interface (PEP 333)

EMC CONFIDENTIAL—INTERNAL USE ONLY 14

 MULTIPLE API CONSUMPTION OPTIONS
• Traditionally, every service has its own
python- project to provide Python API bindings
Horizon
openstackclient and CLI tool
• Within the big tent, CLI tools releases
can be decoupled from service itself
python- Other
*client tools • OpenStackClient project provides
HTTP unified CLI for core IaaS projects with
consistent command structure
• Other language bindings are also
available
API
• Horizon is also technically a unified
Service API
client, but only partial functionality
WSGI
EMC CONFIDENTIAL—INTERNAL USE ONLY 15
WHAT CLIENTS DO DEVELOPERS USE?

Source: OpenStack User Survey, October 2015

EMC CONFIDENTIAL—INTERNAL USE ONLY 16

SERVICE MESSAGING
• Messaging is used for collaboration
between sub-components of a
service, via Oslo RPC library
• Most deployments use an AMQP Oslo default
compliant broker like RabbitMQ or
Qpid
Red Hat
• Broker resiliency and HA is left as
affinity
an exercise for the deployer
• Oslo RPC also supports potentially
brokerless messaging with ZeroMQ
(ØMQ)

EMC CONFIDENTIAL—INTERNAL USE ONLY 17

COMMON MESSAGING PATTERNS
• OpenStack uses both synchronous request-response RPC messaging
and asynchronous notifications
– rpc.call : return message expected
– rpc.cast : fire and forget

• A service may have one or more dedicated queues in a common

message broker infrastructure
• Messages contain ID references to entities in the service DB, to
provide context for actions
3. Retrieve message

AMQP nova-
nova-api 2. Send message with instance schedule
ID to nova-scheduler r

1.Create entry for Nova 4. Look up instance

new instance SQL record by ID and compare
DB with cluster data
EMC CONFIDENTIAL—INTERNAL USE ONLY 18
PERSISTENT DATA (RDBMS)
• Olso DB library uses Python
SQLAlchemy ORM to manage
database interactions
Default
choice of DB • Any database supported by
SQLAlchemy could work

Drop-in • MySQL and similar DB’s (e.g.

replacement MariaDB) are most common choices
for MySQL
• DB resiliency and HA is left as an
exercise for the deployer
Multi-master • Each services manages its own
clustering
database in a shared RDBMS
for mySQL
(except Ceilometer)
EMC CONFIDENTIAL—INTERNAL USE ONLY 19
WHAT DB DO DEPLOYERS USE?

74% of deployments
use some form of MySQL
or MariaDB in production

Source: OpenStack User Survey, October 2015

EMC CONFIDENTIAL—INTERNAL USE ONLY 20

VM REQUEST WORKFLOW STEP 0
Horizon
User fires up browser Keystone
and inputs username
and password into
Horizon

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 21
VM REQUEST WORKFLOW STEP 1
HTTP
Horizon
Keystone
Horizon sends the user
credentials to the Keystone
auth service for authentication
Nova
Nova-
Glance
API compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 22
VM REQUEST WORKFLOW STEP 2
HTTP
Horizon
Keystone

Keystone authenticates user

against its identity backend
Nova and returns auth-token
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 23
 VM REQUEST WORKFLOW STEP 3
Horizon User works the UI to request a
new VM with a particular Keystone
flavor, image, etc. Horizon
HTTP converts input data into Nova
REST API request, with user’s
auth-token
Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 24
VM REQUEST WORKFLOW STEP 4
Horizon Nova API service checks
request and sends auth-token Keystone
to Keystone for validation HTTP

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 25
VM REQUEST WORKFLOW STEP 5
Keystone validates token and
Horizon sends updated authentication
headers with roles and Keystone
permissions

HTTP
Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 26
VM REQUEST WORKFLOW STEP 6
Horizon nova-api validates request data
and checks for DB conflicts. On Keystone
success, it creates initial
database entry for new instance

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
SQL

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 27
VM REQUEST WORKFLOW STEP 7
nova-api sends rpc.cast
Horizon request to queue for nova-
scheduler, referencing new Keystone
instance ID. Call expecting
reply message with updated
instance entry with host ID

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

AMQP Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 28
VM REQUEST WORKFLOW STEP 8
Horizon
Nova-scheduler picks up
Keystone
request from queue for
scheduling of new instance

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova AMQP
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 29
VM REQUEST WORKFLOW STEP 9
Horizon Nova-scheduler queries Nova
DB for instance and cluster Keystone
data, and applies filters to
select a target host. Instance
entry is updated with host ID.

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
SQL

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 30
VM REQUEST WORKFLOW STEP 10
Horizon
Nova-scheduler sends Keystone
rpc.cast request to
queue for nova-compute
on selected host.

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova AMQP
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 31
VM REQUEST WORKFLOW STEP 11
Horizon
Nova-compute picks up Keystone
request message referencing
ID of new instance

Nova
Nova-
Glance
API compute
Nova- driver
scheduler
Swift
AMQP

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 32
VM REQUEST WORKFLOW STEP 12
Horizon Nova-compute sends rpc.call
request to nova-conductor Keystone
for information on VM to be
instantiated

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
AMQP

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 33
VM REQUEST WORKFLOW STEP 13
Horizon
Keystone

Nova
Nova-
Glance
API Nova-conductor
picks up the request compute
Nova- from the queue
driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
AMQP
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 34
VM REQUEST WORKFLOW STEP 14
Horizon
Keystone

Nova
Nova-
Glance
API Nova-conductor
queries the Nova DB compute
Nova- for all relevant data
driver
scheduler
Swift

SQL Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 35
VM REQUEST WORKFLOW STEP 15
Horizon
Keystone

Nova
Nova-
Glance
API Nova-conductor sends
data back to nova- compute
Nova- compute in reply msg
driver
scheduler
Swift
AMQP

Nova-
Neutron
conductor
Nova
DB KVM
AMQP
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 36
VM REQUEST WORKFLOW STEP 16
Nova-compute makes
Horizon REST API call to Glance
to retrieve VM image,
Keystone
providing auth-token

Nova HTTP Glance

Nova-
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 37
VM REQUEST WORKFLOW STEP 17
Horizon Glance API makes request to
Keystone to validate auth- Keystone
token, and receives updated
authorization headers

HTTP
Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 38
VM REQUEST WORKFLOW STEP 18
Horizon
Glance API returns metadata of
Keystone
image, including download URI

Nova HTTP Glance

Nova-
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 39
VM REQUEST WORKFLOW STEP 20
Using image URI, nova-
Horizon compute can download
image file from Swift
Keystone
proxy to local host

Nova
Nova-
Glance
API
compute HTTP
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 40
VM REQUEST WORKFLOW STEP 21
Nova-compute makes
Horizon REST API call to
Neutron to get VM Keystone
networking and IPs,
providing auth-token

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
HTTP Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 41
VM REQUEST WORKFLOW STEP 22
Horizon
Keystone

Nova
Nova-
Glance
API
compute
HTTP
Nova- driver
scheduler Neutron API makes request
to Keystone to validate auth- Swift
token, and receives updated
authorization headers

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 42
VM REQUEST WORKFLOW STEP 23
Horizon
Keystone

Nova
Nova-
Glance
API compue
compute
Nova- driver
scheduler Neutron performs network
setup and configuration and Swift
returns network info to nova- HTTP
compute

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 43
VM REQUEST WORKFLOW STEP 24
Horizon Nova-compute makes REST
API call to Cinder to attach Keystone
volume(s) to the instance,
passing in auth-token

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
HTTP

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 44
VM REQUEST WORKFLOW STEP 25
Horizon
Keystone

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler Cinder API makes request to HTTP
Keystone to validate auth- Swift
token, and receives updated
authorization headers

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 45
VM REQUEST WORKFLOW STEP 26
Horizon
Keystone

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Cinder returns volume Swift
data to nova-compute HTTP

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 46
VM REQUEST WORKFLOW STEP 27
Horizon
Nova-compute sets up host
Keystone
mount for volume if needed

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 47
VM REQUEST WORKFLOW STEP 28
Horizon Nova-compute sets up
data for hypervisor driver Keystone
and executes request to
instantiate new VM

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
libvirt

Nova-
Neutron
conductor
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 48
VM REQUEST WORKFLOW STEP 29
Horizon Hypervisor creates VM and
nova-compute sends message Keystone
to nova-conductor to update
DB with VM state - UP

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
AMQP

SQL Nova-
Neutron
conductor VM
Nova
DB KVM
AMQP
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 49
 VM REQUEST WORKFLOW STEP 30
Horizon Horizon polls Nova API for VM
status and power state. API Keystone
service queries Nova DB and
HTTP returns updated VM status.

Nova
Nova-
Glance
API
compute
Nova- driver
scheduler
Swift
SQL

Nova-
Neutron
conductor VM
Nova
DB KVM
Cinder
Nova queues

Control Node Compute Node

EMC CONFIDENTIAL—INTERNAL USE ONLY 50
IMPLICATIONS FOR DEPLOYMENTS
• Resiliency of the queue and RDBMS services key for availability
of OpenStack control plane (the cloud brain)
– Queue problems compromise flow of control messages (nerves)
– Unavailable DB means no information on resource state and no
way to update it – changes locked out (memory)
• Distribution providers advancing different solutions for control
plane resiliency
– E.g. built-in solution patterns for RabbitMQ clustering, multi-
master MySQL or Galera cluster
• Swift is the odd project out, using neither queuing or RDBMS in the
way of other projects

EMC CONFIDENTIAL—INTERNAL USE ONLY 51

OPENSTACK LOGGING CONVENTIONS
• Each OpenStack service component can generate its own logs, leading
to a proliferation of log files
• Logging managed by oslo.log library and conventionally written to
– /var/log/<project-codename>

• Logging level is set in service .conf file or log.conf file to filter

messages at given severity level or above
– DEBUG < INFO < AUDIT < WARNING < ERROR < CRITICAL < TRACE

EMC CONFIDENTIAL—INTERNAL USE ONLY 52

OPENSTACK CONFIG CONVENTIONS
• Each OpenStack service has a
main service configuration file in
INI text file format
– # marks comment lines
– Statement lines as simple key=value
pairs grouped in sections
– oslo.config library introducing shared
code for config file handling

• By convention, main service conf

file can be found in
– /etc/<codename>/<codename>.conf
– E.g. /etc/nova/nova.conf
– E.g. /etc/keystone/keystone.conf
– E.g. /etc/cinder/cinder.conf

EMC CONFIDENTIAL—INTERNAL USE ONLY 53

 RUNNING A WEB APP ON OPENSTACK
END USER

Load Balancer
Front End Media Files
Transaction
Web Web Object Meta-data
DB
Serve Serve (Swift)
Serve DB
r r
1 2
OS Images r Vol

Compute Compute Compute Storage

(Nova) (Nova) (Nova) (Cinder)

Compute Compute
(Nova) (Nova)
Images Trove
(Glance) (Controller)
Web
Web Public Private
SVC
SVC Network
Nova Cinder
Neutron (Controller) (Controller)

Keystone

© Copyright 2015 EMC Corporation. All rights reserved. 55