P. 1
Mongo Scaling

Mongo Scaling

|Views: 109|Likes:
Publicado porAlvin John Richards
How to build scalable data persistence tier with MongoDB.
How to build scalable data persistence tier with MongoDB.

More info:

Published by: Alvin John Richards on May 02, 2011
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

05/02/2011

pdf

text

original

Scaling with MongoDB Alvin Richards alvin@10gen.

com

Sunday, May 1, 2011

Stuff we are going to cover super fast!
• Vertical Scaling • Horizontal Scaling with MongoDB

• Schema & Index design • Auto Sharding • Replication

Sunday, May 1, 2011

Scaling

• Operations/sec go up • Storage needs go up • Capacity • IOPs • Complexity goes up • Caching
Sunday, May 1, 2011

How do you scale now?
• Optimization & Tuning • Schema & Index Design • O/S tuning • Hardware configuration
$$$

• Vertical scaling • Hardware is expensive • Hard to scale in cloud
throughput
Sunday, May 1, 2011

MongoDB Scaling - Single Node
read

node_a1

write

Sunday, May 1, 2011

Read scaling - add Replicas
read

node_b1 node_a1

write

Sunday, May 1, 2011

Read scaling - add Replicas
read

node_c1 node_b1 node_a1

write

Sunday, May 1, 2011

Write scaling - Sharding
read
shard1
node_c1 node_b1 node_a1

write

Sunday, May 1, 2011

Write scaling - add Shards
read
shard1
node_c1 node_b1 node_a1

shard2
node_c2 node_b2 node_a2

write

Sunday, May 1, 2011

Write scaling - add Shards
read
shard1
node_c1 node_b1 node_a1

shard2
node_c2 node_b2 node_a2

shard3
node_c3 node_b3 node_a3

write

Sunday, May 1, 2011

Scaling with MongoDB
• Schema & Index Design • Sharding • Replication

Sunday, May 1, 2011

Schema
• Data model effects performance

• Embedding versus Linking

• Partial versus full document writes
• Performance problems can be solved by changing
schema

• Roundtrips to database • Disk seek time • Size of data to read & write

Sunday, May 1, 2011

Indexes
• Index common queries • Do not over index

• Right-balanced indexes keep working set small

•(A) and (A,B) are equivalent, choose one

Sunday, May 1, 2011

Query for {a: 7}
With  Index
[-­‐

∞,  5)

[5,  10)

[10,  

∞)

[-­‐

∞,  5)  buckets

[5,  7)

[7,  9)

[9,  10)

[10,  

∞)  buckets

{...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}  {...}

Without  index  -­‐  Scan
Sunday, May 1, 2011

Indexing Embedded Documents & Multikeys
db.posts.save({    title:        “My  First  blog”,    tags:          [“mongodb”,  “cool”],    comments:  [          {author:  “James”,  ts  :  new  Date()}  ] }); db.posts.ensureIndex({“comments.author”:  1}) db.posts.ensureIndex({“tags”:  1})

Sunday, May 1, 2011

Picking an a Index
find({x:  10,  y:  “foo”})

   scan terminate    index  on  x

   index  on  y

remember

Sunday, May 1, 2011

What is Sharding
• Ad-hoc partitioning • Consistent hashing

• Amazon Dynamo • Google BigTable • Yahoo! PNUTS • MongoDB

• Range based partitioning

Sunday, May 1, 2011

MongoDB Sharding
• Automatic partitioning and management • Range based • Convert to sharded system with no downtime • Almost no functionality lost over single master • Fully consistent

Sunday, May 1, 2011

How MongoDB Sharding works
>  db.runCommand(  {  addshard  :  "shard1"  }  );
>  db.runCommand(        {  shardCollection  :  “mydb.blogs”,            key  :  {  age  :  1}  }  )

-∞   +∞  

•Range keys from -∞ to +∞   •Ranges are stored as “chunks”
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.posts.save(  {age:40}  )

-∞   +∞   -∞   40 41 +∞  

•Data in inserted •Ranges are split into more “chunks”
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.posts.save(  {age:40}  ) >  db.posts.save(  {age:50}  )

-∞   +∞   -∞   40 41 +∞   51 +∞  

41 50

•More Data in inserted •Ranges are split into more“chunks”
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.posts.save(  {age:40}  ) >  db.posts.save(  {age:50}  ) >  db.posts.save(  {age:60}  )

-∞   +∞   -∞   40 41 +∞   51 +∞   61 +∞  

41 50

51 60
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.posts.save(  {age:40}  ) >  db.posts.save(  {age:50}  ) >  db.posts.save(  {age:60}  )

-∞   +∞   -∞   40 41 +∞   51 +∞   61 +∞  

41 50

51 60
Sunday, May 1, 2011

How MongoDB Sharding works

shard1 -∞   40 41 50 51 60 61 +∞  
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.runCommand(  {  addshard  :  "shard2"  }  );

-∞   40 41 50 51 60 61 +∞  
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.runCommand(  {  addshard  :  "shard2"  }  );

shard1 -∞   40 41 50 51 60 61 +∞  
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.runCommand(  {  addshard  :  "shard2"  }  );

shard1 -∞   40

shard2 41 50

51 60 61 +∞  
Sunday, May 1, 2011

How MongoDB Sharding works
>  db.runCommand(  {  addshard  :  "shard2"  }  ); >  db.runCommand(  {  addshard  :  "shard3"  }  );

shard1 -∞   40

shard2 41 50

shard3

51 60 61 +∞  
Sunday, May 1, 2011

Sharding Key Examples
{      server  :  "ny153.example.com"  ,      application  :  "apache"  ,      time  :  "2011-­‐01-­‐02T21:21:56.249Z"  ,      level  :  "ERROR"  ,      msg  :  "something  is  broken" }

•Good  :  {server:1} • All data for one server is in a single chunk • Chunk cannot be split any smaller •Better  :  {server:1,time:1} ‣  Chunk  can  be  split  by  millisecond
Sunday, May 1, 2011

Sharding Key Examples
{      server  :  "ny153.example.com"  ,      application  :  "apache"  ,      time  :  "2011-­‐01-­‐02T21:21:56.249Z"  ,      level  :  "ERROR"  ,      msg  :  "something  is  broken" }

•Good  :  {time  :  1} ‣ Time is an increasing number ‣ All data will be first written to a single shard ‣ Data balanced to other shards later •Better  :  {server:1,application:1,time:1} ‣More key values to enable writes to all shards
Sunday, May 1, 2011

Sharding Features
• Shard data without no downtime • Automatic balancing as data is written • Commands routed (switched) to correct node

• Inserts - must have the Shard Key • Updates - must have the Shard Key • Queries • Indexed Queries

• With Shard Key - routed to nodes • Without Shard Key - scatter gather • With Shard Key - routed in order • Without Shard Key - distributed sort merge

Sunday, May 1, 2011

MongoDB Replication
• MongoDB replication like MySQL replication

•Asynchronous master/slave
• Variations:

•Master / slave •Replica Sets

Sunday, May 1, 2011

Replica Set features
• A cluster of N servers • Any (one) node can be primary • Consensus election of primary • Automatic failover • Automatic recovery • All writes to primary • Reads can be to primary (default) or a secondary

Sunday, May 1, 2011

How MongoDB Replication works
Member  1 Member  3

Member  2

•Set is made up of 2 or more nodes
Sunday, May 1, 2011

How MongoDB Replication works
Member  1 Member  3

Member  2 PRIMARY

•Election establishes the PRIMARY •Data replication from PRIMARY to
SECONDARY
Sunday, May 1, 2011

How MongoDB Replication works
Member  1 negotiate   new  master Member  3

Member  2 DOWN

•PRIMARY may fail •Automatic election of new PRIMARY
Sunday, May 1, 2011

How MongoDB Replication works
Member  1

Member  3 PRIMARY

Member  2 DOWN

•New PRIMARY elected •Replication Set re-established
Sunday, May 1, 2011

How MongoDB Replication works
Member  1

Member  3 PRIMARY

RECOVERING

Member  2

•Automatic recovery
Sunday, May 1, 2011

How MongoDB Replication works
Member  1

Member  3 PRIMARY

Member  2

•Replication Set re-established
Sunday, May 1, 2011

Using Replicas

slaveOk() - driver will send read requests to Secondaries - driver will always send writes to Primary Java examples -­‐  DB.slaveOk() -­‐  Collection.slaveOk()

-­‐  find(q).addOption(Bytes.QUERYOPTION_SLAVEOK);

Sunday, May 1, 2011

Creating a Replica Set
>  cfg  =  {        _id  :  "acme_a",        members  :  [            {  _id  :  0,  host  :  "sf1.acme.com"  },            {  _id  :  1,  host  :  "sf2.acme.com"  },            {  _id  :  2,  host  :  "sf3.acme.com"  }  ]  } >  use  admin >  db.runCommand(  {  replSetInitiate  :  cfg  }  )

Sunday, May 1, 2011

Replica Set Member Types
• Normal {priority:1} • Passive {priority:0} • Arbiters

• Cannot be elected as PRIMARY • Can vote in an election • Do not hold any data

• Hidden {hidden:True}

Sunday, May 1, 2011

Replication features
• Reads from Primary are always consistent • Reads from Secondaries are eventually
consistent

• Automatic failover if a Primary fails • Automatic recovery when a node joins the set
Sunday, May 1, 2011

Summary
• Schema & Index design • Simplest way to scale •Sharding • Automatically scale writes •Replication •Automatically scale reads
Sunday, May 1, 2011

download at mongodb.org

We’re Hiring !
alvin@10gen.com
conferences,  appearances,  and  meetups
http://www.10gen.com/events

http://bit.ly/mongoU  

Facebook                    |                  Twitter                  |                  LinkedIn
@mongodb

http://linkd.in/joinmongo

Sunday, May 1, 2011

You're Reading a Free Preview

Descarga
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->