MongoDB for DBA's 1/7.

Introduction
Chapter 1. Introduction to MongoDB, key concepts

Concepts

What were the big differences in hardware over the last few decades that MongoDB attempted to
address?

Parallelism of cores Parallelism of servers Quantum computers Faster clock

speeds More memory

Scaling

Q: When scaling out horizontally (adding more servers to contain your data), what are problems
that arise as you go from, say, 1 commodity server to a few dozen?

Any one server is likelier to fail per given unit of time. The servers must communicate

with one another eating up network bandwidth The need for redundancy increases as the

likelihood of some failure in the system per unit of time increases. Hardware cost per server
is likely to increase.

SQL and Complex Transactions

 What causes significant problems for SQL when you attempt to scale horizontally (to multiple
servers)?

Joins Queries Transactions Inserts Indexes

Documents Overview

What are some advantages of representing our data using a JSON-like format?

JSON presents a flexible and concise framework for specifying queries as well as storing

records. JSON syntax is similar to that of common data structures used in many

programming languages and is, therefore, familiar to developers. JSON is language

independent. JSON is optimized for use with JavaScript.

JSON Types

How many data types are there in JSON?

1 (just strings) 2 4 6

1. String
2. Integer
3. Boolean
4. Null
5. Array
6. Object or documents (subdocuments)

JSON Syntax
 What is the corresponding JSON for the following XML document?
<person>
<name>John</name>
<age>25</age>
<address>
<city>New York</city>
<postalCode>10021</postalCode>
</address>
<phones>
<phone type="home">212-555-1234</phone>
<phone type="mobile">646-555-1234</phone>
</phones>
</person>

{"name":"John",
"age":25,
"address":{"city":"New York","postalCode":"10021"},
"phones":[
{"phone":"212-555-1234","type":"home"},
{"phone":"646-555-1234","type":"mobile"}
]
}

JSON Syntax 2

For the following XML, Is the corresponding JSON example legal json?
<things>
<hat>one</hat>
<coat>z</coat>
<hat>two</hat>
</things>
{
"hat" : "one",
"coat" : "z",
"hat" : "two"
}

Yes No Maybe
Binary JSON

Why do we represent our data as BSON rather than JSON in the system?

Fast machine scanability. Human readability. Stronger typing (and more types)
than JSON

BSON and applications

For a typical client (a python client, for example) that is receiving the results of a query in BSON,
would we convert from BSON to JSON to the client's native data structures (for example, nested
dictionaries and lists in Python), or would we convert from BSON straight to those native data
structures?

BSON -> Native data structures BSON -> JSON -> Native data structures

Dynamic Schema

True or False: MongoDB is schemaless because a schema isn't very important in MongoDB

True False

What is the MongoDB shell?

 Q: By default, which database does the mongo shell connect to?

test localhost mongo help default

Cursors Introduction

In order to query a collection in the mongo shell, we can type which of the following?

db.collection.find() db.find() db.collection.get() db.collection.select(*) db.c

ollection.get(true)

Query Language: Basic concepts

Mongo querys are represented as BSON (JSON)

You have a collection where every document has the same fields, and you want to look at the
value of the “_id”, "name", and “email” fields in order to see their format. Furthermore, you want to
eliminate all other fields from the query results. What query might you write?

db.collection.find({ name : 1, email: 1 }, {} ) db.collection.find( { } , { name: 1 , email: 1 } )

db.collection.find( { name: 1 } , { email : 1 } ) db.collection.find( {name: 1 , email: 1 } )

Query Language: Projection

You want to query the “people” collection, you want the results of the query to include only
documents where age is 50, and you want to look at all fields except “email”. What query should
 you write?

db.people.find( { email : 0, age : 50 } , { } ) db.people.find( { email : 0 }, { age : 50 } )

db.people.find( { age : 50 } , {email: 0 } ) db.people.find( { } , { email : 0 , age : 50 } )

db.people.find( { age : 50 }, {email : 1} )

Query Language: Advantages of a Dynamic Schema

If you want to add a new key: value pair to the documents in the “shapes” collection, what
methods could you use?

db.shapes.alterCollection() db.shapes.update() db.shapes.resizeObject() db.s

hapes.find() db.shapes.save()

Shell: Queries

We have sample documents in our products collection such as:

{
name: "AC1 Case Green",
color: "green",
price: 12.00,
for: "ac1",
type: ["accessory", "case"],
available: true
}
How would we query in the shell for all products that are cases for an ac9 phone? That is,
where type contains the value "case" and for equals "ac9"?

db.products.find({"for":"ac9","type":"case"})

Sorting
 If you want to run a query on the collection, “books,” and sort ASCIIbetically by title on the query,
which of the following will work?

db.books.find( { } , { sort : { title : 1 } } ) db.books.find( { } , { title : "$sort" } )

db.books.find( { } ).sort( { title : -1 } ) db.books.sort( { title : 1 } ).find( { } )

db.books.find().sort( { title : 1 } )

Query Language: Cursors

Recall the documents in the scores collection:

{
"_id" : ObjectId("50844162cb4cf4564b4694f8"),
"student" : 0,
"type" : "exam",
"score" : 75
}
Write a query that retrieves documents of type "exam", sorted by score in descending order,
skipping the first 50 and showing only the next 20.

db.scores.find({"type":"exam"}).sort({"score":-1}).skip(50).limit(20)

MongoDB for DBA's 2/7. Crud. Homeworks

Homework 2.1
We will use the pcat.products collection from week 1. So start
with that; if not already set up, import it:

mongoimport --db pcat -c products < products.json

You can find products.json from the Download Handouts link.

In the shell, if you type:

db.products.count()

should return 11.

Next, download homework2.js from the Download Handouts
link. Run the shell with this script:
mongo --shell pcat homework2.js

First, make a mini-backup of the collection before we start

modifying it. In the shell:

b = db.products_bak; db.products.find().forEach( function(o){ b.insert(o) } )

// check it worked:

b.count()

// should print 11

If you have any issues you can restore from "products_bak"; or,
you can re-import with mongoimport. (You would perhaps need
in that situation to empty the collection first or drop it; see the
--drop option on mongoimport --help.) At the shell ">" prompt
type:

homework.a()

What is the output? (The above will check that products_bak is

populated.)

3.05

Homework 2.2
Add a new product to the products collection of this form:

"_id" : "ac9",

"name" : "AC9 Phone",

"brand" : "ACME",

"type" : "phone",

"price" : 333,

"warranty_years" : 0.25,
 "available" : true

Note: in general because of the automatic line continuation in

the shell, you can cut/paste in the above and shouldn't have to
type it all out. Just enclose it in the proper statement(s) to get
it added.

> myobj = { "_id" : "ac9", "name" : "AC9 Phone", "brand" : "ACME", "type" : "phone",
"price" : 333, "warranty_years" : 0.25, "available" : true }

> db.products.insert(myobj)

WriteResult({ "nInserted" : 1 })

Next, load into a shell variable the object corresponding to

_id : ObjectId("507d95d5719dbef170f15c00")

bb = db.products.find({"_id": ObjectId("507d95d5719dbef170f15c00")})

 Then change term_years to 3 for that document. (And

save that to the database.)
db.products.update({"_id":ObjectId("507d95d5719dbef170f15c00")},
{"$set":{"term_years":3}}
)

 Then change over_rate for sms in limits to 0.01 from

0. Save that too.

db.products.update({"_id":ObjectId("507d95d5719dbef170f15c00")},
{"$set":{"limits.sms.over_rate":0.01}}
)

At the shell prompt type:

homework.b()

What is the output?

0.050.019031
Homework 2.3
How many products have a voice limit? (That is, have a voice
field present in the limits array.)

db.products.find({ "limits" : { "$exists" : "voice" }}).count()

Homework 2.4
Create an index on the field for. You might want to first run the
following to get some context on what is present in that field in
the documents of our collection:
db.products.find({},{for:1})

db.products.ensureIndex({"for":1},{name:"for"})
db.products.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "pcat.products"
},
{
"v" : 1,
"key" : {
"for" : 1
},
"name" : "for",
"ns" : "pcat.products"
}

After creating the index, query products that work with an "ac3"
phone; that is, "ac3" is present in the product's "for" field.

 Q1: How many are there?

 db.products.find({"for":"ac3"}).count()
  Q2: Run an explain plan on the above query. How many
records were scanned?

 db.products.find({"for":"ac3"}).explain()
 {
"cursor" : "BtreeCursor for",
"isMultiKey" : true,
"n" : 4,
"nscannedObjects" : 4,
"nscanned" : 4,
"nscannedObjectsAllPlans" : 4,
"nscannedAllPlans" : 4,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {
"for" : [
[
"ac3",
"ac3"
]
]
},
"server" : "SERVER:27017",
"filterSet" : false

 Q3: Was an index used?

 "cursor" : "BtreeCursor for",

Q1: 0 Q1: 1 Q1: 3

Q1 : 4 Q2 : 1 Q2 : 4 Q2 : 5 Q2 : 12
Q3 : No Q3 : Yes
Homework 2.5
Referring back to 2.4 above, update those products (products that
work with an "ac3" phone) and add 2 to the "price" of each of
those items.

db.products.find({"for":"ac3"})

db.products.update( {"for":"ac3"}, { "$inc" : { "price": 2}}, {"multi" : 1})

89.5954.5
MongoDB for DBA's 3/7. Performance. Homeworks
Homework 3.1
Start a mongod server instance (if you still have a replica set,
that would work too).
Next, download the handout and run:

mongo --shell localhost/performance performance.js

> homework.init()
>

Build an index on the "active" and "tstamp" fields. You can

verify that you've done your job with
db.sensor_readings.getIndexes()

db.sensor_readings.ensureIndex({ "active" :1, "tstamp" :1 },

{"name" :"act_tstamp.idx" })
When you are done, run:
homework.a()

and enter the numeric result below (no spaces).

Note: if you would like to try different indexes, you can
use db.sensor_readings.dropIndexes() to drop your old index before
creating a new one. (For this problem you will only need one
index beyond the _id index which is present by default.)
Result: 3

Homework 3.2
In a mongo shell run homework.b() . This will run in an infinite
loop printing some output as it runs various statements
against the server.
We'll now imagine that on this system a user has complained
of slowness and we suspect there is a slow operation running.
Find the slow operation and terminate it.
In order to do this, you'll want to open a second window (or
tab) and there, run a second instance of the mongo shell, with
something like:
$ mongo --shell localhost/performance performance.js

Keep the other shell with homework.b() going while this is

happening. Once you have eliminated the slow operation, run
(on your second tab):
> db.currentOp()
{
"inprog" : [
{
"opid" : 101,
"active" : true,
"secs_running" : 168,
"microsecs_running" : NumberLong(168262180),
"op" : "update",
"ns" : "performance.sensor_readings",
"query" : {
"$where" : "function(){sleep(500);return false;}"
},
"client" : "127.0.0.1:36365",
"desc" : "conn5",
"threadId" : "0xaf8f7b40",
"connectionId" : 5,
"locks" : {
"^" : "w",
"^performance" : "W"
},
"waitingForLock" : false,
"numYields" : 46,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(330834998)
},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(1771)
}
}
}
]

> db.killOp(101)

{ "info" : "attempting to kill op" }

> db.currentOp()
{ "inprog" : [ ] }
 homework.c()

and enter the output below. Once you have it right and are
ready to move on, ctrl-c (terminate) the shell that is still
running the homework.b() function.

> homework.c()

12
Homework 3.3
Compact the performance.sensor_readings collection,
with paddingFactor set to 1.0. Then
run homework.d() and enter the result below.
Note: If you happen to be running a replica
set, just compact on the primary and run
homework.d() on the primary. You may need
to use the force:true option to run compact
on a primary. You may also want to look
at paddingFactor in the docs. Prior to 2.6,
you would have been able to do this problem
without paddingFactor, but now that
powerOfTwoSizes is on by default, you'll
need to use it in order to fully compact the
collection.

> db.runCommand({"compact" :
"sensor_readings", "paddingFactor" : 1 })
{ "ok" : 1 }
> homework.d()

21

MongoDB for DBA's 4/7. Replication. Homeworks

Homework 4.1
In this chapter’s homework we will create a replica set and
add some data to it.
1. Unpack replication.js from the Download Handout zip file.
2. We will create a three member replica set. Pick a root
working directory to work in. Go to that directory in a console
window.
Given we will have three members in the set, and three
mongod processes, create three data directories:
mkdir 1

mkdir 2

mkdir 3

3. We will now start a single mongod as a standalone server.

Given we will have three mongod processes on our single test
server, we will explicitly specify the port numbers (this
wouldn’t be necessary if we had three real machines or three
virtual machines). We’ll also use the --smallfiles parameter and
--oplogSize so the files are small given we have a lot of server
processes running on our test PC.
$ # starting as a standalone server for problem 1:

$ mongod --dbpath 1 --port 27001 --smallfiles --oplogSize 50

Note: for all mongod startups in the homework this chapter,

you can optionally use --logPath, --logappend, and --fork. Or,
since this is just an exercise on a local PC, you could simply
have a separate terminal window for all and forgo those
settings. Run “mongod --help” for more info on those.
mongod --dbpath 1 --port 27001 --smallfiles --logpath log.1 --logappend
--fork --oplogSize 50
mongod --dbpath 2 --port 27002 --smallfiles --logpath log.2 --logappend
--fork --oplogSize 50
mongod --dbpath 3 --port 27003 --smallfiles --logpath log.3 --logappend
--fork --oplogSize 50

To know pid mongo processes:

ps -Aef | grep mongod

To kill mongod processes

kill pid

4. In a separate terminal window (cmd.exe on Windows), run

the mongo shell with the replication.js file:
mongo --port 27001 --shell replication.js

Then run in the shell:

> homework.init()

This will load a small amount of test data into the database.
Now run:
> homework.a()

and enter the result. This will simply confirm all the above
happened ok.

result: 5001

Homework 4.2
Now convert the mongod instance (the one in the problem 4.1
above, which uses “--dbpath 1”) to a single server replica set.
To do this, you’ll need to stop the mongod (NOT the mongo
shell instance) and restart it with “--replSet” on its command
line. Give the set any name you like.
ps -Aef | grep mongod

mongodb 1009 1 0 09:25 ? 00:00:05 /usr/bin/mongod --config /etc/mongod.conf

user 3260 2009 0 09:42 ? 00:00:02 mongod --dbpath 1 --port 27001 --smallfiles --logpath
log.1 --logappend --fork --oplogSize 50
user 3272 2009 0 09:43 ? 00:00:01 mongod --dbpath 2 --port 27002 --smallfiles --logpath
log.2 --logappend --fork --oplogSize 50
user 3284 2009 0 09:43 ? 00:00:01 mongod --dbpath 3 --port 27003 --smallfiles --logpath
log.3 --logappend --fork --oplogSize 50

user 3360 3221 0 09:48 pts/0 00:00:00 grep --color=auto mongod

To kill mongod processes

kill 3260

mongod --replSet abc --dbpath 1 --port 27001 --smallfiles --logpath log.1

--logappend --fork --oplogSize 50
Then go to the mongo shell. Once there, run
> rs.initiate()

> rs.initiate()
{
"info2" : "no configuration explicitly specified -- making one",
"me" : "SERVER:27001",
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
rs.status()
{
"set" : "abc",
"date" : ISODate("2015-01-30T08:55:02Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "SERVER:27001",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 364,
"optime" : Timestamp(1422607970, 1),
"optimeDate" : ISODate("2015-01-30T08:52:50Z"),
"electionTime" : Timestamp(1422607970, 2),
"electionDate" : ISODate("2015-01-30T08:52:50Z"),
"self" : true
}
],
"ok" : 1
}
abc:PRIMARY> rs.conf()
{
"_id" : "abc",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "SERVER:27001"
}
]
}
When you first ran homework.init(), we loaded some data into
the mongod. You should see it in the replication database. You
can confirm with:
> use replication

> db.foo.find()

Once done with that, run

> homework.b()

in the mongo shell and enter that result below.

result: 5002

Homework 4.3
Now add two more members to the set. Use the 2/ and 3/
directories we created in homework 4.1. Run those two
mongod’s on ports 27002 and 27003 respectively (the exact
numbers could be different).
Remember to use the same replica set name as you used for
the first member.
kill 3272 3484
mongod --replSet abc --dbpath 2 --port 27002 --smallfiles --logpath log.2 --logappend --fork
--oplogSize 50
mongod --replSet abc --dbpath 3 --port 27003 --smallfiles --logpath log.3 --logappend --fork
--oplogSize 50
You will need to add these two new members to your replica
set, which will initially have only one member. In the shell
running on the first member, you can see your replica set
status with
> rs.status()
Initially it will have just that first member. Connecting to the
other members will involve using
rs.add()

. For example,

> rs.add("localhost:27002")

You'll know it's added when you see an

{ "ok" : 1 }

document.
Your machine may or may not be OK with 'localhost'. If it isn't,
try using the name in the "members.name" field in the
document you get by calling rs.status() (but remember to use
the correct port!).
abc:PRIMARY> rs.conf()
{
"_id" : "abc",
"version" : 3,
"members" : [
{
"_id" : 0,
"host" : "SERVER:27001"
},
{
"_id" : 1,
"host" : "SERVER:27002"
},
{
"_id" : 2,
"host" : "SERVER:27003"
}
]
}
Once a secondary has spun up, you can connect to it with a
new mongo shell instance.
mongo --port 27002
Use
rs.slaveOk()

to let the shell know you're OK with (potentially) stale data,

and run some queries. You can also insert data on your
primary and then read it out on your secondary. Once the
servers have sync'd with the primary and are caught up, run
(on your primary):

> homework.c()

and enter the result below.

result: 5

Homework 4.4
We will now remove the first member (@ port 27001) from the
set.
As a first step to doing this we will shut it down. (Given the
rest of the set can maintain a majority, we can still do a
majority reconfiguration if it is down.)
We could simply terminate its mongod process, but if we use
the replSetStepDown command, the failover may be faster.
That is a good practice, though not essential. Connect to
member 1 (port 27001) in the shell and run:
> rs.stepDown()

Then cleanly terminate the mongod process for member 1.

abc:SECONDARY> rs.status()
{
"set" : "abc",
"date" : ISODate("2015-01-30T09:42:11Z"),
"myState" : 2,
"syncingTo" : "SERVER:27003",
"members" : [
{
"_id" : 0,
"name" : "SERVER:27001",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 621,
"optime" : Timestamp(1422610708, 149),
"optimeDate" : ISODate("2015-01-30T09:38:28Z"),
"infoMessage" : "syncing to: SERVER:27003",
"self" : true
},
{
"_id" : 1,
"name" : "SERVER:27002",
"health" : 1,
 "state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 398,
"optime" : Timestamp(1422610708, 149),
"optimeDate" : ISODate("2015-01-30T09:38:28Z"),
"lastHeartbeat" : ISODate("2015-01-30T09:42:10Z"),
"lastHeartbeatRecv" : ISODate("2015-01-30T09:42:11Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "syncing to: SERVER:27001",
"syncingTo" : "SERVER:27001"
},
{
"_id" : 2,
"name" : "SERVER:27003",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 394,
"optime" : Timestamp(1422610708, 149),
"optimeDate" : ISODate("2015-01-30T09:38:28Z"),
"lastHeartbeat" : ISODate("2015-01-30T09:42:11Z"),
"lastHeartbeatRecv" : ISODate("2015-01-30T09:42:11Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1422610890, 1),
"electionDate" : ISODate("2015-01-30T09:41:30Z")
}
],
"ok" : 1
}
Next, go to the new primary of the set. You will probably need
to connect with the mongo shell, which you'll want to run with
'--shell replication.js' since we'll be getting the homework
solution from there.
mongo --port 27003 --shell replication.js
Once you are connected, run rs.status() to check that things
are as you expect. Then reconfigure to remove member 1.
Tip: You can either use rs.reconfig() with your new
configuration that does not contain the first member, or
rs.remove(), specifying the host:port of the server you wish to
remove as a string for the input.
rs.remove("SERVER:27001")
When done, run
> homework.d()

and enter the result.

Tip: If you ran the shell without replication.js on the command
line, restart the shell with it.
 result: 6

Homework 4.5
Note our replica set now has an even number of members, and
that is not a best practice. However, to keep the homework
from getting too long we’ll leave it at that for now, and instead
do one more exercise below involving the oplog.
To get the right answer on this problem, you must perform the
homework questions in order. Otherwise, your oplog may look
different than we expect.
Go to the secondary in the replica set. The shell should say
SECONDARY at the prompt if you've done everything correctly.
mongo --port 27002 --shell replication.js
Switch to the local database and then look at the oplog:
> db.oplog.rs.find()

If you get a blank result, you are not on the right database.
Note: as the local database doesn’t replicate, it will let you
query it without entering “rs.slaveOk()” first.
Next look at the stats on the oplog to get a feel for its size:
> db.oplog.rs.stats()

What result does this expression give when evaluated?

db.oplog.rs.find().sort({$natural:1}).limit(1).next().o.msg[0]

result: R
MongoDB for DBA's 5/7. Replication Part 2.
Homeworks
Homework 5.1
Set up a replica set that includes an arbiter.
To demonstrate that you have done this, what is the text in the
"state" field for the arbiter when you run rs.status()?
Status : 7
Homework 5.2
You have a replica set with two members, having decided that
a total of two copies of your data is sufficient.
You are concerned about robustness, however.
Which of the following are options that will allow you to ensure
that failover can occur if the first server goes down?
Check all that apply.

Add an arbiter. Increase the number of votes for the first

replica set member from one to two. Increase the priority of
the first server from one to two. Add another data bearing
node.

Homework 5.3
At the beginning of the section, "W Timeout and Capacity
Planning", Dwight mentioned something that hasn't yet been
touched upon in this class: Batch Inserts.
He mentions that you can find information on them in the
docs, but he hasn't used them in the shell during this course.
Your job is to look this functionality up in the documentation,
and use it for the following problem:
Please insert into the m102 collection, the following 3
documents, using only one shell query (please use double
quotes around each key in each document in your answer):
 { "a" : 1 }

 { "b" : 2 }

 { "c" : 3 }
Hints: This is not the same as the Bulk() operations, which
were discussed earlier in this course. Also, this does not
involve semicolons, so don't put any into the text box. You
probably want to test this on your machine, and then copy and
paste the insert query in the box below. Do not include the >
sign from the beginning of the shell.

Result: db.m102.insert([{"a":1},{"b":2},{"c":3}])

Homework 5.4
You would like to create a replica set that is robust to data
center failure.
You only have two data centers available. Which
arrangement(s) of servers will allow you to be stay up (as in,
still able to elect a primary) in the event of a failure of either
data center (but not both at once)? Check all that apply.

All 3 servers in data center 1. 2 servers in data center 1,

one server in data center 2. None of the above.

Homework 5.5
Consider the following scenario: You have a two member
replica set, a primary, and a secondary.
The data center with the primary goes down, and is expected
to remain down for the foreseeable future. Your secondary is
now the only copy of your data, and it is not accepting writes.
You want to reconfigure your replica set config to exclude the
primary, and allow your secondary to be elected, but you run
into trouble. Find out the optional parameter that you'll need,
and input it into the box below for your rs.reconfigure(new_cfg,
OPTIONAL PARAMETER).
Hint: You may want to use this documentation page to solve
this problem.
Your answer should be of the form { key : value } (including
brackets).
MongoDB for DBA's 6/7. Scalability. Homeworks
Homework 6.1
For this week's homework we will start with a standalone
MongoDB database, turn it into a sharded cluster with two
shards, and shard one of the collections. We will create a
"dev" environment on our local box: no replica sets, and only
one config server. In production you would almost always use
three config servers and replica sets as part of a sharded
cluster. In the final of the course we'll set up a larger cluster
with replica sets and three config servers.
Download week6.js from Download Handout link.
Start an initially empty mongod database instance.
Connect to it with the shell and week6.js loaded:

mongo --shell localhost/week6 week6.js

Run homework.init() . It will take some time to run as it inserts

quite a few documents. When it is done run

db.trades.stats()

to check the status of the collection.

At this point we have a single mongod and would like to
transform it into a sharded cluster with one shard. (We'll use
this node’s existing week6.trades data in the cluster.)
Stop the mongod process. Now, restart the mongod process
adding the option --shardsvr . If you started mongod with a --
dbpath option, specify that as well.
mongod --shardsvr …

sudo mongod --shardsvr --dbpath /var/lib/mongodb --logpath

/var/log/mongodb/mongod00.log --port 27018 --fork --logappend
--smallfiles --oplogSize 50

Note that with --shardsvr specified the default port for mongod
becomes 27018.
Start a mongo config server:
mongod --configsvr …

sudo mongod --configsvr --dbpath /var/lib/mongodb/configdb --port 27019

--fork --logpath /var/log/mongodb/mongod00.log.cfg0 --logappend

(Note with --configsvr specified the default port for listening

becomes 27019 and the default data directory /data/configdb.
Wherever your data directory is, it is suggested that you verify
that the directory is empty before you begin.)
Start a mongos:
mongos --configdb your_host_name:27019

Connect to mongos with the shell:

mongo --shell localhost/week6 week6.js

Add the first shard (" your_host_name :27018").

sh.addShard("localhost:27018")
{ "shardAdded" : "shard0000", "ok" : 1 }

Verify that the week6.trades data is visible via mongos. Note

at this point the week6 database isn't "sharding enabled" but
its data is still visible via mongos:
> db.trades.find().pretty()

> db.trades.count()

> db.trades.stats()

Run homework.a() and enter the result below. This method will
simply verify that this simple cluster is up and running and
return a result key.

Result: 1000001
Homework 6.2
Now enable sharding for the week6 database. (See sh.help()
for details.)
sh.enableSharding("week6")
{ "ok" : 1 }
Then shard the trades collection on the compound shard key
ticker plus time. Note to shard a collection, you must have an
index on the shard key, so you will need to create the index
first:
> db.trades.ensureIndex( { ticker:1, time:1 } )

> // can now shard the trades collection on the shard key { ticker:1, time:1 }

sh.shardCollection("week6.trades",{ ticker:1, time:1 })

After sharding the collection, look at the chunks which exist:

> use config

> db.chunks.find()

> // or:

> db.chunks.find({}, {min:1,max:1,shard:1,_id:0,ns:1})

Run homework.b() to verify the above and enter the return

value below.

Result : 3

Homework 6.3
Let's now add a new shard. Run another mongod as the new
shard on a new port number. Use --shardsvr.
sudo mongod --shardsvr --dbpath /var/lib/mongodb/a1 --logpath
/var/log/mongodb/mongod01.log --port 27020 --fork --logappend --smallfiles --oplogSize
50 --journal
Then add the shard to the cluster (see sh.help()).
sh.addShard( "localhost:27020")
You can confirm the above worked by running:
homework.check1()

mongos> homework.check1()
db.getSisterDB("config").shards.count() :
2
There are 2 shards in the cluster as expected

Now wait for the balancer to move data among the two shards more
evenly. Periodically run:

> use config

> db.chunks.find( { ns:"week6.trades" },

{min:1,max:1,shard:1,_id:0} ).sort({min:1})

and/or:

db.chunks.aggregate( [

{ $match : { ns : "week6.trades" } } ,

{ $group : { _id : "$shard", n : { $sum : 1 } } }

] )

When done, run homework.c() and enter the result value.That

completes this week's homework. However if you want to
explore more, something to try would be to try some queries
and/or write operations with a single process down to see how
the system behaves in such a situation.

Result: 2

MongoDB for DBA's 7/7. Backup and Recovery

Chapter 7. BACKUP AND RECOVERY: Security, backups and
restoring for backups

Overview of Security
There is a few different ways to run mongoDB and secure it:
1. To run it in a trusted environment: no one has access except for clients.
They have full access to the database. "lock down at the network layer the relevant TCP
ports for the processes on the machines
 2. Enable mongodb authentication with two command line options:
1. --auth : for securing client connections and access
2. --keyFile: intracluster security: the machines in the cluster
authenticating each other among themselves. You can also optionally layer SSL
atop it. To gain encryption of all the communications occurring you have to
compile Mongo with the --SSL make option to do that. More information on the
SSL: docs.mongodb.org/manual/administration/ssl

To run MongoDB with encryption over the wire, you need to:

Compile mongodb with --ssl Run your mongod with

--keyFiles Run your mongod with --auth
Security continued: Authentication and
Authorization
1. Authentication: there are a differents forms supported by mongodb:
1. mongodb challenge response (original authentication mechanism in the
product) with usernames and passwords that are actually stored by the cluster.
2. using X.509 (SSL-style certificates and authentication)
3. Kerberos (only in Enterprise edition)
4. LDAT (only in Enterprise edition)
2. Authorization: access control where you are logged in
3. encryption: It's possible to run mongodb cluster using SSL and encryption for all
the communications through the cluster
4. network setup: making sure the nodes are not accessible on the mongodb ports
from servers or clients that are not allowed to use the cluster. For example, locking it
down not being visible from the public internet
5. auditing: it is not free, only available via mongodb suscription in enterprise
edition
By default, when we run s mongod or a mongos process security is off. It is assumed it is runnint
in a trusted environment.

To turn on the security facilities we can use the option command line --auth:

mongod --dbpath dtabase --auth

mongo localhost/test --> let you connect to mongodb database on localhost because there is not
authentication configured yet for the cluster. We have not created any users or roles yet.

To create users and roles, we have to switch to the Admin database (reserved name because it is a
special database which includes the users and roles for administrators):

<dbname>.system.users for eache database

admin database is special:

mongodb challenge response with usernames and passwords

use admin
var me = { user : "bob", pwd : "robert", roles : [ "userAdminAnyDatabase" ] }

db.createUser( me )

at that moment any try to access database is not authorized because we have not authenticated as
that user we just created yet. And also these user will not have any privileges to reading or writing
to databases, only for database administration.

mongod localhost/admin -u bob -p robert

var you = { user : "will", roles : [ "readWriteAnyDatabase" ] }

db.createUser( you )

mongod localhost/admin -u will

We can create users that have permissions for specific database.

Quick summary of some of the standard roles that are available from v2.6+:
1. read: read any database
2. readWrite: read and write any database
3. dbAdmin: db Admin any database
4. userAdmin: admin any database
5. clusterAdmin: gives one authorization to use the cluster-related mongodb
commands: adding a shard and managing replica sets, etc...
You can also create custom that are user-define roles. More information on the built-in roles in
the mongodb documentation: docs.mongodb.org/manual/security

--auth on the command line: when we are working with a sharded cluster or replica set, we will
use another parameter called --keyFile <filename>: to tell the processes making up the cluster,
the mongod and mongs processes how to authenticate amnong themselves inter-cluster. The
filename will contain a shared secret key that all members of the cluster have available to them to
cross-authenticate so the can coordinate their actions. If we use keyFile, it implies off, but it is
recommended that we list both out explicitly in the config file or command line

Which of the following do you need to run with --auth in order

to run your system with authentication?

driver client mongod mongos mongo

SSL and Keyfiles
 The --keyFile option assures that the members of the cluster are all legitimate and
authenticated as legal members of the cluster. We would have clients which talk to the cluster
and clients applications which also talk to the cluster via mongo drivers. A member of a cluster is
basically a server process that have full privileges to do sort of things. The key is really
authentication rather than authorization.
 The --auth command line option enables the security mechanisms for validating:
 authenticating the client and
 authorizating the client to access a particular database.
In both cases, the traffic that is occurring is not encrypted in terms of data transfer over the TCP
sockets in use, however they are authenticated and the initial authentication hand shake is secure.
The password or the key used for authentication does not go over the wire unencrypted.

We can use SSL with mongo but it does require us to build it ourself using scons--ssl.
more information at docs.mongodb.org/manual/tutorial/configure-ssl

Which are true?

When using --keyFile with a replica set, the keyFile's

content is sent over the network between mongod nodes
unencrypted When using --keyFile with a replica set,
database contents are sent over the network between mongod
nodes unencrypted
Security and Clients
mongod --dbpath database --auth

users are created per database

There are a cuple different kinds of users in terms of the system authorizations.
1. Admin users:
1. can perform admin operations
2. are created in the admin database
3. can access all databases, they ares super user
2. Regular users
1. access a specific database
2. can be read/write or read only
To create a admin user:

use admin --> switch to the special admin database

{ user: "<name>",

pwd: "<cleartext password>",

customData: { <any information> },

roles: [

{ role: "<role>", db: "<database>" } | "<role>",

 ...

userAdmin = { user: "theAdmin", pwd : "admin" }

db.createUSer( userAdmin )
db.auth( "theAdmin" , "admin" ) --> only for admin database. We can have the same username in
two different databases with different passwords.

db.system.users.find()

To create a user for a specific database:

use test
db.createUSer( "pat", "123" ) --> by default it has write permission.

All the drivers provide APIs where you can provide credentiasl to authenticate the client as a
particular user when connecting to the database.

Roles are much more finely grained, and custom user permissions can be
created. More information at docs.mongodb.org/manual/reference/built-in-roles
For MongoDB 2.2 and 2.4, which are true?

The user's password is sent over the network unencrypted. The user's password is

stored on disk unencrypted. You can give a user read-only access to one database and write
access to another database.

Intra-cluster Security
XTo use the -keyFile option, we need to put into a file a key, which is a text string made out of
base64 legal character, so upper and lower case letters, a few symbols and numbers or what are
legal but it is very interesting that put a strong password.

In mongoDB 2.6, --auth is implied by --keyFile.

mongod --dbpath data1 --auth --replSet abc --keyFile filename.txt

rs.initiate()

mongod --port 27002 --dbpath data2 --auth --replSet abc --keyFile filename.txt

> rs.status()
{
"ok" : 0,
"errmsg" : "not authorized on admin to execute command
{ replSetGetStatus: 1.0 }",
"code" : 13
}
>

We need to log in as adminitrator to use that administrative command

Overview of Backing Up
Methods to backup in mongodb for individual replica set or an individual server:

1. mongodump utility: make a dump of all databases on a given server or particular

database.
2. filesystem snapshot
3. backup from a secondary so from a replica set secondary
1. shut down, copy files and restart

Mongodump
Make a dump of all databases on a given server or particular database.
 Use the --oplog option.
 It can be done while the system servicing normal load and operations but it creates some
additional load on the system
 mongorestore utility: to restore from backup later if we needed to. Use the oplogReplay
option : it will allow you to achieve a true point in time snapshot for the particular replica set we
are dumping

Which process would you use to reload data after using

mongodump?

mongorestore mongod mongodump mongos mongoimport

Filesystem Snapshotting
If we have a true snapshot capability, this is a good option for a mongod instance that it is up and
running and hot:

1. we will need journaling enabled: used for crash recovery in the event of an
unexpected crash of a Mongod also mean that any snapshot which includes the journal,
will be valid if it is a true point in time snapshot by the underlying file system
1. if journaling is not enabled, what would happen is you will getting a
snapshot from a point in time where it is possible that an operation was mid-
streamed in terms of its rights to data files and it would not be necessarily
consistent
 2. we need a snapshotting of the entire data directory and file system
2. we could use a feature called db.fsyncLock : it will flush everything to disk and
then lock the system, preventing writes
Snapshots ares generally very fast.

Backing Up a Sharded Cluster

When we use sharding, we back up each shard individually and also the config servers. Steps for
backing up a sharded cluster:

1. turn off the balancer: because it moves data around and if there are chunk
migrations in progress during our backup that could be problematic. In the shell, we
would stop the balancer with sh.stopBalancer() and if it is in the middle of doing
someting that may take a minute before it returns
2. backup config databases
1. using mongodump
2. stop one of our three config servers and copy its files
3. backup each replica set
4. start the balancer: sh.startBalancer()

How to restore it?

We would be to pull back these data files on to the appropiate machines and then start
everything up.
After stopping the balancer on your sharded cluster, you may
need to wait for a live migration to finish before the balancer
stops. Type the command you would use in the mongo shell in
order to stop the balancer.
Hint: It starts with the "sh" helper command...

1
sh.stopBalancer()
Correct
Backup Strategies
To backup a sharded cluster:
1. stop the balancer
1. mongo --host nameMongos --eval "sh.stopBalancer() --> (make sure that
worked)
2. backup config database / a config server
 1. mongodump --host nameMongos_or_nameConfigServer --db config
3. backup all the shards
1. Two ways:
1. shut down a secondary in each replica set and grabbing its data
files. if we have snapshotting, just grabbing a snapshot from each replica
set of one node
2. do a mongodump of each shard:
1. mongodump --host shard_1_srv --oplog
/backups/clusters/shard_1
2. mongodump --host shard_2_srv --oplog
/backups/clusters/shard_2
3. ...
4. mongodump --host shard_n_srv --oplog
/backups/clusters/shard_n
4. turn the balancer back on
1. mongo --host nameMongos --eval "sh.startBalancer()
To take in mind:

1. we might want to check that the cluster is healthy before you even begin.
2. if we have replica sets, they can be up when a single server in the set is down

Additional Features of MongoDB

1. Capped collections: they are circular queues where the data can age out a
least-recently-inserted-order. These is used for mongodb replication oplog which is for
the system profile collection
1. They have to have a preallocated size
2. we cannot delete document in these collections or
grow them via update
3. it can be very fast for doing inserts
2. TTL collections: it autoages out of old documents creating
an special index with the extra parameter named TTL
3. GridFS: meaning grid files system, used to store files in
mongodb. In mongodb the BSON size limit is currently 16Mbytes.
If we store something larger, there is a facility for that called
gridFS (a convention for chunking up large data files or binary
objects in kind of a consistency mechanisme. Most of the drivers
have built-in support for doing that when we need to do that. They
are objects in mongo db collections, where they are broken in
pieces that are say, 1 megabyte chunk if we will. Utilities:
1. mongoFiles: put a file into mongodb into a gridFS
collection or pull one out and reassemble it from its pieces.
You can store videos in mongoDB.
 True False
Restoring Data with MMS Backup

GridFS
If we have a 100 TeraBytes object or file, it can be stored in gridFS.
The drivers for mongo understand gridFS and have support for it.
There are command line tools for it.
Hardware Tips
 fast CPU clock is more helpful than more cores (faster cores rather
than more cores)
 RAM is good. Mongos does not require a lot of ram
 we definitely want 64 bits because mongo uses memory map files
 virtualization is OK but certainly not required. It runs pretty well
on Amazon EC2 and it runs fine on VMWare
 disable NUMA (Non-Uniform Memory Access) machine
 SSD's (SOLID STATE DISKS) are good (reserve some empty space
on the disk (unpartitioned))
 the file system cache is most of mongod's memory usage
 check readahead setting (small value)
The production notes are found
at: docs.mongodb.org/manual/administration/production-notes

If a database is 100GB in size and has an equal amount of

random read and write operations occurring, which hardware
configuration is likely to perform better?

128 GB RAM and a rotating disk storage 32 GB RAM and solid state
disk storage
Additional Resources
 1. docs:
1. mongodb.org
2. driver docs:
1. docs.mongodb.org/ecosystem/drivers/
3. bug database / features
1. jira.mongodb.org
2. support forum
1. google groups
4. IRC
1. irc.freenode.net/#mongodb
2. webchat.freenode.net
5. github.com
1. source code
6. blog:
1. blog.mongodb.org
7. @mongodb
8. MMUGs (mongo meetup groups) in various cities around
the world
9. MMS (Mongo Monitoring Service)
Publicado por Juan José Fajardo (Jadsum) en 16:03 1 comentario:
Enviar por correo electrónicoEscribe un blogCompartir con TwitterCompartir con
FacebookCompartir en Pinterest
Etiquetas: mongoDB_4_DBA
miércoles, 11 de febrero de 2015
MongoDB for DBA's 6/7. Scalability
Chapter 6. SCALABILITY: Sharding setup, sharding monitoring,
shard key selection, inserting large amounts of data

Sharding & Data Distribution

Shards : they are partitions where each one has different data (different documents). One
document will live on one and only one shard and it is possible for documents to move for
rebalancing purposes but in a given point in time the document let someone shard. It is used to
scale-out the system data. We can also have a replication having the same documents in different
members of a replica set for high availability and its safety and disaster recovery.

In mongodb, the data distribution is based on a sharded key, and in a given collection the
documents that have the same sharded key will be on the same shard member and documents
who sharded key are closed to other sharded key because each shard member will have a range of
documents (range based partitioning) corresponding with the sharded key. A given range if given
key range will live on a particural shard.
Example: sharding key = customer name

1. Shard 0 : range [A - Irene]

2. Shard 1 : range [Irene - Maria]
3. Shard 3 : range [Maria - Robert]
4. Shard 4 : range [Robert - Zzzzzz]
We can have sharded key for differents collections.

Why range based ?

 The involve rangers will have an efficiency rate.
 The query will be done on a range efficiently
 Sorting can be quite useful
A field that has the boolean of value of either true or false
would make for a good shard key:

True False

Chunks & Operations

Chunk in mongodb is refered to all the documents in the key range.

Operations with chunks that can have in the background:

1. "Split": split the key range in two different key ranges because the limit of data
has exceeded. That will make sure that there are no huge chunks and the data will be
moved. This operations is unexpensive.
2. "Migrate": it will do when "the balancer" sees there is a lack of balance between
the number of shards on the different shards and decides to do migrations. This
operations is more expensive because the data transfer between members. While the
data is transfered, the system is still live and reads and writes of documents that were in
the key range during the migration will execute.

It is possible to configure the size of each chunk which affects

when they will be split. A large chunk size means chunks will
be split less often as each chunk needs to grow larger before
it is split, and a small chunk size means they will be split more
often, for instance.
Which of the following statements are true regarding a very
small chunk size configuration as compared to a large chunk
size configuration? Check all that apply.

Fewer migrations would have to take place More

migrations would have to take place Each shard would be
very balanced in terms of how much data they contain Each
server would be very unbalanced in terms of how much data
they contain There is no difference in terms of migration
and shard server balance

Sharding Processes
Each shard server will run one or more mongod processes storing data (with replica sets for
instance). We will have multiple mongod's procecesses in multimple mongo servers witin a given
shard.

In a addition, we have config servers (small mongod's) which store the metadata of the clusters
(chunks of the system).

There are one more component: the mongos process --> the data relying component. We can have
any number of these and our clients will connect them to perform operations on the clusters. The
mongos gives a connection from a client, a view of the whole cluster as a single logical entity and
it does not need to worry about if a replica set o shard systems exists.

The mongos will talk about to everything as needed and to the config servers to get metadata and
when a query comes in from a client, the mongos just started up, it will get a relevant metadata it
needs to decide where to send the query from the config servers, and then, it will do the operation
with the mongod's. If it comunicates with more than of those, it will merge data coming back on
as-needed basis.

Properties of these different components:

1. The mongos processes: have no persistent state. There has no data files.
There are really kind of like a load balancer and they will get information they need on
how to get the data and load it and route it from the config servers and they will cache
that in RAM.
2. The mongod's processes: are the data storers (database)
3. The config servers: are the metadata store (contain which shard has the right
data). We can run one config server, if we are running in a development on a
laptop. There are three config servers in a production mongodb cluster in order to get
consensus if one of them get failure. They have identical data with a copy of the metadata
that they keep in sync. As long as one config server is up, the cluster is alive. If all three
were down, the cluster would be down. If all three are not up, metadata changing
operations like splits and migrates cannot happen

db.foo.find(

{ name : { $gt : "Joe", $lt : "John" } } )

is efficient if the collection is sharded on { name : 1 }

True False
Cluster Topology
A Cluster setup minimun will be:

1. how many shards initially --> 4 shards

2. replication factor --> R = 3
it will have 4x3=12 mongod processes in shard servers running from 4 mongos processes

Running on localhost
Example with 4 shards processes and 3 replica sets in one machine

mkdir a0
mkdir a1
mkdir a2
mkdir b0
mkdir b1
mkdir b2
mkdir c0
mkdir c1
mkdir c2
mkdir d0
mkdir d1
mkdir d2

mkdir cfg0
mkdir cfg1
mkdir cfg2

Run the config servers:

mongod --configsvr --dbpath cfg0 --port 26050 --fork --logpath log.cfg0 --logappend
mongod --configsvr --dbpath cfg1 --port 26051 --fork --logpath log.cfg1 --logappend
mongod --configsvr --dbpath cfg2 --port 26052 --fork --logpath log.cfg2 --logappend

Run the shard servers:

mongod --shardsvr --replSet a --dbpath a0 --logpath log.a0 --port 27000 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet a --dbpath a1 --logpath log.a1 --port 27001 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet a --dbpath a2 --logpath log.a2 --port 27002 --fork --logappend
--smallfiles --oplogSize 50

mongod --shardsvr --replSet b --dbpath b0 --logpath log.b0 --port 27100 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet b --dbpath b1 --logpath log.b1 --port 27101 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet b --dbpath b2 --logpath log.b2 --port 27102 --fork --logappend
--smallfiles --oplogSize 50

mongod --shardsvr --replSet c --dbpath c0 --logpath log.c0 --port 27200 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet c --dbpath c1 --logpath log.c1 --port 27201 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet c --dbpath c2 --logpath log.c2 --port 27202 --fork --logappend
--smallfiles --oplogSize 50

mongod --shardsvr --replSet d --dbpath d0 --logpath log.d0 --port 27300 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet d --dbpath d1 --logpath log.d1 --port 27301 --fork --logappend
--smallfiles --oplogSize 50
mongod --shardsvr --replSet d --dbpath d2 --logpath log.d2 --port 27302 --fork --logappend
--smallfiles --oplogSize 50

Run the mongs processes:

mongos --configdb localhost:26050, localhost:26051,localhost:26052 --fork --logappend

--logpath log.mongos0
mongos --configdb localhost:26050, localhost:26051,localhost:26052 --fork --logappend
--logpath log.mongos1 --port 26061
mongos --configdb localhost:26050, localhost:26051,localhost:26052 --fork --logappend
--logpath log.mongos2 --port 26062
mongos --configdb localhost:26050, localhost:26051,localhost:26052 --fork --logappend
--logpath log.mongos3 --port 26063

It is important that clients talk to the mongod default server on 27017 port not directly to the
servers. Best practices:

 Run mongos on the standard mongodb tcp port 27017. It gives the possibility to connect
to the cluester for the rest of the world.
 Do not run shard servers mongod's nor config server on that default port 27017.

How many mongos processes should you run (generally)?

One per config server One per mongod process One per shard One
However many you want but usually much more than one

The Config Database

Commands:

>mongo
>use config
>show collections

chunks
databases
lockpings
locks
mongos
settings
shards
system.indexes
version
Which are true?

To check via the mongo shell what's in the config database of a MongoDB cluster, you must

connect it directly to one of the config servers. Three config servers is typical in a MongoDB

cluster with 1,000 total machines. All the config servers have exactly the same data.

Adding the Initial Shards

For each shard:
1. initiate the replica set connecting to the first shards mongod's
1. ps -A | grep mongod | grep 2700
2. Connect to mongo
1. mongo --port 27000
2. Configuration of replica set:
1. rs.initiate()
2. rs.add("localhost:27001")
3. rs.add("localhost:27002")
4. rs.conf()
3. Disconnect from replica set
2. "add" the shard to the cluster: sh.addShard(<>)
1. Connect to mongos
1. mongo
2. sh.addShard("a/localhost:27000")
3. rs.status()

Enable Sharding for a Collection

1. turn on sharding for the database
2. sh.shardCollection(<collection>,{<field name> : 1 }
In MongoDB v2.6, by default (i.e., if shardCollection hasn't
been invoked) collections are:

On the primary shard and not sharded Not sharded but homed on different shards at

random Automatically sharded on _id

Working with a Sharded Cluster

In general we will want to use the sharded key inquiries whenever possible on a shard collection.
If not, the query will be sent to all the mongod's waiting for an answer. If we have one thousand of
shard servers, we will have an overhead. If we create a new index of the field to query, it will send
to the shard server that have the document using the appropiate chunk.

Cardinality & Monotonic Shard Keys

Thing to consider to select the shard key:
 1. It usually is a field that involved the most of your queries. It is common in queries
for the collection.
2. To make sure that the shard key has suffient cardinality: There is enough values
that ve can spread the data out from the collection to a lot of different shards
3. To make sure that the shard key has good granularity to the shard ley values
4. consider compound of shard keys in order to get granularity
5. Is the key monotonically increasing ? BSON "2object id" fo this

Shard Key Selection Example

To choose shard key in "Orders collection". It depends on what kind of data will access

1. "_id": will be the best and efficiently option: { "_id": 1 }

2. "company": { "company" : 1 }
3. compounds key:
1. { company : 1 , "_id" : 1}
2. { company : 1 , "date" : 1}

Suppose we are building an email server system and have a

collection:

messages
--------
{_id:<message_id>, mailbox_id:___,
sender_id:___, subject:___, date:___,
body:___, read:<bool>, ...
}

We have millions of users (email addresses / mailboxes) so

want to shard this collection. Of the choices listed, what's a
good shard key?

{_id:1} {mailbox_id:1} {sender_id:1}

{sender_id:1,read:1}

Process & Machine Layout

We have:

1. shard servers (mongod --shardsvr)

 2. config servers (mongod --configsvr)
3. mongos processes (mongos): possibilities:
1. put the mongos process in the client to access to all of clusters. In this
case all the traffic between the client and the mongos process will remain inside
the client.
2. run the mongos on every single server. The client would connect to any
member to the cluster at random to the mongos that would be on that default
mongo port 27017 and then, the mongos will talk to anything it needs to talk to in
the whole cluster. Different client machines would connect to different mongos
instances.

If I have 3 shards, 3 member replica sets, 3 config servers, and

6 mongos processes running, how many machines might we
use for that?

While we could use more if we wanted, 9 machines would be just fine. 9 machines works
but is too low for best practice usage

Bulk Inserts & Pre-splitting

Normally we do not need to do but there are some situations where it can be quite useful, in
particulary:

1. sometimes with bulk inicial loads: a case can be when we need to pre-split the
initial key range manually because of the loading data into the shard server one is faster
than we are migrating it out and we are creating a backlog in this shard server 1.

Generally speaking, shard keys with non-uniform key

distributions are:

Ok Ok only if you "pre-split" Not ok

Further Tips & Best Practices

1. Only shards the big collections
2. Pick shard key carefully: You cannot change the shardkey once it is set. You
wuld have to create a new collection and copy the data over to it.
3. Considere pre-splitting some chunks on a bulk load
4. Be aware to monotically increasing shard key values on inserts which
could happen if we are sharded on a timestamp field or BSON Object values. It can be just
fine, but we want to be aware of the behaviour.
5. Adding new shards is easy but takes time because it's an online operation
and it does not disrupt operations of cluster. However, once a shard is added, it's initially
empty, so data has to move over to it. If eachs shard in the system has a few terabytes of
data, it is going to take a while for data to migrate over to that, new shard gradually. We
can not instantly and resources are required for that information to move over
6. Always connect to talk to the cluster through the mongos process, not
directly to the mongod's except for dba work where we really want to talk to a certain
server and do someting that is administratative. It is important no to accidentally connect
to mongod's. In order to prevent that, we can put the mongos on the default port (27017)
7. Keep non-mongos precesses off of 271027 port to avoid mistakes.
8. Use logical config server names for config servers rather than direct ip
addresses or even host names. Otherwise, it is really hard to change the config servers
later or replace one, as every machine in the whole cluster has a memory of what the
config servers are. If we do change config servers, read the documentation on how to do
that, because if some members of the cluster think that the server is a config server, an yet
it is not, it is no good.