MongoDB: 7. Replication (1)

最新的 1.6 版总算提供了 Replica Sets，比起有点莫名其妙的 Replica Pairs，这才是高可用集群所需要的。

1. Replica Sets

Replica Sets 使用 n 个 Mongod 节点，构建具备自动容错转移(auto-failover)、自动恢复(auto-recovery) 的高可用方案。通常使用 3 个 mongod 实例，或者 2 mongod + 1 arbiter 方案。

(1) 首先启动所需的 Mongod 节点。注意使用 replSet 参数指定 Sets Name。

$ sudo mkdir -p /var/mongodb/0
$ sudo mkdir -p /var/mongodb/1
$ sudo mkdir -p /var/mongodb/2

$ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/0 --port 27017 --replSet myset
forked process: 1166
all output going to: /dev/null

$ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/1 --port 27018 --replSet myset
forked process: 1173
all output going to: /dev/null

$ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/2 --port 27019 --replSet myset
forked process: 1180
all output going to: /dev/null
(2) 使用 mongo 配置 Replica Sets。

$ ./mongo

MongoDB shell version: 1.6.1
connecting to: test

> cfg = { _id: "myset", members: [
... { _id:0, host:"localhost:27017" },
... { _id:1, host:"localhost:27018" },
... { _id:2, host:"localhost:27019" }
... ]}

> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}

> rs.conf()
{
"_id" : "myset",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
}
]
}
如此 Replica Sets 就算配置成功。

相关配置数据保存在 local 数据库中。

> show dbs
admin
local

> use local
switched to db local

> show collections
oplog.rs
slaves
system.indexes
system.replset

> db.system.replset.find()
{ "_id" : "myset", "version" : 1, "members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
}
] }
oplog.rs 是一个固定长度的 capped collection，用于记录 Replica Sets 操作日志。

(3) 可以用 isMaster 和 status 命令查看 Replica Sets 状态。

> rs.isMaster()
{
"ismaster" : true,
"secondary" : false,
"hosts" : [
"localhost:27017",
"localhost:27019",
"localhost:27018"
],
"ok" : 1
}

> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:21:13 GMT+0800 (CST)",
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "yuhen-server64:27017",
"health" : 1,
"state" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"uptime" : 280,
"lastHeartbeat" : "Sat Aug 21 2010 15:21:11 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 2,
"uptime" : 284,
"lastHeartbeat" : "Sat Aug 21 2010 15:21:11 GMT+0800 (CST)"
}
],
"ok" : 1
}
在同一时刻，每组 Replica Sets 只有一个 Primary，用于接受写操作。而后会异步复制到其他成员数据库中。一旦 primary 死掉，会自动投票选出接任的 primary 来，原服务器恢复后成为普通成员。如果数据尚未从先前的 primary 复制到成员服务器，有可能会丢失数据。

(4) 为了观察数据复制和容错迁移，我们可以开几个终端，在前台运行 mongod。

$ sudo ./mongod --port 27017 --dbpath /var/mongodb/0 --replSet myset
在 mongo 中向 primary (27017) 插入数据。

> use test
switched to db test

> db.users.insert({name:"user1"})
会在所有 mongod 输出记录中看到相关信息。

# 27017 #
Sat Aug 21 15:35:50 [conn2] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:50 [conn2] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }
Sat Aug 21 15:35:50 [conn2] done for 0 records 0.01secs
Sat Aug 21 15:35:50 [conn12] getmore local.oplog.rs cid:4961370576520295111 getMore: { ts: { $gte: new Date(5507762976880328705) } } bytes:118 nreturned:1 3782ms
Sat Aug 21 15:35:50 [conn2] insert test.users 1029ms

# 27018 #
Sat Aug 21 15:35:51 [rs_sync] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:51 [rs_sync] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }

# 27019 #
Sat Aug 21 15:35:51 [rs_sync] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:51 [rs_sync] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }
我们用 CTRL + C 关掉 primary mongd。

# 27018 #
Sat Aug 21 15:40:16 [ReplSetHealthPollTask] replSet info localhost:27017 is now down (or slow to respond)
Sat Aug 21 15:40:27 [rs_sync] replSet SECONDARY

# 27019 #
Sat Aug 21 15:40:16 [ReplSetHealthPollTask] replSet info localhost:27017 is now down (or slow to respond)
Sat Aug 21 15:40:16 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
Sat Aug 21 15:40:16 [rs Manager] replSet info electSelf 2
Sat Aug 21 15:40:16 [rs Manager] replSet PRIMARY
Mongod 27019 被选为 Primary。

因为 27107 的 socket 关闭，所以 mongo 需要重新连接。

$ ./mongo localhost:27019

MongoDB shell version: 1.6.1
connecting to: localhost:27018/test

> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:41:40 GMT+0800 (CST)",
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 0,
"state" : 1,
"uptime" : 0,
"lastHeartbeat" : "Sat Aug 21 2010 15:40:14 GMT+0800 (CST)",
"errmsg" : "connect/transport error"
},
{
"_id" : 1,
"name" : "yuhen-server64:27018",
"health" : 1,
"state" : 2,
"self" : true
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 1,
"uptime" : 486,
"lastHeartbeat" : "Sat Aug 21 2010 15:41:38 GMT+0800 (CST)"
}
],
"ok" : 1
}
查询先前插入的记录正常。

> db.users.find()
{ "_id" : ObjectId("4c6f81d58e4719f03d5ccc65"), "name" : "user1" }
我们也可以将 mongo 连接到 27018，不过无法查询数据。

重新启动 27017。

# 27017 #
Sat Aug 21 15:44:54 [rs Manager] replSet can't see a majority, will not try to elect self
Sat Aug 21 15:44:56 [ReplSetHealthPollTask] replSet info localhost:27019 is now up
Sat Aug 21 15:44:56 [ReplSetHealthPollTask] replSet info localhost:27018 is now up
Sat Aug 21 15:44:56 [rs_sync] building new index on { _id: 1 } for local.me
Sat Aug 21 15:44:56 [rs_sync] Buildindex local.me idxNo:0 { name: "_id_", ns: "local.me", key: { _id: 1 } }
Sat Aug 21 15:44:56 [rs_sync] done for 0 records 0.002secs
Sat Aug 21 15:44:56 [rs_sync] replSet SECONDARY

# 27018 #
Sat Aug 21 15:44:55 [ReplSetHealthPollTask] replSet info localhost:27017 is now up

# 27019 #
Sat Aug 21 15:44:54 [ReplSetHealthPollTask] replSet info localhost:27017 is now up
rs.status 中 heartbeat 恢复正常，但不再是 Primary。

> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)",
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 2,
"uptime" : 110,
"lastHeartbeat" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)"
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"uptime" : 790,
"lastHeartbeat" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "yuhen-server64:27019",
"health" : 1,
"state" : 1,
"self" : true
}
],
"ok" : 1
}
(5) 我们还可以在运行时添加成员。

$ sudo ./mongod --fork --port 27020 --dbpath /var/mongodb/3 --logpath /dev/null --replSet myset
forked process: 2139
all output going to: /dev/null
注意: 必须连接到 primary 才能添加成员。

$ ./mongo localhost:27019
MongoDB shell version: 1.6.1
connecting to: localhost:27020/test

> rs.add("localhost:27029")
{ "ok" : 1 }

> rs.conf()
{
"_id" : "myset",
"version" : 2,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
},
{
"_id" : 3,
"host" : "localhost:27020"
}
]
}
查看状态。

> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)",
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 2,
"uptime" : 94,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 1,
"uptime" : 94,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "yuhen-server64:27019",
"health" : 1,
"state" : 2,
"self" : true
},
{
"_id" : 3,
"name" : "localhost:27020",
"health" : 1,
"state" : 2,
"uptime" : 92,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
}
],
"ok" : 1
}
(6) 从客户端连接 Replica Sets，需要 drivers 支持。

$ ipython

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
IPython 0.10 -- An enhanced Interactive Python.

In [1]: import pymongo

In [2]: conn = pymongo.Connection(host = ["localhost:27017", "localhost:27018", "localhost:27019", "localhost:27020"])

In [3]: db = conn.test

In [4]: for u in db.users.find(): print u
...:
{u'_id': ObjectId('4c6f81d58e4719f03d5ccc65'), u'name': u'user1'}
{u'_id': ObjectId('4c6f82ad8e4719f03d5ccc66'), u'name': u'user2'}

In [5]: db.users.insert({"name":"user3"})
Out[5]: ObjectId('4c6f8872499b1408b7000000')
从其他终端关掉 primary，看看效果。

In [6]: db.users.count()
---------------------------------------------------------------------------
AutoReconnect Traceback (most recent call last)

AutoReconnect: connection closed
由于连接被中断，引发异常也很正常，不过 pymongo 会自动重新连接，再次操作时一切正常。

In [7]: db.users.count()
Out[7]: 3

In [8]: for u in db.users.find(): print u
...:
{u'_id': ObjectId('4c6f81d58e4719f03d5ccc65'), u'name': u'user1'}
{u'_id': ObjectId('4c6f82ad8e4719f03d5ccc66'), u'name': u'user2'}
{u'_id': ObjectId('4c6f8872499b1408b7000000'), u'name': u'user3'}

因此编码时切记需要做异常保护。