mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-08-14 16:00:25 +02:00
feat: add optional MongoDB replica deployment (#765)
This commit is contained in:
Thomas Durieux
@@ -0,0 +1,200 @@
|
||||
# Optional remote MongoDB replica
|
||||
|
||||
This deployment option converts the existing standalone MongoDB container into
|
||||
a replica set named `rs0` and adds a remote, hidden, non-voting secondary.
|
||||
Normal single-node deployments continue to use `docker-compose.yml` alone.
|
||||
|
||||
The remote member is designed as a recovery and backup source:
|
||||
|
||||
- It cannot become primary.
|
||||
- Its availability does not affect production majority writes.
|
||||
- It is hidden from application traffic.
|
||||
- It is delayed by one hour by default, providing a short recovery window for
|
||||
accidental deletes or a bad migration.
|
||||
|
||||
Replication is not a backup. Destructive changes eventually reach every
|
||||
member. Continue making encrypted, off-host `mongodump` archives from the
|
||||
secondary.
|
||||
|
||||
## Requirements
|
||||
|
||||
1. Take and verify a `mongodump` backup before converting production.
|
||||
2. Install this repository on both servers.
|
||||
3. Connect the servers through a trusted private network such as WireGuard,
|
||||
Tailscale, a private VLAN, or a cloud private network.
|
||||
4. Create stable DNS names that resolve from both servers and from the
|
||||
application container, for example:
|
||||
|
||||
- `mongo-primary.internal`
|
||||
- `mongo-secondary.internal`
|
||||
|
||||
5. Permit TCP port `27017` only between the required private-network hosts.
|
||||
Do not expose MongoDB to the public internet.
|
||||
6. Run the exact same MongoDB version on both servers. Find the current
|
||||
production version with:
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongod --version
|
||||
```
|
||||
|
||||
Set the matching image in both servers' `.env` files:
|
||||
|
||||
```env
|
||||
MONGO_IMAGE=mongo:<exact-version>
|
||||
```
|
||||
|
||||
## 1. Generate the shared member key
|
||||
|
||||
On the primary server:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh generate-key
|
||||
```
|
||||
|
||||
This creates `secrets/mongo-replica-keyfile` without overwriting an existing
|
||||
key. Copy that exact file securely to the same repository-relative path on the
|
||||
secondary server:
|
||||
|
||||
```bash
|
||||
scp secrets/mongo-replica-keyfile \
|
||||
secondary-server:/path/to/anonymous_github/secrets/mongo-replica-keyfile
|
||||
```
|
||||
|
||||
Keep this key outside source control and backups that are accessible to
|
||||
untrusted users. Every replica-set member must share the same key.
|
||||
|
||||
## 2. Configure production
|
||||
|
||||
Add these values to the primary server's `.env`:
|
||||
|
||||
```env
|
||||
# Use the exact version already running in production.
|
||||
MONGO_IMAGE=mongo:<exact-version>
|
||||
|
||||
# The primary server's private/VPN interface.
|
||||
MONGO_BIND_ADDRESS=<primary-private-ip>
|
||||
MONGO_REPLICA_KEYFILE=./secrets/mongo-replica-keyfile
|
||||
|
||||
# Make the replica overlay the default for future Compose commands.
|
||||
COMPOSE_FILE=docker-compose.yml:docker-compose.replica-primary.yml
|
||||
|
||||
# URL-encode special characters in the username and password.
|
||||
MONGODB_URI=mongodb://<user>:<password>@mongo-primary.internal:27017/production?authSource=admin&replicaSet=rs0&retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
Stop application writers, leaving MongoDB available:
|
||||
|
||||
```bash
|
||||
docker compose stop anonymous_github streamer
|
||||
```
|
||||
|
||||
Start MongoDB with replica-set support and initialize the existing database as
|
||||
the primary:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh primary-up \
|
||||
mongo-primary.internal:27017
|
||||
```
|
||||
|
||||
`primary-up` is idempotent. If `rs0` is already initialized, it reports that
|
||||
state instead of replacing the replica-set configuration.
|
||||
|
||||
## 3. Start the remote secondary
|
||||
|
||||
On the secondary server, create a minimal `.env`:
|
||||
|
||||
```env
|
||||
MONGO_IMAGE=mongo:<same-exact-version-as-primary>
|
||||
MONGO_BIND_ADDRESS=<secondary-private-ip>
|
||||
MONGO_REPLICA_KEYFILE=./secrets/mongo-replica-keyfile
|
||||
```
|
||||
|
||||
Start its empty MongoDB data volume:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh secondary-up
|
||||
```
|
||||
|
||||
Do not copy the primary's live WiredTiger volume. MongoDB performs the initial
|
||||
sync after the member is added.
|
||||
|
||||
## 4. Add and verify the secondary
|
||||
|
||||
Back on the primary server, add the remote member with the default one-hour
|
||||
delay:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh add-secondary \
|
||||
mongo-secondary.internal:27017
|
||||
```
|
||||
|
||||
To keep the remote copy current instead, explicitly set a zero-second delay:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh add-secondary \
|
||||
mongo-secondary.internal:27017 0
|
||||
```
|
||||
|
||||
Check initial-sync and replication state:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh status
|
||||
```
|
||||
|
||||
Wait until the remote member reports `SECONDARY`, then restart the application:
|
||||
|
||||
```bash
|
||||
docker compose up -d anonymous_github streamer redis
|
||||
```
|
||||
|
||||
## Operations
|
||||
|
||||
Use the primary overlay for every future operation on the production stack.
|
||||
Keeping `COMPOSE_FILE` in `.env` makes ordinary commands such as
|
||||
`docker compose up -d` and `docker compose logs mongodb` use it automatically.
|
||||
|
||||
Check replica status:
|
||||
|
||||
```bash
|
||||
./scripts/mongodb-replica.sh status
|
||||
```
|
||||
|
||||
If a destructive production operation occurs and the secondary is delayed,
|
||||
stop the secondary container immediately before the delay window elapses:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.replica-secondary.yml stop mongodb-secondary
|
||||
```
|
||||
|
||||
Then take a copy or logical dump of the delayed data before attempting
|
||||
recovery.
|
||||
|
||||
For automatic failover, use three voting data-bearing members instead of
|
||||
turning this two-server recovery topology into a two-voter replica set. Two
|
||||
voters require both servers to acknowledge a majority and can make production
|
||||
unwritable during a network outage.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- All members must use the same replica-set name (`rs0`), MongoDB version, and
|
||||
shared keyfile.
|
||||
- The hostnames stored in `rs.conf()` must resolve from every member.
|
||||
- The application container must also resolve `mongo-primary.internal`. Use
|
||||
private DNS or a Compose `extra_hosts` entry if host DNS is not propagated
|
||||
into Docker.
|
||||
- Check container logs with:
|
||||
|
||||
```bash
|
||||
docker compose logs mongodb
|
||||
docker compose -f docker-compose.replica-secondary.yml logs mongodb-secondary
|
||||
```
|
||||
|
||||
- Re-run `primary-up`, `add-secondary`, or `status` safely; the management
|
||||
operations do not replace existing replica-set members.
|
||||
|
||||
## MongoDB references
|
||||
|
||||
- [Convert a standalone server to a replica set](https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/)
|
||||
- [Deploy a replica set with member authentication](https://www.mongodb.com/docs/v8.0/tutorial/deploy-replica-set-with-keyfile-access-control/)
|
||||
- [Configure a delayed member](https://www.mongodb.com/docs/manual/tutorial/configure-a-delayed-replica-set-member/)
|
||||
- [Hidden replica-set members](https://www.mongodb.com/docs/manual/core/replica-set-hidden-member/)
|
||||
Reference in New Issue
Block a user