mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-09-12 13:48:58 +02:00
fix: encrypt stored GitHub credentials (#797)
This commit is contained in:
Thomas Durieux
@@ -61,6 +61,9 @@ npm i
|
||||
GITHUB_TOKEN=<GITHUB_TOKEN>
|
||||
CLIENT_ID=<CLIENT_ID>
|
||||
CLIENT_SECRET=<CLIENT_SECRET>
|
||||
CREDENTIAL_KEYS='{"2026-09":"<base64-encoded 32-byte random key>"}'
|
||||
CREDENTIAL_ACTIVE_KEY_ID=2026-09
|
||||
CREDENTIAL_LEGACY_READS=false
|
||||
PORT=5000
|
||||
DB_USERNAME=
|
||||
DB_PASSWORD=
|
||||
@@ -68,6 +71,7 @@ AUTH_CALLBACK=http://localhost:5000/github/auth
|
||||
```
|
||||
|
||||
- `GITHUB_TOKEN` — create one at <https://github.com/settings/tokens/new> with the `repo` scope.
|
||||
- `CREDENTIAL_KEYS` / `CREDENTIAL_ACTIVE_KEY_ID` — generate a key with `openssl rand -base64 32`. Existing installations must follow the [credential migration guide](docs/credential-encryption.md) before starting this release.
|
||||
- `CLIENT_ID` / `CLIENT_SECRET` — from a new GitHub App at <https://github.com/settings/applications/new>.
|
||||
- The App's callback must be `https://<host>/github/auth` (matching `AUTH_CALLBACK`).
|
||||
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
# GitHub credential storage and migration
|
||||
|
||||
GitHub tokens live in `credentials`, with a unique `(ownerId, provider)` index.
|
||||
Repositories, gists, and pull requests use their existing `owner` field to resolve
|
||||
credentials. Users and resources have no credential reference fields. API login
|
||||
tokens remain hashed in `users.apiTokens`.
|
||||
|
||||
Each credential contains `ownerId`, `provider`, `updatedAt`, and
|
||||
`encryptedToken: { version, keyId, nonce, ciphertext, tag }`. AES-256-GCM uses a
|
||||
fresh 12-byte nonce and a 16-byte authentication tag. Additional authenticated
|
||||
data binds the ciphertext to the owner, provider, field, and format version.
|
||||
Plaintext exists only in application memory and requests to GitHub or the internal
|
||||
streamer. Use TLS for transport across untrusted networks.
|
||||
|
||||
## Configuration
|
||||
|
||||
Set these secrets on the API, workers, and migration process:
|
||||
|
||||
```dotenv
|
||||
CREDENTIAL_KEYS='{"2026-09":"<base64-encoded 32-byte random key>"}'
|
||||
CREDENTIAL_ACTIVE_KEY_ID=2026-09
|
||||
CREDENTIAL_LEGACY_READS=false
|
||||
```
|
||||
|
||||
Generate the key with `openssl rand -base64 32` and store it in your deployment's
|
||||
secret manager or protected environment file. Keep it out of Git, MongoDB, logs,
|
||||
and database backups. Keep a separately protected recovery copy. The application
|
||||
validates the keyring before connecting to the database; there is no temporary or
|
||||
default encryption key, including in development. The streamer consumes tokens
|
||||
from the API and does not need the key unless it also accesses the database.
|
||||
|
||||
`CREDENTIAL_LEGACY_READS=true` temporarily permits reads from legacy user/resource
|
||||
fields when a credential does not exist. All new writes are encrypted regardless
|
||||
of this setting. A corrupt envelope or missing decryption key fails explicitly;
|
||||
it never falls back to a plaintext token. Legacy tokens are not automatically
|
||||
refreshed: migrate them or log in to create an encrypted credential first.
|
||||
|
||||
## Initial deployment
|
||||
|
||||
This release requires a maintenance window. Do not run old and new writers against
|
||||
the same database during migration. The `--maintenance` flag records the operator's
|
||||
acknowledgement; it does not stop processes or acquire a distributed lock.
|
||||
|
||||
1. Build the release and provision the keyring. Back up MongoDB and verify that
|
||||
the database and separately stored keys can be recovered. Rehearse on a
|
||||
protected copy first.
|
||||
2. Stop all API instances, queue workers, scheduled tasks, and other database
|
||||
writers. Keep MongoDB and Redis running. Do not use the normal rolling deploy
|
||||
script for this initial upgrade. Run the migration as a one-off process using
|
||||
the new image, the existing service environment, and database network access.
|
||||
3. Run the read-only inventory:
|
||||
|
||||
```sh
|
||||
node build/scripts/migrate-credentials.js
|
||||
```
|
||||
|
||||
From a source checkout with development dependencies, the equivalent is
|
||||
`npm run migrate:credentials -- <flags>`.
|
||||
4. Resolve reported `conflicting_token`, `malformed_token`, and `missing_owner`
|
||||
records. Output contains collection names, record IDs, and counts, never token
|
||||
values. Conflicting owners retain all legacy fields and receive no new
|
||||
credential. An existing encrypted credential is never overwritten.
|
||||
|
||||
If differing resource tokens are obsolete, `--prefer-owner-token` explicitly
|
||||
chooses the existing encrypted credential, or otherwise the user's token, over
|
||||
conflicting copies. This can remove access supplied by a different token, so
|
||||
review the inventory first. It never arbitrarily chooses between distinct
|
||||
resource-only tokens. Resource-only tokens migrate when they all agree and
|
||||
their owner exists. Removed accounts do not receive credentials.
|
||||
5. Backfill without deleting the legacy fields:
|
||||
|
||||
```sh
|
||||
node build/scripts/migrate-credentials.js --apply --maintenance
|
||||
```
|
||||
|
||||
Add `--prefer-owner-token` only for the conflict resolution described above.
|
||||
Investigate every nonzero exit status. The cursor processes bounded batches;
|
||||
reruns rescan owners and skip existing credentials, so no checkpoint file or
|
||||
exported plaintext is needed.
|
||||
6. Remove legacy copies after successful backfill:
|
||||
|
||||
```sh
|
||||
node build/scripts/migrate-credentials.js --apply --maintenance --remove-legacy
|
||||
node build/scripts/migrate-credentials.js --verify
|
||||
```
|
||||
|
||||
Use the same conflict-resolution flag if needed. Every owner's envelope is
|
||||
authenticated before their legacy fields are removed. Cleanup spans multiple
|
||||
collections and is not a transaction; keep writers stopped. An interruption is
|
||||
safe to resume with the same command. Verification must report `legacy: 0` and
|
||||
exit successfully; it also authenticates all stored credentials and checks
|
||||
their owners.
|
||||
7. Enforce the storage boundary in MongoDB and purge old Redis sessions:
|
||||
|
||||
```sh
|
||||
node build/scripts/migrate-credentials.js --apply --maintenance --enforce
|
||||
node build/scripts/purge-legacy-sessions.js
|
||||
node build/scripts/purge-legacy-sessions.js --apply
|
||||
```
|
||||
|
||||
Enforcement preserves existing collection validators and rejects legacy token
|
||||
fields on users and resources. Session cleanup scans only `anoGH_session:*`,
|
||||
removes the old object-based Passport session format, and leaves queues and
|
||||
ID-only sessions intact. Rerun its dry run to confirm `found: 0`. The application
|
||||
also rejects old session objects. Existing users must log in again.
|
||||
8. Start only the new release with `CREDENTIAL_LEGACY_READS=false`. Test OAuth
|
||||
login, API-token login, private repositories, gists, pull requests, downloads,
|
||||
and account removal. Confirm startup creates the unique credential index.
|
||||
|
||||
A database-only dump cannot decrypt the credentials. Historical dumps, replica
|
||||
oplogs, Redis snapshots/AOF files, and existing logs may still contain plaintext
|
||||
until their retention expires. Migration does not erase those historical copies.
|
||||
If tokens were exposed previously, revoke/reissue them; encryption cannot undo an
|
||||
exposure.
|
||||
|
||||
## Key rotation and rollback
|
||||
|
||||
Add a new random key to `CREDENTIAL_KEYS` on every reader before switching
|
||||
`CREDENTIAL_ACTIVE_KEY_ID`. New logins and refreshes use the active key; existing
|
||||
records remain readable through their `keyId`. Retain old keys until all records
|
||||
using them have been replaced and any backups needing them have expired. This
|
||||
migration command does not bulk re-encrypt existing credentials.
|
||||
|
||||
After encrypted writes begin, rollback must stay on an encryption-capable release
|
||||
with the same keyring. A pre-encryption release cannot read the new collection and
|
||||
will be rejected by the post-migration validators. Do not decrypt production data
|
||||
as a rollback procedure. If maintenance must be aborted before cutover, keep
|
||||
writers stopped and fix/retry the migration, or restore the pre-upgrade database
|
||||
under the original release as a coordinated recovery.
|
||||
|
||||
## Verification tests
|
||||
|
||||
The regular suite tests cryptography, redaction, session contents, and the
|
||||
credential service. Run MongoDB integration tests against a disposable instance:
|
||||
|
||||
```sh
|
||||
TEST_MONGODB_URI=mongodb://127.0.0.1:27028 npm test
|
||||
```
|
||||
|
||||
The tests create randomly named databases and delete only those test databases.
|
||||
They cover stored ciphertext, hidden projections, concurrent credential writes,
|
||||
refresh, OAuth login, resource lookup, migration conflicts/reruns, missing owners,
|
||||
corruption, and MongoDB validators. Without `TEST_MONGODB_URI`, these integration
|
||||
tests are skipped.
|
||||
|
||||
## Docker Compose command sequence
|
||||
|
||||
Run these blocks in a Bash session on the production host, from the existing
|
||||
Compose project directory, after checking out the reviewed release. Keep the same
|
||||
Compose project name and override files used by production. These commands assume
|
||||
the repository's local `mongodb` service, with its existing root credentials in
|
||||
`MONGO_INITDB_ROOT_USERNAME` and `MONGO_INITDB_ROOT_PASSWORD`. If `MONGODB_URI`
|
||||
points elsewhere, back up that database instead. The host needs Python 3 and GPG.
|
||||
|
||||
1. Set the Compose command, preserve the running streamer count, and build without
|
||||
restarting production:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
set +x
|
||||
umask 077
|
||||
dc=(docker compose)
|
||||
# If production uses the replica override, use instead:
|
||||
# dc=(docker compose -f docker-compose.yml -f docker-compose.replica-primary.yml)
|
||||
streamer_replicas=$("${dc[@]}" ps -q streamer | wc -l | tr -d ' ')
|
||||
test "$streamer_replicas" -gt 0
|
||||
"${dc[@]}" build anonymous_github
|
||||
```
|
||||
|
||||
2. Generate the persistent key directly into `.env`, without printing it or putting
|
||||
it in shell history. This deliberately refuses to replace any existing key
|
||||
configuration. On a resumed migration, keep the existing keys and skip this
|
||||
generation step.
|
||||
|
||||
```bash
|
||||
python3 <<'PY'
|
||||
import base64, os, re, secrets
|
||||
from pathlib import Path
|
||||
env = Path('.env')
|
||||
content = env.read_text()
|
||||
if re.search(r'^\s*(?:export\s+)?CREDENTIAL_(?:KEYS|ACTIVE_KEY_ID|LEGACY_READS)\s*=', content, re.M):
|
||||
raise SystemExit('Credential configuration already exists; preserve it and review before continuing.')
|
||||
key = base64.b64encode(secrets.token_bytes(32)).decode('ascii')
|
||||
os.chmod(env, 0o600)
|
||||
with env.open('a') as output:
|
||||
output.write('\nCREDENTIAL_KEYS=\'{"v1":"' + key + '"}\'\n')
|
||||
output.write('CREDENTIAL_ACTIVE_KEY_ID=v1\nCREDENTIAL_LEGACY_READS=false\n')
|
||||
PY
|
||||
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||
-e 'require("./build/core/credentials").credentialCipher(); console.log("Credential keyring valid")'
|
||||
```
|
||||
|
||||
Save the keyring in your secret manager or a separately protected recovery
|
||||
location before proceeding. Do not use `docker compose config` or `cat .env`
|
||||
in a recorded terminal: those can disclose secrets.
|
||||
|
||||
3. Enter maintenance and stop all application writers. In this Compose file the
|
||||
API starts the queue workers and scheduler. Stop any additional instances or
|
||||
external writers too; do not leave the rolling deploy script running.
|
||||
|
||||
```bash
|
||||
"${dc[@]}" stop -t 120 anonymous_github streamer
|
||||
test -z "$("${dc[@]}" ps --status running -q anonymous_github streamer)"
|
||||
"${dc[@]}" ps mongodb redis
|
||||
```
|
||||
|
||||
4. Take a fresh encrypted dump while writers are stopped. The password is read
|
||||
from the database container's environment into a temporary mode-0600 config in
|
||||
`/dev/shm`; it is never placed in command arguments. GPG prompts for a backup
|
||||
passphrase, which must be stored separately from the archive.
|
||||
|
||||
```bash
|
||||
backup_dir=$(mktemp -d /var/tmp/anonymous-gh-migration.XXXXXXXX)
|
||||
export GPG_TTY=$(tty)
|
||||
"${dc[@]}" exec -T mongodb bash -se <<'SH' | gpg --symmetric --cipher-algo AES256 --output "$backup_dir/mongo.archive.gz.gpg"
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
export CREDENTIAL_DUMP_CONFIG=$(mktemp /dev/shm/credential-dump.XXXXXXXX)
|
||||
trap 'rm -f "$CREDENTIAL_DUMP_CONFIG"' EXIT
|
||||
mongosh --nodb --quiet --eval 'require("fs").writeFileSync(process.env.CREDENTIAL_DUMP_CONFIG, JSON.stringify({password: process.env.MONGO_INITDB_ROOT_PASSWORD}), {mode: 0o600})' >/dev/null
|
||||
mongodump --host 127.0.0.1 --port 27017 \
|
||||
--username "$MONGO_INITDB_ROOT_USERNAME" --authenticationDatabase admin \
|
||||
--config "$CREDENTIAL_DUMP_CONFIG" --archive --gzip
|
||||
SH
|
||||
test -s "$backup_dir/mongo.archive.gz.gpg"
|
||||
gpg --decrypt "$backup_dir/mongo.archive.gz.gpg" >/dev/null
|
||||
printf 'Encrypted backup: %s\n' "$backup_dir/mongo.archive.gz.gpg"
|
||||
```
|
||||
|
||||
The decryption check verifies the encrypted file, not MongoDB restoreability.
|
||||
Confirm your restore rehearsal succeeded on an isolated database before
|
||||
deleting legacy fields. Move this archive to your protected backup storage.
|
||||
|
||||
5. Inventory, backfill, then inventory again:
|
||||
|
||||
```bash
|
||||
migrate() {
|
||||
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||
build/scripts/migrate-credentials.js "$@"
|
||||
}
|
||||
migrate
|
||||
migrate --apply --maintenance
|
||||
migrate
|
||||
```
|
||||
|
||||
Stop on any nonzero exit status or `issues` count. Do not add
|
||||
`--prefer-owner-token` automatically: review the conflict policy above first.
|
||||
After backfill, the second inventory should report `created: 0, issues: 0`.
|
||||
|
||||
6. After the backup/restore check and inventory pass, clean and enforce:
|
||||
|
||||
```bash
|
||||
migrate --apply --maintenance --remove-legacy
|
||||
migrate --verify
|
||||
migrate --apply --maintenance --enforce
|
||||
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||
build/scripts/purge-legacy-sessions.js --apply
|
||||
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||
build/scripts/purge-legacy-sessions.js
|
||||
```
|
||||
|
||||
Require `legacy: 0` from MongoDB verification and `found: 0` from the final
|
||||
session scan. Do not resume traffic on an unresolved error.
|
||||
|
||||
7. Recreate only the application services with the new image and environment:
|
||||
|
||||
```bash
|
||||
"${dc[@]}" up -d --no-deps --force-recreate --wait \
|
||||
--scale "streamer=$streamer_replicas" streamer anonymous_github
|
||||
"${dc[@]}" ps anonymous_github streamer
|
||||
migrate --verify
|
||||
```
|
||||
|
||||
Test login and a private repository/gist/pull-request download before ending
|
||||
maintenance. Existing sessions have been invalidated. If a command fails,
|
||||
keep the application stopped and fix/retry; do not launch the old release
|
||||
against the migrated database.
|
||||
|
||||
The one-off invocation follows Docker's [Compose run documentation](https://docs.docker.com/reference/cli/docker/compose/run/).
|
||||
The backup password handling uses MongoDB's [mongodump configuration-file support](https://www.mongodb.com/docs/database-tools/mongodump/).
|
||||
+2
-1
@@ -14,7 +14,8 @@
|
||||
"dev": "nodemon --transpile-only ./src/server/index.ts",
|
||||
"dev:ui": "node scripts/dev-proxy.js",
|
||||
"build": "rm -rf build && tsc && gulp",
|
||||
"knip": "knip"
|
||||
"knip": "knip",
|
||||
"migrate:credentials": "node -r ts-node/register src/scripts/migrate-credentials.ts"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
|
||||
@@ -2,6 +2,9 @@ import { resolve } from "path";
|
||||
import { randomBytes } from "crypto";
|
||||
|
||||
interface Config {
|
||||
CREDENTIAL_KEYS: string;
|
||||
CREDENTIAL_ACTIVE_KEY_ID: string;
|
||||
CREDENTIAL_LEGACY_READS: boolean;
|
||||
SESSION_SECRET: string;
|
||||
REDIS_PORT: number;
|
||||
REDIS_HOSTNAME: string;
|
||||
@@ -55,6 +58,9 @@ const config: Config = {
|
||||
// Predictable defaults are dangerous: a known SESSION_SECRET lets anyone
|
||||
// forge session cookies. Default to empty and resolve below — random in
|
||||
// dev, required in production. See the post-env block.
|
||||
CREDENTIAL_KEYS: "",
|
||||
CREDENTIAL_ACTIVE_KEY_ID: "",
|
||||
CREDENTIAL_LEGACY_READS: false,
|
||||
SESSION_SECRET: "",
|
||||
CLIENT_ID: "CLIENT_ID",
|
||||
CLIENT_SECRET: "CLIENT_SECRET",
|
||||
|
||||
+2
-21
@@ -1,3 +1,4 @@
|
||||
import { getCredentialToken } from "./credentials";
|
||||
import { RepositoryStatus } from "./types";
|
||||
import User from "./User";
|
||||
import UserModel from "./model/users/users.model";
|
||||
@@ -46,27 +47,7 @@ export default class Gist {
|
||||
}
|
||||
|
||||
async getToken() {
|
||||
let owner = this.owner.model;
|
||||
if (owner && !owner.accessTokens.github) {
|
||||
const temp = await UserModel.findById(owner._id);
|
||||
if (temp) {
|
||||
owner = temp;
|
||||
}
|
||||
}
|
||||
if (owner && owner.accessTokens && owner.accessTokens.github) {
|
||||
if (owner.accessTokens.github != this._model.source.accessToken) {
|
||||
this._model.source.accessToken = owner.accessTokens.github;
|
||||
}
|
||||
return owner.accessTokens.github;
|
||||
}
|
||||
if (this._model.source.accessToken) {
|
||||
try {
|
||||
return this._model.source.accessToken;
|
||||
} catch {
|
||||
logger.warn("invalid token", { code: "invalid_token", httpStatus: 401, gistId: this._model.source.gistId });
|
||||
}
|
||||
}
|
||||
return config.GITHUB_TOKEN;
|
||||
return (await getCredentialToken(this.owner.id, "github", { collection: "anonymizedgists", id: this._model._id })) || config.GITHUB_TOKEN;
|
||||
}
|
||||
|
||||
async download() {
|
||||
|
||||
+20
-44
@@ -4,7 +4,7 @@ import { createClient, RedisClientType } from "redis";
|
||||
|
||||
import AnonymousError from "./AnonymousError";
|
||||
import Repository from "./Repository";
|
||||
import UserModel from "./model/users/users.model";
|
||||
import { getCredential, replaceCredential, getCredentialToken } from "./credentials";
|
||||
import config from "../config";
|
||||
import { createLogger } from "./logger";
|
||||
|
||||
@@ -271,36 +271,21 @@ export async function checkToken(token: string) {
|
||||
}
|
||||
}
|
||||
|
||||
const checkedRepositoryTokens = new WeakMap<Repository, string>();
|
||||
|
||||
export async function getToken(repository: Repository) {
|
||||
logger.debug("getToken", { repoId: repository.repoId });
|
||||
// if (repository.model.source.accessToken) {
|
||||
// // only check the token if the repo has been visited less than 10 minutes ago
|
||||
// if (
|
||||
// repository.status == RepositoryStatus.READY &&
|
||||
// repository.model.lastView > new Date(Date.now() - 1000 * 60 * 10)
|
||||
// ) {
|
||||
// return repository.model.source.accessToken;
|
||||
// } else if (await checkToken(repository.model.source.accessToken)) {
|
||||
// return repository.model.source.accessToken;
|
||||
// }
|
||||
// }
|
||||
if (!repository.owner.model.accessTokens?.github) {
|
||||
const query = await UserModel.findById(repository.owner.id, {
|
||||
accessTokens: 1,
|
||||
accessTokenDates: 1,
|
||||
});
|
||||
if (query?.accessTokens) {
|
||||
repository.owner.model.accessTokens = query.accessTokens;
|
||||
repository.owner.model.accessTokenDates = query.accessTokenDates;
|
||||
}
|
||||
}
|
||||
const ownerAccessToken = repository.owner.model.accessTokens?.github;
|
||||
const credential = await getCredential(repository.owner.id);
|
||||
const ownerAccessToken = credential?.token;
|
||||
if (ownerAccessToken) {
|
||||
const tokenAge = repository.owner.model.accessTokenDates?.github;
|
||||
if (checkedRepositoryTokens.get(repository) === ownerAccessToken) {
|
||||
return ownerAccessToken;
|
||||
}
|
||||
const tokenAge = credential?.updatedAt;
|
||||
// if the token is older than 7 days, refresh it
|
||||
if (
|
||||
!tokenAge ||
|
||||
tokenAge < new Date(Date.now() - 1000 * 60 * 60 * 24 * 7)
|
||||
credential?.persisted &&
|
||||
(!tokenAge || tokenAge < new Date(Date.now() - 1000 * 60 * 60 * 24 * 7))
|
||||
) {
|
||||
const url = `https://api.github.com/applications/${config.CLIENT_ID}/token`;
|
||||
const headers = {
|
||||
@@ -339,23 +324,11 @@ export async function getToken(repository: Repository) {
|
||||
? resBody.token
|
||||
: null;
|
||||
if (refreshed) {
|
||||
repository.owner.model.accessTokens.github = refreshed;
|
||||
if (!repository.owner.model.accessTokenDates) {
|
||||
repository.owner.model.accessTokenDates = { github: new Date() };
|
||||
} else {
|
||||
repository.owner.model.accessTokenDates.github = new Date();
|
||||
if (await replaceCredential(repository.owner.id, ownerAccessToken, refreshed)) {
|
||||
checkedRepositoryTokens.set(repository, refreshed);
|
||||
return refreshed;
|
||||
}
|
||||
await UserModel.updateOne(
|
||||
{ _id: repository.owner.model._id },
|
||||
{
|
||||
$set: {
|
||||
"accessTokens.github": refreshed,
|
||||
"accessTokenDates.github":
|
||||
repository.owner.model.accessTokenDates.github,
|
||||
},
|
||||
}
|
||||
).exec();
|
||||
return refreshed;
|
||||
return (await getCredentialToken(repository.owner.id)) || config.GITHUB_TOKEN;
|
||||
}
|
||||
}
|
||||
logger.warn("token refresh failed; falling back", {
|
||||
@@ -367,9 +340,12 @@ export async function getToken(repository: Repository) {
|
||||
}
|
||||
const check = await checkToken(ownerAccessToken);
|
||||
if (check) {
|
||||
repository.model.source.accessToken = ownerAccessToken;
|
||||
checkedRepositoryTokens.set(repository, ownerAccessToken);
|
||||
return ownerAccessToken;
|
||||
}
|
||||
return config.GITHUB_TOKEN;
|
||||
}
|
||||
return config.GITHUB_TOKEN;
|
||||
return (await getCredentialToken(repository.owner.id, "github", {
|
||||
collection: "anonymizedrepositories", id: repository.model._id,
|
||||
})) || config.GITHUB_TOKEN;
|
||||
}
|
||||
|
||||
+2
-25
@@ -1,3 +1,4 @@
|
||||
import { getCredentialToken } from "./credentials";
|
||||
import { RepositoryStatus } from "./types";
|
||||
import User from "./User";
|
||||
import UserModel from "./model/users/users.model";
|
||||
@@ -24,31 +25,7 @@ export default class PullRequest {
|
||||
}
|
||||
|
||||
async getToken() {
|
||||
let owner = this.owner.model;
|
||||
if (owner && !owner.accessTokens.github) {
|
||||
const temp = await UserModel.findById(owner._id);
|
||||
if (temp) {
|
||||
owner = temp;
|
||||
}
|
||||
}
|
||||
if (owner && owner.accessTokens && owner.accessTokens.github) {
|
||||
if (owner.accessTokens.github != this._model.source.accessToken) {
|
||||
this._model.source.accessToken = owner.accessTokens.github;
|
||||
}
|
||||
return owner.accessTokens.github;
|
||||
}
|
||||
if (this._model.source.accessToken) {
|
||||
try {
|
||||
return this._model.source.accessToken;
|
||||
} catch {
|
||||
logger.warn("invalid token", {
|
||||
code: "invalid_token",
|
||||
httpStatus: 401,
|
||||
pullRequestId: this._model.source.pullRequestId,
|
||||
});
|
||||
}
|
||||
}
|
||||
return config.GITHUB_TOKEN;
|
||||
return (await getCredentialToken(this.owner.id, "github", { collection: "anonymizedpullrequests", id: this._model._id })) || config.GITHUB_TOKEN;
|
||||
}
|
||||
|
||||
async download() {
|
||||
|
||||
+1
-16
@@ -70,23 +70,8 @@ export default class Repository {
|
||||
this.owner.model.isNew = false;
|
||||
}
|
||||
|
||||
private checkedToken: boolean = false;
|
||||
|
||||
async getToken() {
|
||||
if (this.checkedToken) return this._model.source.accessToken as string;
|
||||
const originalToken = this._model.source.accessToken;
|
||||
const token = await getToken(this);
|
||||
if (originalToken != token) {
|
||||
this._model.source.accessToken = token;
|
||||
if (isConnected) {
|
||||
await AnonymizedRepositoryModel.updateOne(
|
||||
{ _id: this._model._id },
|
||||
{ $set: { "source.accessToken": token } }
|
||||
).exec();
|
||||
}
|
||||
}
|
||||
this.checkedToken = true;
|
||||
return token;
|
||||
return getToken(this);
|
||||
}
|
||||
|
||||
get source() {
|
||||
|
||||
+8
-4
@@ -1,3 +1,4 @@
|
||||
import { getCredentialToken } from "./credentials";
|
||||
import AnonymizedRepositoryModel from "./model/anonymizedRepositories/anonymizedRepositories.model";
|
||||
import RepositoryModel from "./model/repositories/repositories.model";
|
||||
import UserModel from "./model/users/users.model";
|
||||
@@ -31,8 +32,8 @@ export default class User {
|
||||
return !!this._model.isAdmin;
|
||||
}
|
||||
|
||||
get accessToken(): string {
|
||||
return this._model.accessTokens.github;
|
||||
async getAccessToken(): Promise<string> {
|
||||
return getCredentialToken(this.id);
|
||||
}
|
||||
|
||||
get photo(): string | undefined {
|
||||
@@ -64,7 +65,7 @@ export default class User {
|
||||
opt?.force === true
|
||||
) {
|
||||
// get the list of repo from github
|
||||
const oct = octokit(this.accessToken);
|
||||
const oct = octokit(await this.getAccessToken());
|
||||
const repositories = (
|
||||
await oct.paginate("GET /user/repos", {
|
||||
visibility: "all",
|
||||
@@ -213,6 +214,9 @@ export default class User {
|
||||
}
|
||||
|
||||
toJSON() {
|
||||
return this._model.toJSON();
|
||||
const value = this._model.toJSON();
|
||||
delete (value as Partial<typeof value>).accessTokens;
|
||||
delete value.apiTokens;
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import { createCipheriv, createDecipheriv, randomBytes } from "crypto";
|
||||
|
||||
export interface EncryptedToken {
|
||||
version: number;
|
||||
keyId: string;
|
||||
nonce: string;
|
||||
ciphertext: string;
|
||||
tag: string;
|
||||
}
|
||||
|
||||
export function createTokenCipher(rawKeys: string, activeKeyId: string) {
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(rawKeys);
|
||||
} catch {
|
||||
throw new Error("CREDENTIAL_KEYS must be a JSON object of base64 keys");
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
throw new Error("CREDENTIAL_KEYS must be a JSON object");
|
||||
}
|
||||
const keys = new Map<string, Buffer>();
|
||||
for (const [id, value] of Object.entries(parsed)) {
|
||||
if (typeof value !== "string" || !id ||
|
||||
Buffer.from(value, "base64").length !== 32 ||
|
||||
Buffer.from(value, "base64").toString("base64") !== value) {
|
||||
throw new Error("Credential keys must be canonical base64-encoded 32-byte keys");
|
||||
}
|
||||
keys.set(id, Buffer.from(value, "base64"));
|
||||
}
|
||||
if (!keys.has(activeKeyId)) throw new Error("CREDENTIAL_ACTIVE_KEY_ID is missing from CREDENTIAL_KEYS");
|
||||
const aad = (ownerId: string, provider: string) =>
|
||||
Buffer.from(JSON.stringify(["credentials", ownerId, provider, "encryptedToken", 1]));
|
||||
return {
|
||||
encrypt(token: string, ownerId: string, provider: string): EncryptedToken {
|
||||
if (!token) throw new Error("Cannot encrypt an empty credential");
|
||||
const nonce = randomBytes(12);
|
||||
const cipher = createCipheriv("aes-256-gcm", keys.get(activeKeyId)!, nonce);
|
||||
cipher.setAAD(aad(ownerId, provider));
|
||||
const ciphertext = Buffer.concat([cipher.update(token, "utf8"), cipher.final()]);
|
||||
return { version: 1, keyId: activeKeyId, nonce: nonce.toString("base64"),
|
||||
ciphertext: ciphertext.toString("base64"), tag: cipher.getAuthTag().toString("base64") };
|
||||
},
|
||||
decrypt(value: EncryptedToken, ownerId: string, provider: string): string {
|
||||
try {
|
||||
if (!value || value.version !== 1 || !keys.has(value.keyId)) throw new Error();
|
||||
const decode = (s: string) => {
|
||||
if (typeof s !== "string" || !s) throw new Error();
|
||||
const b = Buffer.from(s, "base64");
|
||||
if (b.toString("base64") !== s) throw new Error();
|
||||
return b;
|
||||
};
|
||||
const nonce = decode(value.nonce);
|
||||
const tag = decode(value.tag);
|
||||
if (nonce.length !== 12 || tag.length !== 16) throw new Error();
|
||||
const decipher = createDecipheriv("aes-256-gcm", keys.get(value.keyId)!, nonce, { authTagLength: 16 });
|
||||
decipher.setAAD(aad(ownerId, provider));
|
||||
decipher.setAuthTag(tag);
|
||||
return Buffer.concat([decipher.update(decode(value.ciphertext)), decipher.final()]).toString("utf8");
|
||||
} catch {
|
||||
// Never attach the token, envelope, key, or underlying crypto error.
|
||||
throw new Error("Credential decryption failed");
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
import { Types } from "mongoose";
|
||||
import config from "../config";
|
||||
import { createTokenCipher } from "./credential-crypto";
|
||||
import CredentialModel from "./model/credentials/credentials.model";
|
||||
import UserModel from "./model/users/users.model";
|
||||
|
||||
export function credentialCipher() {
|
||||
return createTokenCipher(config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID);
|
||||
}
|
||||
|
||||
export async function getCredential(ownerId: string, provider = "github") {
|
||||
const credential = await CredentialModel.findOne({ ownerId, provider }).select("+encryptedToken").lean();
|
||||
if (credential) {
|
||||
return { token: credentialCipher().decrypt(credential.encryptedToken, String(credential.ownerId), provider),
|
||||
updatedAt: credential.updatedAt, persisted: true };
|
||||
}
|
||||
if (config.CREDENTIAL_LEGACY_READS && provider === "github") {
|
||||
const user = await UserModel.findById(ownerId).select("+accessTokens.github accessTokenDates");
|
||||
const token = user?.accessTokens?.github;
|
||||
if (token) return { token, updatedAt: user?.accessTokenDates?.github, persisted: false };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function getCredentialToken(ownerId: string, provider = "github", resource?: {
|
||||
collection: "anonymizedrepositories" | "anonymizedgists" | "anonymizedpullrequests";
|
||||
id: unknown;
|
||||
}): Promise<string> {
|
||||
const credential = await getCredential(ownerId, provider);
|
||||
if (credential) return credential.token;
|
||||
if (config.CREDENTIAL_LEGACY_READS && resource) {
|
||||
const row = await CredentialModel.db.collection(resource.collection).findOne({
|
||||
_id: resource.id as Types.ObjectId,
|
||||
owner: new Types.ObjectId(ownerId),
|
||||
}, { projection: { "source.accessToken": 1 } });
|
||||
if (typeof row?.source?.accessToken === "string") return row.source.accessToken;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
export async function setCredential(ownerId: string, token: string, provider = "github") {
|
||||
ownerId = new Types.ObjectId(ownerId).toHexString();
|
||||
const encryptedToken = credentialCipher().encrypt(token, ownerId, provider);
|
||||
await CredentialModel.updateOne({ ownerId, provider }, {
|
||||
$set: { encryptedToken, updatedAt: new Date() },
|
||||
}, { upsert: true, runValidators: true });
|
||||
}
|
||||
|
||||
// A refresh must not overwrite a credential replaced by a concurrent login.
|
||||
export async function replaceCredential(ownerId: string, previous: string, token: string) {
|
||||
const current = await CredentialModel.findOne({ ownerId, provider: "github" }).select("+encryptedToken").lean();
|
||||
if (!current) {
|
||||
// Legacy credentials are migrated by the migration script or next login.
|
||||
return false;
|
||||
}
|
||||
const cipher = credentialCipher();
|
||||
if (cipher.decrypt(current.encryptedToken, String(ownerId), "github") !== previous) return false;
|
||||
const result = await CredentialModel.updateOne({ _id: current._id, encryptedToken: current.encryptedToken }, {
|
||||
$set: { encryptedToken: cipher.encrypt(token, String(ownerId), "github"), updatedAt: new Date() },
|
||||
});
|
||||
return result.modifiedCount === 1;
|
||||
}
|
||||
+3
-1
@@ -1,3 +1,4 @@
|
||||
import { redactSecrets } from "./redact-secrets";
|
||||
import { createClient, RedisClientType } from "redis";
|
||||
import config from "../config";
|
||||
|
||||
@@ -251,6 +252,7 @@ function persistError(entry: {
|
||||
|
||||
function emit(level: Level, module: string, args: unknown[]) {
|
||||
if (LEVEL_ORDER[level] < threshold) return;
|
||||
args = args.map(arg => redactSecrets(arg instanceof Error ? serializeError(arg) : arg));
|
||||
const ts = new Date().toISOString();
|
||||
const formatted = args.map(formatArg);
|
||||
const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`;
|
||||
@@ -341,5 +343,5 @@ export function serializeError(err: unknown): Record<string, unknown> {
|
||||
// a stack for handled HTTP errors but keeps debuggability for plain Errors.
|
||||
if (!out.status && !out.httpStatus && e.stack) out.stack = e.stack;
|
||||
|
||||
return out;
|
||||
return redactSecrets(out) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
import { mongo } from "mongoose";
|
||||
import { createTokenCipher, EncryptedToken } from "./credential-crypto";
|
||||
|
||||
type Cipher = ReturnType<typeof createTokenCipher>;
|
||||
const resources = ["anonymizedrepositories", "anonymizedgists", "anonymizedpullrequests"];
|
||||
const legacyQuery = { $or: [{ "source.accessToken": { $exists: true } }, { accessToken: { $exists: true } }] };
|
||||
export interface MigrationOptions {
|
||||
apply?: boolean;
|
||||
removeLegacy?: boolean;
|
||||
preferOwnerToken?: boolean;
|
||||
batchSize?: number;
|
||||
report?: (event: { collection: string; id: string; issue: string }) => void;
|
||||
}
|
||||
|
||||
/** Run with all application writers stopped. Reruns never replace an existing credential. */
|
||||
export async function migrateCredentials(db: mongo.Db, cipher: Cipher, options: MigrationOptions = {}) {
|
||||
const credentials = db.collection("credentials");
|
||||
const users = db.collection("users");
|
||||
const batchSize = options.batchSize || 100;
|
||||
const counts = { owners: 0, created: 0, removed: 0, issues: 0 };
|
||||
const issue = (collection: string, id: unknown, reason: string) => {
|
||||
counts.issues++;
|
||||
options.report?.({ collection, id: String(id), issue: reason });
|
||||
};
|
||||
if (options.apply) await credentials.createIndex({ ownerId: 1, provider: 1 }, { unique: true });
|
||||
for await (const user of users.find({}, { projection: { accessTokens: 1, accessTokenDates: 1, status: 1 } }).batchSize(batchSize)) {
|
||||
counts.owners++;
|
||||
const ownerId = user._id;
|
||||
const existing = await credentials.findOne({ ownerId, provider: "github" });
|
||||
let selected: string | undefined;
|
||||
let valid = true;
|
||||
if (existing) {
|
||||
try { selected = cipher.decrypt(existing.encryptedToken as EncryptedToken, String(ownerId), "github"); }
|
||||
catch { issue("credentials", existing._id, "decryption_failed"); continue; }
|
||||
}
|
||||
const ownerToken = user.accessTokens?.github;
|
||||
const authoritative = !!(selected || (typeof ownerToken === "string" && ownerToken));
|
||||
const inspect = (value: unknown, collection: string, id: unknown) => {
|
||||
if (value === undefined || value === null || value === "") return;
|
||||
if (typeof value !== "string") {
|
||||
issue(collection, id, "malformed_token"); valid = false; return;
|
||||
}
|
||||
if (!selected) selected = value;
|
||||
else if (selected !== value && !(options.preferOwnerToken && authoritative)) {
|
||||
issue(collection, id, "conflicting_token"); valid = false;
|
||||
}
|
||||
};
|
||||
inspect(ownerToken, "users", ownerId);
|
||||
for (const name of resources) {
|
||||
for await (const row of db.collection(name).find({ owner: ownerId, ...legacyQuery }, {
|
||||
projection: { source: 1, accessToken: 1 },
|
||||
}).batchSize(batchSize)) {
|
||||
inspect(row.source?.accessToken, name, row._id);
|
||||
inspect(row.accessToken, name, row._id);
|
||||
}
|
||||
}
|
||||
if (!valid) continue;
|
||||
// Removed accounts must never regain credentials during backfill.
|
||||
if (user.status === "removed") {
|
||||
if (options.apply && options.removeLegacy) await credentials.deleteMany({ ownerId });
|
||||
} else if (selected && !existing) {
|
||||
const encryptedToken = cipher.encrypt(selected, String(ownerId), "github");
|
||||
if (cipher.decrypt(encryptedToken, String(ownerId), "github") !== selected) throw new Error("Credential verification failed");
|
||||
if (options.apply) {
|
||||
await credentials.updateOne({ ownerId, provider: "github" }, { $setOnInsert: {
|
||||
encryptedToken, updatedAt: user.accessTokenDates?.github || new Date(),
|
||||
} }, { upsert: true });
|
||||
const stored = await credentials.findOne({ ownerId, provider: "github" });
|
||||
if (!stored || cipher.decrypt(stored.encryptedToken as EncryptedToken, String(ownerId), "github") !== selected) {
|
||||
issue("users", ownerId, "credential_changed_retry"); continue;
|
||||
}
|
||||
}
|
||||
counts.created++;
|
||||
}
|
||||
if (options.apply && options.removeLegacy) {
|
||||
// The command requires maintenance mode: no login, refresh, deletion, or resource writes.
|
||||
const result = await users.updateOne({ _id: ownerId }, { $unset: { accessTokens: "", accessTokenDates: "" } });
|
||||
counts.removed += result.modifiedCount;
|
||||
for (const name of resources) {
|
||||
const result = await db.collection(name).updateMany({ owner: ownerId, ...legacyQuery }, {
|
||||
$unset: { "source.accessToken": "", accessToken: "" },
|
||||
});
|
||||
counts.removed += result.modifiedCount;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Credentials attached to missing owners cannot be assigned safely.
|
||||
for (const name of resources) {
|
||||
for await (const row of db.collection(name).find(legacyQuery, { projection: { owner: 1 } }).batchSize(batchSize)) {
|
||||
if (!row.owner || !(await users.findOne({ _id: row.owner }, { projection: { _id: 1 } }))) {
|
||||
issue(name, row._id, "missing_owner");
|
||||
}
|
||||
}
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
export async function verifyCredentials(db: mongo.Db, cipher: Cipher) {
|
||||
let checked = 0;
|
||||
for await (const row of db.collection("credentials").find({})) {
|
||||
cipher.decrypt(row.encryptedToken as EncryptedToken, String(row.ownerId), row.provider);
|
||||
if (!(await db.collection("users").findOne({ _id: row.ownerId, status: { $ne: "removed" } }))) {
|
||||
throw new Error("Credential has no active owner");
|
||||
}
|
||||
checked++;
|
||||
}
|
||||
let legacy = await db.collection("users").countDocuments({ accessTokens: { $exists: true } });
|
||||
for (const name of resources) legacy += await db.collection(name).countDocuments(legacyQuery);
|
||||
return { checked, legacy };
|
||||
}
|
||||
|
||||
/** Preserve existing validators while forbidding legacy credential storage. */
|
||||
export async function enforceCredentialStorage(db: mongo.Db, cipher: Cipher) {
|
||||
const verification = await verifyCredentials(db, cipher);
|
||||
if (verification.legacy) throw new Error("Legacy credentials remain");
|
||||
for (const name of ["users", ...resources]) {
|
||||
const info = await db.listCollections({ name }).next();
|
||||
if (!info) await db.createCollection(name);
|
||||
const previous = info && "options" in info ? info.options?.validator : undefined;
|
||||
const absent = name === "users" ? { accessTokens: { $exists: false } } : {
|
||||
"source.accessToken": { $exists: false }, accessToken: { $exists: false },
|
||||
};
|
||||
await db.command({ collMod: name, validator: previous && Object.keys(previous).length ? {
|
||||
$and: [previous, absent],
|
||||
} : absent, validationLevel: "strict", validationAction: "error" });
|
||||
}
|
||||
return verification;
|
||||
}
|
||||
@@ -18,7 +18,7 @@ const AnonymizedGistSchema = new Schema({
|
||||
conference: String,
|
||||
source: {
|
||||
gistId: String,
|
||||
accessToken: String,
|
||||
accessToken: { type: String, select: false },
|
||||
},
|
||||
options: {
|
||||
terms: [String],
|
||||
|
||||
@@ -19,7 +19,7 @@ const AnonymizedPullRequestSchema = new Schema({
|
||||
source: {
|
||||
pullRequestId: Number,
|
||||
repositoryFullName: String,
|
||||
accessToken: String,
|
||||
accessToken: { type: String, select: false },
|
||||
},
|
||||
options: {
|
||||
terms: [String],
|
||||
|
||||
@@ -14,7 +14,7 @@ const AnonymizedRepositorySchema = new Schema({
|
||||
anonymizeDate: Date,
|
||||
lastView: Date,
|
||||
pageView: Number,
|
||||
accessToken: String,
|
||||
accessToken: { type: String, select: false },
|
||||
owner: {
|
||||
type: Schema.Types.ObjectId,
|
||||
ref: "user",
|
||||
@@ -36,7 +36,7 @@ const AnonymizedRepositorySchema = new Schema({
|
||||
commitDate: Date,
|
||||
repositoryId: String,
|
||||
repositoryName: String,
|
||||
accessToken: String,
|
||||
accessToken: { type: String, select: false },
|
||||
},
|
||||
truncatedFolders: {
|
||||
type: [String],
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
import { model, Schema, Types } from "mongoose";
|
||||
import { EncryptedToken } from "../../credential-crypto";
|
||||
|
||||
export interface ICredential {
|
||||
ownerId: Types.ObjectId;
|
||||
provider: string;
|
||||
encryptedToken: EncryptedToken;
|
||||
updatedAt: Date;
|
||||
}
|
||||
const envelope = new Schema({
|
||||
version: { type: Number, required: true, enum: [1] },
|
||||
keyId: { type: String, required: true },
|
||||
nonce: { type: String, required: true },
|
||||
ciphertext: { type: String, required: true },
|
||||
tag: { type: String, required: true },
|
||||
}, { _id: false });
|
||||
const schema = new Schema<ICredential>({
|
||||
ownerId: { type: Schema.Types.ObjectId, required: true, ref: "user" },
|
||||
provider: { type: String, required: true, enum: ["github"] },
|
||||
encryptedToken: { type: envelope, required: true, select: false },
|
||||
updatedAt: { type: Date, required: true },
|
||||
}, { collection: "credentials" });
|
||||
schema.index({ ownerId: 1, provider: 1 }, { unique: true });
|
||||
export default model<ICredential>("Credential", schema);
|
||||
@@ -2,7 +2,7 @@ import { Schema } from "mongoose";
|
||||
|
||||
const UserSchema = new Schema({
|
||||
accessTokens: {
|
||||
github: { type: String },
|
||||
github: { type: String, select: false },
|
||||
},
|
||||
accessTokenDates: {
|
||||
github: { type: Date },
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { Document, Model } from "mongoose";
|
||||
|
||||
export interface IUser {
|
||||
accessTokens: {
|
||||
/** Legacy storage, read only when CREDENTIAL_LEGACY_READS is enabled. */
|
||||
accessTokens?: {
|
||||
github: string;
|
||||
};
|
||||
accessTokenDates?: {
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
const sensitive = /^(?:authorization|proxy-authorization|cookie|set-cookie|token|access_?tokens?|refresh_?token|encryptedToken|ciphertext|nonce|tag|password|client_?secret|CREDENTIAL_KEYS)$/i;
|
||||
export function redactSecrets(value: unknown, seen = new WeakSet<object>()): unknown {
|
||||
if (typeof value === "string") return value
|
||||
.replace(/\b(?:gh[pousr]_[A-Za-z0-9_]+|github_pat_[A-Za-z0-9_]+)\b/g, "[REDACTED]")
|
||||
.replace(/((?:access_token|refresh_token|token)=)[^&\s]+/gi, "$1[REDACTED]")
|
||||
.replace(/\b(Bearer|Basic)\s+[A-Za-z0-9+/=._-]+/gi, "$1 [REDACTED]");
|
||||
if (!value || typeof value !== "object" || value instanceof Date) return value;
|
||||
if (seen.has(value)) return "[Circular]";
|
||||
seen.add(value);
|
||||
if (Array.isArray(value)) return value.map(item => redactSecrets(item, seen));
|
||||
const result: Record<string, unknown> = {};
|
||||
for (const [key, item] of Object.entries(value)) {
|
||||
result[key] = sensitive.test(key) ? "[REDACTED]" : redactSecrets(item, seen);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import "dotenv/config";
|
||||
import mongoose from "mongoose";
|
||||
import config from "../config";
|
||||
import { credentialCipher } from "../core/credentials";
|
||||
import { migrateCredentials, verifyCredentials, enforceCredentialStorage } from "../core/migrate-credentials";
|
||||
|
||||
async function main() {
|
||||
const args = new Set(process.argv.slice(2));
|
||||
for (const arg of args) {
|
||||
if (!["--apply", "--remove-legacy", "--prefer-owner-token", "--maintenance", "--verify", "--enforce"].includes(arg)) {
|
||||
throw new Error("Unknown migration option");
|
||||
}
|
||||
}
|
||||
if (args.has("--apply") && !args.has("--maintenance")) throw new Error("Stop all application writers and pass --maintenance before applying");
|
||||
if (args.has("--enforce") && !args.has("--apply")) throw new Error("--enforce requires --apply");
|
||||
if (args.has("--remove-legacy") && !args.has("--apply")) throw new Error("--remove-legacy requires --apply");
|
||||
const cipher = credentialCipher();
|
||||
const uri = config.MONGODB_URI || `mongodb://${config.DB_USERNAME}:${config.DB_PASSWORD}@${config.DB_HOSTNAME}:27017/production`;
|
||||
await mongoose.connect(uri, config.MONGODB_URI ? {} : { authSource: "admin" });
|
||||
const db = mongoose.connection.db;
|
||||
if (args.has("--enforce")) {
|
||||
process.stdout.write(JSON.stringify(await enforceCredentialStorage(db, cipher)) + "\n");
|
||||
} else if (args.has("--verify")) {
|
||||
const result = await verifyCredentials(db, cipher);
|
||||
process.stdout.write(JSON.stringify(result) + "\n");
|
||||
if (result.legacy) process.exitCode = 1;
|
||||
} else {
|
||||
const result = await migrateCredentials(db, cipher, {
|
||||
apply: args.has("--apply"), removeLegacy: args.has("--remove-legacy"),
|
||||
preferOwnerToken: args.has("--prefer-owner-token"),
|
||||
report: event => process.stdout.write(JSON.stringify(event) + "\n"),
|
||||
});
|
||||
process.stdout.write(JSON.stringify(result) + "\n");
|
||||
if (result.issues) process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
main().catch(() => {
|
||||
// Driver errors can contain document values or connection credentials.
|
||||
process.stderr.write("Credential migration failed; check configuration, connectivity, and encrypted records. No secret values are logged.\n");
|
||||
process.exitCode = 1;
|
||||
}).finally(() => mongoose.disconnect());
|
||||
@@ -0,0 +1,38 @@
|
||||
import "dotenv/config";
|
||||
import { createClient } from "redis";
|
||||
import config from "../config";
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
if (args.some(arg => arg !== "--apply")) throw new Error("Unknown option");
|
||||
const client = createClient({ socket: { host: config.REDIS_HOSTNAME, port: config.REDIS_PORT, reconnectStrategy: false } });
|
||||
client.on("error", () => {});
|
||||
try {
|
||||
await client.connect();
|
||||
let found = 0;
|
||||
let removed = 0;
|
||||
for await (const key of client.scanIterator({ MATCH: "anoGH_session:*", COUNT: 100 })) {
|
||||
const raw = await client.get(key);
|
||||
if (!raw) continue;
|
||||
let legacy = false;
|
||||
try {
|
||||
const value = JSON.parse(raw);
|
||||
legacy = !!value.passport?.user && typeof value.passport.user !== "string";
|
||||
} catch { legacy = true; }
|
||||
if (!legacy) continue;
|
||||
found++;
|
||||
if (args.includes("--apply")) {
|
||||
// Do not remove a session replaced since the scan read it.
|
||||
removed += Number(await client.eval(
|
||||
'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) else return 0 end',
|
||||
{ keys: [key], arguments: [raw] }
|
||||
));
|
||||
}
|
||||
}
|
||||
process.stdout.write(JSON.stringify({ found, removed }) + "\n");
|
||||
} finally { if (client.isOpen) await client.quit(); }
|
||||
}
|
||||
main().catch(() => {
|
||||
process.stderr.write("Legacy session cleanup failed; check Redis configuration and connectivity.\n");
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import { credentialCipher } from "../core/credentials";
|
||||
import CredentialModel from "../core/model/credentials/credentials.model";
|
||||
import mongoose, { ConnectOptions } from "mongoose";
|
||||
import Repository from "../core/Repository";
|
||||
import config from "../config";
|
||||
@@ -20,6 +22,7 @@ export const database = mongoose.connection;
|
||||
export let isConnected = false;
|
||||
|
||||
export async function connect() {
|
||||
credentialCipher(); // Refuse to serve persisted credentials without a valid keyring.
|
||||
mongoose.set("strictQuery", false);
|
||||
const options: ConnectOptions = {
|
||||
appName: "Anonymous GitHub Server",
|
||||
@@ -27,6 +30,7 @@ export async function connect() {
|
||||
};
|
||||
if (!config.MONGODB_URI) options.authSource = "admin";
|
||||
await mongoose.connect(getMongoUrl(), options);
|
||||
await CredentialModel.createIndexes();
|
||||
isConnected = true;
|
||||
|
||||
return database;
|
||||
|
||||
@@ -10,7 +10,7 @@ import config from "../../config";
|
||||
import UserModel from "../../core/model/users/users.model";
|
||||
import { IUserDocument } from "../../core/model/users/users.types";
|
||||
import AnonymousError from "../../core/AnonymousError";
|
||||
import AnonymizedPullRequestModel from "../../core/model/anonymizedPullRequests/anonymizedPullRequests.model";
|
||||
import { setCredential } from "../../core/credentials";
|
||||
import { hashToken } from "./token-auth";
|
||||
import { createLogger, serializeError } from "../../core/logger";
|
||||
import { getLoginToken, isDisabledAccount } from "./auth-utils";
|
||||
@@ -30,13 +30,12 @@ export function ensureAuthenticated(
|
||||
|
||||
const verify = async (
|
||||
accessToken: string,
|
||||
refreshToken: string,
|
||||
_refreshToken: string,
|
||||
profile: Profile,
|
||||
done: OAuth2Strategy.VerifyCallback
|
||||
): Promise<void> => {
|
||||
let user: IUserDocument | null;
|
||||
try {
|
||||
const now = new Date();
|
||||
user = await UserModel.findOne({ "externalIDs.github": profile.id });
|
||||
if (user) {
|
||||
if (isDisabledAccount(user.status)) {
|
||||
@@ -48,20 +47,6 @@ const verify = async (
|
||||
);
|
||||
return;
|
||||
}
|
||||
await UserModel.updateOne(
|
||||
{ _id: user._id },
|
||||
{
|
||||
$set: {
|
||||
"accessTokens.github": accessToken,
|
||||
"accessTokenDates.github": now,
|
||||
},
|
||||
}
|
||||
);
|
||||
await AnonymizedPullRequestModel.updateMany(
|
||||
{ owner: user._id },
|
||||
{ "source.accessToken": accessToken }
|
||||
);
|
||||
user = await UserModel.findById(user._id);
|
||||
} else {
|
||||
// Check if a user with this username already exists (e.g. created
|
||||
// manually without externalIDs.github). Link the GitHub ID to the
|
||||
@@ -87,8 +72,6 @@ const verify = async (
|
||||
{
|
||||
$set: {
|
||||
"externalIDs.github": profile.id,
|
||||
"accessTokens.github": accessToken,
|
||||
"accessTokenDates.github": now,
|
||||
},
|
||||
}
|
||||
);
|
||||
@@ -97,12 +80,6 @@ const verify = async (
|
||||
const photo = profile.photos ? profile.photos[0]?.value : null;
|
||||
user = new UserModel({
|
||||
username: profile.username,
|
||||
accessTokens: {
|
||||
github: accessToken,
|
||||
},
|
||||
accessTokenDates: {
|
||||
github: now,
|
||||
},
|
||||
externalIDs: {
|
||||
github: profile.id,
|
||||
},
|
||||
@@ -126,13 +103,8 @@ const verify = async (
|
||||
);
|
||||
return;
|
||||
}
|
||||
done(null, {
|
||||
username: profile.username,
|
||||
accessToken,
|
||||
refreshToken,
|
||||
profile,
|
||||
user,
|
||||
});
|
||||
await setCredential(String(user!._id), accessToken);
|
||||
done(null, { username: user!.username, user });
|
||||
} catch (error) {
|
||||
logger.error("verify failed", serializeError(error));
|
||||
done(
|
||||
@@ -157,11 +129,20 @@ passport.use(
|
||||
);
|
||||
|
||||
passport.serializeUser((user: Express.User, done) => {
|
||||
done(null, user);
|
||||
const id = (user as { user?: { _id?: unknown } }).user?._id;
|
||||
done(null, String(id));
|
||||
});
|
||||
|
||||
passport.deserializeUser((user: Express.User, done) => {
|
||||
done(null, user);
|
||||
passport.deserializeUser(async (id: string, done) => {
|
||||
// Reject the old session format, which included plaintext credentials.
|
||||
if (typeof id !== "string" || !/^[a-f0-9]{24}$/i.test(id)) return done(null, false);
|
||||
try {
|
||||
const user = await UserModel.findById(id);
|
||||
if (!user || isDisabledAccount(user.status)) return done(null, false);
|
||||
done(null, { username: user.username, user });
|
||||
} catch {
|
||||
done(new Error("Session user lookup failed"));
|
||||
}
|
||||
});
|
||||
|
||||
export function initSession() {
|
||||
@@ -229,7 +210,6 @@ router.post(
|
||||
}
|
||||
const synthUser = {
|
||||
username: model.username,
|
||||
accessToken: model.accessTokens?.github,
|
||||
profile: undefined,
|
||||
user: model,
|
||||
};
|
||||
|
||||
@@ -249,7 +249,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
||||
gist.model.owner = user.id;
|
||||
|
||||
updateGistModel(gist.model, gistUpdate);
|
||||
gist.source.accessToken = user.accessToken;
|
||||
gist.source.gistId = gistUpdate.source.gistId;
|
||||
|
||||
gist.model.conference = gistUpdate.conference;
|
||||
|
||||
@@ -266,7 +266,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
||||
pullRequest.model.owner = user.id;
|
||||
|
||||
updatePullRequestModel(pullRequest.model, pullRequestUpdate);
|
||||
pullRequest.source.accessToken = user.accessToken;
|
||||
pullRequest.source.pullRequestId = pullRequestUpdate.source.pullRequestId;
|
||||
pullRequest.source.repositoryFullName =
|
||||
pullRequestUpdate.source.repositoryFullName;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { getCredentialToken } from "../../core/credentials";
|
||||
import * as express from "express";
|
||||
import { ensureAuthenticated } from "./connection";
|
||||
|
||||
@@ -14,14 +15,12 @@ import { getRepositoryFromGitHub } from "../../core/source/GitHubRepository";
|
||||
import gh = require("parse-github-url");
|
||||
import AnonymizedRepositoryModel from "../../core/model/anonymizedRepositories/anonymizedRepositories.model";
|
||||
import { IAnonymizedRepositoryDocument } from "../../core/model/anonymizedRepositories/anonymizedRepositories.types";
|
||||
import UserModel from "../../core/model/users/users.model";
|
||||
import ConferenceModel from "../../core/model/conference/conferences.model";
|
||||
import AnonymousError from "../../core/AnonymousError";
|
||||
import { addRemovalJob, downloadQueue } from "../../queue";
|
||||
import RepositoryModel from "../../core/model/repositories/repositories.model";
|
||||
import User from "../../core/User";
|
||||
import { RepositoryStatus } from "../../core/types";
|
||||
import { IUserDocument } from "../../core/model/users/users.types";
|
||||
import { checkToken, octokit, getRedisGateResetAt, getToken } from "../../core/GitHubUtils";
|
||||
import { createLogger, serializeError } from "../../core/logger";
|
||||
|
||||
@@ -40,23 +39,14 @@ async function getTokenForAdmin(user: User, req: express.Request) {
|
||||
"source.repositoryName": `${req.params.owner}/${req.params.repo}`,
|
||||
},
|
||||
{
|
||||
"source.accessToken": 1,
|
||||
owner: 1,
|
||||
}
|
||||
).populate({
|
||||
path: "owner",
|
||||
model: UserModel,
|
||||
});
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const user: IUserDocument = existingRepo?.owner as any;
|
||||
if (user instanceof UserModel) {
|
||||
const check = await checkToken(user.accessTokens.github);
|
||||
if (check) {
|
||||
return user.accessTokens.github;
|
||||
}
|
||||
}
|
||||
if (existingRepo) {
|
||||
return existingRepo.source.accessToken;
|
||||
);
|
||||
if (existingRepo?.owner) {
|
||||
const token = await getCredentialToken(String(existingRepo.owner), "github", {
|
||||
collection: "anonymizedrepositories", id: existingRepo._id,
|
||||
});
|
||||
if (token && await checkToken(token)) return token;
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn("getToken lookup failed", serializeError(error));
|
||||
@@ -100,7 +90,7 @@ router.post("/claim", async (req: express.Request, res: express.Response) => {
|
||||
owner: r.owner,
|
||||
repo: r.name,
|
||||
repositoryID: req.query.repositoryID as string,
|
||||
accessToken: user.accessToken,
|
||||
accessToken: await user.getAccessToken(),
|
||||
});
|
||||
if (!repo) {
|
||||
throw new AnonymousError("repo_not_found", {
|
||||
@@ -256,7 +246,7 @@ router.get(
|
||||
async (req: express.Request, res: express.Response) => {
|
||||
try {
|
||||
const user = await getUser(req);
|
||||
let token = user.accessToken;
|
||||
let token = await user.getAccessToken();
|
||||
if (user.isAdmin) {
|
||||
token = (await getTokenForAdmin(user, req)) || token;
|
||||
}
|
||||
@@ -279,7 +269,7 @@ router.get(
|
||||
async (req: express.Request, res: express.Response) => {
|
||||
try {
|
||||
const user = await getUser(req);
|
||||
let token = user.accessToken;
|
||||
let token = await user.getAccessToken();
|
||||
if (user.isAdmin) {
|
||||
token = (await getTokenForAdmin(user, req)) || token;
|
||||
}
|
||||
@@ -307,7 +297,7 @@ router.get(
|
||||
async (req: express.Request, res: express.Response) => {
|
||||
try {
|
||||
const user = await getUser(req);
|
||||
let token = user.accessToken;
|
||||
let token = await user.getAccessToken();
|
||||
if (user.isAdmin) {
|
||||
token = (await getTokenForAdmin(user, req)) || token;
|
||||
}
|
||||
@@ -521,7 +511,7 @@ router.post(
|
||||
});
|
||||
}
|
||||
const repository = await getRepositoryFromGitHub({
|
||||
accessToken: user.accessToken,
|
||||
accessToken: await user.getAccessToken(),
|
||||
owner: parsedRepository.owner,
|
||||
repo: parsedRepository.name,
|
||||
});
|
||||
@@ -533,7 +523,7 @@ router.post(
|
||||
});
|
||||
}
|
||||
await repository.getCommitInfo(repoUpdate.source.commit, {
|
||||
accessToken: user.accessToken,
|
||||
accessToken: await user.getAccessToken(),
|
||||
});
|
||||
repo.model.source.repositoryId = repository.model.id;
|
||||
repo.model.source.repositoryName =
|
||||
@@ -648,7 +638,7 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
||||
});
|
||||
}
|
||||
const repository = await getRepositoryFromGitHub({
|
||||
accessToken: user.accessToken,
|
||||
accessToken: await user.getAccessToken(),
|
||||
owner: r.owner,
|
||||
repo: r.name,
|
||||
});
|
||||
@@ -661,7 +651,7 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
||||
}
|
||||
|
||||
await repository.getCommitInfo(repoUpdate.source.commit, {
|
||||
accessToken: user.accessToken,
|
||||
accessToken: await user.getAccessToken(),
|
||||
});
|
||||
|
||||
const repo = new AnonymizedRepositoryModel();
|
||||
@@ -671,7 +661,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
||||
|
||||
updateRepoModel(repo, repoUpdate);
|
||||
repo.source.type = "GitHubStream";
|
||||
repo.source.accessToken = user.accessToken;
|
||||
repo.source.repositoryId = repository.model.id;
|
||||
repo.source.repositoryName = repoUpdate.fullName;
|
||||
|
||||
@@ -762,7 +751,7 @@ router.post(
|
||||
}
|
||||
|
||||
// verify the GitHub user exists and capture identity fields
|
||||
const oct = octokit(user.accessToken);
|
||||
const oct = octokit(await user.getAccessToken());
|
||||
let ghUser;
|
||||
try {
|
||||
const r = await oct.users.getByUsername({ username });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import CredentialModel from "../../core/model/credentials/credentials.model";
|
||||
import * as express from "express";
|
||||
import got from "got";
|
||||
import config from "../../config";
|
||||
@@ -177,15 +178,15 @@ router.delete("/", async (req: express.Request, res: express.Response) => {
|
||||
await Promise.all([
|
||||
AnonymizedRepositoryModel.updateMany(
|
||||
{ owner: user.model._id },
|
||||
{ $unset: { "source.accessToken": "" } }
|
||||
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||
).exec(),
|
||||
AnonymizedPullRequestModel.updateMany(
|
||||
{ owner: user.model._id },
|
||||
{ $unset: { "source.accessToken": "" } }
|
||||
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||
).exec(),
|
||||
AnonymizedGistModel.updateMany(
|
||||
{ owner: user.model._id },
|
||||
{ $unset: { "source.accessToken": "" } }
|
||||
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||
).exec(),
|
||||
]);
|
||||
|
||||
@@ -199,13 +200,15 @@ router.delete("/", async (req: express.Request, res: express.Response) => {
|
||||
username: config.CLIENT_ID,
|
||||
password: config.CLIENT_SECRET,
|
||||
headers: { accept: "application/vnd.github+json" },
|
||||
json: { access_token: user.accessToken },
|
||||
json: { access_token: await user.getAccessToken() },
|
||||
}
|
||||
);
|
||||
} catch (error) {
|
||||
logger.warn("oauth grant revocation failed", serializeError(error));
|
||||
}
|
||||
|
||||
await CredentialModel.deleteMany({ ownerId: user.model._id });
|
||||
|
||||
await UserModel.updateOne(
|
||||
{ _id: user.model._id },
|
||||
{
|
||||
@@ -296,7 +299,7 @@ router.get(
|
||||
if (!q || q.length < 2) {
|
||||
return res.json([]);
|
||||
}
|
||||
const oct = octokit(user.accessToken);
|
||||
const oct = octokit(await user.getAccessToken());
|
||||
const r = await oct.search.users({ q, per_page: 10 });
|
||||
res.json(
|
||||
r.data.items.map((u) => ({
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
const { expect } = require("chai");
|
||||
require("ts-node/register/transpile-only");
|
||||
const { createTokenCipher } = require("../src/core/credential-crypto");
|
||||
const { redactSecrets } = require("../src/core/redact-secrets");
|
||||
const keys = JSON.stringify({ old: Buffer.alloc(32, 1).toString("base64"), next: Buffer.alloc(32, 2).toString("base64") });
|
||||
|
||||
describe("credential encryption", () => {
|
||||
const cipher = createTokenCipher(keys, "old");
|
||||
it("round trips with independent nonces and no plaintext in the envelope", () => {
|
||||
const a = cipher.encrypt("secret-token", "owner", "github");
|
||||
const b = cipher.encrypt("secret-token", "owner", "github");
|
||||
expect(a.nonce).not.to.equal(b.nonce);
|
||||
expect(JSON.stringify(a)).not.to.include("secret-token");
|
||||
expect(cipher.decrypt(a, "owner", "github")).to.equal("secret-token");
|
||||
});
|
||||
it("rejects altered envelopes and a different owner/provider", () => {
|
||||
const a = cipher.encrypt("secret-token", "owner", "github");
|
||||
for (const field of ["nonce", "tag", "ciphertext"]) {
|
||||
const bytes = Buffer.from(a[field], "base64"); bytes[0] ^= 1;
|
||||
expect(() => cipher.decrypt({ ...a, [field]: bytes.toString("base64") }, "owner", "github")).to.throw("Credential decryption failed");
|
||||
}
|
||||
for (const patch of [{ version: 2 }, { keyId: "missing" }, { tag: "YQ==" }, { nonce: "!!!!" }]) {
|
||||
expect(() => cipher.decrypt({ ...a, ...patch }, "owner", "github")).to.throw();
|
||||
}
|
||||
expect(() => cipher.decrypt(a, "someone-else", "github")).to.throw();
|
||||
expect(() => cipher.decrypt(a, "owner", "other")).to.throw();
|
||||
const wrong = createTokenCipher(JSON.stringify({ old: Buffer.alloc(32, 3).toString("base64") }), "old");
|
||||
expect(() => wrong.decrypt(a, "owner", "github")).to.throw();
|
||||
});
|
||||
it("supports key rotation while retaining reads of old credentials", () => {
|
||||
const rotated = createTokenCipher(keys, "next");
|
||||
expect(rotated.decrypt(cipher.encrypt("secret", "owner", "github"), "owner", "github")).to.equal("secret");
|
||||
expect(rotated.encrypt("secret", "owner", "github").keyId).to.equal("next");
|
||||
});
|
||||
it("refuses missing, malformed, and short keys", () => {
|
||||
for (const raw of ["", "null", "[]", "{}", '{"old":"YQ=="}']) {
|
||||
expect(() => createTokenCipher(raw, "old")).to.throw();
|
||||
}
|
||||
});
|
||||
it("redacts nested credentials, authorization and tokens in URLs/errors", () => {
|
||||
const input = { accessTokens: { github: "plain" }, nested: { authorization: "Bearer plain", encryptedToken: { ciphertext: "abc" } }, message: "failed ghp_abcdef https://example.test/?token=plain" };
|
||||
const result = JSON.stringify(redactSecrets(input));
|
||||
expect(result).not.to.include("plain");
|
||||
expect(result).not.to.include("ghp_abcdef");
|
||||
expect(result).not.to.include("abc");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,214 @@
|
||||
const process = require("process");
|
||||
const { expect } = require("chai");
|
||||
require("ts-node/register/transpile-only");
|
||||
const mongoose = require("mongoose");
|
||||
const { migrateCredentials, verifyCredentials, enforceCredentialStorage } = require("../src/core/migrate-credentials");
|
||||
const { createTokenCipher } = require("../src/core/credential-crypto");
|
||||
|
||||
// Always use a newly named disposable database, never the database in the URI.
|
||||
const describeMongo = process.env.TEST_MONGODB_URI ? describe : describe.skip;
|
||||
describeMongo("credential migration (MongoDB)", function () {
|
||||
this.timeout(20000);
|
||||
let client, db, owner;
|
||||
const cipher = createTokenCipher(JSON.stringify({ test: Buffer.alloc(32, 7).toString("base64") }), "test");
|
||||
const apply = { apply: true, removeLegacy: true };
|
||||
before(async () => {
|
||||
client = new mongoose.mongo.MongoClient(process.env.TEST_MONGODB_URI);
|
||||
await client.connect();
|
||||
db = client.db(`credential_test_${new mongoose.Types.ObjectId()}`);
|
||||
});
|
||||
beforeEach(async () => {
|
||||
await db.dropDatabase();
|
||||
owner = new mongoose.Types.ObjectId();
|
||||
await db.collection("users").insertOne({ _id: owner, username: "test", accessTokens: { github: "secret" } });
|
||||
});
|
||||
after(async () => { if (db) await db.dropDatabase(); if (client) await client.close(); });
|
||||
it("dry run does not modify data", async () => {
|
||||
const result = await migrateCredentials(db, cipher);
|
||||
expect(result.created).to.equal(1);
|
||||
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||
});
|
||||
it("encrypts once, cleans all legacy locations, and is safe to rerun", async () => {
|
||||
for (const name of ["anonymizedrepositories", "anonymizedgists", "anonymizedpullrequests"]) {
|
||||
await db.collection(name).insertOne({ owner, source: { accessToken: "secret" }, accessToken: "secret" });
|
||||
}
|
||||
const first = await migrateCredentials(db, cipher, apply);
|
||||
expect(first.issues).to.equal(0);
|
||||
const credential = await db.collection("credentials").findOne({ ownerId: owner });
|
||||
expect(JSON.stringify(credential)).not.to.include("secret");
|
||||
expect(cipher.decrypt(credential.encryptedToken, String(owner), "github")).to.equal("secret");
|
||||
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||
expect((await migrateCredentials(db, cipher, apply)).created).to.equal(0);
|
||||
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(credential.encryptedToken);
|
||||
});
|
||||
it("reports resource conflicts without deleting either token", async () => {
|
||||
await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "different" } });
|
||||
const events = [];
|
||||
const result = await migrateCredentials(db, cipher, { ...apply, report: e => events.push(e) });
|
||||
expect(result.issues).to.equal(1);
|
||||
expect(JSON.stringify(events)).not.to.include("different");
|
||||
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||
expect((await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true })).issues).to.equal(0);
|
||||
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||
});
|
||||
it("preserves a newer credential and requires explicit conflict resolution", async () => {
|
||||
const envelope = cipher.encrypt("new-login", String(owner), "github");
|
||||
await db.collection("credentials").insertOne({ ownerId: owner, provider: "github", encryptedToken: envelope, updatedAt: new Date() });
|
||||
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(1);
|
||||
await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true });
|
||||
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(envelope);
|
||||
});
|
||||
it("migrates resource-only credentials but never arbitrarily picks between them", async () => {
|
||||
await db.collection("users").updateOne({ _id: owner }, { $unset: { accessTokens: "" } });
|
||||
await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "resource" } });
|
||||
await db.collection("anonymizedpullrequests").insertOne({ owner, source: { accessToken: "other" } });
|
||||
expect((await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true })).issues).to.equal(1);
|
||||
await db.collection("anonymizedpullrequests").deleteMany({});
|
||||
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(0);
|
||||
const row = await db.collection("credentials").findOne({ ownerId: owner });
|
||||
expect(cipher.decrypt(row.encryptedToken, String(owner), "github")).to.equal("resource");
|
||||
});
|
||||
it("reports missing owners and malformed tokens", async () => {
|
||||
await db.collection("anonymizedrepositories").insertOne({ source: { accessToken: "orphan" } });
|
||||
await db.collection("users").updateOne({ _id: owner }, { $set: { "accessTokens.github": { invalid: true } } });
|
||||
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(2);
|
||||
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||
});
|
||||
it("does not recreate credentials for removed accounts", async () => {
|
||||
await db.collection("users").updateOne({ _id: owner }, { $set: { status: "removed" } });
|
||||
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(0);
|
||||
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 0, legacy: 0 });
|
||||
});
|
||||
it("resumes after interruption between backfill and cleanup", async () => {
|
||||
await migrateCredentials(db, cipher, { apply: true });
|
||||
const envelope = (await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken;
|
||||
expect((await verifyCredentials(db, cipher)).legacy).to.equal(1);
|
||||
await migrateCredentials(db, cipher, apply);
|
||||
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(envelope);
|
||||
});
|
||||
it("refuses cleanup when ciphertext cannot be authenticated", async () => {
|
||||
await migrateCredentials(db, cipher, { apply: true });
|
||||
await db.collection("credentials").updateOne({ ownerId: owner }, { $set: { "encryptedToken.tag": Buffer.alloc(16).toString("base64") } });
|
||||
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(1);
|
||||
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||
});
|
||||
it("enforces the unique owner/provider index", async () => {
|
||||
await migrateCredentials(db, cipher, apply);
|
||||
const row = await db.collection("credentials").findOne({ ownerId: owner });
|
||||
delete row._id;
|
||||
try { await db.collection("credentials").insertOne(row); throw new Error("expected duplicate"); }
|
||||
catch (error) { expect(error.code).to.equal(11000); }
|
||||
});
|
||||
it("enforces plaintext rejection in MongoDB after verification", async () => {
|
||||
await migrateCredentials(db, cipher, apply);
|
||||
await enforceCredentialStorage(db, cipher);
|
||||
try { await db.collection("users").updateOne({ _id: owner }, { $set: { "accessTokens.github": "bad" } }); throw new Error("expected rejection"); }
|
||||
catch (error) { expect(error.code).to.equal(121); }
|
||||
try { await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "bad" } }); throw new Error("expected rejection"); }
|
||||
catch (error) { expect(error.code).to.equal(121); }
|
||||
});
|
||||
});
|
||||
|
||||
describeMongo("credential access (MongoDB)", function () {
|
||||
this.timeout(20000);
|
||||
const config = require("../src/config").default;
|
||||
const Credential = require("../src/core/model/credentials/credentials.model").default;
|
||||
const UserModel = require("../src/core/model/users/users.model").default;
|
||||
const RepoModel = require("../src/core/model/anonymizedRepositories/anonymizedRepositories.model").default;
|
||||
const GistModel = require("../src/core/model/anonymizedGists/anonymizedGists.model").default;
|
||||
const PullModel = require("../src/core/model/anonymizedPullRequests/anonymizedPullRequests.model").default;
|
||||
const { setCredential, getCredentialToken, replaceCredential } = require("../src/core/credentials");
|
||||
let settings, owner;
|
||||
before(async () => {
|
||||
settings = [config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS];
|
||||
config.CREDENTIAL_KEYS = JSON.stringify({ test: Buffer.alloc(32, 9).toString("base64") });
|
||||
config.CREDENTIAL_ACTIVE_KEY_ID = "test";
|
||||
config.CREDENTIAL_LEGACY_READS = false;
|
||||
await mongoose.connect(process.env.TEST_MONGODB_URI, { dbName: `credential_access_test_${new mongoose.Types.ObjectId()}` });
|
||||
await Credential.init();
|
||||
});
|
||||
beforeEach(async () => {
|
||||
await Credential.deleteMany({});
|
||||
owner = await UserModel.create({ username: `owner-${new mongoose.Types.ObjectId()}` });
|
||||
});
|
||||
after(async () => {
|
||||
await mongoose.connection.dropDatabase();
|
||||
await mongoose.disconnect();
|
||||
[config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS] = settings;
|
||||
});
|
||||
it("persists only encrypted credentials and hides envelopes in ordinary queries", async () => {
|
||||
await setCredential(owner.id, "real-secret");
|
||||
const raw = await Credential.collection.findOne({ ownerId: owner._id });
|
||||
expect(raw.ownerId.equals(owner._id)).to.equal(true);
|
||||
expect(JSON.stringify(raw)).not.to.include("real-secret");
|
||||
expect((await Credential.findOne({ ownerId: owner._id })).encryptedToken).to.equal(undefined);
|
||||
expect(await getCredentialToken(owner.id)).to.equal("real-secret");
|
||||
expect((await UserModel.collection.findOne({ _id: owner._id })).accessTokens).to.equal(undefined);
|
||||
});
|
||||
it("handles simultaneous logins with one owner/provider row", async () => {
|
||||
await Promise.all(Array.from({ length: 8 }, (_, i) => setCredential(owner.id, `token-${i}`)));
|
||||
expect(await Credential.countDocuments({ ownerId: owner._id })).to.equal(1);
|
||||
expect(await getCredentialToken(owner.id)).to.match(/^token-\d$/);
|
||||
});
|
||||
it("updates the encrypted envelope atomically on refresh", async () => {
|
||||
await setCredential(owner.id, "previous");
|
||||
expect(await replaceCredential(owner.id, "previous", "next")).to.equal(true);
|
||||
expect(await replaceCredential(owner.id, "previous", "stale")).to.equal(false);
|
||||
expect(await getCredentialToken(owner.id)).to.equal("next");
|
||||
});
|
||||
it("resolves gists and pull requests by owner and observes token changes", async () => {
|
||||
const Gist = require("../src/core/Gist").default;
|
||||
const PullRequest = require("../src/core/PullRequest").default;
|
||||
const gist = new Gist(await GistModel.create({ owner: owner._id, source: { gistId: "test" } }));
|
||||
const pull = new PullRequest(await PullModel.create({ owner: owner._id, source: { pullRequestId: "1" } }));
|
||||
await setCredential(owner.id, "first");
|
||||
expect(await gist.getToken()).to.equal("first");
|
||||
expect(await pull.getToken()).to.equal("first");
|
||||
await setCredential(owner.id, "second");
|
||||
expect(await gist.getToken()).to.equal("second");
|
||||
expect(await pull.getToken()).to.equal("second");
|
||||
expect((await GistModel.collection.findOne({ _id: gist.model._id })).source.accessToken).to.equal(undefined);
|
||||
expect((await PullModel.collection.findOne({ _id: pull.model._id })).source.accessToken).to.equal(undefined);
|
||||
});
|
||||
it("refreshes a repository credential without copying it onto the repository", async () => {
|
||||
const Repository = require("../src/core/Repository").default;
|
||||
const repo = new Repository(await RepoModel.create({ owner: owner._id, repoId: "test-repo", source: { type: "GitHubStream" } }));
|
||||
await setCredential(owner.id, "old-token");
|
||||
await Credential.updateOne({ ownerId: owner._id }, { $set: { updatedAt: new Date(0) } });
|
||||
const originalFetch = global.fetch;
|
||||
global.fetch = async (url, options) => {
|
||||
expect(url).to.include("api.github.com/applications/");
|
||||
expect(JSON.parse(options.body).access_token).to.equal("old-token");
|
||||
return { ok: true, json: async () => ({ token: "refreshed-token" }) };
|
||||
};
|
||||
try { expect(await repo.getToken()).to.equal("refreshed-token"); }
|
||||
finally { global.fetch = originalFetch; }
|
||||
expect(await getCredentialToken(owner.id)).to.equal("refreshed-token");
|
||||
expect((await RepoModel.collection.findOne({ _id: repo.model._id })).source.accessToken).to.equal(undefined);
|
||||
});
|
||||
it("OAuth login writes a credential and returns a token-free session user", async () => {
|
||||
const passport = require("passport");
|
||||
require("../src/server/routes/connection");
|
||||
const result = await new Promise((resolve, reject) => passport._strategy("github")._verify("oauth-secret", "refresh-secret", {
|
||||
id: "external-test", username: owner.username, emails: [], photos: [],
|
||||
}, (error, user) => error ? reject(error) : resolve(user)));
|
||||
expect(JSON.stringify(result)).not.to.include("oauth-secret");
|
||||
expect(JSON.stringify(result)).not.to.include("refresh-secret");
|
||||
expect(await getCredentialToken(owner.id)).to.equal("oauth-secret");
|
||||
expect((await UserModel.collection.findOne({ _id: owner._id })).accessTokens).to.equal(undefined);
|
||||
});
|
||||
it("reads hidden legacy resource tokens only during compatibility mode", async () => {
|
||||
const resource = await GistModel.collection.insertOne({ gistId: "legacy-gist", owner: owner._id, source: { accessToken: "legacy-resource" } });
|
||||
const lookup = { collection: "anonymizedgists", id: resource.insertedId };
|
||||
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("");
|
||||
config.CREDENTIAL_LEGACY_READS = true;
|
||||
try {
|
||||
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("legacy-resource");
|
||||
await setCredential(owner.id, "encrypted-wins");
|
||||
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("encrypted-wins");
|
||||
} finally { config.CREDENTIAL_LEGACY_READS = false; }
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,21 @@
|
||||
const { expect } = require("chai");
|
||||
require("ts-node/register/transpile-only");
|
||||
const passport = require("passport");
|
||||
require("../src/server/routes/connection");
|
||||
|
||||
describe("credential-free sessions", () => {
|
||||
it("serializes only the owner ID", done => {
|
||||
passport.serializeUser({ user: { _id: "507f1f77bcf86cd799439011", accessTokens: { github: "secret" } }, accessToken: "secret" }, (error, value) => {
|
||||
expect(error).to.equal(null);
|
||||
expect(value).to.equal("507f1f77bcf86cd799439011");
|
||||
done();
|
||||
});
|
||||
});
|
||||
it("rejects the legacy session object", done => {
|
||||
passport.deserializeUser({ user: { _id: "507f1f77bcf86cd799439011" }, accessToken: "secret" }, (error, value) => {
|
||||
expect(error).to.equal(null);
|
||||
expect(value).to.equal(false);
|
||||
done();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
const { expect } = require("chai");
|
||||
require("ts-node/register/transpile-only");
|
||||
const { Types } = require("mongoose");
|
||||
const config = require("../src/config").default;
|
||||
const Model = require("../src/core/model/credentials/credentials.model").default;
|
||||
const UserModel = require("../src/core/model/users/users.model").default;
|
||||
const { getCredentialToken, setCredential, replaceCredential, credentialCipher } = require("../src/core/credentials");
|
||||
|
||||
describe("credential storage boundary", () => {
|
||||
const owner = new Types.ObjectId().toString();
|
||||
let original, settings, stored;
|
||||
beforeEach(() => {
|
||||
original = { findOne: Model.findOne, updateOne: Model.updateOne, findById: UserModel.findById };
|
||||
settings = [config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS];
|
||||
config.CREDENTIAL_KEYS = JSON.stringify({ test: Buffer.alloc(32, 4).toString("base64") });
|
||||
config.CREDENTIAL_ACTIVE_KEY_ID = "test";
|
||||
config.CREDENTIAL_LEGACY_READS = false;
|
||||
stored = null;
|
||||
Model.findOne = () => ({ select: () => ({ lean: async () => stored }) });
|
||||
Model.updateOne = async (filter, update) => {
|
||||
stored = { _id: new Types.ObjectId(), ownerId: owner, provider: "github", ...update.$set };
|
||||
return { modifiedCount: 1 };
|
||||
};
|
||||
});
|
||||
afterEach(() => {
|
||||
Object.assign(Model, { findOne: original.findOne, updateOne: original.updateOne });
|
||||
UserModel.findById = original.findById;
|
||||
[config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS] = settings;
|
||||
});
|
||||
it("writes only ciphertext and resolves the token by owner/provider", async () => {
|
||||
await setCredential(owner, "github-secret");
|
||||
expect(JSON.stringify(stored)).not.to.include("github-secret");
|
||||
expect(await getCredentialToken(owner)).to.equal("github-secret");
|
||||
});
|
||||
it("refreshes only the credential whose token still matches", async () => {
|
||||
await setCredential(owner, "new-login-token");
|
||||
expect(await replaceCredential(owner, "stale-token", "refresh-token")).to.equal(false);
|
||||
expect(await getCredentialToken(owner)).to.equal("new-login-token");
|
||||
expect(await replaceCredential(owner, "new-login-token", "refresh-token")).to.equal(true);
|
||||
expect(await getCredentialToken(owner)).to.equal("refresh-token");
|
||||
});
|
||||
it("fails closed on corrupt ciphertext even with legacy reads enabled", async () => {
|
||||
config.CREDENTIAL_LEGACY_READS = true;
|
||||
await setCredential(owner, "github-secret");
|
||||
stored.encryptedToken.tag = Buffer.alloc(16).toString("base64");
|
||||
UserModel.findById = () => { throw new Error("must not fall back"); };
|
||||
try { await getCredentialToken(owner); throw new Error("expected failure"); }
|
||||
catch (error) { expect(error.message).to.equal("Credential decryption failed"); }
|
||||
});
|
||||
it("reads legacy users only when explicitly enabled", async () => {
|
||||
let reads = 0;
|
||||
UserModel.findById = () => ({ select: async () => { reads++; return { accessTokens: { github: "legacy" } }; } });
|
||||
expect(await getCredentialToken(owner)).to.equal("");
|
||||
expect(reads).to.equal(0);
|
||||
config.CREDENTIAL_LEGACY_READS = true;
|
||||
expect(await getCredentialToken(owner)).to.equal("legacy");
|
||||
});
|
||||
it("uses a unique owner/provider index and hides envelopes by default", () => {
|
||||
expect(Model.schema.indexes()).to.deep.include([{ ownerId: 1, provider: 1 }, { unique: true, background: true }]);
|
||||
expect(Model.schema.path("encryptedToken").options.select).to.equal(false);
|
||||
expect(credentialCipher()).to.have.property("encrypt");
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user