mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-09-15 23:25:27 +02:00
fix: encrypt stored GitHub credentials (#797)
This commit is contained in:
@@ -61,6 +61,9 @@ npm i
|
|||||||
GITHUB_TOKEN=<GITHUB_TOKEN>
|
GITHUB_TOKEN=<GITHUB_TOKEN>
|
||||||
CLIENT_ID=<CLIENT_ID>
|
CLIENT_ID=<CLIENT_ID>
|
||||||
CLIENT_SECRET=<CLIENT_SECRET>
|
CLIENT_SECRET=<CLIENT_SECRET>
|
||||||
|
CREDENTIAL_KEYS='{"2026-09":"<base64-encoded 32-byte random key>"}'
|
||||||
|
CREDENTIAL_ACTIVE_KEY_ID=2026-09
|
||||||
|
CREDENTIAL_LEGACY_READS=false
|
||||||
PORT=5000
|
PORT=5000
|
||||||
DB_USERNAME=
|
DB_USERNAME=
|
||||||
DB_PASSWORD=
|
DB_PASSWORD=
|
||||||
@@ -68,6 +71,7 @@ AUTH_CALLBACK=http://localhost:5000/github/auth
|
|||||||
```
|
```
|
||||||
|
|
||||||
- `GITHUB_TOKEN` — create one at <https://github.com/settings/tokens/new> with the `repo` scope.
|
- `GITHUB_TOKEN` — create one at <https://github.com/settings/tokens/new> with the `repo` scope.
|
||||||
|
- `CREDENTIAL_KEYS` / `CREDENTIAL_ACTIVE_KEY_ID` — generate a key with `openssl rand -base64 32`. Existing installations must follow the [credential migration guide](docs/credential-encryption.md) before starting this release.
|
||||||
- `CLIENT_ID` / `CLIENT_SECRET` — from a new GitHub App at <https://github.com/settings/applications/new>.
|
- `CLIENT_ID` / `CLIENT_SECRET` — from a new GitHub App at <https://github.com/settings/applications/new>.
|
||||||
- The App's callback must be `https://<host>/github/auth` (matching `AUTH_CALLBACK`).
|
- The App's callback must be `https://<host>/github/auth` (matching `AUTH_CALLBACK`).
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,280 @@
|
|||||||
|
# GitHub credential storage and migration
|
||||||
|
|
||||||
|
GitHub tokens live in `credentials`, with a unique `(ownerId, provider)` index.
|
||||||
|
Repositories, gists, and pull requests use their existing `owner` field to resolve
|
||||||
|
credentials. Users and resources have no credential reference fields. API login
|
||||||
|
tokens remain hashed in `users.apiTokens`.
|
||||||
|
|
||||||
|
Each credential contains `ownerId`, `provider`, `updatedAt`, and
|
||||||
|
`encryptedToken: { version, keyId, nonce, ciphertext, tag }`. AES-256-GCM uses a
|
||||||
|
fresh 12-byte nonce and a 16-byte authentication tag. Additional authenticated
|
||||||
|
data binds the ciphertext to the owner, provider, field, and format version.
|
||||||
|
Plaintext exists only in application memory and requests to GitHub or the internal
|
||||||
|
streamer. Use TLS for transport across untrusted networks.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Set these secrets on the API, workers, and migration process:
|
||||||
|
|
||||||
|
```dotenv
|
||||||
|
CREDENTIAL_KEYS='{"2026-09":"<base64-encoded 32-byte random key>"}'
|
||||||
|
CREDENTIAL_ACTIVE_KEY_ID=2026-09
|
||||||
|
CREDENTIAL_LEGACY_READS=false
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate the key with `openssl rand -base64 32` and store it in your deployment's
|
||||||
|
secret manager or protected environment file. Keep it out of Git, MongoDB, logs,
|
||||||
|
and database backups. Keep a separately protected recovery copy. The application
|
||||||
|
validates the keyring before connecting to the database; there is no temporary or
|
||||||
|
default encryption key, including in development. The streamer consumes tokens
|
||||||
|
from the API and does not need the key unless it also accesses the database.
|
||||||
|
|
||||||
|
`CREDENTIAL_LEGACY_READS=true` temporarily permits reads from legacy user/resource
|
||||||
|
fields when a credential does not exist. All new writes are encrypted regardless
|
||||||
|
of this setting. A corrupt envelope or missing decryption key fails explicitly;
|
||||||
|
it never falls back to a plaintext token. Legacy tokens are not automatically
|
||||||
|
refreshed: migrate them or log in to create an encrypted credential first.
|
||||||
|
|
||||||
|
## Initial deployment
|
||||||
|
|
||||||
|
This release requires a maintenance window. Do not run old and new writers against
|
||||||
|
the same database during migration. The `--maintenance` flag records the operator's
|
||||||
|
acknowledgement; it does not stop processes or acquire a distributed lock.
|
||||||
|
|
||||||
|
1. Build the release and provision the keyring. Back up MongoDB and verify that
|
||||||
|
the database and separately stored keys can be recovered. Rehearse on a
|
||||||
|
protected copy first.
|
||||||
|
2. Stop all API instances, queue workers, scheduled tasks, and other database
|
||||||
|
writers. Keep MongoDB and Redis running. Do not use the normal rolling deploy
|
||||||
|
script for this initial upgrade. Run the migration as a one-off process using
|
||||||
|
the new image, the existing service environment, and database network access.
|
||||||
|
3. Run the read-only inventory:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
node build/scripts/migrate-credentials.js
|
||||||
|
```
|
||||||
|
|
||||||
|
From a source checkout with development dependencies, the equivalent is
|
||||||
|
`npm run migrate:credentials -- <flags>`.
|
||||||
|
4. Resolve reported `conflicting_token`, `malformed_token`, and `missing_owner`
|
||||||
|
records. Output contains collection names, record IDs, and counts, never token
|
||||||
|
values. Conflicting owners retain all legacy fields and receive no new
|
||||||
|
credential. An existing encrypted credential is never overwritten.
|
||||||
|
|
||||||
|
If differing resource tokens are obsolete, `--prefer-owner-token` explicitly
|
||||||
|
chooses the existing encrypted credential, or otherwise the user's token, over
|
||||||
|
conflicting copies. This can remove access supplied by a different token, so
|
||||||
|
review the inventory first. It never arbitrarily chooses between distinct
|
||||||
|
resource-only tokens. Resource-only tokens migrate when they all agree and
|
||||||
|
their owner exists. Removed accounts do not receive credentials.
|
||||||
|
5. Backfill without deleting the legacy fields:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
node build/scripts/migrate-credentials.js --apply --maintenance
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `--prefer-owner-token` only for the conflict resolution described above.
|
||||||
|
Investigate every nonzero exit status. The cursor processes bounded batches;
|
||||||
|
reruns rescan owners and skip existing credentials, so no checkpoint file or
|
||||||
|
exported plaintext is needed.
|
||||||
|
6. Remove legacy copies after successful backfill:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
node build/scripts/migrate-credentials.js --apply --maintenance --remove-legacy
|
||||||
|
node build/scripts/migrate-credentials.js --verify
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the same conflict-resolution flag if needed. Every owner's envelope is
|
||||||
|
authenticated before their legacy fields are removed. Cleanup spans multiple
|
||||||
|
collections and is not a transaction; keep writers stopped. An interruption is
|
||||||
|
safe to resume with the same command. Verification must report `legacy: 0` and
|
||||||
|
exit successfully; it also authenticates all stored credentials and checks
|
||||||
|
their owners.
|
||||||
|
7. Enforce the storage boundary in MongoDB and purge old Redis sessions:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
node build/scripts/migrate-credentials.js --apply --maintenance --enforce
|
||||||
|
node build/scripts/purge-legacy-sessions.js
|
||||||
|
node build/scripts/purge-legacy-sessions.js --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Enforcement preserves existing collection validators and rejects legacy token
|
||||||
|
fields on users and resources. Session cleanup scans only `anoGH_session:*`,
|
||||||
|
removes the old object-based Passport session format, and leaves queues and
|
||||||
|
ID-only sessions intact. Rerun its dry run to confirm `found: 0`. The application
|
||||||
|
also rejects old session objects. Existing users must log in again.
|
||||||
|
8. Start only the new release with `CREDENTIAL_LEGACY_READS=false`. Test OAuth
|
||||||
|
login, API-token login, private repositories, gists, pull requests, downloads,
|
||||||
|
and account removal. Confirm startup creates the unique credential index.
|
||||||
|
|
||||||
|
A database-only dump cannot decrypt the credentials. Historical dumps, replica
|
||||||
|
oplogs, Redis snapshots/AOF files, and existing logs may still contain plaintext
|
||||||
|
until their retention expires. Migration does not erase those historical copies.
|
||||||
|
If tokens were exposed previously, revoke/reissue them; encryption cannot undo an
|
||||||
|
exposure.
|
||||||
|
|
||||||
|
## Key rotation and rollback
|
||||||
|
|
||||||
|
Add a new random key to `CREDENTIAL_KEYS` on every reader before switching
|
||||||
|
`CREDENTIAL_ACTIVE_KEY_ID`. New logins and refreshes use the active key; existing
|
||||||
|
records remain readable through their `keyId`. Retain old keys until all records
|
||||||
|
using them have been replaced and any backups needing them have expired. This
|
||||||
|
migration command does not bulk re-encrypt existing credentials.
|
||||||
|
|
||||||
|
After encrypted writes begin, rollback must stay on an encryption-capable release
|
||||||
|
with the same keyring. A pre-encryption release cannot read the new collection and
|
||||||
|
will be rejected by the post-migration validators. Do not decrypt production data
|
||||||
|
as a rollback procedure. If maintenance must be aborted before cutover, keep
|
||||||
|
writers stopped and fix/retry the migration, or restore the pre-upgrade database
|
||||||
|
under the original release as a coordinated recovery.
|
||||||
|
|
||||||
|
## Verification tests
|
||||||
|
|
||||||
|
The regular suite tests cryptography, redaction, session contents, and the
|
||||||
|
credential service. Run MongoDB integration tests against a disposable instance:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
TEST_MONGODB_URI=mongodb://127.0.0.1:27028 npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
The tests create randomly named databases and delete only those test databases.
|
||||||
|
They cover stored ciphertext, hidden projections, concurrent credential writes,
|
||||||
|
refresh, OAuth login, resource lookup, migration conflicts/reruns, missing owners,
|
||||||
|
corruption, and MongoDB validators. Without `TEST_MONGODB_URI`, these integration
|
||||||
|
tests are skipped.
|
||||||
|
|
||||||
|
## Docker Compose command sequence
|
||||||
|
|
||||||
|
Run these blocks in a Bash session on the production host, from the existing
|
||||||
|
Compose project directory, after checking out the reviewed release. Keep the same
|
||||||
|
Compose project name and override files used by production. These commands assume
|
||||||
|
the repository's local `mongodb` service, with its existing root credentials in
|
||||||
|
`MONGO_INITDB_ROOT_USERNAME` and `MONGO_INITDB_ROOT_PASSWORD`. If `MONGODB_URI`
|
||||||
|
points elsewhere, back up that database instead. The host needs Python 3 and GPG.
|
||||||
|
|
||||||
|
1. Set the Compose command, preserve the running streamer count, and build without
|
||||||
|
restarting production:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
set -euo pipefail
|
||||||
|
set +x
|
||||||
|
umask 077
|
||||||
|
dc=(docker compose)
|
||||||
|
# If production uses the replica override, use instead:
|
||||||
|
# dc=(docker compose -f docker-compose.yml -f docker-compose.replica-primary.yml)
|
||||||
|
streamer_replicas=$("${dc[@]}" ps -q streamer | wc -l | tr -d ' ')
|
||||||
|
test "$streamer_replicas" -gt 0
|
||||||
|
"${dc[@]}" build anonymous_github
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Generate the persistent key directly into `.env`, without printing it or putting
|
||||||
|
it in shell history. This deliberately refuses to replace any existing key
|
||||||
|
configuration. On a resumed migration, keep the existing keys and skip this
|
||||||
|
generation step.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <<'PY'
|
||||||
|
import base64, os, re, secrets
|
||||||
|
from pathlib import Path
|
||||||
|
env = Path('.env')
|
||||||
|
content = env.read_text()
|
||||||
|
if re.search(r'^\s*(?:export\s+)?CREDENTIAL_(?:KEYS|ACTIVE_KEY_ID|LEGACY_READS)\s*=', content, re.M):
|
||||||
|
raise SystemExit('Credential configuration already exists; preserve it and review before continuing.')
|
||||||
|
key = base64.b64encode(secrets.token_bytes(32)).decode('ascii')
|
||||||
|
os.chmod(env, 0o600)
|
||||||
|
with env.open('a') as output:
|
||||||
|
output.write('\nCREDENTIAL_KEYS=\'{"v1":"' + key + '"}\'\n')
|
||||||
|
output.write('CREDENTIAL_ACTIVE_KEY_ID=v1\nCREDENTIAL_LEGACY_READS=false\n')
|
||||||
|
PY
|
||||||
|
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||||
|
-e 'require("./build/core/credentials").credentialCipher(); console.log("Credential keyring valid")'
|
||||||
|
```
|
||||||
|
|
||||||
|
Save the keyring in your secret manager or a separately protected recovery
|
||||||
|
location before proceeding. Do not use `docker compose config` or `cat .env`
|
||||||
|
in a recorded terminal: those can disclose secrets.
|
||||||
|
|
||||||
|
3. Enter maintenance and stop all application writers. In this Compose file the
|
||||||
|
API starts the queue workers and scheduler. Stop any additional instances or
|
||||||
|
external writers too; do not leave the rolling deploy script running.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"${dc[@]}" stop -t 120 anonymous_github streamer
|
||||||
|
test -z "$("${dc[@]}" ps --status running -q anonymous_github streamer)"
|
||||||
|
"${dc[@]}" ps mongodb redis
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Take a fresh encrypted dump while writers are stopped. The password is read
|
||||||
|
from the database container's environment into a temporary mode-0600 config in
|
||||||
|
`/dev/shm`; it is never placed in command arguments. GPG prompts for a backup
|
||||||
|
passphrase, which must be stored separately from the archive.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
backup_dir=$(mktemp -d /var/tmp/anonymous-gh-migration.XXXXXXXX)
|
||||||
|
export GPG_TTY=$(tty)
|
||||||
|
"${dc[@]}" exec -T mongodb bash -se <<'SH' | gpg --symmetric --cipher-algo AES256 --output "$backup_dir/mongo.archive.gz.gpg"
|
||||||
|
set -euo pipefail
|
||||||
|
umask 077
|
||||||
|
export CREDENTIAL_DUMP_CONFIG=$(mktemp /dev/shm/credential-dump.XXXXXXXX)
|
||||||
|
trap 'rm -f "$CREDENTIAL_DUMP_CONFIG"' EXIT
|
||||||
|
mongosh --nodb --quiet --eval 'require("fs").writeFileSync(process.env.CREDENTIAL_DUMP_CONFIG, JSON.stringify({password: process.env.MONGO_INITDB_ROOT_PASSWORD}), {mode: 0o600})' >/dev/null
|
||||||
|
mongodump --host 127.0.0.1 --port 27017 \
|
||||||
|
--username "$MONGO_INITDB_ROOT_USERNAME" --authenticationDatabase admin \
|
||||||
|
--config "$CREDENTIAL_DUMP_CONFIG" --archive --gzip
|
||||||
|
SH
|
||||||
|
test -s "$backup_dir/mongo.archive.gz.gpg"
|
||||||
|
gpg --decrypt "$backup_dir/mongo.archive.gz.gpg" >/dev/null
|
||||||
|
printf 'Encrypted backup: %s\n' "$backup_dir/mongo.archive.gz.gpg"
|
||||||
|
```
|
||||||
|
|
||||||
|
The decryption check verifies the encrypted file, not MongoDB restoreability.
|
||||||
|
Confirm your restore rehearsal succeeded on an isolated database before
|
||||||
|
deleting legacy fields. Move this archive to your protected backup storage.
|
||||||
|
|
||||||
|
5. Inventory, backfill, then inventory again:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
migrate() {
|
||||||
|
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||||
|
build/scripts/migrate-credentials.js "$@"
|
||||||
|
}
|
||||||
|
migrate
|
||||||
|
migrate --apply --maintenance
|
||||||
|
migrate
|
||||||
|
```
|
||||||
|
|
||||||
|
Stop on any nonzero exit status or `issues` count. Do not add
|
||||||
|
`--prefer-owner-token` automatically: review the conflict policy above first.
|
||||||
|
After backfill, the second inventory should report `created: 0, issues: 0`.
|
||||||
|
|
||||||
|
6. After the backup/restore check and inventory pass, clean and enforce:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
migrate --apply --maintenance --remove-legacy
|
||||||
|
migrate --verify
|
||||||
|
migrate --apply --maintenance --enforce
|
||||||
|
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||||
|
build/scripts/purge-legacy-sessions.js --apply
|
||||||
|
"${dc[@]}" run --rm --no-deps -T --entrypoint node anonymous_github \
|
||||||
|
build/scripts/purge-legacy-sessions.js
|
||||||
|
```
|
||||||
|
|
||||||
|
Require `legacy: 0` from MongoDB verification and `found: 0` from the final
|
||||||
|
session scan. Do not resume traffic on an unresolved error.
|
||||||
|
|
||||||
|
7. Recreate only the application services with the new image and environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"${dc[@]}" up -d --no-deps --force-recreate --wait \
|
||||||
|
--scale "streamer=$streamer_replicas" streamer anonymous_github
|
||||||
|
"${dc[@]}" ps anonymous_github streamer
|
||||||
|
migrate --verify
|
||||||
|
```
|
||||||
|
|
||||||
|
Test login and a private repository/gist/pull-request download before ending
|
||||||
|
maintenance. Existing sessions have been invalidated. If a command fails,
|
||||||
|
keep the application stopped and fix/retry; do not launch the old release
|
||||||
|
against the migrated database.
|
||||||
|
|
||||||
|
The one-off invocation follows Docker's [Compose run documentation](https://docs.docker.com/reference/cli/docker/compose/run/).
|
||||||
|
The backup password handling uses MongoDB's [mongodump configuration-file support](https://www.mongodb.com/docs/database-tools/mongodump/).
|
||||||
+2
-1
@@ -14,7 +14,8 @@
|
|||||||
"dev": "nodemon --transpile-only ./src/server/index.ts",
|
"dev": "nodemon --transpile-only ./src/server/index.ts",
|
||||||
"dev:ui": "node scripts/dev-proxy.js",
|
"dev:ui": "node scripts/dev-proxy.js",
|
||||||
"build": "rm -rf build && tsc && gulp",
|
"build": "rm -rf build && tsc && gulp",
|
||||||
"knip": "knip"
|
"knip": "knip",
|
||||||
|
"migrate:credentials": "node -r ts-node/register src/scripts/migrate-credentials.ts"
|
||||||
},
|
},
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
|
|||||||
@@ -2,6 +2,9 @@ import { resolve } from "path";
|
|||||||
import { randomBytes } from "crypto";
|
import { randomBytes } from "crypto";
|
||||||
|
|
||||||
interface Config {
|
interface Config {
|
||||||
|
CREDENTIAL_KEYS: string;
|
||||||
|
CREDENTIAL_ACTIVE_KEY_ID: string;
|
||||||
|
CREDENTIAL_LEGACY_READS: boolean;
|
||||||
SESSION_SECRET: string;
|
SESSION_SECRET: string;
|
||||||
REDIS_PORT: number;
|
REDIS_PORT: number;
|
||||||
REDIS_HOSTNAME: string;
|
REDIS_HOSTNAME: string;
|
||||||
@@ -55,6 +58,9 @@ const config: Config = {
|
|||||||
// Predictable defaults are dangerous: a known SESSION_SECRET lets anyone
|
// Predictable defaults are dangerous: a known SESSION_SECRET lets anyone
|
||||||
// forge session cookies. Default to empty and resolve below — random in
|
// forge session cookies. Default to empty and resolve below — random in
|
||||||
// dev, required in production. See the post-env block.
|
// dev, required in production. See the post-env block.
|
||||||
|
CREDENTIAL_KEYS: "",
|
||||||
|
CREDENTIAL_ACTIVE_KEY_ID: "",
|
||||||
|
CREDENTIAL_LEGACY_READS: false,
|
||||||
SESSION_SECRET: "",
|
SESSION_SECRET: "",
|
||||||
CLIENT_ID: "CLIENT_ID",
|
CLIENT_ID: "CLIENT_ID",
|
||||||
CLIENT_SECRET: "CLIENT_SECRET",
|
CLIENT_SECRET: "CLIENT_SECRET",
|
||||||
|
|||||||
+2
-21
@@ -1,3 +1,4 @@
|
|||||||
|
import { getCredentialToken } from "./credentials";
|
||||||
import { RepositoryStatus } from "./types";
|
import { RepositoryStatus } from "./types";
|
||||||
import User from "./User";
|
import User from "./User";
|
||||||
import UserModel from "./model/users/users.model";
|
import UserModel from "./model/users/users.model";
|
||||||
@@ -46,27 +47,7 @@ export default class Gist {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async getToken() {
|
async getToken() {
|
||||||
let owner = this.owner.model;
|
return (await getCredentialToken(this.owner.id, "github", { collection: "anonymizedgists", id: this._model._id })) || config.GITHUB_TOKEN;
|
||||||
if (owner && !owner.accessTokens.github) {
|
|
||||||
const temp = await UserModel.findById(owner._id);
|
|
||||||
if (temp) {
|
|
||||||
owner = temp;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (owner && owner.accessTokens && owner.accessTokens.github) {
|
|
||||||
if (owner.accessTokens.github != this._model.source.accessToken) {
|
|
||||||
this._model.source.accessToken = owner.accessTokens.github;
|
|
||||||
}
|
|
||||||
return owner.accessTokens.github;
|
|
||||||
}
|
|
||||||
if (this._model.source.accessToken) {
|
|
||||||
try {
|
|
||||||
return this._model.source.accessToken;
|
|
||||||
} catch {
|
|
||||||
logger.warn("invalid token", { code: "invalid_token", httpStatus: 401, gistId: this._model.source.gistId });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return config.GITHUB_TOKEN;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async download() {
|
async download() {
|
||||||
|
|||||||
+20
-44
@@ -4,7 +4,7 @@ import { createClient, RedisClientType } from "redis";
|
|||||||
|
|
||||||
import AnonymousError from "./AnonymousError";
|
import AnonymousError from "./AnonymousError";
|
||||||
import Repository from "./Repository";
|
import Repository from "./Repository";
|
||||||
import UserModel from "./model/users/users.model";
|
import { getCredential, replaceCredential, getCredentialToken } from "./credentials";
|
||||||
import config from "../config";
|
import config from "../config";
|
||||||
import { createLogger } from "./logger";
|
import { createLogger } from "./logger";
|
||||||
|
|
||||||
@@ -271,36 +271,21 @@ export async function checkToken(token: string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const checkedRepositoryTokens = new WeakMap<Repository, string>();
|
||||||
|
|
||||||
export async function getToken(repository: Repository) {
|
export async function getToken(repository: Repository) {
|
||||||
logger.debug("getToken", { repoId: repository.repoId });
|
logger.debug("getToken", { repoId: repository.repoId });
|
||||||
// if (repository.model.source.accessToken) {
|
const credential = await getCredential(repository.owner.id);
|
||||||
// // only check the token if the repo has been visited less than 10 minutes ago
|
const ownerAccessToken = credential?.token;
|
||||||
// if (
|
|
||||||
// repository.status == RepositoryStatus.READY &&
|
|
||||||
// repository.model.lastView > new Date(Date.now() - 1000 * 60 * 10)
|
|
||||||
// ) {
|
|
||||||
// return repository.model.source.accessToken;
|
|
||||||
// } else if (await checkToken(repository.model.source.accessToken)) {
|
|
||||||
// return repository.model.source.accessToken;
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
if (!repository.owner.model.accessTokens?.github) {
|
|
||||||
const query = await UserModel.findById(repository.owner.id, {
|
|
||||||
accessTokens: 1,
|
|
||||||
accessTokenDates: 1,
|
|
||||||
});
|
|
||||||
if (query?.accessTokens) {
|
|
||||||
repository.owner.model.accessTokens = query.accessTokens;
|
|
||||||
repository.owner.model.accessTokenDates = query.accessTokenDates;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const ownerAccessToken = repository.owner.model.accessTokens?.github;
|
|
||||||
if (ownerAccessToken) {
|
if (ownerAccessToken) {
|
||||||
const tokenAge = repository.owner.model.accessTokenDates?.github;
|
if (checkedRepositoryTokens.get(repository) === ownerAccessToken) {
|
||||||
|
return ownerAccessToken;
|
||||||
|
}
|
||||||
|
const tokenAge = credential?.updatedAt;
|
||||||
// if the token is older than 7 days, refresh it
|
// if the token is older than 7 days, refresh it
|
||||||
if (
|
if (
|
||||||
!tokenAge ||
|
credential?.persisted &&
|
||||||
tokenAge < new Date(Date.now() - 1000 * 60 * 60 * 24 * 7)
|
(!tokenAge || tokenAge < new Date(Date.now() - 1000 * 60 * 60 * 24 * 7))
|
||||||
) {
|
) {
|
||||||
const url = `https://api.github.com/applications/${config.CLIENT_ID}/token`;
|
const url = `https://api.github.com/applications/${config.CLIENT_ID}/token`;
|
||||||
const headers = {
|
const headers = {
|
||||||
@@ -339,23 +324,11 @@ export async function getToken(repository: Repository) {
|
|||||||
? resBody.token
|
? resBody.token
|
||||||
: null;
|
: null;
|
||||||
if (refreshed) {
|
if (refreshed) {
|
||||||
repository.owner.model.accessTokens.github = refreshed;
|
if (await replaceCredential(repository.owner.id, ownerAccessToken, refreshed)) {
|
||||||
if (!repository.owner.model.accessTokenDates) {
|
checkedRepositoryTokens.set(repository, refreshed);
|
||||||
repository.owner.model.accessTokenDates = { github: new Date() };
|
return refreshed;
|
||||||
} else {
|
|
||||||
repository.owner.model.accessTokenDates.github = new Date();
|
|
||||||
}
|
}
|
||||||
await UserModel.updateOne(
|
return (await getCredentialToken(repository.owner.id)) || config.GITHUB_TOKEN;
|
||||||
{ _id: repository.owner.model._id },
|
|
||||||
{
|
|
||||||
$set: {
|
|
||||||
"accessTokens.github": refreshed,
|
|
||||||
"accessTokenDates.github":
|
|
||||||
repository.owner.model.accessTokenDates.github,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
).exec();
|
|
||||||
return refreshed;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
logger.warn("token refresh failed; falling back", {
|
logger.warn("token refresh failed; falling back", {
|
||||||
@@ -367,9 +340,12 @@ export async function getToken(repository: Repository) {
|
|||||||
}
|
}
|
||||||
const check = await checkToken(ownerAccessToken);
|
const check = await checkToken(ownerAccessToken);
|
||||||
if (check) {
|
if (check) {
|
||||||
repository.model.source.accessToken = ownerAccessToken;
|
checkedRepositoryTokens.set(repository, ownerAccessToken);
|
||||||
return ownerAccessToken;
|
return ownerAccessToken;
|
||||||
}
|
}
|
||||||
|
return config.GITHUB_TOKEN;
|
||||||
}
|
}
|
||||||
return config.GITHUB_TOKEN;
|
return (await getCredentialToken(repository.owner.id, "github", {
|
||||||
|
collection: "anonymizedrepositories", id: repository.model._id,
|
||||||
|
})) || config.GITHUB_TOKEN;
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-25
@@ -1,3 +1,4 @@
|
|||||||
|
import { getCredentialToken } from "./credentials";
|
||||||
import { RepositoryStatus } from "./types";
|
import { RepositoryStatus } from "./types";
|
||||||
import User from "./User";
|
import User from "./User";
|
||||||
import UserModel from "./model/users/users.model";
|
import UserModel from "./model/users/users.model";
|
||||||
@@ -24,31 +25,7 @@ export default class PullRequest {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async getToken() {
|
async getToken() {
|
||||||
let owner = this.owner.model;
|
return (await getCredentialToken(this.owner.id, "github", { collection: "anonymizedpullrequests", id: this._model._id })) || config.GITHUB_TOKEN;
|
||||||
if (owner && !owner.accessTokens.github) {
|
|
||||||
const temp = await UserModel.findById(owner._id);
|
|
||||||
if (temp) {
|
|
||||||
owner = temp;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (owner && owner.accessTokens && owner.accessTokens.github) {
|
|
||||||
if (owner.accessTokens.github != this._model.source.accessToken) {
|
|
||||||
this._model.source.accessToken = owner.accessTokens.github;
|
|
||||||
}
|
|
||||||
return owner.accessTokens.github;
|
|
||||||
}
|
|
||||||
if (this._model.source.accessToken) {
|
|
||||||
try {
|
|
||||||
return this._model.source.accessToken;
|
|
||||||
} catch {
|
|
||||||
logger.warn("invalid token", {
|
|
||||||
code: "invalid_token",
|
|
||||||
httpStatus: 401,
|
|
||||||
pullRequestId: this._model.source.pullRequestId,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return config.GITHUB_TOKEN;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async download() {
|
async download() {
|
||||||
|
|||||||
+1
-16
@@ -70,23 +70,8 @@ export default class Repository {
|
|||||||
this.owner.model.isNew = false;
|
this.owner.model.isNew = false;
|
||||||
}
|
}
|
||||||
|
|
||||||
private checkedToken: boolean = false;
|
|
||||||
|
|
||||||
async getToken() {
|
async getToken() {
|
||||||
if (this.checkedToken) return this._model.source.accessToken as string;
|
return getToken(this);
|
||||||
const originalToken = this._model.source.accessToken;
|
|
||||||
const token = await getToken(this);
|
|
||||||
if (originalToken != token) {
|
|
||||||
this._model.source.accessToken = token;
|
|
||||||
if (isConnected) {
|
|
||||||
await AnonymizedRepositoryModel.updateOne(
|
|
||||||
{ _id: this._model._id },
|
|
||||||
{ $set: { "source.accessToken": token } }
|
|
||||||
).exec();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
this.checkedToken = true;
|
|
||||||
return token;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
get source() {
|
get source() {
|
||||||
|
|||||||
+8
-4
@@ -1,3 +1,4 @@
|
|||||||
|
import { getCredentialToken } from "./credentials";
|
||||||
import AnonymizedRepositoryModel from "./model/anonymizedRepositories/anonymizedRepositories.model";
|
import AnonymizedRepositoryModel from "./model/anonymizedRepositories/anonymizedRepositories.model";
|
||||||
import RepositoryModel from "./model/repositories/repositories.model";
|
import RepositoryModel from "./model/repositories/repositories.model";
|
||||||
import UserModel from "./model/users/users.model";
|
import UserModel from "./model/users/users.model";
|
||||||
@@ -31,8 +32,8 @@ export default class User {
|
|||||||
return !!this._model.isAdmin;
|
return !!this._model.isAdmin;
|
||||||
}
|
}
|
||||||
|
|
||||||
get accessToken(): string {
|
async getAccessToken(): Promise<string> {
|
||||||
return this._model.accessTokens.github;
|
return getCredentialToken(this.id);
|
||||||
}
|
}
|
||||||
|
|
||||||
get photo(): string | undefined {
|
get photo(): string | undefined {
|
||||||
@@ -64,7 +65,7 @@ export default class User {
|
|||||||
opt?.force === true
|
opt?.force === true
|
||||||
) {
|
) {
|
||||||
// get the list of repo from github
|
// get the list of repo from github
|
||||||
const oct = octokit(this.accessToken);
|
const oct = octokit(await this.getAccessToken());
|
||||||
const repositories = (
|
const repositories = (
|
||||||
await oct.paginate("GET /user/repos", {
|
await oct.paginate("GET /user/repos", {
|
||||||
visibility: "all",
|
visibility: "all",
|
||||||
@@ -213,6 +214,9 @@ export default class User {
|
|||||||
}
|
}
|
||||||
|
|
||||||
toJSON() {
|
toJSON() {
|
||||||
return this._model.toJSON();
|
const value = this._model.toJSON();
|
||||||
|
delete (value as Partial<typeof value>).accessTokens;
|
||||||
|
delete value.apiTokens;
|
||||||
|
return value;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
import { createCipheriv, createDecipheriv, randomBytes } from "crypto";
|
||||||
|
|
||||||
|
export interface EncryptedToken {
|
||||||
|
version: number;
|
||||||
|
keyId: string;
|
||||||
|
nonce: string;
|
||||||
|
ciphertext: string;
|
||||||
|
tag: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createTokenCipher(rawKeys: string, activeKeyId: string) {
|
||||||
|
let parsed: Record<string, unknown>;
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(rawKeys);
|
||||||
|
} catch {
|
||||||
|
throw new Error("CREDENTIAL_KEYS must be a JSON object of base64 keys");
|
||||||
|
}
|
||||||
|
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||||
|
throw new Error("CREDENTIAL_KEYS must be a JSON object");
|
||||||
|
}
|
||||||
|
const keys = new Map<string, Buffer>();
|
||||||
|
for (const [id, value] of Object.entries(parsed)) {
|
||||||
|
if (typeof value !== "string" || !id ||
|
||||||
|
Buffer.from(value, "base64").length !== 32 ||
|
||||||
|
Buffer.from(value, "base64").toString("base64") !== value) {
|
||||||
|
throw new Error("Credential keys must be canonical base64-encoded 32-byte keys");
|
||||||
|
}
|
||||||
|
keys.set(id, Buffer.from(value, "base64"));
|
||||||
|
}
|
||||||
|
if (!keys.has(activeKeyId)) throw new Error("CREDENTIAL_ACTIVE_KEY_ID is missing from CREDENTIAL_KEYS");
|
||||||
|
const aad = (ownerId: string, provider: string) =>
|
||||||
|
Buffer.from(JSON.stringify(["credentials", ownerId, provider, "encryptedToken", 1]));
|
||||||
|
return {
|
||||||
|
encrypt(token: string, ownerId: string, provider: string): EncryptedToken {
|
||||||
|
if (!token) throw new Error("Cannot encrypt an empty credential");
|
||||||
|
const nonce = randomBytes(12);
|
||||||
|
const cipher = createCipheriv("aes-256-gcm", keys.get(activeKeyId)!, nonce);
|
||||||
|
cipher.setAAD(aad(ownerId, provider));
|
||||||
|
const ciphertext = Buffer.concat([cipher.update(token, "utf8"), cipher.final()]);
|
||||||
|
return { version: 1, keyId: activeKeyId, nonce: nonce.toString("base64"),
|
||||||
|
ciphertext: ciphertext.toString("base64"), tag: cipher.getAuthTag().toString("base64") };
|
||||||
|
},
|
||||||
|
decrypt(value: EncryptedToken, ownerId: string, provider: string): string {
|
||||||
|
try {
|
||||||
|
if (!value || value.version !== 1 || !keys.has(value.keyId)) throw new Error();
|
||||||
|
const decode = (s: string) => {
|
||||||
|
if (typeof s !== "string" || !s) throw new Error();
|
||||||
|
const b = Buffer.from(s, "base64");
|
||||||
|
if (b.toString("base64") !== s) throw new Error();
|
||||||
|
return b;
|
||||||
|
};
|
||||||
|
const nonce = decode(value.nonce);
|
||||||
|
const tag = decode(value.tag);
|
||||||
|
if (nonce.length !== 12 || tag.length !== 16) throw new Error();
|
||||||
|
const decipher = createDecipheriv("aes-256-gcm", keys.get(value.keyId)!, nonce, { authTagLength: 16 });
|
||||||
|
decipher.setAAD(aad(ownerId, provider));
|
||||||
|
decipher.setAuthTag(tag);
|
||||||
|
return Buffer.concat([decipher.update(decode(value.ciphertext)), decipher.final()]).toString("utf8");
|
||||||
|
} catch {
|
||||||
|
// Never attach the token, envelope, key, or underlying crypto error.
|
||||||
|
throw new Error("Credential decryption failed");
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
import { Types } from "mongoose";
|
||||||
|
import config from "../config";
|
||||||
|
import { createTokenCipher } from "./credential-crypto";
|
||||||
|
import CredentialModel from "./model/credentials/credentials.model";
|
||||||
|
import UserModel from "./model/users/users.model";
|
||||||
|
|
||||||
|
export function credentialCipher() {
|
||||||
|
return createTokenCipher(config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function getCredential(ownerId: string, provider = "github") {
|
||||||
|
const credential = await CredentialModel.findOne({ ownerId, provider }).select("+encryptedToken").lean();
|
||||||
|
if (credential) {
|
||||||
|
return { token: credentialCipher().decrypt(credential.encryptedToken, String(credential.ownerId), provider),
|
||||||
|
updatedAt: credential.updatedAt, persisted: true };
|
||||||
|
}
|
||||||
|
if (config.CREDENTIAL_LEGACY_READS && provider === "github") {
|
||||||
|
const user = await UserModel.findById(ownerId).select("+accessTokens.github accessTokenDates");
|
||||||
|
const token = user?.accessTokens?.github;
|
||||||
|
if (token) return { token, updatedAt: user?.accessTokenDates?.github, persisted: false };
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function getCredentialToken(ownerId: string, provider = "github", resource?: {
|
||||||
|
collection: "anonymizedrepositories" | "anonymizedgists" | "anonymizedpullrequests";
|
||||||
|
id: unknown;
|
||||||
|
}): Promise<string> {
|
||||||
|
const credential = await getCredential(ownerId, provider);
|
||||||
|
if (credential) return credential.token;
|
||||||
|
if (config.CREDENTIAL_LEGACY_READS && resource) {
|
||||||
|
const row = await CredentialModel.db.collection(resource.collection).findOne({
|
||||||
|
_id: resource.id as Types.ObjectId,
|
||||||
|
owner: new Types.ObjectId(ownerId),
|
||||||
|
}, { projection: { "source.accessToken": 1 } });
|
||||||
|
if (typeof row?.source?.accessToken === "string") return row.source.accessToken;
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function setCredential(ownerId: string, token: string, provider = "github") {
|
||||||
|
ownerId = new Types.ObjectId(ownerId).toHexString();
|
||||||
|
const encryptedToken = credentialCipher().encrypt(token, ownerId, provider);
|
||||||
|
await CredentialModel.updateOne({ ownerId, provider }, {
|
||||||
|
$set: { encryptedToken, updatedAt: new Date() },
|
||||||
|
}, { upsert: true, runValidators: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refresh must not overwrite a credential replaced by a concurrent login.
|
||||||
|
export async function replaceCredential(ownerId: string, previous: string, token: string) {
|
||||||
|
const current = await CredentialModel.findOne({ ownerId, provider: "github" }).select("+encryptedToken").lean();
|
||||||
|
if (!current) {
|
||||||
|
// Legacy credentials are migrated by the migration script or next login.
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const cipher = credentialCipher();
|
||||||
|
if (cipher.decrypt(current.encryptedToken, String(ownerId), "github") !== previous) return false;
|
||||||
|
const result = await CredentialModel.updateOne({ _id: current._id, encryptedToken: current.encryptedToken }, {
|
||||||
|
$set: { encryptedToken: cipher.encrypt(token, String(ownerId), "github"), updatedAt: new Date() },
|
||||||
|
});
|
||||||
|
return result.modifiedCount === 1;
|
||||||
|
}
|
||||||
+3
-1
@@ -1,3 +1,4 @@
|
|||||||
|
import { redactSecrets } from "./redact-secrets";
|
||||||
import { createClient, RedisClientType } from "redis";
|
import { createClient, RedisClientType } from "redis";
|
||||||
import config from "../config";
|
import config from "../config";
|
||||||
|
|
||||||
@@ -251,6 +252,7 @@ function persistError(entry: {
|
|||||||
|
|
||||||
function emit(level: Level, module: string, args: unknown[]) {
|
function emit(level: Level, module: string, args: unknown[]) {
|
||||||
if (LEVEL_ORDER[level] < threshold) return;
|
if (LEVEL_ORDER[level] < threshold) return;
|
||||||
|
args = args.map(arg => redactSecrets(arg instanceof Error ? serializeError(arg) : arg));
|
||||||
const ts = new Date().toISOString();
|
const ts = new Date().toISOString();
|
||||||
const formatted = args.map(formatArg);
|
const formatted = args.map(formatArg);
|
||||||
const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`;
|
const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`;
|
||||||
@@ -341,5 +343,5 @@ export function serializeError(err: unknown): Record<string, unknown> {
|
|||||||
// a stack for handled HTTP errors but keeps debuggability for plain Errors.
|
// a stack for handled HTTP errors but keeps debuggability for plain Errors.
|
||||||
if (!out.status && !out.httpStatus && e.stack) out.stack = e.stack;
|
if (!out.status && !out.httpStatus && e.stack) out.stack = e.stack;
|
||||||
|
|
||||||
return out;
|
return redactSecrets(out) as Record<string, unknown>;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
import { mongo } from "mongoose";
|
||||||
|
import { createTokenCipher, EncryptedToken } from "./credential-crypto";
|
||||||
|
|
||||||
|
type Cipher = ReturnType<typeof createTokenCipher>;
|
||||||
|
const resources = ["anonymizedrepositories", "anonymizedgists", "anonymizedpullrequests"];
|
||||||
|
const legacyQuery = { $or: [{ "source.accessToken": { $exists: true } }, { accessToken: { $exists: true } }] };
|
||||||
|
export interface MigrationOptions {
|
||||||
|
apply?: boolean;
|
||||||
|
removeLegacy?: boolean;
|
||||||
|
preferOwnerToken?: boolean;
|
||||||
|
batchSize?: number;
|
||||||
|
report?: (event: { collection: string; id: string; issue: string }) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Run with all application writers stopped. Reruns never replace an existing credential. */
|
||||||
|
export async function migrateCredentials(db: mongo.Db, cipher: Cipher, options: MigrationOptions = {}) {
|
||||||
|
const credentials = db.collection("credentials");
|
||||||
|
const users = db.collection("users");
|
||||||
|
const batchSize = options.batchSize || 100;
|
||||||
|
const counts = { owners: 0, created: 0, removed: 0, issues: 0 };
|
||||||
|
const issue = (collection: string, id: unknown, reason: string) => {
|
||||||
|
counts.issues++;
|
||||||
|
options.report?.({ collection, id: String(id), issue: reason });
|
||||||
|
};
|
||||||
|
if (options.apply) await credentials.createIndex({ ownerId: 1, provider: 1 }, { unique: true });
|
||||||
|
for await (const user of users.find({}, { projection: { accessTokens: 1, accessTokenDates: 1, status: 1 } }).batchSize(batchSize)) {
|
||||||
|
counts.owners++;
|
||||||
|
const ownerId = user._id;
|
||||||
|
const existing = await credentials.findOne({ ownerId, provider: "github" });
|
||||||
|
let selected: string | undefined;
|
||||||
|
let valid = true;
|
||||||
|
if (existing) {
|
||||||
|
try { selected = cipher.decrypt(existing.encryptedToken as EncryptedToken, String(ownerId), "github"); }
|
||||||
|
catch { issue("credentials", existing._id, "decryption_failed"); continue; }
|
||||||
|
}
|
||||||
|
const ownerToken = user.accessTokens?.github;
|
||||||
|
const authoritative = !!(selected || (typeof ownerToken === "string" && ownerToken));
|
||||||
|
const inspect = (value: unknown, collection: string, id: unknown) => {
|
||||||
|
if (value === undefined || value === null || value === "") return;
|
||||||
|
if (typeof value !== "string") {
|
||||||
|
issue(collection, id, "malformed_token"); valid = false; return;
|
||||||
|
}
|
||||||
|
if (!selected) selected = value;
|
||||||
|
else if (selected !== value && !(options.preferOwnerToken && authoritative)) {
|
||||||
|
issue(collection, id, "conflicting_token"); valid = false;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
inspect(ownerToken, "users", ownerId);
|
||||||
|
for (const name of resources) {
|
||||||
|
for await (const row of db.collection(name).find({ owner: ownerId, ...legacyQuery }, {
|
||||||
|
projection: { source: 1, accessToken: 1 },
|
||||||
|
}).batchSize(batchSize)) {
|
||||||
|
inspect(row.source?.accessToken, name, row._id);
|
||||||
|
inspect(row.accessToken, name, row._id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!valid) continue;
|
||||||
|
// Removed accounts must never regain credentials during backfill.
|
||||||
|
if (user.status === "removed") {
|
||||||
|
if (options.apply && options.removeLegacy) await credentials.deleteMany({ ownerId });
|
||||||
|
} else if (selected && !existing) {
|
||||||
|
const encryptedToken = cipher.encrypt(selected, String(ownerId), "github");
|
||||||
|
if (cipher.decrypt(encryptedToken, String(ownerId), "github") !== selected) throw new Error("Credential verification failed");
|
||||||
|
if (options.apply) {
|
||||||
|
await credentials.updateOne({ ownerId, provider: "github" }, { $setOnInsert: {
|
||||||
|
encryptedToken, updatedAt: user.accessTokenDates?.github || new Date(),
|
||||||
|
} }, { upsert: true });
|
||||||
|
const stored = await credentials.findOne({ ownerId, provider: "github" });
|
||||||
|
if (!stored || cipher.decrypt(stored.encryptedToken as EncryptedToken, String(ownerId), "github") !== selected) {
|
||||||
|
issue("users", ownerId, "credential_changed_retry"); continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
counts.created++;
|
||||||
|
}
|
||||||
|
if (options.apply && options.removeLegacy) {
|
||||||
|
// The command requires maintenance mode: no login, refresh, deletion, or resource writes.
|
||||||
|
const result = await users.updateOne({ _id: ownerId }, { $unset: { accessTokens: "", accessTokenDates: "" } });
|
||||||
|
counts.removed += result.modifiedCount;
|
||||||
|
for (const name of resources) {
|
||||||
|
const result = await db.collection(name).updateMany({ owner: ownerId, ...legacyQuery }, {
|
||||||
|
$unset: { "source.accessToken": "", accessToken: "" },
|
||||||
|
});
|
||||||
|
counts.removed += result.modifiedCount;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Credentials attached to missing owners cannot be assigned safely.
|
||||||
|
for (const name of resources) {
|
||||||
|
for await (const row of db.collection(name).find(legacyQuery, { projection: { owner: 1 } }).batchSize(batchSize)) {
|
||||||
|
if (!row.owner || !(await users.findOne({ _id: row.owner }, { projection: { _id: 1 } }))) {
|
||||||
|
issue(name, row._id, "missing_owner");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return counts;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function verifyCredentials(db: mongo.Db, cipher: Cipher) {
|
||||||
|
let checked = 0;
|
||||||
|
for await (const row of db.collection("credentials").find({})) {
|
||||||
|
cipher.decrypt(row.encryptedToken as EncryptedToken, String(row.ownerId), row.provider);
|
||||||
|
if (!(await db.collection("users").findOne({ _id: row.ownerId, status: { $ne: "removed" } }))) {
|
||||||
|
throw new Error("Credential has no active owner");
|
||||||
|
}
|
||||||
|
checked++;
|
||||||
|
}
|
||||||
|
let legacy = await db.collection("users").countDocuments({ accessTokens: { $exists: true } });
|
||||||
|
for (const name of resources) legacy += await db.collection(name).countDocuments(legacyQuery);
|
||||||
|
return { checked, legacy };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Preserve existing validators while forbidding legacy credential storage. */
|
||||||
|
export async function enforceCredentialStorage(db: mongo.Db, cipher: Cipher) {
|
||||||
|
const verification = await verifyCredentials(db, cipher);
|
||||||
|
if (verification.legacy) throw new Error("Legacy credentials remain");
|
||||||
|
for (const name of ["users", ...resources]) {
|
||||||
|
const info = await db.listCollections({ name }).next();
|
||||||
|
if (!info) await db.createCollection(name);
|
||||||
|
const previous = info && "options" in info ? info.options?.validator : undefined;
|
||||||
|
const absent = name === "users" ? { accessTokens: { $exists: false } } : {
|
||||||
|
"source.accessToken": { $exists: false }, accessToken: { $exists: false },
|
||||||
|
};
|
||||||
|
await db.command({ collMod: name, validator: previous && Object.keys(previous).length ? {
|
||||||
|
$and: [previous, absent],
|
||||||
|
} : absent, validationLevel: "strict", validationAction: "error" });
|
||||||
|
}
|
||||||
|
return verification;
|
||||||
|
}
|
||||||
@@ -18,7 +18,7 @@ const AnonymizedGistSchema = new Schema({
|
|||||||
conference: String,
|
conference: String,
|
||||||
source: {
|
source: {
|
||||||
gistId: String,
|
gistId: String,
|
||||||
accessToken: String,
|
accessToken: { type: String, select: false },
|
||||||
},
|
},
|
||||||
options: {
|
options: {
|
||||||
terms: [String],
|
terms: [String],
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ const AnonymizedPullRequestSchema = new Schema({
|
|||||||
source: {
|
source: {
|
||||||
pullRequestId: Number,
|
pullRequestId: Number,
|
||||||
repositoryFullName: String,
|
repositoryFullName: String,
|
||||||
accessToken: String,
|
accessToken: { type: String, select: false },
|
||||||
},
|
},
|
||||||
options: {
|
options: {
|
||||||
terms: [String],
|
terms: [String],
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ const AnonymizedRepositorySchema = new Schema({
|
|||||||
anonymizeDate: Date,
|
anonymizeDate: Date,
|
||||||
lastView: Date,
|
lastView: Date,
|
||||||
pageView: Number,
|
pageView: Number,
|
||||||
accessToken: String,
|
accessToken: { type: String, select: false },
|
||||||
owner: {
|
owner: {
|
||||||
type: Schema.Types.ObjectId,
|
type: Schema.Types.ObjectId,
|
||||||
ref: "user",
|
ref: "user",
|
||||||
@@ -36,7 +36,7 @@ const AnonymizedRepositorySchema = new Schema({
|
|||||||
commitDate: Date,
|
commitDate: Date,
|
||||||
repositoryId: String,
|
repositoryId: String,
|
||||||
repositoryName: String,
|
repositoryName: String,
|
||||||
accessToken: String,
|
accessToken: { type: String, select: false },
|
||||||
},
|
},
|
||||||
truncatedFolders: {
|
truncatedFolders: {
|
||||||
type: [String],
|
type: [String],
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
import { model, Schema, Types } from "mongoose";
|
||||||
|
import { EncryptedToken } from "../../credential-crypto";
|
||||||
|
|
||||||
|
export interface ICredential {
|
||||||
|
ownerId: Types.ObjectId;
|
||||||
|
provider: string;
|
||||||
|
encryptedToken: EncryptedToken;
|
||||||
|
updatedAt: Date;
|
||||||
|
}
|
||||||
|
const envelope = new Schema({
|
||||||
|
version: { type: Number, required: true, enum: [1] },
|
||||||
|
keyId: { type: String, required: true },
|
||||||
|
nonce: { type: String, required: true },
|
||||||
|
ciphertext: { type: String, required: true },
|
||||||
|
tag: { type: String, required: true },
|
||||||
|
}, { _id: false });
|
||||||
|
const schema = new Schema<ICredential>({
|
||||||
|
ownerId: { type: Schema.Types.ObjectId, required: true, ref: "user" },
|
||||||
|
provider: { type: String, required: true, enum: ["github"] },
|
||||||
|
encryptedToken: { type: envelope, required: true, select: false },
|
||||||
|
updatedAt: { type: Date, required: true },
|
||||||
|
}, { collection: "credentials" });
|
||||||
|
schema.index({ ownerId: 1, provider: 1 }, { unique: true });
|
||||||
|
export default model<ICredential>("Credential", schema);
|
||||||
@@ -2,7 +2,7 @@ import { Schema } from "mongoose";
|
|||||||
|
|
||||||
const UserSchema = new Schema({
|
const UserSchema = new Schema({
|
||||||
accessTokens: {
|
accessTokens: {
|
||||||
github: { type: String },
|
github: { type: String, select: false },
|
||||||
},
|
},
|
||||||
accessTokenDates: {
|
accessTokenDates: {
|
||||||
github: { type: Date },
|
github: { type: Date },
|
||||||
|
|||||||
@@ -1,7 +1,8 @@
|
|||||||
import { Document, Model } from "mongoose";
|
import { Document, Model } from "mongoose";
|
||||||
|
|
||||||
export interface IUser {
|
export interface IUser {
|
||||||
accessTokens: {
|
/** Legacy storage, read only when CREDENTIAL_LEGACY_READS is enabled. */
|
||||||
|
accessTokens?: {
|
||||||
github: string;
|
github: string;
|
||||||
};
|
};
|
||||||
accessTokenDates?: {
|
accessTokenDates?: {
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
const sensitive = /^(?:authorization|proxy-authorization|cookie|set-cookie|token|access_?tokens?|refresh_?token|encryptedToken|ciphertext|nonce|tag|password|client_?secret|CREDENTIAL_KEYS)$/i;
|
||||||
|
export function redactSecrets(value: unknown, seen = new WeakSet<object>()): unknown {
|
||||||
|
if (typeof value === "string") return value
|
||||||
|
.replace(/\b(?:gh[pousr]_[A-Za-z0-9_]+|github_pat_[A-Za-z0-9_]+)\b/g, "[REDACTED]")
|
||||||
|
.replace(/((?:access_token|refresh_token|token)=)[^&\s]+/gi, "$1[REDACTED]")
|
||||||
|
.replace(/\b(Bearer|Basic)\s+[A-Za-z0-9+/=._-]+/gi, "$1 [REDACTED]");
|
||||||
|
if (!value || typeof value !== "object" || value instanceof Date) return value;
|
||||||
|
if (seen.has(value)) return "[Circular]";
|
||||||
|
seen.add(value);
|
||||||
|
if (Array.isArray(value)) return value.map(item => redactSecrets(item, seen));
|
||||||
|
const result: Record<string, unknown> = {};
|
||||||
|
for (const [key, item] of Object.entries(value)) {
|
||||||
|
result[key] = sensitive.test(key) ? "[REDACTED]" : redactSecrets(item, seen);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
import "dotenv/config";
|
||||||
|
import mongoose from "mongoose";
|
||||||
|
import config from "../config";
|
||||||
|
import { credentialCipher } from "../core/credentials";
|
||||||
|
import { migrateCredentials, verifyCredentials, enforceCredentialStorage } from "../core/migrate-credentials";
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const args = new Set(process.argv.slice(2));
|
||||||
|
for (const arg of args) {
|
||||||
|
if (!["--apply", "--remove-legacy", "--prefer-owner-token", "--maintenance", "--verify", "--enforce"].includes(arg)) {
|
||||||
|
throw new Error("Unknown migration option");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (args.has("--apply") && !args.has("--maintenance")) throw new Error("Stop all application writers and pass --maintenance before applying");
|
||||||
|
if (args.has("--enforce") && !args.has("--apply")) throw new Error("--enforce requires --apply");
|
||||||
|
if (args.has("--remove-legacy") && !args.has("--apply")) throw new Error("--remove-legacy requires --apply");
|
||||||
|
const cipher = credentialCipher();
|
||||||
|
const uri = config.MONGODB_URI || `mongodb://${config.DB_USERNAME}:${config.DB_PASSWORD}@${config.DB_HOSTNAME}:27017/production`;
|
||||||
|
await mongoose.connect(uri, config.MONGODB_URI ? {} : { authSource: "admin" });
|
||||||
|
const db = mongoose.connection.db;
|
||||||
|
if (args.has("--enforce")) {
|
||||||
|
process.stdout.write(JSON.stringify(await enforceCredentialStorage(db, cipher)) + "\n");
|
||||||
|
} else if (args.has("--verify")) {
|
||||||
|
const result = await verifyCredentials(db, cipher);
|
||||||
|
process.stdout.write(JSON.stringify(result) + "\n");
|
||||||
|
if (result.legacy) process.exitCode = 1;
|
||||||
|
} else {
|
||||||
|
const result = await migrateCredentials(db, cipher, {
|
||||||
|
apply: args.has("--apply"), removeLegacy: args.has("--remove-legacy"),
|
||||||
|
preferOwnerToken: args.has("--prefer-owner-token"),
|
||||||
|
report: event => process.stdout.write(JSON.stringify(event) + "\n"),
|
||||||
|
});
|
||||||
|
process.stdout.write(JSON.stringify(result) + "\n");
|
||||||
|
if (result.issues) process.exitCode = 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
main().catch(() => {
|
||||||
|
// Driver errors can contain document values or connection credentials.
|
||||||
|
process.stderr.write("Credential migration failed; check configuration, connectivity, and encrypted records. No secret values are logged.\n");
|
||||||
|
process.exitCode = 1;
|
||||||
|
}).finally(() => mongoose.disconnect());
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
import "dotenv/config";
|
||||||
|
import { createClient } from "redis";
|
||||||
|
import config from "../config";
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
if (args.some(arg => arg !== "--apply")) throw new Error("Unknown option");
|
||||||
|
const client = createClient({ socket: { host: config.REDIS_HOSTNAME, port: config.REDIS_PORT, reconnectStrategy: false } });
|
||||||
|
client.on("error", () => {});
|
||||||
|
try {
|
||||||
|
await client.connect();
|
||||||
|
let found = 0;
|
||||||
|
let removed = 0;
|
||||||
|
for await (const key of client.scanIterator({ MATCH: "anoGH_session:*", COUNT: 100 })) {
|
||||||
|
const raw = await client.get(key);
|
||||||
|
if (!raw) continue;
|
||||||
|
let legacy = false;
|
||||||
|
try {
|
||||||
|
const value = JSON.parse(raw);
|
||||||
|
legacy = !!value.passport?.user && typeof value.passport.user !== "string";
|
||||||
|
} catch { legacy = true; }
|
||||||
|
if (!legacy) continue;
|
||||||
|
found++;
|
||||||
|
if (args.includes("--apply")) {
|
||||||
|
// Do not remove a session replaced since the scan read it.
|
||||||
|
removed += Number(await client.eval(
|
||||||
|
'if redis.call("GET", KEYS[1]) == ARGV[1] then return redis.call("DEL", KEYS[1]) else return 0 end',
|
||||||
|
{ keys: [key], arguments: [raw] }
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
process.stdout.write(JSON.stringify({ found, removed }) + "\n");
|
||||||
|
} finally { if (client.isOpen) await client.quit(); }
|
||||||
|
}
|
||||||
|
main().catch(() => {
|
||||||
|
process.stderr.write("Legacy session cleanup failed; check Redis configuration and connectivity.\n");
|
||||||
|
process.exitCode = 1;
|
||||||
|
});
|
||||||
@@ -1,3 +1,5 @@
|
|||||||
|
import { credentialCipher } from "../core/credentials";
|
||||||
|
import CredentialModel from "../core/model/credentials/credentials.model";
|
||||||
import mongoose, { ConnectOptions } from "mongoose";
|
import mongoose, { ConnectOptions } from "mongoose";
|
||||||
import Repository from "../core/Repository";
|
import Repository from "../core/Repository";
|
||||||
import config from "../config";
|
import config from "../config";
|
||||||
@@ -20,6 +22,7 @@ export const database = mongoose.connection;
|
|||||||
export let isConnected = false;
|
export let isConnected = false;
|
||||||
|
|
||||||
export async function connect() {
|
export async function connect() {
|
||||||
|
credentialCipher(); // Refuse to serve persisted credentials without a valid keyring.
|
||||||
mongoose.set("strictQuery", false);
|
mongoose.set("strictQuery", false);
|
||||||
const options: ConnectOptions = {
|
const options: ConnectOptions = {
|
||||||
appName: "Anonymous GitHub Server",
|
appName: "Anonymous GitHub Server",
|
||||||
@@ -27,6 +30,7 @@ export async function connect() {
|
|||||||
};
|
};
|
||||||
if (!config.MONGODB_URI) options.authSource = "admin";
|
if (!config.MONGODB_URI) options.authSource = "admin";
|
||||||
await mongoose.connect(getMongoUrl(), options);
|
await mongoose.connect(getMongoUrl(), options);
|
||||||
|
await CredentialModel.createIndexes();
|
||||||
isConnected = true;
|
isConnected = true;
|
||||||
|
|
||||||
return database;
|
return database;
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import config from "../../config";
|
|||||||
import UserModel from "../../core/model/users/users.model";
|
import UserModel from "../../core/model/users/users.model";
|
||||||
import { IUserDocument } from "../../core/model/users/users.types";
|
import { IUserDocument } from "../../core/model/users/users.types";
|
||||||
import AnonymousError from "../../core/AnonymousError";
|
import AnonymousError from "../../core/AnonymousError";
|
||||||
import AnonymizedPullRequestModel from "../../core/model/anonymizedPullRequests/anonymizedPullRequests.model";
|
import { setCredential } from "../../core/credentials";
|
||||||
import { hashToken } from "./token-auth";
|
import { hashToken } from "./token-auth";
|
||||||
import { createLogger, serializeError } from "../../core/logger";
|
import { createLogger, serializeError } from "../../core/logger";
|
||||||
import { getLoginToken, isDisabledAccount } from "./auth-utils";
|
import { getLoginToken, isDisabledAccount } from "./auth-utils";
|
||||||
@@ -30,13 +30,12 @@ export function ensureAuthenticated(
|
|||||||
|
|
||||||
const verify = async (
|
const verify = async (
|
||||||
accessToken: string,
|
accessToken: string,
|
||||||
refreshToken: string,
|
_refreshToken: string,
|
||||||
profile: Profile,
|
profile: Profile,
|
||||||
done: OAuth2Strategy.VerifyCallback
|
done: OAuth2Strategy.VerifyCallback
|
||||||
): Promise<void> => {
|
): Promise<void> => {
|
||||||
let user: IUserDocument | null;
|
let user: IUserDocument | null;
|
||||||
try {
|
try {
|
||||||
const now = new Date();
|
|
||||||
user = await UserModel.findOne({ "externalIDs.github": profile.id });
|
user = await UserModel.findOne({ "externalIDs.github": profile.id });
|
||||||
if (user) {
|
if (user) {
|
||||||
if (isDisabledAccount(user.status)) {
|
if (isDisabledAccount(user.status)) {
|
||||||
@@ -48,20 +47,6 @@ const verify = async (
|
|||||||
);
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
await UserModel.updateOne(
|
|
||||||
{ _id: user._id },
|
|
||||||
{
|
|
||||||
$set: {
|
|
||||||
"accessTokens.github": accessToken,
|
|
||||||
"accessTokenDates.github": now,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
);
|
|
||||||
await AnonymizedPullRequestModel.updateMany(
|
|
||||||
{ owner: user._id },
|
|
||||||
{ "source.accessToken": accessToken }
|
|
||||||
);
|
|
||||||
user = await UserModel.findById(user._id);
|
|
||||||
} else {
|
} else {
|
||||||
// Check if a user with this username already exists (e.g. created
|
// Check if a user with this username already exists (e.g. created
|
||||||
// manually without externalIDs.github). Link the GitHub ID to the
|
// manually without externalIDs.github). Link the GitHub ID to the
|
||||||
@@ -87,8 +72,6 @@ const verify = async (
|
|||||||
{
|
{
|
||||||
$set: {
|
$set: {
|
||||||
"externalIDs.github": profile.id,
|
"externalIDs.github": profile.id,
|
||||||
"accessTokens.github": accessToken,
|
|
||||||
"accessTokenDates.github": now,
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
@@ -97,12 +80,6 @@ const verify = async (
|
|||||||
const photo = profile.photos ? profile.photos[0]?.value : null;
|
const photo = profile.photos ? profile.photos[0]?.value : null;
|
||||||
user = new UserModel({
|
user = new UserModel({
|
||||||
username: profile.username,
|
username: profile.username,
|
||||||
accessTokens: {
|
|
||||||
github: accessToken,
|
|
||||||
},
|
|
||||||
accessTokenDates: {
|
|
||||||
github: now,
|
|
||||||
},
|
|
||||||
externalIDs: {
|
externalIDs: {
|
||||||
github: profile.id,
|
github: profile.id,
|
||||||
},
|
},
|
||||||
@@ -126,13 +103,8 @@ const verify = async (
|
|||||||
);
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
done(null, {
|
await setCredential(String(user!._id), accessToken);
|
||||||
username: profile.username,
|
done(null, { username: user!.username, user });
|
||||||
accessToken,
|
|
||||||
refreshToken,
|
|
||||||
profile,
|
|
||||||
user,
|
|
||||||
});
|
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
logger.error("verify failed", serializeError(error));
|
logger.error("verify failed", serializeError(error));
|
||||||
done(
|
done(
|
||||||
@@ -157,11 +129,20 @@ passport.use(
|
|||||||
);
|
);
|
||||||
|
|
||||||
passport.serializeUser((user: Express.User, done) => {
|
passport.serializeUser((user: Express.User, done) => {
|
||||||
done(null, user);
|
const id = (user as { user?: { _id?: unknown } }).user?._id;
|
||||||
|
done(null, String(id));
|
||||||
});
|
});
|
||||||
|
|
||||||
passport.deserializeUser((user: Express.User, done) => {
|
passport.deserializeUser(async (id: string, done) => {
|
||||||
done(null, user);
|
// Reject the old session format, which included plaintext credentials.
|
||||||
|
if (typeof id !== "string" || !/^[a-f0-9]{24}$/i.test(id)) return done(null, false);
|
||||||
|
try {
|
||||||
|
const user = await UserModel.findById(id);
|
||||||
|
if (!user || isDisabledAccount(user.status)) return done(null, false);
|
||||||
|
done(null, { username: user.username, user });
|
||||||
|
} catch {
|
||||||
|
done(new Error("Session user lookup failed"));
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
export function initSession() {
|
export function initSession() {
|
||||||
@@ -229,7 +210,6 @@ router.post(
|
|||||||
}
|
}
|
||||||
const synthUser = {
|
const synthUser = {
|
||||||
username: model.username,
|
username: model.username,
|
||||||
accessToken: model.accessTokens?.github,
|
|
||||||
profile: undefined,
|
profile: undefined,
|
||||||
user: model,
|
user: model,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -249,7 +249,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
|||||||
gist.model.owner = user.id;
|
gist.model.owner = user.id;
|
||||||
|
|
||||||
updateGistModel(gist.model, gistUpdate);
|
updateGistModel(gist.model, gistUpdate);
|
||||||
gist.source.accessToken = user.accessToken;
|
|
||||||
gist.source.gistId = gistUpdate.source.gistId;
|
gist.source.gistId = gistUpdate.source.gistId;
|
||||||
|
|
||||||
gist.model.conference = gistUpdate.conference;
|
gist.model.conference = gistUpdate.conference;
|
||||||
|
|||||||
@@ -266,7 +266,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
|||||||
pullRequest.model.owner = user.id;
|
pullRequest.model.owner = user.id;
|
||||||
|
|
||||||
updatePullRequestModel(pullRequest.model, pullRequestUpdate);
|
updatePullRequestModel(pullRequest.model, pullRequestUpdate);
|
||||||
pullRequest.source.accessToken = user.accessToken;
|
|
||||||
pullRequest.source.pullRequestId = pullRequestUpdate.source.pullRequestId;
|
pullRequest.source.pullRequestId = pullRequestUpdate.source.pullRequestId;
|
||||||
pullRequest.source.repositoryFullName =
|
pullRequest.source.repositoryFullName =
|
||||||
pullRequestUpdate.source.repositoryFullName;
|
pullRequestUpdate.source.repositoryFullName;
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { getCredentialToken } from "../../core/credentials";
|
||||||
import * as express from "express";
|
import * as express from "express";
|
||||||
import { ensureAuthenticated } from "./connection";
|
import { ensureAuthenticated } from "./connection";
|
||||||
|
|
||||||
@@ -14,14 +15,12 @@ import { getRepositoryFromGitHub } from "../../core/source/GitHubRepository";
|
|||||||
import gh = require("parse-github-url");
|
import gh = require("parse-github-url");
|
||||||
import AnonymizedRepositoryModel from "../../core/model/anonymizedRepositories/anonymizedRepositories.model";
|
import AnonymizedRepositoryModel from "../../core/model/anonymizedRepositories/anonymizedRepositories.model";
|
||||||
import { IAnonymizedRepositoryDocument } from "../../core/model/anonymizedRepositories/anonymizedRepositories.types";
|
import { IAnonymizedRepositoryDocument } from "../../core/model/anonymizedRepositories/anonymizedRepositories.types";
|
||||||
import UserModel from "../../core/model/users/users.model";
|
|
||||||
import ConferenceModel from "../../core/model/conference/conferences.model";
|
import ConferenceModel from "../../core/model/conference/conferences.model";
|
||||||
import AnonymousError from "../../core/AnonymousError";
|
import AnonymousError from "../../core/AnonymousError";
|
||||||
import { addRemovalJob, downloadQueue } from "../../queue";
|
import { addRemovalJob, downloadQueue } from "../../queue";
|
||||||
import RepositoryModel from "../../core/model/repositories/repositories.model";
|
import RepositoryModel from "../../core/model/repositories/repositories.model";
|
||||||
import User from "../../core/User";
|
import User from "../../core/User";
|
||||||
import { RepositoryStatus } from "../../core/types";
|
import { RepositoryStatus } from "../../core/types";
|
||||||
import { IUserDocument } from "../../core/model/users/users.types";
|
|
||||||
import { checkToken, octokit, getRedisGateResetAt, getToken } from "../../core/GitHubUtils";
|
import { checkToken, octokit, getRedisGateResetAt, getToken } from "../../core/GitHubUtils";
|
||||||
import { createLogger, serializeError } from "../../core/logger";
|
import { createLogger, serializeError } from "../../core/logger";
|
||||||
|
|
||||||
@@ -40,23 +39,14 @@ async function getTokenForAdmin(user: User, req: express.Request) {
|
|||||||
"source.repositoryName": `${req.params.owner}/${req.params.repo}`,
|
"source.repositoryName": `${req.params.owner}/${req.params.repo}`,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"source.accessToken": 1,
|
|
||||||
owner: 1,
|
owner: 1,
|
||||||
}
|
}
|
||||||
).populate({
|
);
|
||||||
path: "owner",
|
if (existingRepo?.owner) {
|
||||||
model: UserModel,
|
const token = await getCredentialToken(String(existingRepo.owner), "github", {
|
||||||
});
|
collection: "anonymizedrepositories", id: existingRepo._id,
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
});
|
||||||
const user: IUserDocument = existingRepo?.owner as any;
|
if (token && await checkToken(token)) return token;
|
||||||
if (user instanceof UserModel) {
|
|
||||||
const check = await checkToken(user.accessTokens.github);
|
|
||||||
if (check) {
|
|
||||||
return user.accessTokens.github;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (existingRepo) {
|
|
||||||
return existingRepo.source.accessToken;
|
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
logger.warn("getToken lookup failed", serializeError(error));
|
logger.warn("getToken lookup failed", serializeError(error));
|
||||||
@@ -100,7 +90,7 @@ router.post("/claim", async (req: express.Request, res: express.Response) => {
|
|||||||
owner: r.owner,
|
owner: r.owner,
|
||||||
repo: r.name,
|
repo: r.name,
|
||||||
repositoryID: req.query.repositoryID as string,
|
repositoryID: req.query.repositoryID as string,
|
||||||
accessToken: user.accessToken,
|
accessToken: await user.getAccessToken(),
|
||||||
});
|
});
|
||||||
if (!repo) {
|
if (!repo) {
|
||||||
throw new AnonymousError("repo_not_found", {
|
throw new AnonymousError("repo_not_found", {
|
||||||
@@ -256,7 +246,7 @@ router.get(
|
|||||||
async (req: express.Request, res: express.Response) => {
|
async (req: express.Request, res: express.Response) => {
|
||||||
try {
|
try {
|
||||||
const user = await getUser(req);
|
const user = await getUser(req);
|
||||||
let token = user.accessToken;
|
let token = await user.getAccessToken();
|
||||||
if (user.isAdmin) {
|
if (user.isAdmin) {
|
||||||
token = (await getTokenForAdmin(user, req)) || token;
|
token = (await getTokenForAdmin(user, req)) || token;
|
||||||
}
|
}
|
||||||
@@ -279,7 +269,7 @@ router.get(
|
|||||||
async (req: express.Request, res: express.Response) => {
|
async (req: express.Request, res: express.Response) => {
|
||||||
try {
|
try {
|
||||||
const user = await getUser(req);
|
const user = await getUser(req);
|
||||||
let token = user.accessToken;
|
let token = await user.getAccessToken();
|
||||||
if (user.isAdmin) {
|
if (user.isAdmin) {
|
||||||
token = (await getTokenForAdmin(user, req)) || token;
|
token = (await getTokenForAdmin(user, req)) || token;
|
||||||
}
|
}
|
||||||
@@ -307,7 +297,7 @@ router.get(
|
|||||||
async (req: express.Request, res: express.Response) => {
|
async (req: express.Request, res: express.Response) => {
|
||||||
try {
|
try {
|
||||||
const user = await getUser(req);
|
const user = await getUser(req);
|
||||||
let token = user.accessToken;
|
let token = await user.getAccessToken();
|
||||||
if (user.isAdmin) {
|
if (user.isAdmin) {
|
||||||
token = (await getTokenForAdmin(user, req)) || token;
|
token = (await getTokenForAdmin(user, req)) || token;
|
||||||
}
|
}
|
||||||
@@ -521,7 +511,7 @@ router.post(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
const repository = await getRepositoryFromGitHub({
|
const repository = await getRepositoryFromGitHub({
|
||||||
accessToken: user.accessToken,
|
accessToken: await user.getAccessToken(),
|
||||||
owner: parsedRepository.owner,
|
owner: parsedRepository.owner,
|
||||||
repo: parsedRepository.name,
|
repo: parsedRepository.name,
|
||||||
});
|
});
|
||||||
@@ -533,7 +523,7 @@ router.post(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
await repository.getCommitInfo(repoUpdate.source.commit, {
|
await repository.getCommitInfo(repoUpdate.source.commit, {
|
||||||
accessToken: user.accessToken,
|
accessToken: await user.getAccessToken(),
|
||||||
});
|
});
|
||||||
repo.model.source.repositoryId = repository.model.id;
|
repo.model.source.repositoryId = repository.model.id;
|
||||||
repo.model.source.repositoryName =
|
repo.model.source.repositoryName =
|
||||||
@@ -648,7 +638,7 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
const repository = await getRepositoryFromGitHub({
|
const repository = await getRepositoryFromGitHub({
|
||||||
accessToken: user.accessToken,
|
accessToken: await user.getAccessToken(),
|
||||||
owner: r.owner,
|
owner: r.owner,
|
||||||
repo: r.name,
|
repo: r.name,
|
||||||
});
|
});
|
||||||
@@ -661,7 +651,7 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
await repository.getCommitInfo(repoUpdate.source.commit, {
|
await repository.getCommitInfo(repoUpdate.source.commit, {
|
||||||
accessToken: user.accessToken,
|
accessToken: await user.getAccessToken(),
|
||||||
});
|
});
|
||||||
|
|
||||||
const repo = new AnonymizedRepositoryModel();
|
const repo = new AnonymizedRepositoryModel();
|
||||||
@@ -671,7 +661,6 @@ router.post("/", async (req: express.Request, res: express.Response) => {
|
|||||||
|
|
||||||
updateRepoModel(repo, repoUpdate);
|
updateRepoModel(repo, repoUpdate);
|
||||||
repo.source.type = "GitHubStream";
|
repo.source.type = "GitHubStream";
|
||||||
repo.source.accessToken = user.accessToken;
|
|
||||||
repo.source.repositoryId = repository.model.id;
|
repo.source.repositoryId = repository.model.id;
|
||||||
repo.source.repositoryName = repoUpdate.fullName;
|
repo.source.repositoryName = repoUpdate.fullName;
|
||||||
|
|
||||||
@@ -762,7 +751,7 @@ router.post(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// verify the GitHub user exists and capture identity fields
|
// verify the GitHub user exists and capture identity fields
|
||||||
const oct = octokit(user.accessToken);
|
const oct = octokit(await user.getAccessToken());
|
||||||
let ghUser;
|
let ghUser;
|
||||||
try {
|
try {
|
||||||
const r = await oct.users.getByUsername({ username });
|
const r = await oct.users.getByUsername({ username });
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import CredentialModel from "../../core/model/credentials/credentials.model";
|
||||||
import * as express from "express";
|
import * as express from "express";
|
||||||
import got from "got";
|
import got from "got";
|
||||||
import config from "../../config";
|
import config from "../../config";
|
||||||
@@ -177,15 +178,15 @@ router.delete("/", async (req: express.Request, res: express.Response) => {
|
|||||||
await Promise.all([
|
await Promise.all([
|
||||||
AnonymizedRepositoryModel.updateMany(
|
AnonymizedRepositoryModel.updateMany(
|
||||||
{ owner: user.model._id },
|
{ owner: user.model._id },
|
||||||
{ $unset: { "source.accessToken": "" } }
|
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||||
).exec(),
|
).exec(),
|
||||||
AnonymizedPullRequestModel.updateMany(
|
AnonymizedPullRequestModel.updateMany(
|
||||||
{ owner: user.model._id },
|
{ owner: user.model._id },
|
||||||
{ $unset: { "source.accessToken": "" } }
|
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||||
).exec(),
|
).exec(),
|
||||||
AnonymizedGistModel.updateMany(
|
AnonymizedGistModel.updateMany(
|
||||||
{ owner: user.model._id },
|
{ owner: user.model._id },
|
||||||
{ $unset: { "source.accessToken": "" } }
|
{ $unset: { "source.accessToken": "", accessToken: "" } }
|
||||||
).exec(),
|
).exec(),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
@@ -199,13 +200,15 @@ router.delete("/", async (req: express.Request, res: express.Response) => {
|
|||||||
username: config.CLIENT_ID,
|
username: config.CLIENT_ID,
|
||||||
password: config.CLIENT_SECRET,
|
password: config.CLIENT_SECRET,
|
||||||
headers: { accept: "application/vnd.github+json" },
|
headers: { accept: "application/vnd.github+json" },
|
||||||
json: { access_token: user.accessToken },
|
json: { access_token: await user.getAccessToken() },
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
logger.warn("oauth grant revocation failed", serializeError(error));
|
logger.warn("oauth grant revocation failed", serializeError(error));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
await CredentialModel.deleteMany({ ownerId: user.model._id });
|
||||||
|
|
||||||
await UserModel.updateOne(
|
await UserModel.updateOne(
|
||||||
{ _id: user.model._id },
|
{ _id: user.model._id },
|
||||||
{
|
{
|
||||||
@@ -296,7 +299,7 @@ router.get(
|
|||||||
if (!q || q.length < 2) {
|
if (!q || q.length < 2) {
|
||||||
return res.json([]);
|
return res.json([]);
|
||||||
}
|
}
|
||||||
const oct = octokit(user.accessToken);
|
const oct = octokit(await user.getAccessToken());
|
||||||
const r = await oct.search.users({ q, per_page: 10 });
|
const r = await oct.search.users({ q, per_page: 10 });
|
||||||
res.json(
|
res.json(
|
||||||
r.data.items.map((u) => ({
|
r.data.items.map((u) => ({
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
const { expect } = require("chai");
|
||||||
|
require("ts-node/register/transpile-only");
|
||||||
|
const { createTokenCipher } = require("../src/core/credential-crypto");
|
||||||
|
const { redactSecrets } = require("../src/core/redact-secrets");
|
||||||
|
const keys = JSON.stringify({ old: Buffer.alloc(32, 1).toString("base64"), next: Buffer.alloc(32, 2).toString("base64") });
|
||||||
|
|
||||||
|
describe("credential encryption", () => {
|
||||||
|
const cipher = createTokenCipher(keys, "old");
|
||||||
|
it("round trips with independent nonces and no plaintext in the envelope", () => {
|
||||||
|
const a = cipher.encrypt("secret-token", "owner", "github");
|
||||||
|
const b = cipher.encrypt("secret-token", "owner", "github");
|
||||||
|
expect(a.nonce).not.to.equal(b.nonce);
|
||||||
|
expect(JSON.stringify(a)).not.to.include("secret-token");
|
||||||
|
expect(cipher.decrypt(a, "owner", "github")).to.equal("secret-token");
|
||||||
|
});
|
||||||
|
it("rejects altered envelopes and a different owner/provider", () => {
|
||||||
|
const a = cipher.encrypt("secret-token", "owner", "github");
|
||||||
|
for (const field of ["nonce", "tag", "ciphertext"]) {
|
||||||
|
const bytes = Buffer.from(a[field], "base64"); bytes[0] ^= 1;
|
||||||
|
expect(() => cipher.decrypt({ ...a, [field]: bytes.toString("base64") }, "owner", "github")).to.throw("Credential decryption failed");
|
||||||
|
}
|
||||||
|
for (const patch of [{ version: 2 }, { keyId: "missing" }, { tag: "YQ==" }, { nonce: "!!!!" }]) {
|
||||||
|
expect(() => cipher.decrypt({ ...a, ...patch }, "owner", "github")).to.throw();
|
||||||
|
}
|
||||||
|
expect(() => cipher.decrypt(a, "someone-else", "github")).to.throw();
|
||||||
|
expect(() => cipher.decrypt(a, "owner", "other")).to.throw();
|
||||||
|
const wrong = createTokenCipher(JSON.stringify({ old: Buffer.alloc(32, 3).toString("base64") }), "old");
|
||||||
|
expect(() => wrong.decrypt(a, "owner", "github")).to.throw();
|
||||||
|
});
|
||||||
|
it("supports key rotation while retaining reads of old credentials", () => {
|
||||||
|
const rotated = createTokenCipher(keys, "next");
|
||||||
|
expect(rotated.decrypt(cipher.encrypt("secret", "owner", "github"), "owner", "github")).to.equal("secret");
|
||||||
|
expect(rotated.encrypt("secret", "owner", "github").keyId).to.equal("next");
|
||||||
|
});
|
||||||
|
it("refuses missing, malformed, and short keys", () => {
|
||||||
|
for (const raw of ["", "null", "[]", "{}", '{"old":"YQ=="}']) {
|
||||||
|
expect(() => createTokenCipher(raw, "old")).to.throw();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
it("redacts nested credentials, authorization and tokens in URLs/errors", () => {
|
||||||
|
const input = { accessTokens: { github: "plain" }, nested: { authorization: "Bearer plain", encryptedToken: { ciphertext: "abc" } }, message: "failed ghp_abcdef https://example.test/?token=plain" };
|
||||||
|
const result = JSON.stringify(redactSecrets(input));
|
||||||
|
expect(result).not.to.include("plain");
|
||||||
|
expect(result).not.to.include("ghp_abcdef");
|
||||||
|
expect(result).not.to.include("abc");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,214 @@
|
|||||||
|
const process = require("process");
|
||||||
|
const { expect } = require("chai");
|
||||||
|
require("ts-node/register/transpile-only");
|
||||||
|
const mongoose = require("mongoose");
|
||||||
|
const { migrateCredentials, verifyCredentials, enforceCredentialStorage } = require("../src/core/migrate-credentials");
|
||||||
|
const { createTokenCipher } = require("../src/core/credential-crypto");
|
||||||
|
|
||||||
|
// Always use a newly named disposable database, never the database in the URI.
|
||||||
|
const describeMongo = process.env.TEST_MONGODB_URI ? describe : describe.skip;
|
||||||
|
describeMongo("credential migration (MongoDB)", function () {
|
||||||
|
this.timeout(20000);
|
||||||
|
let client, db, owner;
|
||||||
|
const cipher = createTokenCipher(JSON.stringify({ test: Buffer.alloc(32, 7).toString("base64") }), "test");
|
||||||
|
const apply = { apply: true, removeLegacy: true };
|
||||||
|
before(async () => {
|
||||||
|
client = new mongoose.mongo.MongoClient(process.env.TEST_MONGODB_URI);
|
||||||
|
await client.connect();
|
||||||
|
db = client.db(`credential_test_${new mongoose.Types.ObjectId()}`);
|
||||||
|
});
|
||||||
|
beforeEach(async () => {
|
||||||
|
await db.dropDatabase();
|
||||||
|
owner = new mongoose.Types.ObjectId();
|
||||||
|
await db.collection("users").insertOne({ _id: owner, username: "test", accessTokens: { github: "secret" } });
|
||||||
|
});
|
||||||
|
after(async () => { if (db) await db.dropDatabase(); if (client) await client.close(); });
|
||||||
|
it("dry run does not modify data", async () => {
|
||||||
|
const result = await migrateCredentials(db, cipher);
|
||||||
|
expect(result.created).to.equal(1);
|
||||||
|
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||||
|
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||||
|
});
|
||||||
|
it("encrypts once, cleans all legacy locations, and is safe to rerun", async () => {
|
||||||
|
for (const name of ["anonymizedrepositories", "anonymizedgists", "anonymizedpullrequests"]) {
|
||||||
|
await db.collection(name).insertOne({ owner, source: { accessToken: "secret" }, accessToken: "secret" });
|
||||||
|
}
|
||||||
|
const first = await migrateCredentials(db, cipher, apply);
|
||||||
|
expect(first.issues).to.equal(0);
|
||||||
|
const credential = await db.collection("credentials").findOne({ ownerId: owner });
|
||||||
|
expect(JSON.stringify(credential)).not.to.include("secret");
|
||||||
|
expect(cipher.decrypt(credential.encryptedToken, String(owner), "github")).to.equal("secret");
|
||||||
|
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).created).to.equal(0);
|
||||||
|
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(credential.encryptedToken);
|
||||||
|
});
|
||||||
|
it("reports resource conflicts without deleting either token", async () => {
|
||||||
|
await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "different" } });
|
||||||
|
const events = [];
|
||||||
|
const result = await migrateCredentials(db, cipher, { ...apply, report: e => events.push(e) });
|
||||||
|
expect(result.issues).to.equal(1);
|
||||||
|
expect(JSON.stringify(events)).not.to.include("different");
|
||||||
|
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||||
|
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||||
|
expect((await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true })).issues).to.equal(0);
|
||||||
|
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||||
|
});
|
||||||
|
it("preserves a newer credential and requires explicit conflict resolution", async () => {
|
||||||
|
const envelope = cipher.encrypt("new-login", String(owner), "github");
|
||||||
|
await db.collection("credentials").insertOne({ ownerId: owner, provider: "github", encryptedToken: envelope, updatedAt: new Date() });
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(1);
|
||||||
|
await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true });
|
||||||
|
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(envelope);
|
||||||
|
});
|
||||||
|
it("migrates resource-only credentials but never arbitrarily picks between them", async () => {
|
||||||
|
await db.collection("users").updateOne({ _id: owner }, { $unset: { accessTokens: "" } });
|
||||||
|
await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "resource" } });
|
||||||
|
await db.collection("anonymizedpullrequests").insertOne({ owner, source: { accessToken: "other" } });
|
||||||
|
expect((await migrateCredentials(db, cipher, { ...apply, preferOwnerToken: true })).issues).to.equal(1);
|
||||||
|
await db.collection("anonymizedpullrequests").deleteMany({});
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(0);
|
||||||
|
const row = await db.collection("credentials").findOne({ ownerId: owner });
|
||||||
|
expect(cipher.decrypt(row.encryptedToken, String(owner), "github")).to.equal("resource");
|
||||||
|
});
|
||||||
|
it("reports missing owners and malformed tokens", async () => {
|
||||||
|
await db.collection("anonymizedrepositories").insertOne({ source: { accessToken: "orphan" } });
|
||||||
|
await db.collection("users").updateOne({ _id: owner }, { $set: { "accessTokens.github": { invalid: true } } });
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(2);
|
||||||
|
expect(await db.collection("credentials").countDocuments()).to.equal(0);
|
||||||
|
});
|
||||||
|
it("does not recreate credentials for removed accounts", async () => {
|
||||||
|
await db.collection("users").updateOne({ _id: owner }, { $set: { status: "removed" } });
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(0);
|
||||||
|
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 0, legacy: 0 });
|
||||||
|
});
|
||||||
|
it("resumes after interruption between backfill and cleanup", async () => {
|
||||||
|
await migrateCredentials(db, cipher, { apply: true });
|
||||||
|
const envelope = (await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken;
|
||||||
|
expect((await verifyCredentials(db, cipher)).legacy).to.equal(1);
|
||||||
|
await migrateCredentials(db, cipher, apply);
|
||||||
|
expect(await verifyCredentials(db, cipher)).to.deep.equal({ checked: 1, legacy: 0 });
|
||||||
|
expect((await db.collection("credentials").findOne({ ownerId: owner })).encryptedToken).to.deep.equal(envelope);
|
||||||
|
});
|
||||||
|
it("refuses cleanup when ciphertext cannot be authenticated", async () => {
|
||||||
|
await migrateCredentials(db, cipher, { apply: true });
|
||||||
|
await db.collection("credentials").updateOne({ ownerId: owner }, { $set: { "encryptedToken.tag": Buffer.alloc(16).toString("base64") } });
|
||||||
|
expect((await migrateCredentials(db, cipher, apply)).issues).to.equal(1);
|
||||||
|
expect((await db.collection("users").findOne({ _id: owner })).accessTokens.github).to.equal("secret");
|
||||||
|
});
|
||||||
|
it("enforces the unique owner/provider index", async () => {
|
||||||
|
await migrateCredentials(db, cipher, apply);
|
||||||
|
const row = await db.collection("credentials").findOne({ ownerId: owner });
|
||||||
|
delete row._id;
|
||||||
|
try { await db.collection("credentials").insertOne(row); throw new Error("expected duplicate"); }
|
||||||
|
catch (error) { expect(error.code).to.equal(11000); }
|
||||||
|
});
|
||||||
|
it("enforces plaintext rejection in MongoDB after verification", async () => {
|
||||||
|
await migrateCredentials(db, cipher, apply);
|
||||||
|
await enforceCredentialStorage(db, cipher);
|
||||||
|
try { await db.collection("users").updateOne({ _id: owner }, { $set: { "accessTokens.github": "bad" } }); throw new Error("expected rejection"); }
|
||||||
|
catch (error) { expect(error.code).to.equal(121); }
|
||||||
|
try { await db.collection("anonymizedgists").insertOne({ owner, source: { accessToken: "bad" } }); throw new Error("expected rejection"); }
|
||||||
|
catch (error) { expect(error.code).to.equal(121); }
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describeMongo("credential access (MongoDB)", function () {
|
||||||
|
this.timeout(20000);
|
||||||
|
const config = require("../src/config").default;
|
||||||
|
const Credential = require("../src/core/model/credentials/credentials.model").default;
|
||||||
|
const UserModel = require("../src/core/model/users/users.model").default;
|
||||||
|
const RepoModel = require("../src/core/model/anonymizedRepositories/anonymizedRepositories.model").default;
|
||||||
|
const GistModel = require("../src/core/model/anonymizedGists/anonymizedGists.model").default;
|
||||||
|
const PullModel = require("../src/core/model/anonymizedPullRequests/anonymizedPullRequests.model").default;
|
||||||
|
const { setCredential, getCredentialToken, replaceCredential } = require("../src/core/credentials");
|
||||||
|
let settings, owner;
|
||||||
|
before(async () => {
|
||||||
|
settings = [config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS];
|
||||||
|
config.CREDENTIAL_KEYS = JSON.stringify({ test: Buffer.alloc(32, 9).toString("base64") });
|
||||||
|
config.CREDENTIAL_ACTIVE_KEY_ID = "test";
|
||||||
|
config.CREDENTIAL_LEGACY_READS = false;
|
||||||
|
await mongoose.connect(process.env.TEST_MONGODB_URI, { dbName: `credential_access_test_${new mongoose.Types.ObjectId()}` });
|
||||||
|
await Credential.init();
|
||||||
|
});
|
||||||
|
beforeEach(async () => {
|
||||||
|
await Credential.deleteMany({});
|
||||||
|
owner = await UserModel.create({ username: `owner-${new mongoose.Types.ObjectId()}` });
|
||||||
|
});
|
||||||
|
after(async () => {
|
||||||
|
await mongoose.connection.dropDatabase();
|
||||||
|
await mongoose.disconnect();
|
||||||
|
[config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS] = settings;
|
||||||
|
});
|
||||||
|
it("persists only encrypted credentials and hides envelopes in ordinary queries", async () => {
|
||||||
|
await setCredential(owner.id, "real-secret");
|
||||||
|
const raw = await Credential.collection.findOne({ ownerId: owner._id });
|
||||||
|
expect(raw.ownerId.equals(owner._id)).to.equal(true);
|
||||||
|
expect(JSON.stringify(raw)).not.to.include("real-secret");
|
||||||
|
expect((await Credential.findOne({ ownerId: owner._id })).encryptedToken).to.equal(undefined);
|
||||||
|
expect(await getCredentialToken(owner.id)).to.equal("real-secret");
|
||||||
|
expect((await UserModel.collection.findOne({ _id: owner._id })).accessTokens).to.equal(undefined);
|
||||||
|
});
|
||||||
|
it("handles simultaneous logins with one owner/provider row", async () => {
|
||||||
|
await Promise.all(Array.from({ length: 8 }, (_, i) => setCredential(owner.id, `token-${i}`)));
|
||||||
|
expect(await Credential.countDocuments({ ownerId: owner._id })).to.equal(1);
|
||||||
|
expect(await getCredentialToken(owner.id)).to.match(/^token-\d$/);
|
||||||
|
});
|
||||||
|
it("updates the encrypted envelope atomically on refresh", async () => {
|
||||||
|
await setCredential(owner.id, "previous");
|
||||||
|
expect(await replaceCredential(owner.id, "previous", "next")).to.equal(true);
|
||||||
|
expect(await replaceCredential(owner.id, "previous", "stale")).to.equal(false);
|
||||||
|
expect(await getCredentialToken(owner.id)).to.equal("next");
|
||||||
|
});
|
||||||
|
it("resolves gists and pull requests by owner and observes token changes", async () => {
|
||||||
|
const Gist = require("../src/core/Gist").default;
|
||||||
|
const PullRequest = require("../src/core/PullRequest").default;
|
||||||
|
const gist = new Gist(await GistModel.create({ owner: owner._id, source: { gistId: "test" } }));
|
||||||
|
const pull = new PullRequest(await PullModel.create({ owner: owner._id, source: { pullRequestId: "1" } }));
|
||||||
|
await setCredential(owner.id, "first");
|
||||||
|
expect(await gist.getToken()).to.equal("first");
|
||||||
|
expect(await pull.getToken()).to.equal("first");
|
||||||
|
await setCredential(owner.id, "second");
|
||||||
|
expect(await gist.getToken()).to.equal("second");
|
||||||
|
expect(await pull.getToken()).to.equal("second");
|
||||||
|
expect((await GistModel.collection.findOne({ _id: gist.model._id })).source.accessToken).to.equal(undefined);
|
||||||
|
expect((await PullModel.collection.findOne({ _id: pull.model._id })).source.accessToken).to.equal(undefined);
|
||||||
|
});
|
||||||
|
it("refreshes a repository credential without copying it onto the repository", async () => {
|
||||||
|
const Repository = require("../src/core/Repository").default;
|
||||||
|
const repo = new Repository(await RepoModel.create({ owner: owner._id, repoId: "test-repo", source: { type: "GitHubStream" } }));
|
||||||
|
await setCredential(owner.id, "old-token");
|
||||||
|
await Credential.updateOne({ ownerId: owner._id }, { $set: { updatedAt: new Date(0) } });
|
||||||
|
const originalFetch = global.fetch;
|
||||||
|
global.fetch = async (url, options) => {
|
||||||
|
expect(url).to.include("api.github.com/applications/");
|
||||||
|
expect(JSON.parse(options.body).access_token).to.equal("old-token");
|
||||||
|
return { ok: true, json: async () => ({ token: "refreshed-token" }) };
|
||||||
|
};
|
||||||
|
try { expect(await repo.getToken()).to.equal("refreshed-token"); }
|
||||||
|
finally { global.fetch = originalFetch; }
|
||||||
|
expect(await getCredentialToken(owner.id)).to.equal("refreshed-token");
|
||||||
|
expect((await RepoModel.collection.findOne({ _id: repo.model._id })).source.accessToken).to.equal(undefined);
|
||||||
|
});
|
||||||
|
it("OAuth login writes a credential and returns a token-free session user", async () => {
|
||||||
|
const passport = require("passport");
|
||||||
|
require("../src/server/routes/connection");
|
||||||
|
const result = await new Promise((resolve, reject) => passport._strategy("github")._verify("oauth-secret", "refresh-secret", {
|
||||||
|
id: "external-test", username: owner.username, emails: [], photos: [],
|
||||||
|
}, (error, user) => error ? reject(error) : resolve(user)));
|
||||||
|
expect(JSON.stringify(result)).not.to.include("oauth-secret");
|
||||||
|
expect(JSON.stringify(result)).not.to.include("refresh-secret");
|
||||||
|
expect(await getCredentialToken(owner.id)).to.equal("oauth-secret");
|
||||||
|
expect((await UserModel.collection.findOne({ _id: owner._id })).accessTokens).to.equal(undefined);
|
||||||
|
});
|
||||||
|
it("reads hidden legacy resource tokens only during compatibility mode", async () => {
|
||||||
|
const resource = await GistModel.collection.insertOne({ gistId: "legacy-gist", owner: owner._id, source: { accessToken: "legacy-resource" } });
|
||||||
|
const lookup = { collection: "anonymizedgists", id: resource.insertedId };
|
||||||
|
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("");
|
||||||
|
config.CREDENTIAL_LEGACY_READS = true;
|
||||||
|
try {
|
||||||
|
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("legacy-resource");
|
||||||
|
await setCredential(owner.id, "encrypted-wins");
|
||||||
|
expect(await getCredentialToken(owner.id, "github", lookup)).to.equal("encrypted-wins");
|
||||||
|
} finally { config.CREDENTIAL_LEGACY_READS = false; }
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
const { expect } = require("chai");
|
||||||
|
require("ts-node/register/transpile-only");
|
||||||
|
const passport = require("passport");
|
||||||
|
require("../src/server/routes/connection");
|
||||||
|
|
||||||
|
describe("credential-free sessions", () => {
|
||||||
|
it("serializes only the owner ID", done => {
|
||||||
|
passport.serializeUser({ user: { _id: "507f1f77bcf86cd799439011", accessTokens: { github: "secret" } }, accessToken: "secret" }, (error, value) => {
|
||||||
|
expect(error).to.equal(null);
|
||||||
|
expect(value).to.equal("507f1f77bcf86cd799439011");
|
||||||
|
done();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
it("rejects the legacy session object", done => {
|
||||||
|
passport.deserializeUser({ user: { _id: "507f1f77bcf86cd799439011" }, accessToken: "secret" }, (error, value) => {
|
||||||
|
expect(error).to.equal(null);
|
||||||
|
expect(value).to.equal(false);
|
||||||
|
done();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
const { expect } = require("chai");
|
||||||
|
require("ts-node/register/transpile-only");
|
||||||
|
const { Types } = require("mongoose");
|
||||||
|
const config = require("../src/config").default;
|
||||||
|
const Model = require("../src/core/model/credentials/credentials.model").default;
|
||||||
|
const UserModel = require("../src/core/model/users/users.model").default;
|
||||||
|
const { getCredentialToken, setCredential, replaceCredential, credentialCipher } = require("../src/core/credentials");
|
||||||
|
|
||||||
|
describe("credential storage boundary", () => {
|
||||||
|
const owner = new Types.ObjectId().toString();
|
||||||
|
let original, settings, stored;
|
||||||
|
beforeEach(() => {
|
||||||
|
original = { findOne: Model.findOne, updateOne: Model.updateOne, findById: UserModel.findById };
|
||||||
|
settings = [config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS];
|
||||||
|
config.CREDENTIAL_KEYS = JSON.stringify({ test: Buffer.alloc(32, 4).toString("base64") });
|
||||||
|
config.CREDENTIAL_ACTIVE_KEY_ID = "test";
|
||||||
|
config.CREDENTIAL_LEGACY_READS = false;
|
||||||
|
stored = null;
|
||||||
|
Model.findOne = () => ({ select: () => ({ lean: async () => stored }) });
|
||||||
|
Model.updateOne = async (filter, update) => {
|
||||||
|
stored = { _id: new Types.ObjectId(), ownerId: owner, provider: "github", ...update.$set };
|
||||||
|
return { modifiedCount: 1 };
|
||||||
|
};
|
||||||
|
});
|
||||||
|
afterEach(() => {
|
||||||
|
Object.assign(Model, { findOne: original.findOne, updateOne: original.updateOne });
|
||||||
|
UserModel.findById = original.findById;
|
||||||
|
[config.CREDENTIAL_KEYS, config.CREDENTIAL_ACTIVE_KEY_ID, config.CREDENTIAL_LEGACY_READS] = settings;
|
||||||
|
});
|
||||||
|
it("writes only ciphertext and resolves the token by owner/provider", async () => {
|
||||||
|
await setCredential(owner, "github-secret");
|
||||||
|
expect(JSON.stringify(stored)).not.to.include("github-secret");
|
||||||
|
expect(await getCredentialToken(owner)).to.equal("github-secret");
|
||||||
|
});
|
||||||
|
it("refreshes only the credential whose token still matches", async () => {
|
||||||
|
await setCredential(owner, "new-login-token");
|
||||||
|
expect(await replaceCredential(owner, "stale-token", "refresh-token")).to.equal(false);
|
||||||
|
expect(await getCredentialToken(owner)).to.equal("new-login-token");
|
||||||
|
expect(await replaceCredential(owner, "new-login-token", "refresh-token")).to.equal(true);
|
||||||
|
expect(await getCredentialToken(owner)).to.equal("refresh-token");
|
||||||
|
});
|
||||||
|
it("fails closed on corrupt ciphertext even with legacy reads enabled", async () => {
|
||||||
|
config.CREDENTIAL_LEGACY_READS = true;
|
||||||
|
await setCredential(owner, "github-secret");
|
||||||
|
stored.encryptedToken.tag = Buffer.alloc(16).toString("base64");
|
||||||
|
UserModel.findById = () => { throw new Error("must not fall back"); };
|
||||||
|
try { await getCredentialToken(owner); throw new Error("expected failure"); }
|
||||||
|
catch (error) { expect(error.message).to.equal("Credential decryption failed"); }
|
||||||
|
});
|
||||||
|
it("reads legacy users only when explicitly enabled", async () => {
|
||||||
|
let reads = 0;
|
||||||
|
UserModel.findById = () => ({ select: async () => { reads++; return { accessTokens: { github: "legacy" } }; } });
|
||||||
|
expect(await getCredentialToken(owner)).to.equal("");
|
||||||
|
expect(reads).to.equal(0);
|
||||||
|
config.CREDENTIAL_LEGACY_READS = true;
|
||||||
|
expect(await getCredentialToken(owner)).to.equal("legacy");
|
||||||
|
});
|
||||||
|
it("uses a unique owner/provider index and hides envelopes by default", () => {
|
||||||
|
expect(Model.schema.indexes()).to.deep.include([{ ownerId: 1, provider: 1 }, { unique: true, background: true }]);
|
||||||
|
expect(Model.schema.path("encryptedToken").options.select).to.equal(false);
|
||||||
|
expect(credentialCipher()).to.have.property("encrypt");
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user