mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-10-19 22:27:20 +02:00
fix: retain recipient suppression after access recovery (#897)
* fix: retain recipient suppression after access recovery * fix: migrate cold starts and fence resource deactivation * docs: stop every writer before alert reconciliation
This commit is contained in:
1 parent
445a479339
commit
772e40debd
12 files changed
+429
-122
No files matched your search
+1
-1
@@ -26,7 +26,7 @@ RUN --mount=type=cache,id=npm-prod,target=/root/.npm,sharing=locked \
|
|||||||
|
|
||||||
# Stage 4: minimal runtime
|
# Stage 4: minimal runtime
|
||||||
FROM node:22-alpine AS runtime
|
FROM node:22-alpine AS runtime
|
||||||
LABEL science.4open.owner-alert-protocol="recipient-v1"
|
LABEL science.4open.owner-alert-protocol="recipient-v2"
|
||||||
ENV NODE_ENV=production
|
ENV NODE_ENV=production
|
||||||
ENV PORT=5000
|
ENV PORT=5000
|
||||||
EXPOSE 5000
|
EXPOSE 5000
|
||||||
|
|||||||
@@ -116,20 +116,15 @@ Alerts cover invalid GitHub credentials and repository access errors encountered
|
|||||||
while reading or refreshing content. No periodic access scan is performed.
|
while reading or refreshing content. No periodic access scan is performed.
|
||||||
Emails contain no repository names, anonymous URLs, or credentials.
|
Emails contain no repository names, anonymous URLs, or credentials.
|
||||||
|
|
||||||
An atomic recipient claim allows one email attempt across repositories, pull
|
An atomic recipient claim allows one email attempt per normalized address across
|
||||||
requests, gists and accounts sharing the same address. Sign-in does not reopen
|
repositories, pull requests, gists and accounts. Suppression has no time expiry.
|
||||||
alerts. An explicit reconnect or resource change prepares recovery, but the
|
Sign-in, reconnecting, successful access recovery, resource removal or expiration,
|
||||||
recipient stays blocked until a fresh GitHub read confirms access to the resource
|
and disabling the original account retain suppression for that address. Inactive
|
||||||
that triggered the alert. A fresh read during an in-flight send is retained and
|
resources cannot trigger new alerts. Changing to an address that has never been
|
||||||
applied when that send completes. A newer reset or source change invalidates
|
notified permits its first alert.
|
||||||
that recovery evidence. Cached reads and successful reads of other resources
|
|
||||||
cannot reopen it. Removed or expired resources and disabled owners no longer
|
|
||||||
block future alerts after their send has completed.
|
|
||||||
|
|
||||||
Dispatched recipient guards have no time expiry, so there are no daily reminders.
|
Failed deliveries and interrupted sends whose outcome is unknown also stay
|
||||||
Failed deliveries remain blocked until confirmed recovery. An interrupted send
|
blocked to avoid duplicate delivery. Unsent claims can be cancelled by confirmed recovery before dispatch. Bookkeeping failures
|
||||||
whose outcome is unknown stays blocked to avoid duplicate delivery. Unsent claims
|
|
||||||
can be cancelled by confirmed recovery before dispatch. Bookkeeping failures
|
|
||||||
release claims when no provider request has started. Pending claims interrupted
|
release claims when no provider request has started. Pending claims interrupted
|
||||||
before dispatch can be reclaimed after a 60-second lease during startup or a new
|
before dispatch can be reclaimed after a 60-second lease during startup or a new
|
||||||
notification attempt. Claim identity and state checks prevent a paused sender
|
notification attempt. Claim identity and state checks prevent a paused sender
|
||||||
@@ -150,22 +145,31 @@ Recovery recorded during an active reset is retried when reset finishes or its
|
|||||||
lease expires, with the original source and revision checks.
|
lease expires, with the original source and revision checks.
|
||||||
Provider idempotency keys
|
Provider idempotency keys
|
||||||
and redacted attempt records support auditing without storing recipient addresses
|
and redacted attempt records support auditing without storing recipient addresses
|
||||||
or repository names in the audit log. Attempt records are saved before dispatch,
|
or repository names in the audit log. Records include owner and resource IDs,
|
||||||
|
resource kind, generation and a categorized access failure to identify the
|
||||||
|
trigger without exposing the address. Attempt records are saved before dispatch,
|
||||||
with completion recorded separately. Existing resource claims are migrated to
|
with completion recorded separately. Existing resource claims are migrated to
|
||||||
recipient guards before new application instances serve requests using an indexed
|
recipient guards before new application instances serve requests using an indexed
|
||||||
scan with resumable checkpoints. Other instances wait while its lease is healthy,
|
scan with resumable checkpoints. Other instances wait while its lease is healthy,
|
||||||
and take over if the lease expires.
|
and take over if the lease expires.
|
||||||
Every existing resource claim is retained behind the recipient guard. Recovering
|
Every existing resource claim is retained behind the recipient guard. Recovery
|
||||||
one resource atomically marks its claim recovered; other claims keep the recipient
|
marks claims recovered without authorizing another email. Completed provider
|
||||||
blocked until they recover or are retired. A newer owner reset makes its matching
|
results are recorded before guard updates, with bounded retries and reconciliation
|
||||||
claims require another fresh read. Completed provider results are recorded before
|
on startup or a later notification. The upgrade restores missing recipient guards
|
||||||
guard updates, with bounded retries and reconciliation on startup or a later
|
from immutable provider completion records, including guards previously released
|
||||||
notification. Retirement also checks the latest owner reset revision.
|
by recovery or resource removal. This history scan uses an index and resumable
|
||||||
|
checkpoints and does not send email. Its `recipient-alerts-v2` database marker is
|
||||||
|
independent of the v1 marker, so cold starts also restore missing suppression.
|
||||||
|
Resource lifecycle mutations advance the owner dispatch fence and block new
|
||||||
|
attempts until the status write completes. Concurrent mutations keep separate
|
||||||
|
reservations. Failed or ambiguous status writes retain suppression for that
|
||||||
|
owner; run the offline reconciliation command below after stopping all writers
|
||||||
|
to clear interrupted reservations safely.
|
||||||
|
|
||||||
Use `scripts/deploy.sh` for the first upgrade to recipient alerts. The new image
|
Use `scripts/deploy.sh` when upgrading recipient alert suppression. The new image
|
||||||
has the `science.4open.owner-alert-protocol=recipient-v1` label. When running
|
has the `science.4open.owner-alert-protocol=recipient-v2` label. When running
|
||||||
application containers lack that label, the script builds the replacement, stops
|
application containers have an older or missing protocol label, the script builds
|
||||||
and verifies all old API and streamer containers, then prepares a full claim
|
the replacement, stops and verifies all old API and streamer containers, then prepares a full claim
|
||||||
reconciliation and starts the new streamers and API. Stopped legacy containers
|
reconciliation and starts the new streamers and API. Stopped legacy containers
|
||||||
from a rollback also trigger reconciliation, so claims written after an earlier
|
from a rollback also trigger reconciliation, so claims written after an earlier
|
||||||
migration are included. This path also runs when retrying with an already-built
|
migration are included. This path also runs when retrying with an already-built
|
||||||
@@ -174,7 +178,7 @@ unsent pending claims and clears their matching legacy fields before backfill.
|
|||||||
This upgrade pauses traffic while claims migrate and allows up to 15
|
This upgrade pauses traffic while claims migrate and allows up to 15
|
||||||
minutes for startup, configurable with `MIGRATION_HEALTH_TIMEOUT`. Subsequent
|
minutes for startup, configurable with `MIGRATION_HEALTH_TIMEOUT`. Subsequent
|
||||||
deployments keep the usual rolling streamer replacement. Manual upgrades must
|
deployments keep the usual rolling streamer replacement. Manual upgrades must
|
||||||
stop all legacy application writers, run
|
stop all application writers, run
|
||||||
`docker compose run --rm --no-deps --entrypoint node anonymous_github ./build/scripts/reconcile-legacy-owner-alerts.js`
|
`docker compose run --rm --no-deps --entrypoint node anonymous_github ./build/scripts/reconcile-legacy-owner-alerts.js`
|
||||||
using the new image, and then start the new application instances. Run this
|
using the new image, and then start the new application instances. Run this
|
||||||
preparation after legacy writers have operated, even if their containers were
|
preparation after legacy writers have operated, even if their containers were
|
||||||
|
|||||||
+4
-4
@@ -37,15 +37,15 @@ list_streamers() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# Capture application writers, including stopped containers from a rollback.
|
# Capture application writers, including stopped containers from a rollback.
|
||||||
# Legacy instances do not participate in recipient claims, so they must stop
|
# Older instances can release recipient guards, so they must stop before
|
||||||
# before a new instance marks the one-time claim migration complete.
|
# a new instance marks permanent-suppression migration complete.
|
||||||
application_ids="$(docker compose ps --all -q anonymous_github streamer)"
|
application_ids="$(docker compose ps --all -q anonymous_github streamer)"
|
||||||
apps=()
|
apps=()
|
||||||
if [[ -n $application_ids ]]; then mapfile -t apps <<<"$application_ids"; fi
|
if [[ -n $application_ids ]]; then mapfile -t apps <<<"$application_ids"; fi
|
||||||
legacy_alerts=0
|
legacy_alerts=0
|
||||||
for app in "${apps[@]}"; do
|
for app in "${apps[@]}"; do
|
||||||
protocol="$(docker inspect -f '{{index .Config.Labels "science.4open.owner-alert-protocol"}}' "$app")"
|
protocol="$(docker inspect -f '{{index .Config.Labels "science.4open.owner-alert-protocol"}}' "$app")"
|
||||||
[[ $protocol == recipient-v1 ]] || legacy_alerts=1
|
[[ $protocol == recipient-v2 ]] || legacy_alerts=1
|
||||||
done
|
done
|
||||||
|
|
||||||
# 1. Build
|
# 1. Build
|
||||||
@@ -56,7 +56,7 @@ after="$(image_id "$IMAGE")"
|
|||||||
|
|
||||||
if [[ $legacy_alerts == 1 ]]; then
|
if [[ $legacy_alerts == 1 ]]; then
|
||||||
protocol="$(docker image inspect -f '{{index .Config.Labels "science.4open.owner-alert-protocol"}}' "$IMAGE")"
|
protocol="$(docker image inspect -f '{{index .Config.Labels "science.4open.owner-alert-protocol"}}' "$IMAGE")"
|
||||||
[[ $protocol == recipient-v1 ]] || { warn "replacement image lacks recipient alert migration support"; exit 1; }
|
[[ $protocol == recipient-v2 ]] || { warn "replacement image lacks recipient alert migration support"; exit 1; }
|
||||||
TIMEOUT="${MIGRATION_HEALTH_TIMEOUT:-900}"
|
TIMEOUT="${MIGRATION_HEALTH_TIMEOUT:-900}"
|
||||||
log "Pausing legacy application writers before recipient alert migration"
|
log "Pausing legacy application writers before recipient alert migration"
|
||||||
docker compose stop -t 30 anonymous_github streamer
|
docker compose stop -t 30 anonymous_github streamer
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { ownerAlertLifecycle } from "../owner-alert-lifecycle.schema";
|
||||||
import { Schema } from "mongoose";
|
import { Schema } from "mongoose";
|
||||||
|
|
||||||
const AnonymizedGistSchema = new Schema({
|
const AnonymizedGistSchema = new Schema({
|
||||||
@@ -74,4 +75,6 @@ const AnonymizedGistSchema = new Schema({
|
|||||||
|
|
||||||
AnonymizedGistSchema.index({ status: 1, "options.expirationDate": 1 });
|
AnonymizedGistSchema.index({ status: 1, "options.expirationDate": 1 });
|
||||||
|
|
||||||
|
AnonymizedGistSchema.plugin(ownerAlertLifecycle);
|
||||||
|
|
||||||
export default AnonymizedGistSchema;
|
export default AnonymizedGistSchema;
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { ownerAlertLifecycle } from "../owner-alert-lifecycle.schema";
|
||||||
import { repositoryAccessSchema } from "../repository-access.schema";
|
import { repositoryAccessSchema } from "../repository-access.schema";
|
||||||
import { Schema } from "mongoose";
|
import { Schema } from "mongoose";
|
||||||
|
|
||||||
@@ -71,4 +72,6 @@ const AnonymizedPullRequestSchema = new Schema({
|
|||||||
|
|
||||||
AnonymizedPullRequestSchema.index({ status: 1, "options.expirationDate": 1 });
|
AnonymizedPullRequestSchema.index({ status: 1, "options.expirationDate": 1 });
|
||||||
|
|
||||||
|
AnonymizedPullRequestSchema.plugin(ownerAlertLifecycle);
|
||||||
|
|
||||||
export default AnonymizedPullRequestSchema;
|
export default AnonymizedPullRequestSchema;
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { ownerAlertLifecycle } from "../owner-alert-lifecycle.schema";
|
||||||
import { repositoryAccessSchema } from "../repository-access.schema";
|
import { repositoryAccessSchema } from "../repository-access.schema";
|
||||||
import { Schema } from "mongoose";
|
import { Schema } from "mongoose";
|
||||||
|
|
||||||
@@ -123,4 +124,6 @@ AnonymizedRepositorySchema.index({
|
|||||||
"options.expirationDate": 1,
|
"options.expirationDate": 1,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
AnonymizedRepositorySchema.plugin(ownerAlertLifecycle);
|
||||||
|
|
||||||
export default AnonymizedRepositorySchema;
|
export default AnonymizedRepositorySchema;
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import { randomUUID } from "crypto";
|
||||||
|
import { Document, Query, Schema, Types, mongo } from "mongoose";
|
||||||
|
|
||||||
|
const inactive = new Set(["archived", "expired", "expiring", "removed", "removing"]);
|
||||||
|
type Owner = { _id: Types.ObjectId; accessAlertResetRevision?: number; accessAlertDeactivations?: string[] };
|
||||||
|
type Fence = { users: mongo.Collection<Owner>; owners: Types.ObjectId[]; token: string };
|
||||||
|
|
||||||
|
/** Lifecycle mutations and email dispatch compete on the owner document. */
|
||||||
|
export function ownerAlertLifecycle(schema: Schema): void {
|
||||||
|
const fences = new WeakMap<object, Fence>();
|
||||||
|
const begin = async (operation: object, db: { collection(name: string): unknown }, owners: Types.ObjectId[]) => {
|
||||||
|
if (!owners.length) return;
|
||||||
|
const users = db.collection("users") as mongo.Collection<Owner>;
|
||||||
|
const token = randomUUID();
|
||||||
|
// Do not expire this reservation: an interrupted database mutation may still
|
||||||
|
// commit. Offline reconciliation can clear it after all writers have stopped.
|
||||||
|
await users.updateMany({ _id: { $in: owners } }, {
|
||||||
|
$inc: { accessAlertResetRevision: 1 }, $addToSet: { accessAlertDeactivations: token },
|
||||||
|
});
|
||||||
|
fences.set(operation, { users, owners, token });
|
||||||
|
};
|
||||||
|
const finish = async (operation: object) => {
|
||||||
|
const fence = fences.get(operation);
|
||||||
|
if (!fence) return;
|
||||||
|
await fence.users.updateMany({ _id: { $in: fence.owners } }, { $pull: { accessAlertDeactivations: fence.token } });
|
||||||
|
fences.delete(operation);
|
||||||
|
};
|
||||||
|
schema.pre(["updateOne", "updateMany", "findOneAndUpdate"], async function (this: Query<unknown, unknown>) {
|
||||||
|
const update = this.getUpdate();
|
||||||
|
if (!update) return;
|
||||||
|
const stages = (Array.isArray(update) ? update : [update]) as { $set?: { status?: unknown }; status?: unknown }[];
|
||||||
|
if (!stages.some(stage => { const status = stage.$set?.status ?? stage.status; return typeof status === "string" && inactive.has(status); })) return;
|
||||||
|
const filter = this.getFilter();
|
||||||
|
const lookup = this.model.distinct("owner", filter);
|
||||||
|
const collation = this.getOptions().collation;
|
||||||
|
if (collation) lookup.collation(collation);
|
||||||
|
const owners = (await lookup.exec()).filter(id => id && Types.ObjectId.isValid(id));
|
||||||
|
await begin(this, this.model.db, owners);
|
||||||
|
// Preserve the owner snapshot: reassignment must not move this mutation to
|
||||||
|
// an account whose dispatch fence was not advanced.
|
||||||
|
this.setQuery({ $and: [filter, { $or: [{ owner: { $in: owners } }, { owner: null }] }] });
|
||||||
|
});
|
||||||
|
schema.post(["updateOne", "updateMany", "findOneAndUpdate"], async function () { await finish(this); });
|
||||||
|
schema.pre("save", async function (this: Document & { status?: string; owner?: Types.ObjectId; $where?: Record<string, unknown> }) {
|
||||||
|
if (!inactive.has(this.status || "") || !this.isModified("status")) return;
|
||||||
|
const current = this.isNew ? null : await this.collection.findOne({ _id: this._id }, { projection: { owner: 1 } });
|
||||||
|
const owners = [current?.owner, this.owner].filter((id): id is Types.ObjectId => Boolean(id && Types.ObjectId.isValid(id)));
|
||||||
|
await begin(this, this.db, owners);
|
||||||
|
if (current) this.$where = { $and: [this.$where || {}, { owner: current.owner ?? null }] };
|
||||||
|
});
|
||||||
|
schema.post("save", async function () { await finish(this); });
|
||||||
|
// Failed or ambiguous mutations retain their reservation. Releasing one here
|
||||||
|
// could permit dispatch while the resource write subsequently completes.
|
||||||
|
}
|
||||||
@@ -21,6 +21,7 @@ const UserSchema = new Schema({
|
|||||||
accessAlertResetRevision: { type: Number, select: false },
|
accessAlertResetRevision: { type: Number, select: false },
|
||||||
accessAlertResetUntil: { type: Date, select: false },
|
accessAlertResetUntil: { type: Date, select: false },
|
||||||
accessAlertFenceRevision: { type: Number, select: false },
|
accessAlertFenceRevision: { type: Number, select: false },
|
||||||
|
accessAlertDeactivations: { type: [String], default: undefined, select: false },
|
||||||
accessAlertDispatches: { type: Schema.Types.Mixed, select: false },
|
accessAlertDispatches: { type: Schema.Types.Mixed, select: false },
|
||||||
emails: [
|
emails: [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -64,9 +64,10 @@ type AlertRecord = { _id: string; state?: string; lease?: string; leaseUntil?: D
|
|||||||
generation?: string | null; attemptId?: string; claimedAt?: Date; completedAt?: Date;
|
generation?: string | null; attemptId?: string; claimedAt?: Date; completedAt?: Date;
|
||||||
recipientKey?: string; resetRevision?: number; httpStatus?: number; startedAt?: Date;
|
recipientKey?: string; resetRevision?: number; httpStatus?: number; startedAt?: Date;
|
||||||
progress?: Record<string, { claimedAt: Date; id: Types.ObjectId }>; legacyClaims?: LegacyClaim[];
|
progress?: Record<string, { claimedAt: Date; id: Types.ObjectId }>; legacyClaims?: LegacyClaim[];
|
||||||
resourceRevision?: number; deferredRecoveries?: DeferredRecovery[]; dispatchPrepared?: boolean };
|
resourceRevision?: number; deferredRecoveries?: DeferredRecovery[]; dispatchPrepared?: boolean;
|
||||||
|
deliveryProgress?: { completedAt: Date; id: string }; failureCode?: string };
|
||||||
const alertCollection = (name = RECIPIENT_ALERTS) => UserModel.db.collection(name) as unknown as mongo.Collection<AlertRecord>;
|
const alertCollection = (name = RECIPIENT_ALERTS) => UserModel.db.collection(name) as unknown as mongo.Collection<AlertRecord>;
|
||||||
const MIGRATION = "recipient-alerts-v1";
|
const MIGRATION = "recipient-alerts-v2";
|
||||||
const PENDING_LEASE_MS = 60000;
|
const PENDING_LEASE_MS = 60000;
|
||||||
const expiredPending = () => ({ state: "pending", claimedAt: { $lte: new Date(Date.now() - PENDING_LEASE_MS) } });
|
const expiredPending = () => ({ state: "pending", claimedAt: { $lte: new Date(Date.now() - PENDING_LEASE_MS) } });
|
||||||
const expiredPreparation = () => ({ state: { $in: ["pending", "sending"] }, claimedAt: { $lte: new Date(Date.now() - PENDING_LEASE_MS) } });
|
const expiredPreparation = () => ({ state: { $in: ["pending", "sending"] }, claimedAt: { $lte: new Date(Date.now() - PENDING_LEASE_MS) } });
|
||||||
@@ -82,6 +83,7 @@ async function prepareOwnerDispatch(record: AlertRecord, revision: number | null
|
|||||||
const path = `accessAlertDispatches.${record.attemptId}`;
|
const path = `accessAlertDispatches.${record.attemptId}`;
|
||||||
const prepared = await UserModel.db.collection("users").updateOne({ _id: record.owner,
|
const prepared = await UserModel.db.collection("users").updateOne({ _id: record.owner,
|
||||||
accessAlertResetRevision: generationFilter(revision),
|
accessAlertResetRevision: generationFilter(revision),
|
||||||
|
"accessAlertDeactivations.0": { $exists: false },
|
||||||
$or: [{ accessAlertResetUntil: { $exists: false } }, { accessAlertResetUntil: { $lte: new Date() } }],
|
$or: [{ accessAlertResetUntil: { $exists: false } }, { accessAlertResetUntil: { $lte: new Date() } }],
|
||||||
}, [{ $set: { accessAlertDispatches: { $arrayToObject: { $filter: {
|
}, [{ $set: { accessAlertDispatches: { $arrayToObject: { $filter: {
|
||||||
input: { $objectToArray: { $ifNull: ["$accessAlertDispatches", {}] } }, as: "dispatch",
|
input: { $objectToArray: { $ifNull: ["$accessAlertDispatches", {}] } }, as: "dispatch",
|
||||||
@@ -101,6 +103,7 @@ async function commitOwnerRevision(owner: Types.ObjectId, revision: number | nul
|
|||||||
const path = `accessAlertDispatches.${record?.attemptId}`;
|
const path = `accessAlertDispatches.${record?.attemptId}`;
|
||||||
const committed = await UserModel.db.collection("users").updateOne({ _id: owner,
|
const committed = await UserModel.db.collection("users").updateOne({ _id: owner,
|
||||||
accessAlertResetRevision: generationFilter(revision),
|
accessAlertResetRevision: generationFilter(revision),
|
||||||
|
"accessAlertDeactivations.0": { $exists: false },
|
||||||
...(record ? { [`${path}.state`]: "prepared" } : {}),
|
...(record ? { [`${path}.state`]: "prepared" } : {}),
|
||||||
$or: [{ accessAlertResetUntil: { $exists: false } }, { accessAlertResetUntil: { $lte: new Date() } }],
|
$or: [{ accessAlertResetUntil: { $exists: false } }, { accessAlertResetUntil: { $lte: new Date() } }],
|
||||||
}, record ? { $set: { [`${path}.state`]: "committed" } } : { $inc: { accessAlertFenceRevision: 1 } });
|
}, record ? { $set: { [`${path}.state`]: "committed" } } : { $inc: { accessAlertFenceRevision: 1 } });
|
||||||
@@ -152,11 +155,14 @@ async function reconcileUnsentAlerts(writersStopped = false): Promise<void> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Call only after all legacy application writers have stopped. */
|
/** Call only after all application writers have stopped. */
|
||||||
export async function prepareLegacyOwnerAlertMigration(): Promise<void> {
|
export async function prepareLegacyOwnerAlertMigration(): Promise<void> {
|
||||||
await reconcileUnsentAlerts(true);
|
await reconcileUnsentAlerts(true);
|
||||||
|
// Interrupted lifecycle mutations cannot resume once every writer is stopped.
|
||||||
|
await UserModel.db.collection("users").updateMany({ accessAlertDeactivations: { $exists: true } },
|
||||||
|
{ $unset: { accessAlertDeactivations: "" }, $inc: { accessAlertResetRevision: 1 } });
|
||||||
await alertCollection().updateOne({ _id: MIGRATION }, { $set: { state: "new" },
|
await alertCollection().updateOne({ _id: MIGRATION }, { $set: { state: "new" },
|
||||||
$unset: { progress: "", lease: "", leaseUntil: "" } }, { upsert: true });
|
$unset: { progress: "", deliveryProgress: "", lease: "", leaseUntil: "" } }, { upsert: true });
|
||||||
}
|
}
|
||||||
|
|
||||||
async function reconcileCompletedSend(record: AlertRecord): Promise<void> {
|
async function reconcileCompletedSend(record: AlertRecord): Promise<void> {
|
||||||
@@ -219,6 +225,43 @@ export async function initializeOwnerAccessAlerts(): Promise<void> {
|
|||||||
.catch(error => { heartbeatError = error; });
|
.catch(error => { heartbeatError = error; });
|
||||||
}, 5000);
|
}, 5000);
|
||||||
try {
|
try {
|
||||||
|
// Delivery evidence survives v1 guard recovery and resource deletion. Restore
|
||||||
|
// missing guards once, before traffic, so those addresses cannot be emailed again.
|
||||||
|
const events = alertCollection("owner_access_alert_events");
|
||||||
|
const index = "owner_access_alert_delivery_history";
|
||||||
|
await events.createIndex({ completedAt: 1, _id: 1 }, { name: index,
|
||||||
|
partialFilterExpression: { completedAt: { $exists: true } } });
|
||||||
|
const checkpoint = (await alerts.findOne({ _id: MIGRATION, lease }))?.deliveryProgress;
|
||||||
|
const filter = { state: { $in: ["accepted", "failed"] }, completedAt: { $exists: true },
|
||||||
|
...(checkpoint ? { $or: [{ completedAt: { $gt: checkpoint.completedAt } },
|
||||||
|
{ completedAt: checkpoint.completedAt, _id: { $gt: checkpoint.id } }] } : {}) };
|
||||||
|
let lastDelivery: { completedAt: Date; id: string } | undefined;
|
||||||
|
let delivered = 0;
|
||||||
|
const saveDeliveries = async () => {
|
||||||
|
if (!lastDelivery) return;
|
||||||
|
const saved = await alerts.updateOne({ _id: MIGRATION, lease }, { $set: { deliveryProgress: lastDelivery } });
|
||||||
|
if (!saved.matchedCount) throw new Error("owner_alert_migration_lease_lost");
|
||||||
|
};
|
||||||
|
for await (const event of events.find(filter).sort({ completedAt: 1, _id: 1 }).hint(index).batchSize(100)) {
|
||||||
|
if (heartbeatError) throw heartbeatError;
|
||||||
|
if (event.recipientKey && /^[0-9a-f]{64}$/.test(event.recipientKey)) {
|
||||||
|
const previous = await alerts.findOne({ _id: event.recipientKey });
|
||||||
|
if (previous && (["pending", "cancelled"].includes(previous.state || "")
|
||||||
|
|| (previous.state === "sending" && previous.dispatchPrepared))) {
|
||||||
|
// A v1 repeat may still be preparing after its earlier guard was
|
||||||
|
// released. Cancel known-unsent work even while its lease is fresh;
|
||||||
|
// committed or ambiguous dispatches remain blocked instead.
|
||||||
|
await releaseUnsentAlert(previous, { state: previous.state });
|
||||||
|
}
|
||||||
|
await alerts.updateOne({ _id: event.recipientKey }, { $setOnInsert: {
|
||||||
|
state: "recipient_notified", attemptId: event.attemptId ?? randomUUID(),
|
||||||
|
claimedAt: event.completedAt, completedAt: event.completedAt, httpStatus: event.httpStatus,
|
||||||
|
} }, { upsert: true });
|
||||||
|
}
|
||||||
|
lastDelivery = { completedAt: event.completedAt!, id: event._id };
|
||||||
|
if (++delivered % 100 === 0) await saveDeliveries();
|
||||||
|
}
|
||||||
|
await saveDeliveries();
|
||||||
for (const [kind, name] of Object.entries(resourceCollections)) {
|
for (const [kind, name] of Object.entries(resourceCollections)) {
|
||||||
const resources = UserModel.db.collection(name);
|
const resources = UserModel.db.collection(name);
|
||||||
const index = "owner_access_alert_backfill";
|
const index = "owner_access_alert_backfill";
|
||||||
@@ -293,7 +336,8 @@ async function completeRecovery(candidate: AlertRecord, recovery: DeferredRecove
|
|||||||
} else if (candidate.state === "pending") {
|
} else if (candidate.state === "pending") {
|
||||||
await releaseUnsentAlert(candidate, { ...fence, state: "pending" });
|
await releaseUnsentAlert(candidate, { ...fence, state: "pending" });
|
||||||
} else {
|
} else {
|
||||||
await alertCollection().deleteOne(fence);
|
// Recovery does not authorize another generic email to the same address.
|
||||||
|
await alertCollection().updateOne(fence, { $set: { state: "recovered" } });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -337,7 +381,7 @@ async function applyDeferredRecoveries(key: string, attemptId?: string): Promise
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Only a fresh upstream read after owner action may reopen a recipient alert. */
|
/** Confirm recovery after owner action; only known-unsent claims may be released. */
|
||||||
export async function withOwnerAccessRecovery<T>(ownerId: string, resource: AlertResource, work: () => Promise<T>, expectedSource: Record<string, unknown> = {}, confirmed: (result: T) => boolean = () => true): Promise<T> {
|
export async function withOwnerAccessRecovery<T>(ownerId: string, resource: AlertResource, work: () => Promise<T>, expectedSource: Record<string, unknown> = {}, confirmed: (result: T) => boolean = () => true): Promise<T> {
|
||||||
let snapshot: { accessAlertGeneration?: string } | null = null;
|
let snapshot: { accessAlertGeneration?: string } | null = null;
|
||||||
let candidates: AlertRecord[] = [];
|
let candidates: AlertRecord[] = [];
|
||||||
@@ -436,10 +480,11 @@ export async function notifyOwnerAccessProblem(ownerId: string, error: unknown,
|
|||||||
try {
|
try {
|
||||||
if (!resource || !Types.ObjectId.isValid(resource.id) || !Types.ObjectId.isValid(ownerId)) return;
|
if (!resource || !Types.ObjectId.isValid(resource.id) || !Types.ObjectId.isValid(ownerId)) return;
|
||||||
const collection = UserModel.db.collection(resourceCollections[resource.kind]);
|
const collection = UserModel.db.collection(resourceCollections[resource.kind]);
|
||||||
const identity = { _id: new Types.ObjectId(resource.id), owner: new Types.ObjectId(ownerId) };
|
const identity = { _id: new Types.ObjectId(resource.id), owner: new Types.ObjectId(ownerId),
|
||||||
|
status: { $nin: ["archived", "expired", "expiring", "removed", "removing"] } };
|
||||||
const ownerState = await UserModel.db.collection("users").findOne({ _id: identity.owner },
|
const ownerState = await UserModel.db.collection("users").findOne({ _id: identity.owner },
|
||||||
{ projection: { accessAlertResetRevision: 1, accessAlertResetUntil: 1 } });
|
{ projection: { accessAlertResetRevision: 1, accessAlertResetUntil: 1, accessAlertDeactivations: 1 } });
|
||||||
if (!ownerState || ownerState.accessAlertResetUntil?.getTime() > Date.now()) return;
|
if (!ownerState || ownerState.accessAlertDeactivations?.length || ownerState.accessAlertResetUntil?.getTime() > Date.now()) return;
|
||||||
const ownerRevision = ownerState.accessAlertResetRevision ?? null;
|
const ownerRevision = ownerState.accessAlertResetRevision ?? null;
|
||||||
const snapshot = await collection.findOne(identity, { projection: { accessAlertGeneration: 1, accessAlertClaimedAt: 1, accessAlertClaimRevision: 1 } });
|
const snapshot = await collection.findOne(identity, { projection: { accessAlertGeneration: 1, accessAlertClaimedAt: 1, accessAlertClaimRevision: 1 } });
|
||||||
if (!snapshot) return;
|
if (!snapshot) return;
|
||||||
@@ -486,16 +531,9 @@ export async function notifyOwnerAccessProblem(ownerId: string, error: unknown,
|
|||||||
}
|
}
|
||||||
retired = await releaseUnsentAlert(previous, previous.state === "cancelled" ? retirement : { ...retirement, ...expiredPending() });
|
retired = await releaseUnsentAlert(previous, previous.state === "cancelled" ? retirement : { ...retirement, ...expiredPending() });
|
||||||
} else {
|
} else {
|
||||||
const claims = previous.state === "recovered" ? [] : previous.legacyClaims?.filter(claim => !claim.recovered) ?? [previous as LegacyClaim];
|
// Delivered, legacy, recovered and ambiguous sends stay blocked permanently,
|
||||||
for (const claim of claims) {
|
// even if their original resource or account no longer exists.
|
||||||
const previousResource = await UserModel.db.collection(resourceCollections[claim.resourceKind]).findOne(
|
return;
|
||||||
{ _id: claim.resourceId, owner: claim.owner }, { projection: { status: 1 } });
|
|
||||||
const previousOwner = await UserModel.findById(claim.owner).select("status").lean();
|
|
||||||
if (previousOwner && !["banned", "removed"].includes(previousOwner.status || "")
|
|
||||||
&& previousResource && !["removed", "expired"].includes(previousResource.status)) return;
|
|
||||||
}
|
|
||||||
if (previous.legacyClaims) retirement.legacyClaims = previous.legacyClaims;
|
|
||||||
retired = Boolean((await alerts.deleteOne(retirement)).deletedCount);
|
|
||||||
}
|
}
|
||||||
if (!retired) return;
|
if (!retired) return;
|
||||||
}
|
}
|
||||||
@@ -516,7 +554,10 @@ export async function notifyOwnerAccessProblem(ownerId: string, error: unknown,
|
|||||||
const base = /^https?:\/\//.test(config.APP_HOSTNAME) ? config.APP_HOSTNAME : `https://${config.APP_HOSTNAME}`;
|
const base = /^https?:\/\//.test(config.APP_HOSTNAME) ? config.APP_HOSTNAME : `https://${config.APP_HOSTNAME}`;
|
||||||
const url = new URL("/connections", base).href;
|
const url = new URL("/connections", base).href;
|
||||||
await alertCollection("owner_access_alert_events").insertOne({ _id: attemptId, attemptId,
|
await alertCollection("owner_access_alert_events").insertOne({ _id: attemptId, attemptId,
|
||||||
recipientKey: key, state: "prepared", startedAt: new Date() });
|
recipientKey: key, state: "prepared", startedAt: new Date(), owner: identity.owner,
|
||||||
|
resourceKind: resource.kind, resourceId: identity._id, generation: record.generation,
|
||||||
|
failureCode: accessErrors.has(typeof error === "string" ? error : error instanceof Error ? error.message : "")
|
||||||
|
? typeof error === "string" ? error : (error as Error).message : "upstream_access_failure" });
|
||||||
const claim = await collection.updateOne({ ...identity,
|
const claim = await collection.updateOne({ ...identity,
|
||||||
accessAlertGeneration: generationFilter(record.generation),
|
accessAlertGeneration: generationFilter(record.generation),
|
||||||
accessAlertClaimRevision: revisionFilter(record.resourceRevision ?? 0),
|
accessAlertClaimRevision: revisionFilter(record.resourceRevision ?? 0),
|
||||||
@@ -557,7 +598,8 @@ export async function notifyOwnerAccessProblem(ownerId: string, error: unknown,
|
|||||||
try {
|
try {
|
||||||
await retryAlertWrite(async () => {
|
await retryAlertWrite(async () => {
|
||||||
try { await alertCollection("owner_access_alert_events").insertOne({ _id: `${attemptId}/completion`, attemptId,
|
try { await alertCollection("owner_access_alert_events").insertOne({ _id: `${attemptId}/completion`, attemptId,
|
||||||
recipientKey: key, state, httpStatus, completedAt }); }
|
recipientKey: key, state, httpStatus, completedAt, owner: identity.owner,
|
||||||
|
resourceKind: resource.kind, resourceId: identity._id, generation: record.generation }); }
|
||||||
catch (error) { if ((error as { code?: number }).code !== 11000) throw error; }
|
catch (error) { if ((error as { code?: number }).code !== 11000) throw error; }
|
||||||
});
|
});
|
||||||
} catch { logger.warn("owner access email audit unavailable", { attemptId }); }
|
} catch { logger.warn("owner access email audit unavailable", { attemptId }); }
|
||||||
|
|||||||
@@ -18,11 +18,11 @@ const args = process.argv.slice(2), state = JSON.parse(fs.readFileSync(process.e
|
|||||||
fs.appendFileSync(process.env.TASK_DOCKER_LOG, JSON.stringify(args) + "\\n");
|
fs.appendFileSync(process.env.TASK_DOCKER_LOG, JSON.stringify(args) + "\\n");
|
||||||
const finish = (output = "", code = 0) => { fs.writeFileSync(process.env.TASK_DOCKER_STATE, JSON.stringify(state)); process.stdout.write(output); process.exit(code); };
|
const finish = (output = "", code = 0) => { fs.writeFileSync(process.env.TASK_DOCKER_STATE, JSON.stringify(state)); process.stdout.write(output); process.exit(code); };
|
||||||
if (args[0] === "image" && args[1] === "inspect") {
|
if (args[0] === "image" && args[1] === "inspect") {
|
||||||
if (args.some(a => a.includes("Labels"))) finish(state.badProtocol ? "<no value>" : "recipient-v1");
|
if (args.some(a => a.includes("Labels"))) finish(state.badProtocol ? "<no value>" : "recipient-v2");
|
||||||
finish(state.built ? "sha:new" : "sha:old");
|
finish(state.built ? "sha:new" : "sha:old");
|
||||||
}
|
}
|
||||||
if (args[0] === "inspect") {
|
if (args[0] === "inspect") {
|
||||||
if (args.some(a => a.includes("Labels"))) finish("<no value>");
|
if (args.some(a => a.includes("Labels"))) finish(state.oldProtocol || "<no value>");
|
||||||
if (args.includes("{{.State.Running}}")) finish(state.stopped && !state.refuseStop ? "false" : "true");
|
if (args.includes("{{.State.Running}}")) finish(state.stopped && !state.refuseStop ? "false" : "true");
|
||||||
finish("healthy");
|
finish("healthy");
|
||||||
}
|
}
|
||||||
@@ -68,6 +68,14 @@ finish("unexpected docker call", 2);
|
|||||||
expect(result.calls.slice(up + 1, api).some(args => args[0] === "inspect" && args.at(-1) === name)).to.equal(true);
|
expect(result.calls.slice(up + 1, api).some(args => args[0] === "inspect" && args.at(-1) === name)).to.equal(true);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
it("stops recipient-v1 writers before restoring permanent recipient suppression", () => {
|
||||||
|
const result = deploy({ oldProtocol: "recipient-v1" });
|
||||||
|
expect(result.status, result.stderr).to.equal(0);
|
||||||
|
const stop = result.calls.findIndex(args => args[0] === "compose" && args[1] === "stop");
|
||||||
|
const reconcile = result.calls.findIndex(args => args[0] === "compose" && args[1] === "run");
|
||||||
|
const up = result.calls.findIndex(args => args[0] === "compose" && args[1] === "up");
|
||||||
|
expect(stop).to.be.greaterThan(-1); expect(reconcile).to.be.greaterThan(stop); expect(up).to.be.greaterThan(reconcile);
|
||||||
|
});
|
||||||
it("aborts before migration starts if a legacy writer remains running", () => {
|
it("aborts before migration starts if a legacy writer remains running", () => {
|
||||||
const result = deploy({ refuseStop: true });
|
const result = deploy({ refuseStop: true });
|
||||||
expect(result.status).not.to.equal(0);
|
expect(result.status).not.to.equal(0);
|
||||||
|
|||||||
@@ -154,7 +154,7 @@ describe("owner access notifications", () => {
|
|||||||
expect(events[1].state).to.equal("accepted");
|
expect(events[1].state).to.equal("accepted");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reopens an alert only after owner action and successful access to its triggering resource", async () => {
|
it("keeps the recipient suppressed after owner action and successful access", async () => {
|
||||||
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
||||||
await notifyOwnerAccessProblem(ownerId, "token_expired", other);
|
await notifyOwnerAccessProblem(ownerId, "token_expired", other);
|
||||||
expect(sent).to.have.length(1);
|
expect(sent).to.have.length(1);
|
||||||
@@ -165,8 +165,9 @@ describe("owner access notifications", () => {
|
|||||||
expect(recipients.size).to.equal(1);
|
expect(recipients.size).to.equal(1);
|
||||||
await notifications.withOwnerAccessRecovery(ownerId, resource, async () => {});
|
await notifications.withOwnerAccessRecovery(ownerId, resource, async () => {});
|
||||||
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
||||||
expect(sent).to.have.length(2);
|
expect(sent).to.have.length(1);
|
||||||
expect(events).to.have.length(4);
|
expect([...recipients.values()][0].state).to.equal("recovered");
|
||||||
|
expect(events).to.have.length(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("cannot clear another owner's recipient guard", async () => {
|
it("cannot clear another owner's recipient guard", async () => {
|
||||||
@@ -217,7 +218,7 @@ describe("owner access notifications", () => {
|
|||||||
expect(sent).to.have.length(1);
|
expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
for (const failure of ["rejected", "timeout"]) it(`retains the claim after ${failure} until confirmed recovery`, async () => {
|
for (const failure of ["rejected", "timeout"]) it(`retains the claim after ${failure} including confirmed recovery`, async () => {
|
||||||
let attempts = 0;
|
let attempts = 0;
|
||||||
stub(global, "fetch", async () => { attempts++; if (failure === "timeout") throw new Error("timeout"); return { ok: false }; });
|
stub(global, "fetch", async () => { attempts++; if (failure === "timeout") throw new Error("timeout"); return { ok: false }; });
|
||||||
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
||||||
@@ -228,7 +229,7 @@ describe("owner access notifications", () => {
|
|||||||
expect(attempts).to.equal(1);
|
expect(attempts).to.equal(1);
|
||||||
await notifications.withOwnerAccessRecovery(ownerId, resource, async () => {});
|
await notifications.withOwnerAccessRecovery(ownerId, resource, async () => {});
|
||||||
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
await notifyOwnerAccessProblem(ownerId, "token_expired", resource);
|
||||||
expect(attempts).to.equal(2);
|
expect(attempts).to.equal(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not claim an alert without configuration, an active owner, and an address", async () => {
|
it("does not claim an alert without configuration, an active owner, and an address", async () => {
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
const guards = () => mongoose.connection.collection(alerts.RECIPIENT_ALERTS);
|
const guards = () => mongoose.connection.collection(alerts.RECIPIENT_ALERTS);
|
||||||
const recipient = () => alerts.recipientAlertKey("owner@example.com");
|
const recipient = () => alerts.recipientAlertKey("owner@example.com");
|
||||||
async function create(ownerId, kind = "repository") {
|
async function create(ownerId, kind = "repository") {
|
||||||
const id = new mongoose.Types.ObjectId(); await coll(kind).insertOne({ _id: id, owner: ownerId, repoId: String(id), status: "ready" }); return { kind, id: String(id) };
|
const id = new mongoose.Types.ObjectId(); await coll(kind).insertOne({ _id: id, owner: ownerId, [{ repository: "repoId", gist: "gistId", "pull-request": "pullRequestId" }[kind]]: String(id), status: "ready" }); return { kind, id: String(id) };
|
||||||
}
|
}
|
||||||
before(async () => {
|
before(async () => {
|
||||||
nativeFetch = global.fetch; originalConfig = { ...config };
|
nativeFetch = global.fetch; originalConfig = { ...config };
|
||||||
@@ -35,6 +35,189 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect(sent).to.have.length(1); expect(await guards().countDocuments({ _id: recipient() })).to.equal(1);
|
expect(sent).to.have.length(1); expect(await guards().countDocuments({ _id: recipient() })).to.equal(1);
|
||||||
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments({ state: "accepted" })).to.equal(1);
|
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments({ state: "accepted" })).to.equal(1);
|
||||||
});
|
});
|
||||||
|
for (const kind of ["repository", "gist", "pull-request"]) {
|
||||||
|
for (const status of ["archived", "expired", "expiring", "removed", "removing"]) it(`does not notify for ${status} ${kind}`, async () => {
|
||||||
|
const inactive = await create(owner._id, kind);
|
||||||
|
await coll(kind).updateOne({ _id: new mongoose.Types.ObjectId(inactive.id) }, { $set: { status } });
|
||||||
|
await notify(owner.id, "token_expired", inactive);
|
||||||
|
expect(sent).to.have.length(0); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
||||||
|
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments()).to.equal(0);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
it("cancels a send when the source is removed during audit preparation", async () => {
|
||||||
|
const events = mongoose.connection.collection("owner_access_alert_events"), insert = events.insertOne;
|
||||||
|
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
|
events.insertOne = async function (...args) { entered(); await new Promise(resolve => { release = resolve; }); return insert.apply(this, args); };
|
||||||
|
const pending = notify(owner.id, "token_expired", resource);
|
||||||
|
try {
|
||||||
|
await gate;
|
||||||
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
||||||
|
} finally { events.insertOne = insert; release(); await pending; }
|
||||||
|
expect(sent).to.have.length(0); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
||||||
|
expect((await coll(resource.kind).findOne({ _id: new mongoose.Types.ObjectId(resource.id) })).accessAlertClaimedAt).to.equal(undefined);
|
||||||
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
it("records the triggering resource and categorized failure without its address or source name", async () => {
|
||||||
|
await notify(owner.id, "token_expired", resource);
|
||||||
|
const records = await mongoose.connection.collection("owner_access_alert_events").find().toArray();
|
||||||
|
expect(records).to.have.length(2);
|
||||||
|
for (const record of records) {
|
||||||
|
expect(String(record.owner)).to.equal(owner.id); expect(String(record.resourceId)).to.equal(resource.id);
|
||||||
|
expect(record.resourceKind).to.equal(resource.kind); expect(record.generation).to.equal(null);
|
||||||
|
expect(JSON.stringify(record)).not.to.include("owner@example.com");
|
||||||
|
}
|
||||||
|
expect(records.find(record => record.state === "prepared").failureCode).to.equal("token_expired");
|
||||||
|
});
|
||||||
|
for (const state of ["accepted", "failed"]) it(`restores a missing guard from ${state} provider history before serving traffic`, async () => {
|
||||||
|
await mongoose.connection.collection("owner_access_alert_events").insertOne({ _id: "old/completion", attemptId: "old",
|
||||||
|
recipientKey: recipient(), state, completedAt: new Date(1) });
|
||||||
|
await Promise.all(Array.from({ length: 5 }, () => alerts.initializeOwnerAccessAlerts()));
|
||||||
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recipient_notified");
|
||||||
|
await notify(owner.id, "token_expired", resource); await notify(otherOwner.id, "token_expired", other);
|
||||||
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
|
await alerts.initializeOwnerAccessAlerts(); expect(sent).to.have.length(0);
|
||||||
|
});
|
||||||
|
it("does not restore a delivery guard from an uncompleted preparation event", async () => {
|
||||||
|
await mongoose.connection.collection("owner_access_alert_events").insertOne({ _id: "prepared", recipientKey: recipient(), state: "prepared", startedAt: new Date(1) });
|
||||||
|
await alerts.initializeOwnerAccessAlerts(); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
it("resumes provider history restoration after a checkpointed startup failure", async () => {
|
||||||
|
const events = mongoose.connection.collection("owner_access_alert_events");
|
||||||
|
const keys = Array.from({ length: 105 }, (_, i) => alerts.recipientAlertKey(`old${i}@example.com`));
|
||||||
|
await events.insertMany(keys.map((recipientKey, i) => ({ _id: String(i).padStart(3, "0") + "/completion",
|
||||||
|
recipientKey, attemptId: String(i), state: "accepted", completedAt: new Date(1) })));
|
||||||
|
const collection = guards(), update = collection.updateOne; let restored = 0;
|
||||||
|
collection.updateOne = async function (filter, operation, ...args) {
|
||||||
|
if (operation.$setOnInsert?.state === "recipient_notified" && ++restored === 103) throw new Error("interrupted history backfill");
|
||||||
|
return update.call(this, filter, operation, ...args);
|
||||||
|
};
|
||||||
|
try { await alerts.initializeOwnerAccessAlerts(); throw new Error("Expected migration failure"); }
|
||||||
|
catch (error) { expect(error.message).to.equal("interrupted history backfill"); }
|
||||||
|
finally { collection.updateOne = update; }
|
||||||
|
expect((await guards().findOne({ _id: "recipient-alerts-v2" })).deliveryProgress.id).to.equal("099/completion");
|
||||||
|
await guards().updateOne({ _id: "recipient-alerts-v2" }, { $set: { leaseUntil: new Date(0) } });
|
||||||
|
await alerts.initializeOwnerAccessAlerts();
|
||||||
|
expect(await guards().countDocuments({ state: "recipient_notified" })).to.equal(105);
|
||||||
|
expect((await guards().findOne({ _id: "recipient-alerts-v2" })).deliveryProgress.id).to.equal("104/completion");
|
||||||
|
expect((await events.indexes()).some(index => index.name === "owner_access_alert_delivery_history" && index.partialFilterExpression)).to.equal(true);
|
||||||
|
expect(sent).to.have.length(0);
|
||||||
|
});
|
||||||
|
it("runs the v2 history migration on a cold start with only a ready v1 marker", async () => {
|
||||||
|
await guards().insertOne({ _id: "recipient-alerts-v1", state: "ready" });
|
||||||
|
await mongoose.connection.collection("owner_access_alert_events").insertOne({ _id: "v1/completion", attemptId: "v1",
|
||||||
|
recipientKey: recipient(), state: "accepted", completedAt: new Date(1) });
|
||||||
|
await alerts.initializeOwnerAccessAlerts();
|
||||||
|
expect((await guards().findOne({ _id: "recipient-alerts-v2" })).state).to.equal("ready");
|
||||||
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recipient_notified");
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
|
});
|
||||||
|
it("cancels a fresh unsent v1 repeat before restoring its delivery suppression", async () => {
|
||||||
|
const attemptId = "v1-unsent-repeat";
|
||||||
|
await guards().insertMany([{ _id: "recipient-alerts-v1", state: "ready" }, {
|
||||||
|
_id: recipient(), owner: owner._id, resourceKind: resource.kind, resourceId: new mongoose.Types.ObjectId(resource.id),
|
||||||
|
attemptId, state: "pending", dispatchPrepared: true, resourceRevision: 0, claimedAt: new Date(),
|
||||||
|
}]);
|
||||||
|
await mongoose.connection.collection("users").updateOne({ _id: owner._id }, { $set: {
|
||||||
|
[`accessAlertDispatches.${attemptId}`]: { state: "prepared", claimedAt: new Date() },
|
||||||
|
} });
|
||||||
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: {
|
||||||
|
accessAlertClaimedAt: new Date(), accessAlertAttemptId: attemptId, accessAlertClaimRevision: 1,
|
||||||
|
} });
|
||||||
|
await mongoose.connection.collection("owner_access_alert_events").insertOne({ _id: "v1-delivered/completion", attemptId: "v1-delivered",
|
||||||
|
recipientKey: recipient(), state: "accepted", completedAt: new Date(1) });
|
||||||
|
await alerts.initializeOwnerAccessAlerts();
|
||||||
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recipient_notified");
|
||||||
|
expect((await coll(resource.kind).findOne({ _id: new mongoose.Types.ObjectId(resource.id) })).accessAlertClaimedAt).to.equal(undefined);
|
||||||
|
expect((await mongoose.connection.collection("users").findOne({ _id: owner._id })).accessAlertDispatches[attemptId]).to.equal(undefined);
|
||||||
|
await alerts.initializeOwnerAccessAlerts(); await notify(owner.id, "token_expired", resource);
|
||||||
|
expect(sent).to.have.length(0);
|
||||||
|
});
|
||||||
|
const models = () => ({
|
||||||
|
repository: require("../src/core/model/anonymizedRepositories/anonymizedRepositories.model").default,
|
||||||
|
gist: require("../src/core/model/anonymizedGists/anonymizedGists.model").default,
|
||||||
|
"pull-request": require("../src/core/model/anonymizedPullRequests/anonymizedPullRequests.model").default,
|
||||||
|
});
|
||||||
|
for (const kind of ["repository", "gist", "pull-request"]) {
|
||||||
|
it(`fences ${kind} deletion after the sender's last resource check`, async () => {
|
||||||
|
const source = await create(owner._id, kind);
|
||||||
|
const collection = guards(), update = collection.updateOne;
|
||||||
|
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
|
collection.updateOne = async function (filter, operation, ...args) {
|
||||||
|
if (operation.$set?.state === "sending") { entered(); await new Promise(resolve => { release = resolve; }); }
|
||||||
|
return update.call(this, filter, operation, ...args);
|
||||||
|
};
|
||||||
|
const pending = notify(owner.id, "token_expired", source);
|
||||||
|
try {
|
||||||
|
await gate;
|
||||||
|
await models()[kind].updateOne({ _id: source.id }, { $set: { status: "removing" } }).exec();
|
||||||
|
} finally { collection.updateOne = update; release(); await pending; }
|
||||||
|
expect(sent).to.have.length(0); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
||||||
|
expect((await mongoose.connection.collection("users").findOne({ _id: owner._id })).accessAlertDeactivations).to.deep.equal([]);
|
||||||
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
it("does not deactivate a resource reassigned after the lifecycle owner snapshot", async () => {
|
||||||
|
const Model = models().repository, users = mongoose.connection.collection("users"), update = users.updateMany;
|
||||||
|
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
|
users.updateMany = async function (filter, operation, ...args) {
|
||||||
|
if (operation.$addToSet?.accessAlertDeactivations) { entered(); await new Promise(resolve => { release = resolve; }); }
|
||||||
|
return update.call(this, filter, operation, ...args);
|
||||||
|
};
|
||||||
|
const pending = Model.updateOne({ _id: resource.id }, { $set: { status: "removing" } }).exec();
|
||||||
|
let result;
|
||||||
|
try {
|
||||||
|
await gate;
|
||||||
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { owner: otherOwner._id } });
|
||||||
|
} finally { users.updateMany = update; release(); result = await pending; }
|
||||||
|
expect(result.matchedCount).to.equal(0);
|
||||||
|
expect((await coll(resource.kind).findOne({ _id: new mongoose.Types.ObjectId(resource.id) })).status).to.equal("ready");
|
||||||
|
await notify(otherOwner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
it("blocks new senders until all concurrent lifecycle mutations finish", async () => {
|
||||||
|
const second = await create(owner._id), active = await create(owner._id);
|
||||||
|
const Model = models().repository, collection = Model.collection, update = collection.updateOne;
|
||||||
|
const releases = new Map(); let entered; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
|
collection.updateOne = async function (filter, operation, ...args) {
|
||||||
|
if (operation.$set?.status === "removing") {
|
||||||
|
await new Promise(resolve => { releases.set(String(filter.$and[0]._id), resolve); if (releases.size === 2) entered(); });
|
||||||
|
}
|
||||||
|
return update.call(this, filter, operation, ...args);
|
||||||
|
};
|
||||||
|
const first = Model.updateOne({ _id: resource.id }, { $set: { status: "removing" } }).exec();
|
||||||
|
const last = Model.updateOne({ _id: second.id }, { $set: { status: "removing" } }).exec();
|
||||||
|
try {
|
||||||
|
await gate;
|
||||||
|
expect((await mongoose.connection.collection("users").findOne({ _id: owner._id })).accessAlertDeactivations).to.have.length(2);
|
||||||
|
await notify(owner.id, "token_expired", active); expect(sent).to.have.length(0);
|
||||||
|
releases.get(second.id)(); await last;
|
||||||
|
await notify(owner.id, "token_expired", active); expect(sent).to.have.length(0);
|
||||||
|
} finally { collection.updateOne = update; for (const release of releases.values()) release(); await Promise.all([first, last]); }
|
||||||
|
await notify(owner.id, "token_expired", active); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
it("retains an ambiguous lifecycle reservation until offline reconciliation", async () => {
|
||||||
|
const Model = models().repository, collection = Model.collection, update = collection.updateOne;
|
||||||
|
collection.updateOne = async () => { throw new Error("lost database acknowledgement"); };
|
||||||
|
try { await Model.updateOne({ _id: resource.id }, { $set: { status: "removing" } }).exec(); throw new Error("Expected mutation failure"); }
|
||||||
|
catch (error) { expect(error.message).to.equal("lost database acknowledgement"); }
|
||||||
|
finally { collection.updateOne = update; }
|
||||||
|
await alerts.initializeOwnerAccessAlerts();
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
|
await alerts.prepareLegacyOwnerAlertMigration(); await alerts.initializeOwnerAccessAlerts();
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
|
});
|
||||||
|
for (const operation of ["updateMany", "findOneAndUpdate", "save"]) it(`fences inactive lifecycle status via ${operation}`, async () => {
|
||||||
|
const Model = models().repository;
|
||||||
|
const before = await mongoose.connection.collection("users").findOne({ _id: owner._id });
|
||||||
|
if (operation === "save") {
|
||||||
|
const doc = await Model.findById(resource.id); doc.status = "expired"; await doc.save();
|
||||||
|
} else await Model[operation]({ _id: resource.id }, { $set: { status: "expired" } }).exec();
|
||||||
|
const after = await mongoose.connection.collection("users").findOne({ _id: owner._id });
|
||||||
|
expect(after.accessAlertResetRevision).to.equal((before.accessAlertResetRevision || 0) + 1);
|
||||||
|
expect(after.accessAlertDeactivations).to.deep.equal([]);
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
|
});
|
||||||
it("does not reopen after reconnect alone, unrelated resource success, or failed recovery", async () => {
|
it("does not reopen after reconnect alone, unrelated resource success, or failed recovery", async () => {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id);
|
||||||
await recover(otherOwner.id, other, async () => {});
|
await recover(otherOwner.id, other, async () => {});
|
||||||
@@ -42,14 +225,17 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(owner.id, "token_expired", resource); await notify(otherOwner.id, "token_expired", other);
|
await notify(owner.id, "token_expired", resource); await notify(otherOwner.id, "token_expired", other);
|
||||||
expect(sent).to.have.length(1);
|
expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("requires owner action before a successful upstream read can reopen the recipient", async () => {
|
it("keeps suppression after recovery but allows a first alert to a new address", async () => {
|
||||||
await notify(owner.id, "token_expired", resource);
|
await notify(owner.id, "token_expired", resource);
|
||||||
await recover(owner.id, resource, async () => {}); expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
|
||||||
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
|
await Users.updateOne({ _id: owner._id }, { $set: { notificationEmail: "new@example.com" } });
|
||||||
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(2);
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(2);
|
||||||
expect(sent[0].headers["Idempotency-Key"]).not.to.equal(sent[1].headers["Idempotency-Key"]);
|
expect(sent[0].headers["Idempotency-Key"]).not.to.equal(sent[1].headers["Idempotency-Key"]);
|
||||||
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments()).to.equal(4);
|
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments()).to.equal(4);
|
||||||
|
await Users.updateOne({ _id: owner._id }, { $set: { notificationEmail: "OWNER@example.com" } });
|
||||||
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(2);
|
||||||
});
|
});
|
||||||
it("does not release a pending send when recovery races with provider delivery", async () => {
|
it("does not release a pending send when recovery races with provider delivery", async () => {
|
||||||
let release, entered; const gate = new Promise(resolve => { entered = resolve; });
|
let release, entered; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
@@ -77,14 +263,17 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
});
|
});
|
||||||
it("fences a second owner action that races after recovery's final resource check", async () => {
|
it("fences a second owner action that races after recovery's final resource check", async () => {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
const collection = guards(), remove = collection.deleteOne;
|
const collection = guards(), update = collection.updateOne;
|
||||||
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
collection.deleteOne = async function (...args) { entered(); await new Promise(resolve => { release = resolve; }); return remove.apply(this, args); };
|
collection.updateOne = async function (filter, operation, ...args) {
|
||||||
|
if (operation.$set?.state === "recovered") { entered(); await new Promise(resolve => { release = resolve; }); }
|
||||||
|
return update.call(this, filter, operation, ...args);
|
||||||
|
};
|
||||||
try {
|
try {
|
||||||
const pending = recover(owner.id, resource, async () => {}); await gate;
|
const pending = recover(owner.id, resource, async () => {}); await gate;
|
||||||
await reset(owner.id, resource); release(); await pending;
|
await reset(owner.id, resource); release(); await pending;
|
||||||
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted");
|
||||||
} finally { collection.deleteOne = remove; }
|
} finally { collection.updateOne = update; }
|
||||||
});
|
});
|
||||||
it("does not accept a successful read of a stale source binding", async () => {
|
it("does not accept a successful read of a stale source binding", async () => {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
@@ -98,11 +287,11 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(otherOwner.id, "repo_not_found", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "repo_not_found", other); expect(sent).to.have.length(1);
|
||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("failed");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("failed");
|
||||||
});
|
});
|
||||||
it("releases a guard for a removed resource without erasing delivery history", async () => {
|
it("keeps a removed resource's recipient guard and delivery history", async () => {
|
||||||
await notify(owner.id, "token_expired", resource);
|
await notify(owner.id, "token_expired", resource);
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments()).to.equal(4);
|
expect(await mongoose.connection.collection("owner_access_alert_events").countDocuments()).to.equal(2);
|
||||||
});
|
});
|
||||||
for (const step of ["resource update", "resource verification", "audit insertion", "dispatch transition"]) it(`releases an unsent guard after ${step} fails`, async () => {
|
for (const step of ["resource update", "resource verification", "audit insertion", "dispatch transition"]) it(`releases an unsent guard after ${step} fails`, async () => {
|
||||||
const collection = step.startsWith("resource") ? coll(resource.kind) : step === "audit insertion" ? mongoose.connection.collection("owner_access_alert_events") : guards();
|
const collection = step.startsWith("resource") ? coll(resource.kind) : step === "audit insertion" ? mongoose.connection.collection("owner_access_alert_events") : guards();
|
||||||
@@ -165,7 +354,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("sending");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("sending");
|
||||||
});
|
});
|
||||||
it("reaps only expired unsent claims during startup and never reseeds them", async () => {
|
it("reaps only expired unsent claims during startup and never reseeds them", async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "ready" });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "ready" });
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(0), accessAlertAttemptId: recipient() } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(0), accessAlertAttemptId: recipient() } });
|
||||||
for (const [key, state, date] of [[recipient(), "pending", new Date(0)], ["recent", "pending", new Date()],
|
for (const [key, state, date] of [[recipient(), "pending", new Date(0)], ["recent", "pending", new Date()],
|
||||||
["sending", "sending", new Date(0)], ["accepted", "accepted", new Date(0)], ["failed", "failed", new Date(0)]]) {
|
["sending", "sending", new Date(0)], ["accepted", "accepted", new Date(0)], ["failed", "failed", new Date(0)]]) {
|
||||||
@@ -226,7 +415,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
for (const changed of ["generation", "attempt"]) it(`preserves newer legacy fields when the ${changed} no longer matches the cancelled attempt`, async () => {
|
for (const changed of ["generation", "attempt"]) it(`preserves newer legacy fields when the ${changed} no longer matches the cancelled attempt`, async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "ready" });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "ready" });
|
||||||
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
||||||
resourceId: new mongoose.Types.ObjectId(resource.id), generation: "old-generation", attemptId: "old-attempt", state: "pending", claimedAt: new Date(0) });
|
resourceId: new mongoose.Types.ObjectId(resource.id), generation: "old-generation", attemptId: "old-attempt", state: "pending", claimedAt: new Date(0) });
|
||||||
const generation = changed === "generation" ? "new-generation" : "old-generation";
|
const generation = changed === "generation" ? "new-generation" : "old-generation";
|
||||||
@@ -256,7 +445,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("cleans recent pending legacy fields once deployment has stopped every writer", async () => {
|
it("cleans recent pending legacy fields once deployment has stopped every writer", async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "ready" });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "ready" });
|
||||||
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
||||||
resourceId: new mongoose.Types.ObjectId(resource.id), attemptId: "recent-unsent", state: "pending", claimedAt: new Date() });
|
resourceId: new mongoose.Types.ObjectId(resource.id), attemptId: "recent-unsent", state: "pending", claimedAt: new Date() });
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: {
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: {
|
||||||
@@ -267,7 +456,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect((await coll(resource.kind).findOne({ _id: new mongoose.Types.ObjectId(resource.id) })).accessAlertClaimedAt).to.equal(undefined);
|
expect((await coll(resource.kind).findOne({ _id: new mongoose.Types.ObjectId(resource.id) })).accessAlertClaimedAt).to.equal(undefined);
|
||||||
});
|
});
|
||||||
it("preserves an untagged legacy delivery written during rollback while cancelling a native pending attempt", async () => {
|
it("preserves an untagged legacy delivery written during rollback while cancelling a native pending attempt", async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "ready" });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "ready" });
|
||||||
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
await guards().insertOne({ _id: recipient(), owner: owner._id, resourceKind: resource.kind,
|
||||||
resourceId: new mongoose.Types.ObjectId(resource.id), attemptId: "native-unsent", state: "pending", claimedAt: new Date() });
|
resourceId: new mongoose.Types.ObjectId(resource.id), attemptId: "native-unsent", state: "pending", claimedAt: new Date() });
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date() } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date() } });
|
||||||
@@ -275,7 +464,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("legacy");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("legacy");
|
||||||
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(1);
|
await notify(owner.id, "token_expired", resource); expect(sent).to.have.length(0);
|
||||||
});
|
});
|
||||||
it("blocks dispatch while an owner reset has advanced the guard revision but not the resource generation", async () => {
|
it("blocks dispatch while an owner reset has advanced the guard revision but not the resource generation", async () => {
|
||||||
const events = mongoose.connection.collection("owner_access_alert_events"), insert = events.insertOne;
|
const events = mongoose.connection.collection("owner_access_alert_events"), insert = events.insertOne;
|
||||||
@@ -363,8 +552,8 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect(record.deferredRecoveries).to.have.length(1);
|
expect(record.deferredRecoveries).to.have.length(1);
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
} finally { release(); await pending; }
|
} finally { release(); await pending; }
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("invalidates in-flight recovery evidence after a newer reset", async () => {
|
it("invalidates in-flight recovery evidence after a newer reset", async () => {
|
||||||
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
||||||
@@ -379,7 +568,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted");
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
await recover(owner.id, resource, async () => {});
|
await recover(owner.id, resource, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("invalidates deferred recovery if its source binding changes before delivery completes", async () => {
|
it("invalidates deferred recovery if its source binding changes before delivery completes", async () => {
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { source: { repositoryName: "owner/repo" } } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { source: { repositoryName: "owner/repo" } } });
|
||||||
@@ -396,7 +585,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
} finally { release(); await pending; }
|
} finally { release(); await pending; }
|
||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted");
|
||||||
await recover(owner.id, resource, async () => {}, { "source.repositoryName": "new/repo" });
|
await recover(owner.id, resource, async () => {}, { "source.repositoryName": "new/repo" });
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("rejects dispatch when reset starts after the guard reservation but before the owner commitment", async () => {
|
it("rejects dispatch when reset starts after the guard reservation but before the owner commitment", async () => {
|
||||||
const collection = guards(), update = collection.updateOne, updateMany = collection.updateMany;
|
const collection = guards(), update = collection.updateOne, updateMany = collection.updateMany;
|
||||||
@@ -433,21 +622,21 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await gate; await recover(owner.id, resource, async () => {});
|
await gate; await recover(owner.id, resource, async () => {});
|
||||||
expect((await guards().findOne({ _id: recipient() })).deferredRecoveries).to.have.length(1);
|
expect((await guards().findOne({ _id: recipient() })).deferredRecoveries).to.have.length(1);
|
||||||
} finally { users.updateOne = update; release(); await action; }
|
} finally { users.updateOne = update; release(); await action; }
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("retries saved recovery after a crashed reset lease expires", async () => {
|
it("retries saved recovery after a crashed reset lease expires", async () => {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
await mongoose.connection.collection("users").updateOne({ _id: owner._id }, { $set: { accessAlertResetUntil: new Date(Date.now() + 150) } });
|
await mongoose.connection.collection("users").updateOne({ _id: owner._id }, { $set: { accessAlertResetUntil: new Date(Date.now() + 150) } });
|
||||||
await recover(owner.id, resource, async () => {});
|
await recover(owner.id, resource, async () => {});
|
||||||
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
||||||
for (let i = 0; i < 100 && await guards().findOne({ _id: recipient() }); i++) await new Promise(resolve => global.setTimeout(resolve, 20));
|
for (let i = 0; i < 100 && (await guards().findOne({ _id: recipient() }))?.state !== "recovered"; i++) await new Promise(resolve => global.setTimeout(resolve, 20));
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
for (const state of ["prepared", "cancelled", "committed"]) it(`reconciles interrupted guard reservation with a ${state} owner dispatch record`, async () => {
|
for (const state of ["prepared", "cancelled", "committed"]) it(`reconciles interrupted guard reservation with a ${state} owner dispatch record`, async () => {
|
||||||
const attemptId = `interrupted-${state}`;
|
const attemptId = `interrupted-${state}`;
|
||||||
await guards().insertMany([{ _id: "recipient-alerts-v1", state: "ready" }, {
|
await guards().insertMany([{ _id: "recipient-alerts-v2", state: "ready" }, {
|
||||||
_id: recipient(), owner: owner._id, resourceKind: resource.kind, resourceId: new mongoose.Types.ObjectId(resource.id),
|
_id: recipient(), owner: owner._id, resourceKind: resource.kind, resourceId: new mongoose.Types.ObjectId(resource.id),
|
||||||
attemptId, state: "sending", dispatchPrepared: true, resourceRevision: 0, claimedAt: new Date(0),
|
attemptId, state: "sending", dispatchPrepared: true, resourceRevision: 0, claimedAt: new Date(0),
|
||||||
}]);
|
}]);
|
||||||
@@ -477,22 +666,22 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
const account = await mongoose.connection.collection("users").findOne({ _id: owner._id });
|
const account = await mongoose.connection.collection("users").findOne({ _id: owner._id });
|
||||||
expect(Object.keys(account.accessAlertDispatches)).to.deep.equal(["unknown"]);
|
expect(Object.keys(account.accessAlertDispatches)).to.deep.equal(["unknown"]);
|
||||||
});
|
});
|
||||||
it("allows active accounts to receive alerts when a shared-address guard belongs to a banned owner", async () => {
|
it("keeps shared-address suppression when the original owner is banned", async () => {
|
||||||
await notify(owner.id, "token_expired", resource);
|
await notify(owner.id, "token_expired", resource);
|
||||||
await Users.updateOne({ _id: owner._id }, { $set: { status: "banned" } });
|
await Users.updateOne({ _id: owner._id }, { $set: { status: "banned" } });
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("resumes an indexed migration from its last completed checkpoint", async () => {
|
it("resumes an indexed migration from its last completed checkpoint", async () => {
|
||||||
await Users.updateOne({ _id: otherOwner._id }, { $set: { notificationEmail: "second@example.com" } });
|
await Users.updateOne({ _id: otherOwner._id }, { $set: { notificationEmail: "second@example.com" } });
|
||||||
const second = await create(otherOwner._id);
|
const second = await create(otherOwner._id);
|
||||||
await coll("repository").updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
await coll("repository").updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
||||||
await coll("repository").updateOne({ _id: new mongoose.Types.ObjectId(second.id) }, { $set: { accessAlertClaimedAt: new Date(2) } });
|
await coll("repository").updateOne({ _id: new mongoose.Types.ObjectId(second.id) }, { $set: { accessAlertClaimedAt: new Date(2) } });
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "new", leaseUntil: new Date(0),
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "new", leaseUntil: new Date(0),
|
||||||
progress: { repository: { claimedAt: new Date(1), id: new mongoose.Types.ObjectId(resource.id) } } });
|
progress: { repository: { claimedAt: new Date(1), id: new mongoose.Types.ObjectId(resource.id) } } });
|
||||||
await alerts.initializeOwnerAccessAlerts();
|
await alerts.initializeOwnerAccessAlerts();
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
||||||
expect(await guards().findOne({ _id: alerts.recipientAlertKey("second@example.com") })).not.to.equal(null);
|
expect(await guards().findOne({ _id: alerts.recipientAlertKey("second@example.com") })).not.to.equal(null);
|
||||||
expect((await guards().findOne({ _id: "recipient-alerts-v1" })).progress.repository.claimedAt.getTime()).to.equal(2);
|
expect((await guards().findOne({ _id: "recipient-alerts-v2" })).progress.repository.claimedAt.getTime()).to.equal(2);
|
||||||
expect((await coll("repository").indexes()).some(i => i.name === "owner_access_alert_backfill" && i.partialFilterExpression)).to.equal(true);
|
expect((await coll("repository").indexes()).some(i => i.name === "owner_access_alert_backfill" && i.partialFilterExpression)).to.equal(true);
|
||||||
});
|
});
|
||||||
it("confirms recovery on a fresh uncached file read but never on a cache hit", async () => {
|
it("confirms recovery on a fresh uncached file read but never on a cache hit", async () => {
|
||||||
@@ -512,9 +701,11 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
const source = new GitHubStream({ repoId: resource.id, organization: "owner", repoName: "repo", commit: "pinned", getToken: async () => "mock-token" });
|
const source = new GitHubStream({ repoId: resource.id, organization: "owner", repoName: "repo", commit: "pinned", getToken: async () => "mock-token" });
|
||||||
source.downloadWithFallback = async () => Readable.from(["fresh-bytes"]);
|
source.downloadWithFallback = async () => Readable.from(["fresh-bytes"]);
|
||||||
const read = () => source.getFileContentCache("a.txt", resource.id, () => ({ sha: "a".repeat(40), size: 11 }));
|
const read = () => source.getFileContentCache("a.txt", resource.id, () => ({ sha: "a".repeat(40), size: 11 }));
|
||||||
(await read()).destroy(); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
(await read()).destroy(); expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
|
const secondKey = alerts.recipientAlertKey("second@example.com");
|
||||||
|
await Users.updateOne({ _id: owner._id }, { $set: { notificationEmail: "second@example.com" } });
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
(await read()).destroy(); expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
(await read()).destroy(); expect((await guards().findOne({ _id: secondKey })).state).to.equal("accepted");
|
||||||
} finally { database.isConnected = originals.isConnected; storage.fileInfo = originals.fileInfo; storage.write = originals.write; storage.read = originals.read; }
|
} finally { database.isConnected = originals.isConnected; storage.fileInfo = originals.fileInfo; storage.write = originals.write; storage.read = originals.read; }
|
||||||
});
|
});
|
||||||
it("confirms recovery when a successful scheduled refresh finds the same commit", async () => {
|
it("confirms recovery when a successful scheduled refresh finds the same commit", async () => {
|
||||||
@@ -532,7 +723,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
try {
|
try {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
await repo.updateIfNeeded({ force: false });
|
await repo.updateIfNeeded({ force: false });
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
} finally { github.getRepositoryFromGitHub = original; }
|
} finally { github.getRepositoryFromGitHub = original; }
|
||||||
});
|
});
|
||||||
it("seeds old claims once while five instances start concurrently", async () => {
|
it("seeds old claims once while five instances start concurrently", async () => {
|
||||||
@@ -553,7 +744,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
});
|
});
|
||||||
it("does not retire a legacy group when only its first triggering resource is removed", async () => {
|
it("does not retire a legacy group when only its first triggering resource is removed", async () => {
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
||||||
@@ -562,9 +753,9 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
});
|
});
|
||||||
it("atomically releases the final legacy guard during concurrent recoveries", async () => {
|
it("atomically marks concurrent legacy recoveries without releasing suppression", async () => {
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
||||||
await coll(other.kind).updateOne({ _id: new mongoose.Types.ObjectId(other.id) }, { $set: { accessAlertClaimedAt: new Date(2) } });
|
await coll(other.kind).updateOne({ _id: new mongoose.Types.ObjectId(other.id) }, { $set: { accessAlertClaimedAt: new Date(2) } });
|
||||||
await alerts.initializeOwnerAccessAlerts(); await reset(owner.id, resource); await reset(otherOwner.id, other);
|
await alerts.initializeOwnerAccessAlerts(); await reset(owner.id, resource); await reset(otherOwner.id, other);
|
||||||
@@ -580,7 +771,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("legacy");
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("legacy");
|
||||||
await recover(owner.id, resource, async () => {});
|
await recover(owner.id, resource, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
});
|
});
|
||||||
it("fences legacy claim mutation when reset races after the final resource check", async () => {
|
it("fences legacy claim mutation when reset races after the final resource check", async () => {
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { accessAlertClaimedAt: new Date(1) } });
|
||||||
@@ -603,11 +794,11 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await alerts.initializeOwnerAccessAlerts(); expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
await alerts.initializeOwnerAccessAlerts(); expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
await coll(other.kind).updateOne({ _id: new mongoose.Types.ObjectId(other.id) }, { $set: { accessAlertClaimedAt: new Date() } });
|
await coll(other.kind).updateOne({ _id: new mongoose.Types.ObjectId(other.id) }, { $set: { accessAlertClaimedAt: new Date() } });
|
||||||
await alerts.prepareLegacyOwnerAlertMigration();
|
await alerts.prepareLegacyOwnerAlertMigration();
|
||||||
const marker = await guards().findOne({ _id: "recipient-alerts-v1" }); expect(marker.state).to.equal("new"); expect(marker.progress).to.equal(undefined);
|
const marker = await guards().findOne({ _id: "recipient-alerts-v2" }); expect(marker.state).to.equal("new"); expect(marker.progress).to.equal(undefined);
|
||||||
await alerts.initializeOwnerAccessAlerts(); await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
await alerts.initializeOwnerAccessAlerts(); await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
expect((await guards().findOne({ _id: recipient() })).legacyClaims.filter(claim => !claim.recovered)).to.have.length(1);
|
expect((await guards().findOne({ _id: recipient() })).legacyClaims.filter(claim => !claim.recovered)).to.have.length(1);
|
||||||
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(0);
|
||||||
});
|
});
|
||||||
it("preserves a native guard's origin when rollback reconciliation adds another legacy claim", async () => {
|
it("preserves a native guard's origin when rollback reconciliation adds another legacy claim", async () => {
|
||||||
await alerts.initializeOwnerAccessAlerts(); await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await alerts.initializeOwnerAccessAlerts(); await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
@@ -617,22 +808,17 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
await reset(otherOwner.id, other); await recover(otherOwner.id, other, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
await recover(owner.id, resource, async () => {});
|
await recover(owner.id, resource, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("fences completed-guard retirement against a concurrent reactivation and reset", async () => {
|
it("keeps suppression through removal, reactivation, reset and recovery", async () => {
|
||||||
await notify(owner.id, "token_expired", resource);
|
await notify(owner.id, "token_expired", resource);
|
||||||
|
const original = await guards().findOne({ _id: recipient() });
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "removed" } });
|
||||||
const collection = guards(), remove = collection.deleteOne;
|
await notify(otherOwner.id, "token_expired", other);
|
||||||
let entered, release; const gate = new Promise(resolve => { entered = resolve; });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "ready" } });
|
||||||
collection.deleteOne = async function (...args) { entered(); await new Promise(resolve => { release = resolve; }); return remove.apply(this, args); };
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
const pending = notify(otherOwner.id, "token_expired", other);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
try {
|
expect((await guards().findOne({ _id: recipient() })).attemptId).to.equal(original.attemptId);
|
||||||
await gate;
|
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { status: "ready" } });
|
|
||||||
await reset(owner.id, resource);
|
|
||||||
} finally { collection.deleteOne = remove; release(); await pending; }
|
|
||||||
expect(sent).to.have.length(1); expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
|
||||||
await recover(owner.id, resource, async () => {}); await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
|
||||||
});
|
});
|
||||||
for (const accepted of [true, false]) it(`retries the guard update after a definite ${accepted ? "accepted" : "rejected"} provider response`, async () => {
|
for (const accepted of [true, false]) it(`retries the guard update after a definite ${accepted ? "accepted" : "rejected"} provider response`, async () => {
|
||||||
const collection = guards(), update = collection.updateOne; let writes = 0;
|
const collection = guards(), update = collection.updateOne; let writes = 0;
|
||||||
@@ -662,7 +848,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
if (startup) await alerts.initializeOwnerAccessAlerts(); else await notify(otherOwner.id, "token_expired", other);
|
if (startup) await alerts.initializeOwnerAccessAlerts(); else await notify(otherOwner.id, "token_expired", other);
|
||||||
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted"); expect(sent).to.have.length(1);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("accepted"); expect(sent).to.have.length(1);
|
||||||
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(2);
|
await notify(otherOwner.id, "token_expired", other); expect(sent).to.have.length(1);
|
||||||
});
|
});
|
||||||
it("records a provider completion once after its insertion acknowledgement is lost", async () => {
|
it("records a provider completion once after its insertion acknowledgement is lost", async () => {
|
||||||
const collection = mongoose.connection.collection("owner_access_alert_events"), insert = collection.insertOne;
|
const collection = mongoose.connection.collection("owner_access_alert_events"), insert = collection.insertOne;
|
||||||
@@ -690,7 +876,7 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
await gate; await notify(otherOwner.id, "token_expired", other);
|
await gate; await notify(otherOwner.id, "token_expired", other);
|
||||||
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
await reset(owner.id, resource); await recover(owner.id, resource, async () => {});
|
||||||
} finally { collection.updateOne = update; release(); await pending; }
|
} finally { collection.updateOne = update; release(); await pending; }
|
||||||
expect(sent).to.have.length(1); expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect(sent).to.have.length(1); expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
});
|
});
|
||||||
it("confirms truncated-file metadata recovery only on a successful fresh matching-source probe", async () => {
|
it("confirms truncated-file metadata recovery only on a successful fresh matching-source probe", async () => {
|
||||||
const database = require("../src/server/database");
|
const database = require("../src/server/database");
|
||||||
@@ -705,31 +891,33 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
try {
|
try {
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
expect((await source.fetchFileInfoFromPath("a.txt")).sha).to.equal("a".repeat(40));
|
expect((await source.fetchFileInfoFromPath("a.txt")).sha).to.equal("a".repeat(40));
|
||||||
expect(await guards().findOne({ _id: recipient() })).to.equal(null);
|
expect((await guards().findOne({ _id: recipient() })).state).to.equal("recovered");
|
||||||
|
const secondKey = alerts.recipientAlertKey("second@example.com");
|
||||||
|
await Users.updateOne({ _id: owner._id }, { $set: { notificationEmail: "second@example.com" } });
|
||||||
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
await notify(owner.id, "token_expired", resource); await reset(owner.id, resource);
|
||||||
response = { type: "dir" }; expect(await source.fetchFileInfoFromPath("a.txt")).to.equal(null);
|
response = { type: "dir" }; expect(await source.fetchFileInfoFromPath("a.txt")).to.equal(null);
|
||||||
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
expect((await guards().findOne({ _id: secondKey })).state).to.equal("accepted");
|
||||||
response = Object.assign(new Error("Forbidden"), { status: 403 });
|
response = Object.assign(new Error("Forbidden"), { status: 403 });
|
||||||
try { await source.fetchFileInfoFromPath("a.txt"); throw new Error("Expected a failed probe"); }
|
try { await source.fetchFileInfoFromPath("a.txt"); throw new Error("Expected a failed probe"); }
|
||||||
catch (error) { expect(error.message).to.equal("Forbidden"); }
|
catch (error) { expect(error.message).to.equal("Forbidden"); }
|
||||||
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
expect((await guards().findOne({ _id: secondKey })).state).to.equal("accepted");
|
||||||
response = { type: "file", sha: "a".repeat(40), size: 11 };
|
response = { type: "file", sha: "a".repeat(40), size: 11 };
|
||||||
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { "source.repositoryName": "different/source" } });
|
await coll(resource.kind).updateOne({ _id: new mongoose.Types.ObjectId(resource.id) }, { $set: { "source.repositoryName": "different/source" } });
|
||||||
await source.fetchFileInfoFromPath("a.txt");
|
await source.fetchFileInfoFromPath("a.txt");
|
||||||
expect(await guards().findOne({ _id: recipient() })).not.to.equal(null);
|
expect((await guards().findOne({ _id: secondKey })).state).to.equal("accepted");
|
||||||
} finally { database.isConnected = originals.isConnected; utils.octokit = originals.octokit; }
|
} finally { database.isConnected = originals.isConnected; utils.octokit = originals.octokit; }
|
||||||
});
|
});
|
||||||
it("resumes a migration after a startup crashed and its lease expired", async () => {
|
it("resumes a migration after a startup crashed and its lease expired", async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "new", lease: "crashed", leaseUntil: new Date(0) });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "new", lease: "crashed", leaseUntil: new Date(0) });
|
||||||
await alerts.initializeOwnerAccessAlerts(); expect((await guards().findOne({ _id: "recipient-alerts-v1" })).state).to.equal("ready");
|
await alerts.initializeOwnerAccessAlerts(); expect((await guards().findOne({ _id: "recipient-alerts-v2" })).state).to.equal("ready");
|
||||||
});
|
});
|
||||||
it("continues waiting beyond 300 polls while another migration holds a healthy lease", async () => {
|
it("continues waiting beyond 300 polls while another migration holds a healthy lease", async () => {
|
||||||
await guards().insertOne({ _id: "recipient-alerts-v1", state: "new", lease: "active-migration", leaseUntil: new Date(Date.now() + 600000) });
|
await guards().insertOne({ _id: "recipient-alerts-v2", state: "new", lease: "active-migration", leaseUntil: new Date(Date.now() + 600000) });
|
||||||
const collection = guards(), update = collection.updateOne, timeout = global.setTimeout;
|
const collection = guards(), update = collection.updateOne, timeout = global.setTimeout;
|
||||||
let polls = 0;
|
let polls = 0;
|
||||||
collection.updateOne = async function (filter, ...args) {
|
collection.updateOne = async function (filter, ...args) {
|
||||||
if (filter._id === "recipient-alerts-v1" && filter.$or && ++polls === 305) {
|
if (filter._id === "recipient-alerts-v2" && filter.$or && ++polls === 305) {
|
||||||
await update.call(this, { _id: "recipient-alerts-v1" }, { $set: { state: "ready" } });
|
await update.call(this, { _id: "recipient-alerts-v2" }, { $set: { state: "ready" } });
|
||||||
}
|
}
|
||||||
return update.call(this, filter, ...args);
|
return update.call(this, filter, ...args);
|
||||||
};
|
};
|
||||||
@@ -737,6 +925,6 @@ describeMongo("recipient access alerts in MongoDB", function () {
|
|||||||
try { await alerts.initializeOwnerAccessAlerts(); }
|
try { await alerts.initializeOwnerAccessAlerts(); }
|
||||||
finally { collection.updateOne = update; global.setTimeout = timeout; }
|
finally { collection.updateOne = update; global.setTimeout = timeout; }
|
||||||
expect(polls).to.equal(305);
|
expect(polls).to.equal(305);
|
||||||
expect((await guards().findOne({ _id: "recipient-alerts-v1" })).state).to.equal("ready");
|
expect((await guards().findOne({ _id: "recipient-alerts-v2" })).state).to.equal("ready");
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
Reference in new issue
Block a user