Self-hosted platform for standing up and running software factories on AWS: complete DevSecOps environments with secure delivery pipelines, supply-chain controls, compliance evidence and multi-tenant governance — in minutes, not months.
Compute is the one cost a tenant can run up without limit. Every other expensive dimension is metered — API calls, AI calls, automation calls, plugin builds by count — but nothing charges for how long that work occupies a core.
Status: PLAN ONLY — nothing here is built.
Direction: a new per-period quota metered at the work boundary, sized per
tier, with a stackable pack for more. Named buildMinutes rather than
“CPU credits”, for the reason in D1.
Not from infrastructure metrics. container_cpu_usage_seconds_total is
scraped (with container_spec_cpu_quota, the CFS throttle counters and
machine_cpu_cores), so CPU seconds exist — labelled by container and pod,
which is the wrong axis. Tenant work does not run in per-tenant containers:
No PromQL recovers an organization from a metric whose finest label is a pod shared by every organization.
So meter at the work boundary instead — when a build finishes, charge its duration to the org that asked for it. Two consequences, both good:
plugin_build_duration_seconds{org_id} is
already observed per org, and plugin_builds_total{status, org_id} already
counts per org. The measurement is not new work; the accounting is.Adding a quota type is compiler-guided. QuotaType is a union and
QUOTA_TYPE_LABELS, QuotaTierLimits, DEFAULT_TIER_LIMITS and the frontend’s
QUOTA_TYPE_LABEL are all Record<QuotaType, …> — add the member and tsc
enumerates every table that must be filled. Nothing silently defaults.
Everything downstream is generic over VALID_QUOTA_TYPES and comes along for
free: the 80%/100% threshold notices (inbox + email + webhook, with the existing
opt-out), the usage ledger and its CSV, pooling at the account root, the at-risk
views, the 429 with an upgradeUrl, and the add-on/bundle plumbing.
This is the opposite of the storage quota’s position (see the storage plan): storage is a stock that fits the per-period model badly and needs special handling everywhere. Build minutes are a per-period flow, exactly like every other quota here, so they inherit the whole apparatus.
buildMinutes, not cpuCreditsDecision: buildMinutes, scoped to plugin builds, for the first release.
“CPU credits” promises a measurement we cannot make and invites an expectation
the scope violates: pipeline executions run in AWS and are not metered at all
— pipeline_executions_total does not even carry org_id. A quota called
“CPU credits” that ignores the largest compute an org can trigger would charge
the org that builds plugins and let the org running hundreds of pipeline
executions pay nothing. That is not a rounding error; it is a fairness inversion
customers will notice and be right about.
buildMinutes is self-documenting: it says what it counts, and nobody
reasonably expects pipeline runs to be in it. Ingesting CodeBuild is then an
honest follow-on (Phase 4) rather than a correction to a promise already made.
Decision: a credit is a vCPU-minute; a build is charged
ceil(duration_seconds × buildkit_vcpu / 60).
Per-build CPU is not observable — buildkit is a shared sidecar — but slot occupancy is, and it is what the platform actually pays for. An org that holds a build slot for ten minutes costs that slot whether it pegs the core or not, so occupancy is both the measurable quantity and the fairer basis.
buildkit_vcpu comes from the deployment: the minikube tree sets the buildkitd
container to 1500m, i.e. 1.5 vCPU, so a 10-minute build costs 15
buildMinutes. It must be read from configuration rather than hardcoded, since
an operator who raises the limit raises the real cost per minute.
Decision: reserve a pessimistic estimate at admission; settle the difference on completion. This replaces an earlier draft of this plan that said “meter after, refuse the next one” — that was wrong, and the reason is worth keeping.
Build cost is unknown until the build ends, which argues for charging afterwards. But a budget checked at admission and charged at completion bounds nothing: N builds can all be admitted while the budget allows one. A Developer org with 15 credits left can still have twenty builds admitted; at a 10-minute build (15 credits) that is 300 credits spent against 15 remaining — a 20x overshoot, not a rounding error. Every other expensive quota here avoids exactly this by reserving atomically before the work.
So:
withQuotaReservation, the same primitive the
plugins quota already uses on this path. Over budget, refuse with the
standard 429 + upgradeUrl (sendQuotaReserveDenied). Concurrent overshoot
is then bounded by estimate x concurrency rather than unbounded.overshoot”. At one
unit per API call that is immaterial; a build’s true-up is 15–100 units in one
increment, so a partial fit would record the overrun as overshoot and never
charge it. An org that overran would get the overrun free. Reservations are
the exact path and stay synchronous for exactly this reason, which is why
plugins, pipelines and aiCalls all use them.resetAt into the true-up (Q2). Builds are the
longest-running quota’d operation in the system — 20–60 minutes against a
boundary that arrives every 3 days — so a build straddling a period reset is
inevitable at volume, not exotic. The refund/settle path already threads
reservation.quota.resetAt to stop a rollover double-charging; the true-up must
do the same, or it settles against the wrong period.The two gates must unwind together. upload-plugin.ts already wraps the
upload in a plugins reservation, with the discipline spelled out in its own
comment: a throw refunds automatically, and markConsumed() at hand-off stops
that refund once the queued job owns the slot. Adding buildMinutes makes two
reservations on one submission, so a refusal of either must release the other —
state the order, and unwind both in one place. Two budgets reserved together and
released separately is the defect shape that leaks a slot on every refusal.
Decision: a build is charged for the time it ran, whatever its outcome — with one carve-out.
createBuildFailedHandler is a separate path from completedHandler, and a
build that fails after eight minutes consumed eight minutes. If only successes
are charged, the cheapest way to obtain free compute is to fail — so both
handlers meter, and a cancelled build is charged for the time it ran.
The carve-out is whose fault it was. A failure caused by tenant content (a broken Dockerfile, a failing smoke test, a refused scan) is the tenant’s compute and is charged. A retry forced by infrastructure — a worker eviction, a transient registry error — is the platform’s, and charging an org three times for one submission because the platform retried it twice is indefensible.
The field for that line already exists: FailureCategory is
'retryable' | 'permanent' (api/plugin/src/helpers/plugin-helpers.ts), carried
on the job and set by the failure handler. retryable is the platform’s — do not
charge the re-attempt; permanent is the tenant’s. The multiplier is real rather
than hypothetical: PLUGIN_BUILD_MAX_ATTEMPTS defaults to 2 on the main queue,
so one submission can burn two attempts with no further human action. Phase 2
should name the field, not restate the principle.
The charged-on-failure rule wants saying out loud in the product copy, because it is the part of this scheme a customer is most likely to find surprising.
Decision: cap what one build may cost; do not carve the budget per team.
Build minutes pool at the account root like every flow quota, so one team’s
runaway build loop consumes the whole account’s compute — and before D7 cut
the period to a day, that meant up to three days in which no team in the account
could build.
apiCalls tolerates this because it is cheap and large (25k–1.5M); one spike is
absorbed. Build minutes are few, expensive, and slow to refill, so the same
pooling behaves much worse.
Two mitigations, and only one is worth building now:
A per-build ceiling (do this). Refuse any single build projected beyond N credits, and stop a pathological build from consuming a tier’s budget on its own. Cheap, and it addresses the common case — one broken Dockerfile looping, not deliberate abuse.
The ceiling must be ≤ the smallest tier’s period budget, or a build that hits it can never be admitted on that tier at all. At Developer’s 3,000/month (D7) a 60-minute ceiling at 1.5 vCPU is 90 credits, which fits comfortably — but the margin is a property of the WINDOW, not of the ceiling: on the one-day window an earlier draft chose, the same ceiling was 90 against 100 and the constraint was nearly binding. Pick N against the smallest row of the Phase 1 table, and re-check it if the window ever shortens.
Per-team sub-allocation (defer). Predictable, and the same option the storage plan deferred for the same reason: it is an administration surface nobody has asked for. Revisit if accounts actually contend.
The residual risk after the ceiling is many legitimate builds from one team crowding out another. That is the pooled model working as designed, and it is visible — the at-risk views and the 80% notice name the account, not the team.
Decision: refuse the build when the budget cannot be read.
Quota checks “time out fast (fail open)” after QUOTA_CHECK_TIMEOUT_MS (300 ms),
counted on quota_fail_open_total, which alerts. That is right for apiCalls:
breaking the console because the quota service blinked is worse than serving a
few unmetered requests.
storageBytes is deliberately the exception — a longer read budget behind a
fail-closed push gate, on the reasoning that an unverifiable cap must not let
an over-quota org consume. Build minutes sit on that side of the line: expensive,
deferrable by seconds without harm, and the thing being protected is real
compute. A build refused because the budget could not be read is a retry; a build
admitted on an unreadable budget is capacity given away.
Give the check the longer budget storage gets rather than the 300 ms hot-path one — an admission check runs once per submission, not once per request, so it can afford to wait for an answer instead of guessing.
Decision: give buildMinutes its own period and make it a calendar month,
aligned to the subscription. Leave QUOTA_RESET_DAYS alone for every other
quota.
This settles what an earlier draft left open (“accept the three-day window, or argue for a per-type period”). Three findings decided that it needs its OWN period; D10 decides the LENGTH, and a one-day window — which two earlier drafts of this decision chose — is the wrong answer for a reason neither of them weighed. The reasoning is kept below rather than deleted, because the two structural findings are what make a per-type period possible at all and they are unchanged.
The invariant that was supposed to make the window harmless does not hold.
Phase 1 used to claim plugins “stays the primary limit on every tier by
construction: a tier runs out of builds before it runs out of minutes”. That
is false on the failure path, and the failure path is the one that matters.
releasePluginQuota (api/plugin/src/queue/build-quota.ts) refunds the
plugins slot on every terminal failure, while D4 charges buildMinutes for a
tenant-caused one. So a broken Dockerfile costs minutes and no plugins. On
Developer’s old 300, a 15-minute failing build at 1.5 vCPU is ~22 credits, so
roughly thirteen failed iterations exhausted the budget with the plugins
counter still at 0/10 — and then nobody in the account built for up to three
days.
The two quotas cannot be reconciled, either. Charging plugins for failures
would give a Developer ten attempts per period, which is worse for the exact
workflow Phase 1 already flags as that tier’s weak point (someone iterating on a
Dockerfile, no warm cache). Refunding buildMinutes is the free-compute hole D4
exists to close. So they diverge permanently, the claim is struck, and window
length becomes the only lever.
“Aligned to the subscription” is not implementable as worded, and that is a
real constraint rather than a quibble. nextQuotaResetDate does DAY
arithmetic (date.setDate(date.getDate() + days)), and the Mongo rollover adds
unit: 'day'. Worse, the reset is LAZY — it fires on the first increment after
expiry — so the boundary follows the org’s own activity rather than a calendar.
At three days nobody notices; over a month two things become visible and
arguable:
Two honest options. Either accept a 30-day rolling window and say so in the
docs and the UI copy — in which case drop “aligned to the subscription”, because
it will not be — or switch the reset to month arithmetic ($dateAdd does
support unit: 'month') and make periodDays carry a unit rather than a day
count. The second is the better product and the larger change: it is a contract
change on the field the dashboard reads, not just a different number in it.
Decide before Phase 1 ships, because the field shape is what Phase 1 moves into
QuotaSummary.
A per-type period costs much less than assumed. The earlier draft called it
“a change to the quota service, not a config tweak”. It is nearer the latter:
resetDays is already a parameter of both rollover stages
(resetIfExpiredStage(usagePath, amount, resetDays) and its pooled twin in
api/quota/src/services/quota-service.ts), and resetAt is already stored per
counter, not per org (api/quota/src/models/organization.ts). Rollover is lazy
and evaluated per type inside the Mongo pipeline, so there is no sweep to change
and no schema migration. The real work is one response-contract change:
periodDays is a single top-level field on OrgQuotaResponse that the
dashboard’s “per N days” copy and its usage forecast read, and it has to move
into each QuotaSummary.
Why a MONTH and not a day, which is where the earlier drafts went wrong. Shortening the window was argued as “same throughput, a third of the lockout”: a day gives the same minutes over three days and caps the worst-case wait at one day. That is true and it is not the whole trade. A window does two jobs — it bounds the burn AND it decides how much of your allowance you may spend at once — and the earlier reasoning only priced the first.
Build work is bursty by nature, and this plan says so elsewhere without drawing the conclusion: changing a base image rebuilds everything that sits on it, while publishing another version costs little. So the expensive day is a base-image bump, a CVE sweep or a release — a day on which an organization wants its whole catalogue rebuilt at once.
Price that against a daily window at Pro: a full catalogue rebuild is 25 builds x 24 wall-minutes x 1.5 = 900 credits in one day against a 300/day cap. The account is refused on the first morning of its release while holding 8,100 unspent credits for the month. A daily window does not ration that work, it forbids it — and the remedy it leaves (buy a pack) charges a customer for capacity it has already paid for and cannot reach.
A monthly window is the same total capacity with the accumulation restored, and D10 is the requirement that settles it: the budget has to cover a month of normal activity, which it cannot do if a normal month’s shape is refused.
What a month costs, and the mitigations that make it affordable. The honest objection is the one D7 was built on: a long window means a long lockout. Four things bound it, and the last is new:
QUOTA_CHECK_DENIED_CACHE_MS).QuotaCard already renders UsageForecast, which
extrapolates the burn rate to the period boundary. On a one-day window that is
noise; on a month it is the sentence that matters — “at this rate you run out
on the 22nd” — and it arrives with weeks to act rather than hours.Two riders this creates, both small and both real:
plugins every three days,
buildMinutes monthly. The page already renders a per-type resetAt (it is on
every QuotaSummary), so this surfaces correctly once periodDays is
per-type; it is a copy problem, not a mechanism one. It also removes the last
reason to describe the two quotas as a ladder.Also extend the existing grace band. enforcedQuotaLimit
(packages/api-core/src/types/quota-tiers.ts) already implements a soft overrun
above the nominal cap, and it was built for this complaint in these words: “a
self-mutating pipeline that hard-stopped at 100% stayed stopped until the period
reset, up to 3 days.” Today it is a hardcoded quotaType !== 'automationCalls'.
Generalise it to a per-type grace lookup fed by QUOTA_AUTOMATION_GRACE_PCT and
a new QUOTA_BUILD_MINUTES_GRACE_PCT, rather than adding a second special case
to the predicate. The band does not solve a lockout — it is 10% — but it removes
the cliff at exactly 100%, which is the half of the complaint it was designed for.
Decision: do not mirror automationCalls. Add a trigger label to the build
metric, and revisit only if the data says to.
The symmetry argument is tempting and was worth writing down: automationCalls
exists “so a fleet of pipelines or CI scripts can’t lock people out of the
console”, and CI-triggered builds could equally lock out a human iterating on a
plugin. But the analogy breaks on what each split protects.
automationCalls separates two different surfaces. Automation spends on
writes; what is being defended is a human’s ability to read the console — a
different activity, which the split genuinely protects. Build minutes have only
one surface: building. A separate automation budget does not give the human a
build slot (both contend for the same buildkitd regardless), it only gives them a
smaller private budget. That is a sub-allocation, and D5 already deferred
per-team sub-allocation for precisely this reason: it is an administration
surface nobody has asked for.
It is also not the cheap mirror it looks like. The apiCalls /
automationCalls choice is made in middleware, where the principal is in hand. A
build’s charge lands in a queue handler, and PluginBuildJobData
(api/plugin/src/helpers/plugin-helpers.ts) carries orgId, userId and an
access snapshot but no principal type. The split needs a new payload field
threaded through the upload route, the DLQ replay and the failed-build retry —
paths already carrying delicate snapshot semantics (reservedResetAt,
quotaReleased).
And the deciding evidence is one label away. The earlier draft said the
org_id-labelled build metrics “could answer today” how much volume is
automated. They cannot, quite: plugin_builds_total carries {status, org_id}
only (api/plugin/src/queue/plugin-build-queue.ts,
api/plugin/src/queue/failure-handler.ts). Adding a bounded
trigger: 'user' | 'automation' label is one line in each of those two call
sites and makes the question answerable before anyone builds a second budget,
two more tier numbers, a second pack and two more frontend rows. Do that in
Phase 2 and leave the split unbuilt.
If the data ever justifies it, note that the cheaper remedy comes first: a per-build ceiling (D5) already bounds one runaway automated build, and the grace band (D7) already removes the hard stop that makes a shared budget feel punitive.
Decision: no tier’s daily budget may exceed what the deployment’s buildkitd can actually deliver, and the ladder below must be checked against that before it is sold.
This is the one decision that changes the plan’s conclusions rather than its
details, and it was missed because the Phase 1 table is derived from the
plugins cap — a DEMAND model — with nothing on the other side of the equation.
The arithmetic, and there are TWO numbers — a floor and a ceiling. A credit
is a vCPU-minute (D2). deploy/*/k8s/plugin.yaml gives the buildkitd sidecar
limits.cpu: 1500m but requests.cpu: 300m, one sidecar per plugin pod, with
replicas: 1. Kubernetes schedules on REQUESTS, so:
Ceiling (the limit, available only when the node is otherwise idle):
1.5 vCPU x 1440 min x 30 days = 64,800 vCPU-minutes per monthFloor (the request, which is what is actually reserved):
0.3 vCPU x 1440 min x 30 days = 12,960 vCPU-minutes per month
The ceiling is the number a first draft of this decision used, and it is the optimistic one: a build can burst to 1.5 vCPU only if nothing else on the node wants the cores. The guaranteed figure is the floor, and against it even Team (24,000) is already 185% — not just the top two rows. Quote both, because selling against the ceiling is selling capacity the scheduler never reserved.
The first fix is therefore to raise requests, not to argue about the limit.
A quota that promises 1.5 vCPU of throughput against a 0.3 vCPU reservation is
promising what the scheduler has not set aside; making requests match the
capacity being sold is what turns the ceiling into the floor.
against the demand-derived ladder (the ratios are period-independent, so this reads the same whatever window D7 picks):
| Tier | Cap/month | Share of the ENTIRE pool |
|---|---|---|
| Developer | 3,000 | 5% |
| Pro | 9,000 | 14% |
| Team | 24,000 | 37% |
| Business | 60,000 | 93% |
| Enterprise | 135,000 | 208% |
One Enterprise tenant’s cap is twice the platform’s total capacity — undeliverable with no other tenant on the system at all. One Business tenant leaves 4,800 vCPU-minutes a month for everybody else. Two Team tenants take 74%.
This is not ordinary oversubscription. Selling more than you have is normal
and healthy where nobody uses their whole cap — that is exactly the reasoning
behind the storage plan’s StorageCommitmentExceedsCapacity, which fires only
past ~10x. Here the top tier’s cap cannot be met in isolation, and the failure
mode is different in kind: a full disk stops everyone, while a saturated build
pool QUEUES everyone. The quota as designed caps a VOLUME and says nothing about
concurrency, so two tenants with budget to spare still wait behind one buildkitd.
Scaling the ladder down does not work, and that is the useful finding. Hold
Enterprise to half the pool (32,400/month) and the whole table scales by 0.24:
Developer lands at 720 credits/month, which at 20 wall-minutes and 1.5 vCPU is
24 builds a month against a plugins cap of 100 — so buildMinutes would
become the binding limit on the cheapest tier, which is exactly the inversion
Phase 1 rejected when it rejected the first draft’s numbers. The ladder and the
shipped capacity are not reconcilable by re-picking numbers.
So SUPPLY is the variable that has to move, and the plan must say by how much:
135000 / (1440 x 30) = 3.125 vCPU of
buildkitd running flat out, with zero headroom and nothing left for any other
tenant.And here is the part that reframes this whole decision: the fix is cheap.
Priced at us-east-1 on-demand (checked October 2026), a vCPU-hour is ~$0.036 on
Graviton (c7g.large, $0.0725/hr for 2 vCPU) and ~$0.045 on x86
(c7i.large, $0.0892/hr); EKS Auto Mode adds ~12% on top, and charges it at the
on-demand rate even under a Savings Plan. So:
| Reserved capacity | Serves | Cost/month (c7i, +Auto Mode) |
|---|---|---|
| 1.5 vCPU (today’s limit) | Business, as sole tenant | ~$55 |
| 3.125 vCPU | Enterprise, as sole tenant | ~$114 |
| 6.25 vCPU | Enterprise at half the pool | ~$228 |
Removing this constraint entirely costs on the order of $150–180 a month more than the platform spends today, against a single Enterprise subscription at $899. So D9 is a PROVISIONING decision with a known and small price, not a product limit — and it should be read that way rather than as a reason to shrink the ladder. The revenue side makes the same point: the ceiling’s 64,800 credits/month sold as packs at $4.99 per 500 is ~$647/month against ~$55 of cost.
What stays true is the discipline, not the pessimism: the ladder must not be published without the check below, because the failure mode of getting it wrong is selling a cap nobody can serve.
What to build, rather than what to assert:
periodSupply = buildkit_vcpu x 1440 x periodDays from it.max(tier buildMinutes) <= periodSupply x
SINGLE_TENANT_SHARE. A deployment whose ladder exceeds its own capacity
should refuse to sell the tiers it cannot serve, or at minimum say so loudly
at startup — the same shape as the tier-config echo already in the quota
service’s boot path.quota_buildminutes_committed over periodSupply, warning
only, not routed to any actuator. Compute cannot be auto-expanded the way a
PVC can, so there is nothing for an actuator to do.One wrinkle for whoever sizes it. buildkitd is a sidecar of the plugin pod, so supply rises either by raising that container’s limit (a bigger node — and rootless buildkit already constrains the node choice: see the Fargate note in the deploy docs) or by scaling plugin replicas, which multiplies buildkitd instances. The second breaks D2’s “shared sidecar” premise: per-pod capacity becomes the unit, and a tenant’s builds may land on either, so the ceiling is per-pod while the quota is per-account. Prefer the bigger limit until that is designed.
Decision: size every tier so a standard month of work fits with roughly 3x headroom, and state the activity model rather than the number.
The ladder was derived from the plugins cap — one internal number from
another — which says nothing about whether a customer can get a month’s work
done. This is the requirement that makes the table answerable, and it follows
AUTOMATION_CALLS_SIZING’s example of writing the arithmetic down “so the next
person argues with the arithmetic, not with the numbers”.
What a normal month looks like. A tier’s plugins cap is a per-3-day build
RATE, so it already asserts a monthly build count (x10). Normal activity is not
the cap — the cap exists for peaks — so take normal as 30% utilisation:
| Tier | Builds/month at the cap | Normal (30%) | x wall-min x 1.5 vCPU | Normal credits/month |
|---|---|---|---|---|
| Developer | 100 | 30 | 20 | 900 |
| Pro | 250 | 75 | 24 | 2,700 |
| Team | 500 | 150 | 32 | 7,200 |
| Business | 1,000 | 300 | 40 | 18,000 |
| Enterprise | 1,500 | 450 | 60 | 40,500 |
Against the Phase 1 ladder those land at a uniform 3.3x headroom — the same factor on every tier, because both sides are linear in the same two assumptions. So the answer to “is it enough for a month of standard activity?” is yes, in aggregate, with room for a month that runs three times hot.
The aggregate was never the problem; the SHAPE was. The same figures under a one-day window refuse a Pro account’s release day while it holds 8,100 unspent credits — see D7, which this decision reverses to a monthly period for exactly this reason. A budget that covers a month of normal activity has to be spendable the way a month of normal activity actually arrives: unevenly.
Two consequences worth naming:
build_minutes_charged_total per org against the tier cap gives real
utilisation, and the ladder should be re-derived from it rather than from 30%.
Until then, 30% is a stated guess and should be labelled as one in the docs.buildMinutes to QuotaType and VALID_QUOTA_TYPES, then fill the tables
tsc names: QUOTA_TYPE_LABELS (“Build minute”), QUOTA_UPGRADE_PATH,
QuotaTierLimits, DEFAULT_TIER_LIMITS, and the frontend’s
QUOTA_TYPE_LABEL.DISPLAYED_QUOTA_TYPES so it shows on the quotas page from day one.
(Storage’s absence from that list is a live bug; do not create a second one.)VALID_QUOTA_TYPES, but storage is the standing counter-example
of a quota that looked generic and needed a carve-out in pooled-quota.ts.
Confirm two things before relying on it: that a TEAM-triggered build increments
the ROOT’s counter through resolveIncrementRoot, and that the admission check
reads the pooled limit rather than a team’s own -1. Getting the second wrong
reproduces the storage bug exactly — every team spending past the root’s cap,
because the gate reads -1 as unlimited.resetDays is already a parameter of
resetIfExpiredStage and its pooled twin, and resetAt is already stored per
counter, so this is a per-type lookup threaded to those call sites — not a
schema change and not a sweep. The contract work is moving periodDays off
OrgQuotaResponse and into each QuotaSummary, which the dashboard’s “per N
days” copy and usage forecast read.enforcedQuotaLimit hardcodes
quotaType !== 'automationCalls'; replace the predicate with a per-type grace
lookup rather than adding a second special case. Both the conditional
increments and the status read already route through it, so nothing else moves.docs/environment-variables.md — every variable read
through envInt/envBool/envStr must appear there or env-documented
fails the build. The per-tier override QUOTA_TIER_<TIER>_BUILD_MINUTES comes
free with tierLimitEnv; QUOTA_RESET_DAYS_BUILD_MINUTES (the per-type
period), QUOTA_BUILD_MINUTES_GRACE_PCT (the band) and the buildkit vCPU
figure each need their own entry.Tier sizing — DERIVED, so the argument is about the principle. An earlier
draft picked a number per tier and asserted it was “generous relative to the
plugins count quota”. Checked against the real limits (10 / 25 / 50 / 100 /
150) that was true on four tiers and false on the one that matters: it gave
Developer 8 wall-minutes per permitted build against 24–67 everywhere else,
so the new quota became the binding constraint on the only tier that cannot pay —
and on the tier where builds are slowest, with no warm cache and someone
iterating on a Dockerfile.
So the ladder is derived rather than chosen. The budget is per month (D7)
while plugins is per three days, so the derivation goes through builds per
month:
monthly credits = (plugins_limit x 10) x per-build wall-minutes x buildkit_vcpu
| Tier | plugins / 3 days |
Builds / month | Wall-min per build | buildMinutes / month |
|---|---|---|---|---|
| Developer | 10 | 100 | 20 | 3,000 |
| Pro | 25 | 250 | 24 | 9,000 |
| Team | 50 | 500 | 32 | 24,000 |
| Business | 100 | 1,000 | 40 | 60,000 |
| Enterprise | 150 | 1,500 | 60 | 135,000 |
| Unlimited | — | — | — | -1 |
Every row is exact at buildkit_vcpu = 1.5. D10 checks these against a month of
normal activity and finds a uniform 3.3x headroom, which is the point of the
sizing: a standard month fits, and a month that runs three times hot still fits.
But the table is a DEMAND model and is not sellable as it stands. Nothing in this derivation looks at supply, and D9 shows the top two rows exceed what the shipped buildkitd can deliver — Business takes 93% of the whole pool and Enterprise 208% of it. The ratios are period-independent, so moving to a monthly window changes nothing about that: 1.5 vCPU delivers 64,800 credits a month, and Enterprise’s 135,000 is still twice it. Treat the numbers here as the shape of the ladder and D9’s check as the gate on publishing it.
The only judgement left is the wall-minutes column — how long a build on each tier is allowed to take on average — which is a question somebody can actually hold an opinion about. It rises with tier because larger customers build larger images, not because larger customers deserve more slack.
What this table does NOT give you is a guarantee that plugins binds first.
An earlier draft claimed it did, “by construction”. D7 records why that is false:
a terminal build failure refunds the plugins slot and still charges the
minutes, so the two quotas diverge on exactly the path a Developer hits most
(iterating on a Dockerfile). Size the table for normal use, and read the
“how we would know it worked” check as a detector of mis-sizing rather than as
a proof it cannot happen.
Developer deserves a sanity pass as a row, not as a cell. It already carries 10 builds per period and 2 GB of registry storage — about four images at typical size. So a Developer can build ten times and store four. Adding a third, differently-shaped cap to the same workflow is worth reviewing against the other two rather than in isolation.
completedHandler and in createBuildFailedHandler, from
job.processedOn / finishedOn, both of which are already read there for the
duration histogram. Best-effort, like every other increment: never fail or slow
a build because accounting failed.plugins reservation, so an
org over budget is told at submission rather than discovering it later.build_minutes_charged_total{org_id, outcome} so the charge is
auditable against the duration histogram already there — if the two disagree,
the accounting is wrong.failureCategory. BullMQ retries each set
processedOn / finishedOn, so a job retried three times charges three times
unless something says otherwise, and PLUGIN_BUILD_MAX_ATTEMPTS is 2 by
default. Charge a permanent failure (tenant content); do not charge the
re-attempt of a retryable one (the platform’s). The field is already on the
job — this is a condition at the meter, not a new taxonomy.trigger: 'user' | 'automation' to the two plugin_builds_total call sites
(plugin-build-queue.ts, failure-handler.ts). This is the one line that
makes “should automated builds have their own budget?” answerable from data
instead of from argument, and D8 declines to build the split until it is.system org
and setup-load images are built at install; neither should spend a tier budget.
One condition, but without it a fresh install spends its own first budget on
its own seed data.b('build_minutes_pack', 'Build Minutes Pack (+500 min)',
'500 additional build minutes / month', { buildMinutes: 500 },
499, PAID, <sort>,
{ volumeTiers: SEAT_STYLE_VOLUME, maxQuantity: <from D9 supply> })
+500 minutes a month, granular, with seat-style volume breaks and a quantity cap. Each of those four is a decision the earlier drafts got wrong or left open; the reasoning is below. A pack grant raises the cap per period, so the monthly period (D7) sets the unit: +500/month is about fourteen extra builds at Pro’s assumed 24 wall-minutes.
Three things to get right, all learned from the existing catalogue:
plugin_pack, api_pack and automation_pack
all carry CAPACITY_VOLUME (10/20/30% at 4/10/25 units). storage_pack is the
one capacity pack without it, at the highest price — an omission, not a
decision. Do not copy it.Keep the grant GRANULAR, which is what buys headroom on the only size
constraint that binds. RATE_PACKS refuses a pack granting more than half
the cap of any tier that may buy it (grant > cap / 2). A coarse pack sized
at a sixth of Pro’s budget sat exactly on that bound in an earlier draft —
passing only because the comparison is strict, and coupling pack size to the
wall-minutes column that same draft called “the only judgement left”. At
+500 against Pro’s 9,000 the pack is 5.6% of the cap, so Pro’s row can be
retuned without breaking a test. The no-crossing rule is comfortable too
(9,000 + 500 is nowhere near Team’s 24,000).
An earlier draft also justified its size with a DEVELOPER boundary
(“100 + 150 = 250, under 300”), which does not apply once the pack is
PAID-only — and PAID-only is not optional: a separate test asserts
storage_pack is the only capacity pack sold on Developer.
CAPACITY_VOLUME. The catalogue encodes
the rule and an earlier draft read it backwards: CAPACITY_VOLUME’s 10/20/30%
at 4/10/25 units is sized for a COARSE pack, while seat — the one
granular pack — carries its own, much higher 5/15/40 breaks precisely because
granularity moves them. Copy seat. With 4/10/25 on a +500 pack, a Pro
account merely doubling its budget buys 18 units and takes 20% off, and
anyone topping up seriously clears 25 units and takes 30% off permanently
— on the one dimension the platform pays for per unit.stackable defaults to true and no maxQuantity is set,
so an account could buy arbitrarily many minutes against a pool that delivers
64,800 a month (D9). D5 added a per-BUILD ceiling for exactly this reasoning;
this is its pricing-side twin, and without it the catalogue can sell capacity
the platform has no way to produce. Derive the cap from supply, not from a
round number.AddonGrid already
takes an exact typed quantity and divides the grant by
quotaPackUnit(dimension).per — which is why storage is typed in GB rather
than in packs. A buildMinutes catalog entry with
packUnit: { name: 'min/month', per: 1 } makes “enter 2000” mean 2,000 more
minutes a month, which is the number a buyer actually has in mind. F2 below
asks only for a LABEL; it needs both.PRICE IT AT $4.99/month, derived rather than picked. Cost does not set it: 500 vCPU-minutes is 8.3 vCPU-hours, which at us-east-1 on-demand (~$0.036/hr Graviton, ~$0.045 x86, +~12% for EKS Auto Mode) is $0.30–$0.47 of compute. Even as the one dimension with genuine marginal cost, the pack is value-priced at ~93% gross margin, and anyone reasoning from “real cost” to “expensive” has the wrong instinct.
What sets it is the method the seat pack already uses: sit between the
ladder’s adjacent-step rates so upgrading stays better value. Per 500 credits
those steps imply Developer→Pro $3.25, Pro→Team $1.33 (cheapest),
Team→Business $2.36, Business→Enterprise $4.33 (dearest). The binding
requirement is that the pack must not undercut the CHEAPEST step after its
deepest volume discount: at $4.99 the 30% break lands at $3.49, still well
clear of $1.33. A sibling cross-check agrees — api_pack grants 25% of Pro’s
cap for $11.99, and a +500 pack is 5.6% of it, which scales to $2.69 — so the
defensible band is $2.69–$4.99 and this takes the top of it, because
compute is the one dimension with a hard supply ceiling (D9) and packs should
not be the cheap way around right-sizing.
It is also coherently cheaper than the $11.99 siblings because the grant is finer. A smaller step costs less, which is what makes “enter 2000 minutes” read as buying a quantity rather than a product.
b() defaults annual to
monthly x 10 — two months free, ~17% off, the catalogue’s steepest standing
discount. On a consumable with a supply ceiling, an annual pre-pay commits the
platform to reserving capacity for twelve months at a discount, which is the
opposite of what the ceiling wants. Set annual to monthly x 12 ($59.88) and
let the volume curve be the only discount on this dimension.bundle-sizing imposes NO price floor on this pack, and an earlier draft
claimed otherwise. buildMinutes still has to be added to both DIMS and
RATE_PACKS — neither is derived from VALID_QUOTA_TYPES — but read what the
test does with it: closeGap runs per dimension and the costs are SUMMED into
one basket, compared once against the tier delta. Adding a dimension therefore
makes basket > delta easier, and since the suite passes today without
buildMinutes the property cannot be broken at any price.
The “~$11.11 floor” (and the ~$13.34 before the daily rescale) came from treating the buildMinutes row as if it alone had to exceed the $40 Pro→Team delta. It does not, so the worry that $11.99 clears the floor by only 7% was about a constraint that does not exist. The price is a margin decision, not a test-satisfying one — which matters more here than elsewhere, because this is the one dimension the platform pays for per unit.
DEFAULT_BUNDLE_DIMENSION_MAP must
gain build_minutes_pack, and a test holds the catalogue and the map in sync.
The map’s own comment records why: stakeholder_reports shipped missing from
it and was therefore “granted to the customer and then NEVER metered on
Marketplace, i.e. given away”. Of every bundle to give away by omission, the
only one with real marginal cost is the worst.throughput_bundle. It is plugin_pack + api_pack + ai_pack
at $25.99 against a $35.97 member basket. Build minutes are the
build-throughput dimension, so either they join it — repricing the combo, with
warnOnNonDiscountCombos requiring it stay below the new basket — or the plan
says why the throughput combo omits the throughput that costs money.b() defaults annual to
monthly x 10: two months free, ~17% off, the steepest standing discount in
the catalogue. Inheriting it on the only dimension with per-unit cost should be
a decision rather than a default.Why granularity was the decision behind all four. Every add-on is
quantity x a fixed grant, so the grant size is simply the step the buyer types:
seat grants 1 and typing 8 means 8 seats; storage_pack grants 10 GB and is
typed in GB. Choosing a coarse grant is what put the earlier draft on the ratio
bound, made CAPACITY_VOLUME look right, and left the typed unit unstated. One
choice fixed three symptoms, which is usually the sign it was the real decision.
Adding a dimension that is DERIVED from another one silently breaks the packs
that grant the other one. The formula is
credits/month = (plugins x 10) x wall-minutes x vcpu, so:
| Pack | Grants | Build minutes it needs | Grants today |
|---|---|---|---|
plugin_pack |
+10 plugin builds / period | ~3,600/month | 0 |
pipeline_pack |
+5 pipelines, +25 plugins (Team) | ~12,000/month | 0 |
A buyer of either gets builds they have no minutes to run. This repository has already made this exact mistake twice and fixed it twice, and both comments are still in the catalogue:
pipeline_pack grants plugins alongside pipelines because “a pipelines-only
pack broke that ratio for anyone who bought it … so the pipelines it sold could
not be built for”;seat grants API and AI capacity because “a seats-only pack broke that
invariant for anyone who bought it … and nothing told the buyer. Granting the
per-seat share keeps a bought seat worth what a tier seat is worth.”So extend both grants, and add the invariant test beside the one that already holds “every tier keeps 5 plugin builds per pipeline, packs included” — a minutes-per-plugin-build ratio is the same guard on the same relationship.
One conflict to resolve first: RATE_PACKS requires a pack to grant exactly
ONE dimension (so BILLING_BUNDLE_<ID>_GRANT keeps working), which is why
pipeline_pack is classed as a differentiator and exempt. A plugin_pack that
also granted minutes would have to leave RATE_PACKS the same way — losing the
ratio bound and the free-tier guard with it. Either accept that trade, or decide
deliberately that plugin builds and their minutes are sold separately and say so
where a buyer can read it.
Build minutes are the first quota with genuine per-unit marginal cost. Every other dimension is a rate limit protecting shared infrastructure; this one is compute the platform actually pays for. That invites a pricing model the plan never weighs: meter it and bill overage, rather than cap it and sell packs.
Capped is still right, for a reason worth writing down rather than assuming: there is no overage-billing machinery in this codebase at all (the storage plan recorded the same thing — “there is no overage billing”). A cap plus packs reuses the whole bundle apparatus; metering is a new subsystem with its own invoicing, disputes and proration. Revisit only if build minutes become a material share of revenue.
buildMinutescheckEntitlementOvercap refuses a downgrade or pack removal that would strand
an account over a cap it cannot shrink into. It guards seats, plugins,
pipelines, listings and storageBytes, and excludes the per-period rate
quotas with a stated reason: they reset, so landing over a reduced cap costs at
most one period.
The monthly window (D7) invalidates that reasoning for this dimension. “At
most one period” is three days for apiCalls and a month for
buildMinutes — so a mid-month downgrade, or cancelling a Build Minutes Pack,
can refuse every build for the rest of the month. That is a month-long outage
caused by a billing click, which is precisely the failure the guard exists to
prevent.
So add buildMinutes to the guarded list. Note that this is the same correction
storage needed, for the same reason and with the same shape: storage was grouped
with the rate quotas on the “they reset” rule, the rule did not apply to it, and
the fix was one entry in that list. The lesson generalises — the exclusion is
about period LENGTH, not about being a rate — and the guard’s comment should
say so, or the next long-window quota repeats it.
One consequence to design, not just assert: unlike plugins, build minutes are
spent continuously rather than held, so “current usage” against a reduced cap is
a point-in-time reading that will keep rising. Refusing the downgrade outright is
the conservative answer; the alternative — allow it and let the grace band plus
the forecast carry the account to the period boundary — is defensible but needs
the notice to say what is about to happen.
The honest completion of D1. AWS reports duration and compute type per build, and
pipeline-manager already talks to AWS, so the data is reachable. Once pipeline
execution compute is metered:
org_id to pipeline_executions_total — one label, and executions become
attributable by count even before any of this;cpuCredits becomes accurate, and the rename is a migration
with a reason rather than a promise made early.Until then the quota is honestly scoped and honestly named.
The two questions this plan used to leave open — the three-day period (Q3) and a separate budget for automated builds (Q5) — are settled in D7 and D8. Each was resolved by checking the code rather than by picking a side: Q3 turned on finding that the invariant meant to make it harmless is false and that a per-type period is far cheaper than assumed; Q5 turned on finding that the surface being protected is not analogous and that the payload carries no principal to split on.
The remaining judgement is the wall-minutes column of the Phase 1 table, which is flagged there and is a product opinion rather than a question about the code.
seat carries a per-seat capacity invariant — every paid tier holds exactly
100,000 API calls and 200 AI calls per seat, and the pack grants that share so a
bought seat is worth a tier seat. The obvious question is whether it must now
grant build minutes too.
It must not, because there is no invariant to preserve: per-seat minutes are not constant across the ladder — Pro 9,000, Team 8,000, Business 7,500, Enterprise 9,000. Recorded so nobody re-raises it.
Worth noticing while here, though: Pro and Enterprise already sit at exactly
9,000 per seat. Re-deriving Team to 27,000 and Business to 72,000 would make
the whole column constant, let seat carry build minutes like it carries the
other two, and bring the dimension under the existing invariant test for free.
That is an improvement rather than a defect — and it collides head-on with D9,
since 72,000/month is above what the shipped pool can deliver at all. Decide the
supply question first; this one is downstream of it.
Anonymous submissions build on a separate quarantine buildkitd (1000m),
and their jobs carry “nothing but a submission id” — no orgId — so that compute
falls on the platform rather than a tenant. It is deliberately not charged, and
it is not a hole: the isolated pool is small by design and runs one build at a
time, which bounds it without a quota.
Recorded for the same reason the storage plan records that quarantine storage is bounded by an unconditional 30-day sweep — so the next reader does not have to re-derive the answer.
Enforced contracts, not polish — the same list the storage plan carries:
| Item | Contract |
|---|---|
| New env vars | docs/environment-variables.md, or env-documented.test.ts fails |
| Any doc edit | the help corpus is SHA-pinned to its source doc; regenerate or help-corpus.test.ts fails |
| Quota type | the Record<QuotaType, …> tables are compiler-forced; the frontend label map is one of them |
An earlier draft called this work “free” twice and was wrong twice. Two of the
five deliverables have since become nearly free, for a reason worth
recording: the quota display catalog
(packages/api-core/src/types/quota-catalog.ts) was built after that draft and
collapsed the seven label/unit maps the frontend used to carry. F1 and F2 are now
catalog entries rather than new mechanisms; F3–F5 are the real work.
F1 — a minutes render mode (now one entry, compiler-enforced). The draft
said QuotaCard formats everything through fmtNum and that build minutes would
be the second quota needing unit-aware display. That display is built:
QuotaRender is 'count' | 'bytes', formatQuotaValue switches on it
exhaustively, and QuotaCard, UsageCard, TeamUsageCard and both at-risk
lists all route through it. So adding 'minutes' to the union makes tsc name
every surface that must handle it — the case the exhaustive switch was written
for. The work is the mode plus its formatter (“2h 30m”), not a new pathway.
F2 — a packUnit on the catalog entry. The draft described PACK_UNITS as a
hardcoded map with one member (storageBytes → GB), and framed adding build
minutes as a judgement its comment had already set up. That map is gone; the unit
is packUnit on the catalog entry and AddonGrid reads it through
quotaPackUnit. The judgement still stands and still falls the same way — a
+500/month pack reading “500 minutes” says far more than “1 pack” — and with
packUnit declared the field is TYPED in minutes, so the buyer enters the number
they have in mind rather than a pack count. It is a catalog field now, not a map
to extend.
F3 — show in-flight reservations, or the number appears to go backwards.
D3 reserves an estimate at admission and settles on completion, so while a build
runs, used includes a reservation that may be refunded if the build beats
the estimate. The page will show usage climb and then drop. No quota does that
today — AI reservations settle in seconds, builds in minutes — so there is no
existing affordance for it, and without one it reads as a bug and generates
support traffic about wrong numbers. Either show “N in flight” explicitly or
render the reserved portion distinctly from the settled portion. This is a
requirement on D3, not a frontend nicety: a quota that trues up on a
human-visible timescale has to say so.
Post-purchase latency is bounded, and worth saying (Q6). A denied check
reading is cached until the period resets, capped at
QUOTA_CHECK_DENIED_CACHE_MS (60 s), explicitly “so an upgrade, a bundle or an
operator reset is honoured within a minute”. So an org that buys a pack to
unblock a build may wait up to a minute. That is deliberate and bounded, not a
defect — but the purchase confirmation should say so, or it will be reported as
one.
F4 — two refusals, two messages. D5 adds a per-build ceiling, so there are now two distinct rejections: budget exhausted (buy a pack, or wait for reset) and this single build is too expensive (fix the build — a pack will not help). One shared 429 message sends people to buy capacity that cannot solve their problem.
F5 — a cost hint before the build, modelled on AIQuotaHint. The closest
precedent in the codebase, built for exactly this: “the one line every AI surface
shows BEFORE a generation: how many AI calls are left and when they reset, said
up front instead of discovered as a 429”. It already handles the not-charged
case, disables the action on the same predicate the caller uses, and fails open
when the quota cannot be read.
A build submission deserves it more than an AI call does: one build is ~15 credits where an AI call is one unit, and a base-image bump submits dozens at once. The monthly period (D7) changes what the hint should SAY rather than whether it is needed: not “15 of your 85 remaining today” — which a daily window would have made the honest phrasing — but “about 15 of your 2,900 remaining this month”, alongside the forecast that D7 leans on as its main safety mechanism. Highest-value frontend item here, and the one an earlier draft of this plan omitted entirely while calling the frontend work free.
A build-minutes section in docs/quotas.md: the unit and how it is derived, the
per-tier table, that failed builds are charged (and which retries are not),
the per-build ceiling, and what the pack buys.
build_minutes_charged_total summed per org
should track plugin_build_duration_seconds{org_id} × the vCPU figure. Drift
means the accounting is wrong, and this is the check that catches it.buildMinutes refusals outnumber
plugins refusals on any tier, that tier’s per-build allowance in the derived
table is too low — this quota is meant to catch pathological duration, not
normal use. Watch Developer first: it is the tier where the two are closest.| Phase | Size | Why |
|---|---|---|
| 1 | Small–Medium | A union member plus compiler-named tables; D7 adds a per-type period (a lookup threaded to two call sites) and moves periodDays into QuotaSummary, which touches the dashboard. D10’s utilisation figure is a guess until Phase 2 measures it |
| Frontend | Medium | F1/F2 are now catalog entries (the display catalog landed); the work is in-flight reservations, two refusal messages, and the pre-build cost hint |
| 2 | Medium | Two metering hooks, a reserve-and-true-up settle shared with the plugins gate, retry pricing, exemptions, ledger attribution |
| 3 | Small–Medium | One catalogue entry, two test-list additions, a Marketplace dimension, a maxQuantity, and the grant-granularity decision (which also sets the volume breaks) |
| 3b | Small | Extending plugin_pack / pipeline_pack grants, plus the ratio invariant test — but it needs the RATE_PACKS single-dimension conflict resolved first |
| 3d | Hours | One entry in checkEntitlementOvercap, plus the notice copy for what a refused downgrade means |
| 4 | Largest | A new AWS ingestion path, plus a rename migration |
trigger label on the build metric so the question can be answered from data
if anyone asks it again.QUOTA_RESET_DAYS for any other quota (D7). The daily
period is buildMinutes-only; every other counter keeps the global window.D9 FIRST, before anything is published. It is the only item that can invalidate the tier table rather than refine it: at the shipped 1.5 vCPU the top two rows are undeliverable, and no amount of Phase 1–3 work makes a cap the hardware cannot serve into a real product. Sizing the pool, or trimming the ladder to it, is the gate on the rest.
D10 decides what the numbers mean, and D7’s window decides whether they are reachable. Both are paper decisions — settle them with D9 rather than after it, because all three move the same table and re-deriving it three times is how a ladder ends up internally inconsistent.
Then 1 → 2 → 3 (with 3b), then 4 when pipeline compute matters enough.
One ordering note that only became visible once the window was monthly:
Phase 2 is what turns D10’s 30% utilisation guess into a measurement. Until
build_minutes_charged_total has run for a month, every number in the Phase 1
table rests on an assumption nobody has tested, so treat the first month after
Phase 2 as the point at which the ladder is re-derived — not as confirmation
that it was right.
Phase 1 alone is inert — a quota with no meter. Phase 2 makes it real. Phase 3 without Phase 2 would sell capacity against a limit nothing enforces, which is worse than not selling it.
Phase 4 is independent of 1–3 in mechanism but dependent in meaning: it is what
turns an honestly-scoped build quota into a complete compute quota. Its first
step (an org_id label on pipeline_executions_total) is worth landing early
regardless, because it is one label and it makes the gap measurable.