--- title: "Incidents" url: https://services.dev/docs/operate/incidents captured: [operate-incidents.txt] --- # Incidents URL: https://services.dev/docs/operate/incidents > Four controls for when something breaks, run first by the response your files name. Hold one step, reroute it to people, drain a Service of new orders, or stop the whole brand. Then find every job the cause touched, redo it at no charge, lift without sending a job back to the cause, and close the incident on its record, cost included. An incident answers one question: what stops, and what keeps going? A hold never moves a date the world sets, never touches pay a member has earned, and never charges a buyer for work that didn't happen. ## The four controls | Control | CLI | What stops | What keeps going | |---|---|---|---| | Hold a step | `hold .` | every job reaching the step waits there, with `job.held` | every other step, every ask and clock | | Reroute it to people | `hold . --to [--pay ]` | the AI at that step: each job posts a Task to the role instead | the job, which goes on once the person's act commits | | Drain a Service | `hold --drain` | new orders, which answer `not_live` with the waitlist | every job already ordered, to its end | | Stop the brand | `stop --live [--except ]` | every AI call, outside send, capture, new Task and new order | clocks and their notices, accrued pay and payouts, sends already out, every read, and each step `--except` names | A hold can narrow to the jobs it names, such as `--where jurisdiction=US-NJ`, so one bad leg stops while the rest file. ## Respond before you read Each alarm can run the control you wrote for it, in seconds, before anyone reads it. `respond` in `brand.ts` maps an alarm to its hold: ```ts title="brand.ts" respond: { 'quality.alarm': { hold: 'draft', to: 'attorney', pay: 120, expires: '8h' }, 'spend.alarm': { hold: 'draft' }, 'pool.alarm:no_eligible': { raise: 'sooner' }, }, ``` A response records `hold.placed { by: 'respond', alarm }` and lists under the console's **done for you**, with its lift. `check` refuses a response that reroutes a reserved act to a role without the act's credential. Responses run on [Team](https://services.dev/docs/money/plans.md#team). ## A model that changed overnight At 09:40 on a Monday, signers are sending back nearly a third of api.lawyer's morning trademark responses, all at the AI's draft. By the time the owner opens the console, the file has already rerouted the draft: ```bash open https://api.lawyer/console # 9 Nov, 09:40 ``` ```txt title="magic values: none" alarm quality.alarm · trademark-response · send-back budget burning 14x over the last hour and 5 minutes · signers sent back 9 of 31 responses since 06:00 (29%), against an objective of 5% every one at draft · since 06:12 decision.made names the family claude-x at a new build done 09:40:07 · respond in brand.ts held the draft and rerouted it to attorneys, $120.00 an act, for up to 8h · hld_2Lp4 · hold.placed { by: 'respond', alarm: 'quality.alarm' } controls lift the hold · change its pay or its hours · drain the Service · stop the brand ``` ```bash npx services.dev holds ``` ```txt title="magic values: none" api.lawyer · live · 1 hold hld_2Lp4 trademark-response.draft → attorney since 09:40 by respond 22 drafts posted, 22 claimed, p50 41m to claim expires 17:40 ``` The draft named only the family `claude-x`, and the provider moved its build at 06:12. A rollback wouldn't help, because both versions named the same family. With the draft [pinned to a build](https://services.dev/docs/operate/deploy.md#pin-the-model-to-a-build), a new build reaches buyers only through a deploy, and [a staged one](https://services.dev/docs/operate/deploy.md#live-in-stages) shows it in the first cohort. ## Which control for which cause | The cause | The control | |---|---| | your own bundle | `rollback`, or `jobs repin` the jobs in flight: [Versions](https://services.dev/docs/operate/versions.md#rolling-back) | | the model, under a family pin | reroute the step, then deploy a build pin | | an outside system | none: the submit step's [fallback](https://services.dev/docs/service/submit.md) runs with no hold | | the Pool | [Pool health](https://services.dev/docs/operate/pool-health.md)'s moves | | a key that may have leaked | `keys rotate --all --live` first; stop only if you can't tell what the key signed | ## What it already touched ```bash npx services.dev jobs affected --where decision.model_build=claude-x@2026-11-09 --since 06:12 --live ``` ```txt title="magic values: none" trademark-response · 31 matters drafted on claude-x@2026-11-09 since 06:12 sent back 9 by their signers at respond · each redrafts at no charge in flight 14 held at draft · drafted by attorneys under hld_2Lp4 awaiting 2 signed, filed and delivered · the client hasn't accepted accepted 6 signed, filed and accepted run it npx services.dev jobs send-back --affected "decision.model_build=claude-x@2026-11-09 status=delivered,accepted" --from draft ``` ```bash npx services.dev jobs send-back --affected "decision.model_build=claude-x@2026-11-09 status=delivered,accepted" --from draft --reason incident ``` ```txt title="magic values: none" ← 8 matters back at draft · by: 'incident' · each redrafts on the fixed build, then goes to its attorney again clients pay nothing more · each matter's timeline says "redone: we found a problem in our draft" attorneys each second act is paid again, $175.00 from your line: $1,400.00 · a supplemental response files where one is owed event job.sent_back { by: 'incident' } on each ``` ## Lift it without going back A deploy at 13:55 pins the draft to the build from the day before. Matters still pinned to the old version would draft on the moved build again, so the lift's dry run sorts them first: ```bash npx services.dev hold lift hld_2Lp4 --dry-run ``` ```txt title="magic values: none" hld_2Lp4 · trademark-response.draft → attorney · dry run, nothing changes release 4 matters on 2026-11-09.1, ordered since 13:55 · AI drafts each on claude-x@2026-11-08 kept 6 matters on 2026-11-02.1, not yet claimed · they stay with attorneys: 2026-11-02.1 names only the family finish 3 matters claimed by attorneys · each finishes on the terms it was claimed under run it npx services.dev hold lift hld_2Lp4 --repin-to 2026-11-09.1 ``` ```bash npx services.dev hold lift hld_2Lp4 --repin-to 2026-11-09.1 ``` ```txt title="magic values: none" ← lifted 14:20 · after 4h 40m · 22 drafts by attorneys, $2,640.00 · hold.lifted repinned 6 matters to 2026-11-09.1, at draft · job.repinned { by: 'hold.lift' } on each released 10 matters to AI on claude-x@2026-11-08 · 3 finish with their attorneys ``` ## After it The incident opens on its first alarm or hold and gathers every alarm, hold, deploy, repin, redo and refund until its last hold lifts. Its record is the postmortem's source: ```bash npx services.dev incidents get inc_4Lp1 ``` ```txt title="magic values: none" inc_4Lp1 · trademark-response · opened 09:40 by quality.alarm · closed 14:20 when hld_2Lp4 lifted · 4h 40m cause the draft named the family claude-x, and its build moved at 06:12 · a rollback wouldn't have helped: 2026-10-19.3 named the same family timeline 09:40 quality.alarm · 09:40 hld_2Lp4 by respond · 13:41 evl_3Lq7, no lens finds fewer · 13:55 2026-11-09.1 live, draft pinned to claude-x@2026-11-08 · 14:20 lifted, 6 repinned touched 31 matters · 9 sent back by signers · 8 sent back by: 'incident' · 14 drafted by attorneys · 0 dates missed cost $2,640.00 reroute · $1,400.00 second acts · AI $5.68 on 8 redrafts · $0.00 to clients · $4,045.68 in all status no buyer-facing delay: every matter kept its stated time · nothing posted on api.lawyer/status report npx services.dev incidents report inc_4Lp1 writes the postmortem from this record ``` Every brand's Site serves `/status`. A drain, a stop, or a hold that pushes a job past its stated time shows there and on the job's timeline, in the buyer's words, never naming the model or the cause. ## Drain, or stop everything Draining keeps the Service visible and takes no new orders, the way a Service with `status: 'waitlist'` does: ```bash npx services.dev hold trademark-response --drain --reason "the model's new build" ``` ```txt title="magic values: none" ← hld_2Lp5 · placed 09:46 · trademark-response drains · live orders answer not_live with the waitlist, as a Service with status: 'waitlist' does · no buyer is charged in flight 38 matters run to their ends on the version and the holds they have status api.lawyer/status: "New trademark responses are paused. Matters already ordered keep their dates." lifted each waitlisted buyer gets the link to order ``` The stop is for the day something is wrong everywhere. It prints what it costs if it stands, and `--except` keeps a dated step running: ```bash npx services.dev stop --live --except respond --reason "every AI step answering out of shape" ``` ```txt title="magic values: none" ← api.lawyer · stopped 16 Nov, 10:02 · live stops every AI call, outside send, capture, new review and new order · 214 matters wait where they are keeps respond, as --except asks: 9 office dates inside 30 days still reach their attorneys · clocks and their notices · accrued pay and payouts on schedule · sends already out · every read expires 11 card authorizations lapse inside 7 days if the stop stands · each renews on start, or the buyer is asked again asks a party's link says the matter is paused, and keeps what they already did status api.lawyer/status: "We've paused new work. Every date you were given still holds." start npx services.dev start --live ``` ## Who can act Your files first, then your coding agent within the ceilings [you set](https://services.dev/docs/operate.md#the-file-first-the-agent-second-you-last), then a staff role your [custom roles](https://services.dev/docs/people/staff.md) allow to hold, then you. Every hold, lift, stop and start is on the [audit log](https://services.dev/docs/operate/observe.md#the-audit-log) with who placed it and the reason. Every control runs on every plan. ## Controls | Call (CLI) | API | Event | On the console | Plan | |---|---|---|---|---| | `hold . [--where ]` | `POST /owner/holds { at, where }` | `hold.placed`, `job.held` | `/console/holds` | every plan | | `hold . --to [--pay ]` | `POST /owner/holds { at, to, pay }` | `hold.placed` | `/console/holds` | every plan | | `hold --drain` | `POST /owner/holds { at, drain: true }` | `hold.placed` | `/console/holds` | every plan | | `holds` | `GET /owner/holds` | none: it reads | `/console/holds` | every plan | | `jobs affected --where --since