We usually describe features by what they do when everything goes right.
Upload a file.
Publish an article.
Run a report.
Deploy a release.
Send a payment.
Import records.
Schedule a job.
The description ends at success.
That is understandable. Success is the reason the feature exists.
But production systems spend part of their lives outside the happy path.
Connections disappear.
Processes restart.
Users close browsers.
Workers crash.
Credentials expire.
Dependencies respond slowly.
Two people act at the same time.
A job completes halfway.
A request succeeds but its acknowledgement is lost.
The feature still exists in those moments.
The question is whether it remains understandable.
This is why I increasingly think of recovery as part of the feature itself.
Not a support procedure added later.
Not an emergency document.
Not the responsibility of whoever happens to be awake when something breaks.
If a feature can fail, then the ability to recover from that failure belongs in its design.
Success Is Only One Transition
Yesterday’s article focused on making state visible before automating it.
Once state is visible, another question immediately appears:
What happens when the transition does not complete normally?
Suppose an article is:
Approved
The scheduler begins publishing it.
The publishing service creates the public version.
Before the scheduler receives confirmation, its process restarts.
What state should the article be in now?
If the workflow understands only:
Approved
and:
Published
it has a problem.
The article may already be public.
The scheduler may still think publication has not happened.
Retrying blindly may create duplicate effects.
Marking it failed may be false.
Doing nothing may leave the workflow permanently stuck.
Recovery begins with admitting that execution can stop between meaningful states.
Failures Should Leave Evidence
A weak failure often looks like this:
Something went wrong.
That sentence may be technically true.
Operationally, it is nearly empty.
What went wrong?
Before or after the authoritative change?
Was anything committed?
Can the operation be retried?
Does the user need to do something?
Does an operator need to intervene?
Can the system verify the current state?
A recoverable system preserves enough evidence to answer those questions.
For example:
Operation ID: PUB-98142
Previous state: Approved
Attempt started: 09:00:03
Delivery request sent: yes
Acknowledgement received: no
Authoritative delivery state: unknown
Recovery action: verify publication by operation ID before retrying
Now the failure is not a fog.
It is a condition.
And conditions can be designed for.
Recovery Begins Before Failure
This is the unintuitive part.
Recovery is mostly designed while the system is healthy.
Once a failure has happened, your options depend on decisions already made.
Did the operation have a unique identity?
Was the previous revision preserved?
Are writes atomic?
Can the process be restarted?
Was temporary data retained?
Are logs correlated?
Is there a rollback point?
Can the system inspect authoritative state?
Were side effects separated?
If the answers are no, recovery may require manual archaeology.
We search logs.
Compare databases.
Ask users what they saw.
Inspect timestamps.
Guess what happened.
Sometimes we get lucky.
That is not a recovery strategy.
Preserve the Last Known Good State
A reliable system should know what was good before it attempted something risky.
This is obvious in deployment.
Before replacing version 17 with version 18, preserve enough information to return to version 17.
But the same principle applies in smaller workflows.
Before overwriting a document, preserve the previous revision.
Before migrating a schema, know the current version.
Before applying a configuration change, preserve the prior configuration.
Before replacing a generated artifact, retain its predecessor long enough to validate the new one.
Recovery becomes dramatically easier when the system can answer:
What was the last known good state?
Without that, rollback becomes reconstruction.
Those are very different activities.
Rollback Is Not Always the Right Recovery
Engineers often treat rollback as the universal solution.
Something failed.
Go back.
Sometimes that is exactly right.
But not every operation is reversible.
Suppose a system has already sent an email.
You cannot unsend it.
Suppose an external payment has settled.
Rolling back your local database does not reverse the payment.
Suppose content was publicly visible for ten minutes.
Returning to the previous version does not erase the fact that people may already have seen it.
Suppose data was exported to another organization.
The information has crossed a boundary.
Recovery may require compensation rather than rollback.
This distinction is important.
Reversal and Compensation Are Different
A reversible operation can often return directly to its earlier state.
For example:
Version 18 fails health checks.
Redeploy version 17.
That is rollback.
A non-reversible side effect needs a compensating action.
For example:
A duplicate invoice was issued.
Create a cancellation or correction.
A notification was sent with incorrect information.
Send a correction.
A payment reservation was created.
Release it through the external provider’s supported process.
The goal is no longer to pretend the original event never happened.
It is to restore the system to an acceptable state using another valid operation.
This is a more honest model of recovery.
Partial Success Is a Real Outcome
Many complex operations contain multiple steps.
Imagine:
- Save article revision.
- Mark approved.
- Generate public representation.
- Publish to blog.
- Update search index.
- Send notification.
Now imagine step 5 fails.
Did publication fail?
Not entirely.
The article may already be live.
Calling the whole process Failed hides important information.
Calling it Succeeded is also misleading.
The system may need to represent:
Primary outcome: Published
Secondary outcome: Search indexing failed
Recovery: Retry indexing
This separation is powerful.
It prevents one recoverable secondary failure from forcing the entire workflow backward.
It also tells operators what actually needs attention.
Separate Critical and Non-Critical Side Effects
This leads to a useful design question:
Which parts of an operation define success?
Suppose publishing an article must:
make the content public,
update an index,
send analytics,
notify subscribers.
Does failure of analytics mean publication should fail?
Probably not.
Does failure of content persistence mean publication should continue?
Probably not.
The difference should be explicit.
A strong workflow distinguishes:
critical steps,
recoverable secondary steps,
optional side effects.
This keeps recovery proportionate.
Without that distinction, systems either fail too easily or pretend secondary failures do not matter.
Recovery Should Be Scoped
A good recovery path should repair only what is broken.
If search indexing failed, retry indexing.
Do not republish the article.
If thumbnail generation failed, regenerate thumbnails.
Do not recreate the entire media object.
If one imported row failed validation, perhaps isolate that row rather than rerunning the whole import.
This is one reason modular workflows are easier to operate.
Smaller states create smaller recovery actions.
A giant transaction may look elegant until something fails near the end.
Then the only recovery option is repeating everything.
Checkpoints Reduce the Cost of Interruption
Long-running operations benefit from checkpoints.
Suppose an import processes 100,000 records.
If it fails at record 82,000, should it restart from the beginning?
Perhaps.
But if processing is expensive and safe checkpoints are possible, the system can preserve progress.
For example:
Batch 1: complete
Batch 2: complete
Batch 3: complete
Batch 4: partial
Now recovery can begin from the last confirmed checkpoint.
This idea appears in many systems:
file uploads,
data imports,
backups,
migrations,
video processing,
large reports,
synchronization jobs.
The goal is not always maximum speed.
It is reducing the amount of successful work that must be repeated after interruption.
A Checkpoint Must Be Trustworthy
Not every progress marker is a real checkpoint.
Suppose a job writes:
Processed 80%.
Then crashes.
Can it safely resume from 80%?
Only if the system knows exactly what that percentage means.
A trustworthy checkpoint should represent a state that is complete and durable.
For example:
Records 1–80,000 committed.
That is useful.
Or:
Chunk 17 uploaded and verified.
Useful.
Or:
Schema migration step 4 committed successfully.
Useful.
The checkpoint should correspond to something the system can rely on after restart.
Otherwise it is merely progress display.
Recovery Needs Idempotency
Recovery frequently involves retry.
Retry becomes much safer when operations are idempotent.
If the same logical request runs twice, the final result should not accidentally double.
This matters for:
publishing,
payments,
imports,
notifications,
job creation,
synchronization,
and API calls.
A good recovery path often depends on a stable operation identity.
The system can ask:
Has operation X already completed?
If yes, return the existing result.
If no, continue.
This turns retry from a gamble into a controlled action.
Retry Should Have a Policy
Retrying forever is not resilience.
It is denial.
Some failures are transient.
A network timeout may disappear.
A dependency may recover.
A lock may be released.
Other failures are persistent.
Invalid credentials.
Malformed data.
Permission errors.
Schema incompatibility.
No amount of retry changes those conditions.
A good system distinguishes retryable from non-retryable failures.
For example:
Retryable: connection timeout
Retryable with delay: service temporarily unavailable
Not retryable: validation error
Not retryable: authorization denied
Manual review: authoritative state uncertain
The recovery path becomes much clearer when retry policy is part of the design.
Backoff Protects the System
When retry is appropriate, timing matters.
A failing dependency does not need 10,000 clients hammering it every second.
A simple backoff strategy can reduce pressure.
Retry after:
1 minute,
then 5,
then 15,
then 30.
The exact policy varies.
The principle is that recovery should not make the incident worse.
This is particularly important in automated systems.
Humans naturally pause when repeated attempts fail.
Machines will enthusiastically repeat a bad idea forever unless told otherwise.
Give Up Gracefully
A retry policy also needs an end.
After a certain number of attempts, or after a certain amount of time, the system should move into a state that requires different handling.
For example:
Manual Review Required
or:
Failed Permanently
or:
Waiting for Configuration
This is not defeat.
It is clarity.
The system has learned that automatic recovery is no longer appropriate.
The next action belongs to a person or another process.
The worst state is often not failure.
It is endless invisible retry.
Recovery Requires Ownership
When automation reaches the end of what it can safely do, someone must own the exception.
Who receives the alert?
Who can inspect the state?
Who has permission to repair it?
Who decides whether to retry, compensate, or abandon?
A system that generates an error but assigns no responsibility has not finished designing recovery.
This connects directly to the earlier article on handoffs.
Recovery is a handoff from normal automation to exceptional handling.
The same rules apply.
Preserve context.
Name the owner.
Define the next action.
Carry evidence.
Error Messages Should Support Recovery
User-facing error messages often describe the problem but not the next safe action.
Compare:
Upload failed.
with:
Upload stopped after 73%. Your completed chunks were preserved. Resume when the connection returns.
The second message changes the user’s mental state.
They know their work is not lost.
They know what happened.
They know what to do.
Or compare:
Publication error.
with:
We could not confirm publication. Do not publish again yet. We are checking whether the public revision was created.
Again, the message supports recovery.
A useful error message answers at least one question:
What should I do now?
Do Not Blame the User for System Uncertainty
Some error messages turn technical uncertainty into user responsibility.
Something went wrong. Please try again.
Sometimes that is harmless.
Sometimes it asks the user to make a dangerous decision without enough information.
If retrying may duplicate an action, the system should not casually outsource that risk.
If the system can verify state itself, it should.
If manual action is required, explain why.
The interface should guide the user toward the safest recovery path, not merely acknowledge discomfort.
Preserve User Work Aggressively
There are few more frustrating experiences than:
You spend twenty minutes editing.
Save fails.
The page reloads.
Everything disappears.
The technical failure may have lasted one second.
The human loss is twenty minutes.
Good recovery design protects the value the user has already created.
Local drafts.
Autosave.
Temporary storage.
Unsaved-change warnings.
Revision history.
Resumable upload.
These are not luxury features.
They are ways of respecting the asymmetry between machine time and human time.
A server can rerun a request cheaply.
A person cannot always recreate lost thought cheaply.
Recovery Should Be Testable
Teams often test success carefully.
Failure paths receive less attention.
Yet recovery logic is exactly the kind of logic you do not want to discover during a real incident.
Test:
What happens if the database connection disappears after validation?
What happens if the response is lost after commit?
What happens if the worker restarts mid-job?
What happens if the second of five steps fails?
What happens if the same request is submitted twice?
What happens if the previous revision changed?
What happens if rollback itself fails?
These tests reveal whether recovery exists in reality or only in documentation.
Chaos Is Sometimes Useful, but Start Smaller
There is an entire discipline around deliberately injecting failures into systems.
That can be valuable.
But resilience does not require dramatic chaos experiments on day one.
Start with controlled questions.
Stop a worker halfway through a job.
Disconnect a test dependency.
Force a timeout.
Submit the same request twice.
Use an invalid credential.
Simulate a stale revision.
Restart the application during background execution.
Observe.
Can the system explain what happened?
Can it recover?
Can an operator tell what to do?
Small failure exercises often reveal enormous weaknesses.
Runbooks Are Interfaces for Humans
Not every recovery can be fully automated.
For operational incidents, a runbook can act as a human-facing recovery interface.
A useful runbook should not be a long narrative.
It should answer:
What condition are we in?
How do I verify it?
What actions are safe?
What actions are dangerous?
How do I know recovery succeeded?
When should I escalate?
For example:
Condition: Worker restarted during publication.
Verify: Query publication by operation ID.
If revision exists: Mark scheduler job completed.
If revision does not exist: Retry with the same operation ID.
If state cannot be determined: Escalate. Do not generate a new operation ID.
That is actionable.
The runbook preserves reasoning under pressure.
Recovery Documentation Should Match the Real System
Outdated runbooks are dangerous.
If documentation says:
Restart service A
but service A no longer owns the operation, recovery becomes guesswork.
This is another reason to keep recovery close to architecture.
When the workflow changes, recovery behavior should change with it.
Tests can help.
Automation can help.
Embedded operational guidance can help.
The goal is to reduce the gap between how the system actually behaves and how people believe it behaves.
Roll Forward Can Be Better Than Roll Back
Sometimes the safest recovery is fixing forward.
Suppose a database migration adds a column and application version 18 depends on it.
Deployment fails after the schema change.
Rolling the schema backward may risk data loss.
Deploying a corrected application version may be safer.
This is roll-forward recovery.
It is common in systems where:
data changes are irreversible,
external side effects have occurred,
or older versions cannot safely interpret the new state.
Good recovery design does not worship rollback.
It chooses the safest path back to a valid system.
Sometimes that path goes backward.
Sometimes forward.
Recovery Time Is a Design Metric
Teams measure uptime.
Latency.
Error rate.
Throughput.
They should also care about:
How long does recovery take?
Two systems may fail equally often.
System A recovers automatically in thirty seconds.
System B requires a specialist, three terminal sessions, and ninety minutes of investigation.
Their operational reliability is not equivalent.
Mean time to recovery is not merely an operations statistic.
It reflects architecture.
Observable state.
Clear ownership.
Good checkpoints.
Reversible changes.
Useful logs.
Safe automation.
These design choices shape recovery time long before the incident.
Recovery Confidence Matters Too
Speed is not everything.
A fast recovery that leaves everyone uncertain is fragile.
Did we repair the actual problem?
Did we duplicate anything?
Did we lose data?
Can we trust the current state?
A strong recovery ends with verification.
Not:
The command ran successfully.
But:
The system is healthy again, authoritative state matches expectation, and no duplicate effects were created.
The recovery itself needs an acceptance condition.
Otherwise we may simply move from one uncertain state into another.
Build Verification Into the End
Every important recovery action should have a closing check.
After rollback:
Is the previous version healthy?
After retry:
Did the operation complete exactly once?
After resuming:
Were all checkpoints processed?
After compensation:
Does the resulting state satisfy the business rule?
After restoring data:
Do row counts, revisions, or checksums match expectations?
Recovery without verification is an assumption.
We should know when we are finished.
Recovery Paths Reveal Architecture Quality
A system can look beautifully designed on the happy path.
The real architecture often becomes visible when something goes wrong.
Can one component fail without corrupting another?
Can work resume?
Can side effects be identified?
Can ownership move?
Can we inspect the current state?
Can we undo or compensate?
Can the system distinguish what it knows from what it does not know?
Failures expose hidden coupling.
Recovery exposes whether those couplings were designed consciously.
Simple Systems Recover More Easily
This connects to the earlier article on small improvements that can survive.
Every extra dependency adds another place recovery may need to reason about.
Every duplicate source of truth adds reconciliation.
Every hidden exception adds another branch.
Every unclear owner adds delay.
Every irreversible side effect adds complexity.
This is one of the strongest arguments for architectural simplicity.
Simple systems do not merely cost less to build.
They often cost less to recover.
Personal Work Needs Recovery Too
The same idea applies outside software.
A thesis draft is corrupted.
Can you recover yesterday’s revision?
A computer fails.
Are important files somewhere else?
Internet access disappears.
Can you continue offline?
A work session is interrupted.
Can you resume without reconstructing your thinking?
A bicycle breaks.
Is there an alternative transport path?
A schedule collapses.
Do you know which commitment matters most?
Personal resilience is also about preserving state and having recovery paths.
The exact mechanisms differ.
The principle does not.
Redundancy Is Not the Same as Recovery
Having two copies of something feels safe.
Sometimes it is.
But redundancy without a recovery process can create confusion.
Two databases.
Which is authoritative?
Two backups.
Can either actually be restored?
Two servers.
Does traffic fail over correctly?
Two copies of a document.
Which is current?
Recovery needs more than duplicates.
It needs a known procedure for returning to a trusted state.
A backup you have never restored is a possibility, not yet a proven recovery mechanism.
Recovery Should Be Practiced
Fire drills exist because the first time you discover an emergency procedure should not be during the emergency.
Systems deserve similar thinking.
Occasionally test:
Can we restore the backup?
Can we deploy the previous release?
Can we resume the interrupted job?
Can we rebuild the search index?
Can we rotate a credential?
Can another team member follow the runbook?
Practice turns theoretical recovery into operational confidence.
It also reveals stale assumptions.
Design for the 2 A.M. Version of Yourself
There is a simple test I like.
Imagine encountering this failure while tired, under pressure, and without the person who built the feature.
Would the system help you understand what happened?
Would the state be visible?
Would the operation have an identity?
Would the logs connect?
Would the safe next step be obvious?
Could you make things worse accidentally?
That imaginary operator is a useful design persona.
Because real incidents rarely arrive when everyone is rested, fully staffed, and holding perfect context.
Good recovery design reduces the amount of brilliance required during bad moments.
Recovery Is a Form of Respect
There is also a human dimension.
A recoverable system respects:
the user’s time,
the operator’s attention,
the developer who comes later,
the organization’s continuity.
It does not assume perfect connectivity.
It does not assume nobody makes mistakes.
It does not assume the original author will always be available.
It accepts that interruptions happen.
That is not pessimistic engineering.
It is realistic engineering.
From Visible State to Recoverable State
Yesterday’s article argued:
Make state visible before you automate it.
Today adds the next requirement:
Make state recoverable before you depend on the automation.
Visibility tells us where the work is.
Recovery tells us how to move from a bad or uncertain state back to a valid one.
Together, they create systems that are easier to trust.
A visible failure with no recovery path is still painful.
A recovery path with no visible state is dangerous.
We need both.
The Feature Is Not Finished at Success
When designing a feature, ask the happy-path questions.
What should happen?
What should the user see?
How fast should it be?
What data should be created?
Then keep going.
What can fail?
What state exists if it fails halfway?
What must be preserved?
Can we retry?
Can we resume?
Can we roll back?
Do we need compensation?
Who owns recovery?
How do we verify that recovery worked?
These questions do not make software gloomy.
They make it dependable.
The happy path tells us what the feature can do.
The recovery path tells us whether we can trust it enough to use seriously.
And that is the deeper test.
A feature is not complete when it succeeds once.
It becomes mature when failure does not erase our understanding of what happened next.
Recovery is not the appendix.
Recovery is part of the feature.