Untested Incident Response Plan? Ask for the After Action Report
An untested incident response plan produces no evidence. A rehearsed one produces an after action report, the document NIST defines as the findings and recommendations from an exercise. Ask your provider for the last one and the list of things it changed. A plan nobody ran cannot supply it.
- Only 25 percent of businesses hold a formal incident response plan, and the government survey that counted them found only a minority had tested theirs comprehensively.
- A test and an exercise are different things. A restore log proves the data comes back. It says nothing about who declares an incident.
- Every exercise produces an after action report. Ask for the last one and the list of items it changed, because an unrehearsed plan has neither.
- CISA publishes over 100 ready-made tabletop exercise packages, and the document set includes an After Action Report.
- Print the plan and the contact list. The shared folder holding them is often the thing that is down.
What is the paper plan problem?
The paper plan problem is a continuity or incident plan that exists as a document and has never been run. It is well formatted. It sits in a shared folder. Nobody has rehearsed it, so the first time anyone reads it under time pressure is during the incident.
The scale of it is measured, not guessed. The UK government's annual survey of organizations found formal incident response plans in only 25 percent of businesses and 19 percent of charities, falling to 21 percent of micro businesses against 76 percent of large ones. Only 39 percent assigned roles or responsibilities to individuals, which is the cheapest preparation there is. And for the smallest firms the trend runs backwards: small businesses holding business continuity plans that address cyber security fell to 44 percent, down from 53 percent in 2024/2025.
The sharper finding sits in the same report. Holding a plan and having run it are counted separately, and the gap between them is the subject of this article: only a minority of organisations reported comprehensive testing of incident response plans. The researchers then name the consequence in their own words, which is the sentence worth quoting to anyone who says the plan is fine. This false sense of preparedness, by possessing an incident response plan that lacks testing, could lead to organisational risk. One interviewed manufacturer put it more plainly still, saying its plan was not fully documented and has not been tested. This keeps me awake at night. That is the paper plan problem described by a government statistician rather than by a vendor with software to sell.
This guide covers the document and the rehearsal around it. For the drill itself, read Test Your MSP's Incident Response. For backup proof specifically, read When Was Your Backup Last Restored.
A test and an exercise are not the same thing
A test checks whether a system works. An exercise checks whether people know what to do. NIST separates the two in Special Publication 800-84, its guide to test, training and exercise programs, and that distinction decides what the evidence on your desk actually proves.
Start with the published wording. NIST defines a test as an evaluation tool that uses quantifiable metrics to validate the operability of a system or system component. It defines a tabletop exercise as a discussion-based exercise where personnel with roles and responsibilities in a particular IT plan meet in a classroom setting or in breakout groups to validate the content of the plan, with a facilitator initiating the discussion by presenting a scenario. A functional exercise goes further, letting personnel with operational responsibilities validate their IT plans and their operational readiness for emergencies in a simulated operational environment.
The rungs differ in cost and disruption. Ready.gov, the federal preparedness site, calls tabletop exercises cost-effective tools to validate plans and capabilities and notes that many run for a few hours. A full-scale exercise uses real equipment and real people on location, and the same page calls it a lengthy exercise. A 20-person business has no reason to climb past the tabletop.
Now apply the vocabulary to what your provider sends you. A monthly backup success email is monitoring. A quarterly restore log is a test, and a good one, because it proves your provider can restore the data. Neither is an exercise. Neither says who is allowed to declare an incident, who notifies your customers, or who calls the insurer at 3am. Providers answer the exercise question with test evidence because test evidence is the kind they already produce.
An exercise produces a document, and that document is your evidence
An exercise produces an after action report. NIST defines it as a document containing findings and recommendations from an exercise or a test. For a buyer that record is the whole point, because it exists only if the rehearsal actually happened.
Ready.gov sets out the sequence that produces it. Run the exercise, hold a facilitated hot wash at the end, collect written feedback on evaluation forms, then compile an after-action report that documents suggested improvements and distribute it to management, and address those improvements through the organization's corrective action program. The report is not paperwork. It is the mechanism that changes the plan.
That creates an asymmetry you can use on Tuesday. Ask a provider whether the plan has been tested and you get a yes, because the word test honestly covers a backup job. Ask for the after action report from the last exercise and the list of items it changed, and a plan nobody has run produces nothing. There is no way to supply the output of a meeting that never took place.
A second check costs one email. A rehearsed plan carries a revision history, because an exercise always finds something. An unrehearsed plan has one version, dated the day it was handed over. So ask for the plan's current revision date and the last change made to it. The after action report and the revision history settle it between them. Neither can be faked by a document nobody has opened.
What published guidance already asks for
Published guidance already sets an annual floor and names who has to be in the room. CISA's Cross-Sector Cybersecurity Performance Goals say it in one line. Goal 1.C states that IR plans should be reviewed and drilled, at a minimum, on an annual basis, and tells organizations to ensure drills are realistic and include all relevant stakeholders and to regularly exercise IR plans for threats common to their sector and locality.
One regulator has written the same thing into law, and it is worth reading even where it does not bind you. New York's financial regulator requires covered firms to test their incident response and business continuity and disaster recovery plans periodically, but at a minimum annually, and to do so with all staff and management critical to the response. The same rule lists what the plan must address, including preparation of root cause analysis that describes how and why the event occurred. Those obligations apply to New York financial institutions. For everyone else the rule is a free, well-drafted model written by a party with nothing to sell you.
NIST supplies the starting document at no charge. Special Publication 800-34 Revision 1, its contingency planning guide for federal systems, publishes contingency plan templates for low, moderate and high impact systems alongside a business impact analysis template. A template is a sound skeleton, and it is not a rehearsal.
Two intervals are enough to hold a provider to. Review and drill the plan once a year, and again after any change of provider, contract or key staff. If the first run finds nothing, the scenario was too easy.
The plan is stored in the folder that is down
Store the plan where the incident cannot reach it, because the usual hiding place is the first casualty. A continuity plan saved on the shared drive, or a contact list living in a mailbox, is unavailable in exactly the event it was written for.
Federal guidance is blunt about this. The StopRansomware Guide from CISA and MS-ISAC tells organizations to ensure a hard copy of the plan and an offline version is available, and during a response to use out-of-band communication methods such as phone calls so attackers do not learn they have been spotted. Its response contacts table expects 24x7 contact information and roles for IT and security staff, leadership, law enforcement, managed service providers and cyber insurers. That table is the single page you most want printed.
New York's rule anticipated the same failure. Current copies of the plans, or the relevant portions of them, must be distributed or otherwise accessible, including during a cybersecurity event. CISA goes one step further on recovery, expecting organizations to keep essential functions running in a degraded mode and naming paper-based operations or radio communications as examples.
You also have less time than the plan assumes. Sophos, working from 661 incident response cases, reports that once an attacker is inside an organization it takes them just 3.4 hours to get to the AD server, that 88 percent of ransomware payloads are deployed during non-business hours, and that median dwell time declined to three days. AD is the Active Directory server that holds your logins. The first read of the plan happens at 3am, on a phone, by whoever answered it.
Run the ten-minute version of this today, without involving your provider. Open the plan on your phone and assume Microsoft 365 is unavailable. Find the out-of-hours number for your provider, your insurer and your bank. If any of the three lives only inside the system you are pretending is down, you have just found your first finding.
The cheapest real exercise for a 20-person business
To run a real exercise in a 20-person business, book 90 minutes, put five people in a room, work one scenario against a clock, and write the after action report before anybody leaves. Nothing in production is touched, and nothing needs buying.
Five seats cover the whole decision chain.
- The decision maker who can authorize spending without asking anyone else.
- The internal owner of IT, even where that is the office manager.
- The provider's incident responder, not the account manager.
- The person who speaks to customers, because that call lands on day one.
- A facilitator who reads the scenario aloud, keeps the clock and takes notes.
Take the scenario rather than writing one. CISA publishes over 100 tabletop exercise packages, each one customizable and including template exercise objectives, scenarios, and discussion questions plus a collection of references. The document set covers role guidance for planners, facilitators and participants, an invitation template, a slide deck, a feedback form and an After Action Report. Pick a boring scenario. A finance mailbox sending fake invoices to suppliers overnight teaches more than a nation-state attack, because it is the one that turns up.
Close the session with the report rather than with a good feeling. Each finding needs a named owner and a date, and the plan gets a new version number the same week. Then schedule the next run. The report you produce is the document you hand your insurer, your auditor and the next provider you interview.
A provider who hands you the after action report is operating the plan. A provider who sends you the plan again is selling the document.
Six questions to send your provider on Tuesday
Send six questions in one email, then read the replies beside the answers you get from any provider you are considering.
- Send me the after action report from the last exercise of our plan.
- List the items that report changed, with an owner and a date against each.
- Tell me the plan's current revision date and the last change made to it.
- Name who from your team would sit in a 90-minute tabletop exercise this quarter.
- Show me where a hard copy and an offline copy of the plan are held.
- Confirm that the out-of-hours numbers in the plan were dialled in the last 12 months.
None of this makes an incident go well, and no honest provider claims otherwise. It moves the discovery of your gaps from the worst possible day to a Tuesday afternoon. Ask three providers the same six questions. The one who sends a dated report with an owner against each finding is the one doing the work.
FAQ
What is an after action report?
An after action report is the written output of an exercise. NIST defines it as a document containing findings and recommendations from an exercise or a test. It records what the rehearsal found, what should change, and who owns each change. CISA includes an After Action Report in the documentation set of its tabletop exercise packages.
How often should an incident response plan be tested?
At least once a year. CISA's performance goals state that IR plans should be reviewed and drilled, at a minimum, on an annual basis, and New York's financial regulator requires covered firms to test the plans periodically, but at a minimum annually with all staff and management critical to the response. Test again after any change of provider, contract or key staff.
Does a backup restore test count as rehearsing the plan?
No. A restore test proves the data comes back. NIST classes a test as an evaluation tool that uses quantifiable metrics to validate the operability of a system or system component, while an exercise validates whether people can run the plan. Both are worth having. Only the exercise produces an after action report.
What is the difference between a tabletop exercise and a functional exercise?
A tabletop exercise is discussion-based. NIST describes personnel meeting in a classroom setting or in breakout groups to validate the content of the plan while a facilitator presents a scenario. A functional exercise has personnel with operational responsibilities validate their operational readiness in a simulated operational environment. Start with the tabletop, which Ready.gov calls cost-effective.
Where should the incident response plan be stored?
Somewhere the incident cannot reach. CISA and MS-ISAC tell organizations to ensure a hard copy of the plan and an offline version is available and to use out-of-band communication methods such as phone calls during a response. New York's rule requires current copies to be distributed or otherwise accessible, including during a cybersecurity event. A plan in the folder that is down is not a plan.
Best IT MSP is an independent directory of managed IT and cybersecurity providers across the US and Canada. We rank on verified rating and firmographic data, we label paid placement, and we do not sell IT services. Ask three merit-ranked firms in your city for their last after action report and compare what comes back.