- › An investigation can only reach as far back as your oldest surviving log, and that boundary was set by a storage budget.
- › When the trail ends at your retention edge, you cannot prove the intrusion started there, only that your visibility did.
- › That distinction decides what you tell regulators, customers, and an insurer, so it is a disclosure problem and not only a technical one.
- › Retention is not uniform, and the shortest window in the chain is the one that caps the whole investigation.
- › Pick the handful of log sources that establish scope and fund those for a year, instead of trying to keep everything.
Picture the moment an investigation stops. You have been walking an intrusion backward, and it has been going well. The lateral movement is mapped, the credential use is clear, and you have traced the activity from the finance server back to a workstation, and from that workstation back to a VPN session in early June.
So you pull the VPN logs for May, because the June session did not come from nowhere. The console returns nothing. Not an empty result meaning no activity, but nothing at all, because that data aged out five weeks ago and the retention was 90 days.
The investigation is over. Not finished, because you have not answered the question, but over, since there is nowhere left to look. Everything you will ever know about this intrusion is now fixed, and it was fixed by a storage decision somebody made in a budget meeting two years ago that nobody in the room called a security decision.
What the Retention Edge Actually Costs You
The technical loss is obvious enough. What tends to get missed is that the edge of your retention also changes what you are able to honestly say afterward.
When the trail runs out at day 90, you have two candidate explanations and no way to choose between them. Either the intrusion genuinely began around that date, or it began earlier and you cannot see it. Those produce very different disclosures, and the honest answer is that you do not know.
Now consider who asks. A regulator wants to know when unauthorized access began, because the clock attaches to the acquisition rather than to your discovery. A customer wants to know whether their data was in scope during the period they care about. An insurer wants the same facts, and an attorney on the other side will ask what you did to find out.
Saying “our evidence begins on June 3” invites the obvious follow-up about what happened before June 3, and “we do not retain that” is a complete answer that satisfies nobody. It also reads, fairly or not, as a choice you made.
The organizations that handle this well are the ones that decided in advance how far back they wanted to be able to see, and funded that. The ones that handle it badly discover their answer during the worst week of the year.
Why the Number Is Almost Always 30 or 90
Nobody sat down and reasoned their way to 90 days. The number arrives from somewhere else entirely.
It is frequently the default in whatever platform you bought, or the tier your license covers before ingestion costs step up, or the figure a compliance framework names as a floor that then became the ceiling. PCI DSS asks for a year with 90 days immediately available, and a great many organizations read that as 90 days and stopped reading.
The economics push the same direction. Log volume grows with every new endpoint, service, and cloud account, and the bill grows with it, so the retention conversation happens annually as a cost conversation. Security is in that meeting arguing for a longer window against a specific number on a specific line, and the counter-argument is always that nobody has needed 180 days yet.
I want to be fair to the finance side of this, because the costs are real and they compound. Keeping everything for a year is genuinely expensive and mostly wasteful, since the overwhelming majority of what you ingest will never be read by anybody. The mistake is treating retention as one dial for all data instead of a question asked separately of each source.
The Shortest Window Is the Real One
Here is the part that surprises teams who believe they have this handled. Your real retention is the shortest window anywhere in the chain that matters for a given question, whatever headline figure your SIEM carries.
Walk a typical stack through and the picture changes quickly. Your SIEM holds 12 months, which sounds comfortable. The endpoint agent only forwards a subset of events and keeps 30 days locally. The identity provider retains sign-in logs for 30 days on your license tier. Your cloud provider’s data-plane logging was never enabled, because it is off by default and costs extra. The VPN appliance writes to local disk and rotates when the disk fills, which happens every few weeks.
You have 12 months of some things and 30 days of the things that establish who logged in from where. Investigations run on the second category, because authentication events are what let you distinguish legitimate use from an attacker holding valid credentials, which is the whole problem with the living off the land technique set.
So the useful exercise is to pick one question and ask how far back you could answer it, which measures something quite different from how long you keep logs.
The Question to Test Against
Here is the test I use, and it is close to what you will actually be asked in the first hour. Write it down, then trace what answering it would require of your data.
A named account was used to access a sensitive system yesterday. Establish every authentication that account performed, every host it touched, and every resource it read, going back six months.
Answering that needs four things at minimum: identity provider sign-in logs for the authentication trail, endpoint telemetry for process and network activity on each host, cloud audit logs for resource access, and network flow data to catch the movement your endpoint agent missed because it was never installed on that server.
Take each of those four and write down the real retention, not the policy figure. Go and look, because the gap between what the policy says and what the platform is actually holding is a finding on its own, and it usually comes from a tier change or a silent ingestion cap nobody was alerted to.
Whatever the smallest number is, that is how far back you can see. It is one number, it is knowable this afternoon, and almost nobody has it written down.
Fixing It Without Buying a Year of Everything
The instinct after a scare like this is to extend retention across the board, and that is the version finance rejects, correctly. The useful move is narrower and considerably cheaper.
Sort your sources by whether they establish scope. A small number of log types answer the questions that matter during an incident, which are who authenticated, from where, and what did they touch. Identity provider logs, cloud control-plane audit logs, endpoint process and network telemetry, and VPN or remote access records do most of that work. Debug logs from an internal application almost never do. Fund the first group for a year and let the rest age out on whatever the default is.
Use cheap storage for the cold copy. The expensive thing is usually indexed, searchable retention rather than storage. Writing the raw events to object storage costs a fraction of keeping them hot, and an investigation can tolerate the day it takes to rehydrate six month old data. Slow answers beat no answers by a wide margin.
Turn on the logging that exists and is not enabled. Cloud data-plane logging, identity provider audit categories above the default tier, and command-line process auditing on endpoints are all commonly off. Each one costs money to ingest and nothing to switch on, and the absence of any of them shows up as a hole at exactly the wrong moment.
Alert when a source goes quiet. A log source that stops arriving is a gap that will be invisible until somebody goes looking for that month. Monitoring for the absence of expected telemetry is unglamorous and it catches both the misconfiguration and the attacker who turned it off.
Write the retention figure into your incident response plan. Put the actual per-source numbers in the plan so the team knows the boundary before the incident, rather than discovering it at two in the morning. It also turns an abstract budget argument into a specific sentence a board member can read.
The Argument That Works
Technical arguments for retention lose to budget arguments, reliably, because the technical case is about a capability and the budget case is about a number. You will not win that exchange by describing what a SIEM does.
The argument that works reframes it as the answer you will be able to give. Tell leadership that today, if asked when an intrusion began, the organization can answer for the last 30 days and cannot answer at all beyond that. Then ask whether 30 days is the window they want to be able to speak to in front of a regulator, a major customer, or a court.
That question gets a different answer from the one about storage costs, and it is the same decision. The difference is that this version is being made deliberately, by the people who will have to live with it, before anything has happened.
Chris Armour is Director of Information Security at Grab The Axe.
Operating on the philosophy that 'you can't build a secure system if you don't know how to break it,' Chris leads our engineering division. A top 1% National Cyber League competitor, he hardens our digital infrastructure against the very exploits he has mastered.
View Author Page →
