The $500,000 line item nobody talks about
Look at a typical Splunk deployment for a mid-sized security team. Five hundred gigabytes of daily ingest, roughly $600,000 a year in license costs, $1.1M to $1.3M in fully loaded TCO once infrastructure and staffing are counted in. Somewhere between 60 and 70 percent of that volume is firewall allow traffic. If those ratios are accurate for your environment, you are spending roughly $400,000 to $500,000 a year to index data that nobody searches, nobody alerts on, and nobody asks an auditor to produce except in narrow circumstances.
This is not an exotic finding. Splunk Community threads, Reddit's r/splunk, vendor whitepapers, and consulting engagements have all reported the same distribution for years. Firewall allow logs are the largest single category of ingest in almost every Splunk deployment we have audited. They are also the least interrogated.
This post walks through why that is, what the detection and compliance rationale actually supports, and what a routing layer changes once you accept that most of what your SIEM receives is not for your SIEM.
What a firewall allow log actually is
Every packet that traverses a firewall produces a record. A Palo Alto session generates a pan_traffic log. A FortiGate produces an fgt_traffic entry in key-value format. A Cisco ASA emits a %ASA-6-302013 built-connection message. The allow vs deny distinction in the log is literal: the action field is set to allow, accept, permit, or the vendor-specific equivalent.
An allow log says a session was permitted and completed. It contains the five-tuple, bytes transferred, user if identity is enabled, and protocol metadata. If the session was unremarkable, that is where the record ends. If it was remarkable, the same session produces additional logs: a threat log, an intrusion event, a user-ID correlation, a URL filtering decision. Those additional logs are the ones your detections actually fire on.
The allow record itself is a session manifest. It proves a connection happened. It does not imply the connection was interesting.
Why allows dominate the ingest volume
A corporate perimeter sees an enormous amount of routine traffic every second. DNS queries. Software update checks. Heartbeats to SaaS platforms. Cloud storage syncs. Each of those is a permitted session and each generates an allow log. The ratio of permitted to denied traffic in a typical enterprise sits somewhere around 90 to 10, and often higher. Firewall policy is explicit by design: you permit what you need, you deny everything else, and the default deny rule barely fires because traffic that would hit it is usually blocked earlier on the rule chain.
When the firewall forwards logs to Splunk, both categories of session show up in the same stream. Allow and deny mix together. By the time they hit the indexer, they are being charged at the same per-GB rate. The volume split is structural: DNS alone can generate hundreds of thousands of allow sessions per hour on a busy network, each one producing a log entry Splunk will charge you to store and parse.
The detection rules that do not fire on them
Open your Splunk Enterprise Security correlation searches or your custom alerts and look at the filter logic. The vast majority of firewall-based detections fall into one of two categories.
The first is pattern matching on deny or threat actions. "Alert when a host has more than N denied outbound connections in five minutes" is an exfiltration heuristic. "Alert when threat engine severity is high" is an intrusion detection signature. "Alert on IPS signatures for specific CVE ranges" is a known-bad matcher. All of these filter for action != "allow" or for enriched threat metadata that lives on a separate log type.
The second is behavioral anomaly detection on aggregate session counts: spikes in outbound data volume, new destination IPs, unusual port usage. These could theoretically use allow logs as input, but most implementations derive them from NetFlow, VPC flow logs, or endpoint telemetry rather than firewall syslog. The cardinality of firewall allows makes them expensive to aggregate inside Splunk at analytics-tier pricing when the same summary is available from lower-cost sources.
The practical outcome is that allow logs sit in the indexer, get parsed, get written to a tsidx file, get replicated across the cluster per your replication factor and search factor, and are never touched by a saved search.
The compliance archive question
Before getting to the fix, the honest part. There are two legitimate reasons allow logs stay in the SIEM even when nothing fires on them.
The first is incident response. When an analyst is investigating a known compromise, the question "what did this host talk to during the window in question" is sometimes answered by allow logs. A browser fetch to a C2 server looks like an allow until threat intel enriches it. A lateral movement attempt to an internal system that did not trigger host-based detection shows up as an allow from the firewall's perspective. For a narrow class of investigations, allow logs are the corroborating record.
The second is compliance retention. PCI-DSS, HIPAA audit controls, SOC 2 CC7, and most sector-specific frameworks require logs of "security-relevant events" for a defined retention period, typically one year online and seven years archived. Firewall logs fall inside that scope at most organizations.
Neither of these reasons requires the logs to live in your SIEM's analytics tier. They require the logs to be retrievable and queryable within an acceptable window. An S3 bucket with lifecycle policies and a search layer on top meets both requirements at a fraction of the cost of Splunk indexing.
What the Splunk bill looks like
Splunk's pricing at 500 GB per day lands around $600,000 a year in license fees, before infrastructure and staffing. The renewal uplift compounds at 9 percent annually on the per-GB term, which Splunk itself documents in its Pricing FAQ. A $600K deployment signs a renewal at $654K, the next at $713K, the third at $777K, and so on, assuming volume stays flat. Volume never stays flat. Log volumes grow around 50 percent year over year across the industry.
Add a dedicated Splunk administrator, typically $120K to $170K in salary, who spends a large share of their time on props.conf and transforms.conf tuning, Technology Add-on updates, and forwarder management for exactly the data types we are discussing. A team running 20 MSP clients might have two of these administrators full-time.
Now apply the 60 to 70 percent ratio. At 500 GB per day, firewall allow logs account for 300 to 350 GB daily. That is $360,000 to $420,000 a year in license alone, not counting the infrastructure they occupy, the storage they replicate across, or the admin cycles spent keeping their parsers consistent. Once infrastructure and staffing are weighted in, the annual cost of indexing allow logs at analytics-tier pricing clears half a million dollars for a mid-sized deployment.
The routing split
A Cribl pipeline sits between the firewall collector and the Splunk HTTP Event Collector. The pipeline has two routes.
Route one sends every event, without filtering, to an S3 bucket. The bucket has a lifecycle policy: hot storage for the first thirty days, Infrequent Access for the next year, Glacier for the remainder of the retention requirement. S3 hot-tier cost lands at roughly $0.023 per GB per month and falls by an order of magnitude in Glacier. Full-fidelity storage for 300 GB per day of firewall data over a full year is measured in thousands of dollars, not hundreds of thousands.
Route two is conditional. Events where action != "allow" pass a filter and continue on to Splunk HEC. Allow events terminate after the S3 route. Splunk receives every deny, every threat log, every IPS hit, every URL filtering block, every authentication event, every configuration change. Those are the events your detections actually use. They arrive pre-parsed and field-extracted by Cribl, which means Splunk does less work on the indexer side.
The pipeline also deduplicates. Cribl's Suppress function removes duplicate events within a configurable key and time window, which typically trims another 10 to 30 percent off the volume that does reach Splunk. That is a property the native Splunk pipeline cannot offer on its own.
What stays in Splunk
Everything that powers a detection. Every event that populates a dashboard. Every record an investigator pulls during triage. Your correlation searches, your saved searches, your SPL dashboards, and your existing Technology Add-on integrations all continue to function against the same schema they have always used. Nothing changes on the Splunk side except the volume the indexer sees.
For the narrow class of incident response where allow logs are needed, Cribl Search queries the S3 archive using the same field model. The analyst workflow becomes: query the last ninety days from Splunk for denies and threats, then query the full retention window from S3 for corresponding allows if the case warrants it. Most investigations never need the second query.
What lives in S3
Every log the firewall produced, at full fidelity, stored for the full retention period, queryable within the audit response window. Auditors get the archive. Compliance teams get the retention. The SOC gets the investigation depth on the rare cases that need it. The CFO gets a cost profile that matches the actual utility of the data.
A thirty-day pilot
Route a single firewall through this split for thirty days. Measure three things. The percentage of events suppressed by the allow filter. The percentage of detection rules that still fire correctly with the reduced input stream. The change in daily Splunk ingest attributable to that source.
One source is enough to prove the architecture. If the detections still fire, if the compliance path still resolves, and if the volume reduction matches what Splunk's own sourcetype breakdown predicted, the rollout to the rest of the firewall fleet is mechanical. Most engagements hit 40 to 50 percent total ingest reduction within the first quarter, and 60 to 70 percent once the allow split is extended to authentication success events, DNS queries, and DHCP.
The structural takeaway
Your SIEM is an analytics platform. You should be paying for the analytics you use, not the storage you do not. Firewall allow logs are the clearest example of a category where the cost model and the utility model diverge. Routing them out of the SIEM and into object storage does not change your security posture, does not break compliance, and does not require analyst retraining. It moves a line item off the analytics-tier bill.
$500,000 a year for data nobody queries is a defensible number only if the data is genuinely unrecoverable from anywhere else. For firewall allows, it is not. The pipeline architecture that fixes this is not experimental, and the components involved have been in production for years. The only unusual thing about this conversation is how rarely it happens before a renewal cycle forces it.