Extracting and visualizing HTTP status code frequency with Python

When a website starts serving traffic from thousands of unique visitors each day, the raw access logs hold more value than most operators realise. A well-crafted Python script can transform those otherwise noisy text files into clear charts that reveal which pages are healthy, which are throwing errors, and where the infrastructure is being abused. Status code analysis also feeds straight into search engine optimisation work, alerting teams to broken redirects or accidental 404s that harm visibility on Australian search results. Learn more about Testing Kodos With Thousands Of Regex Patterns To Find Performance Bottlenecks.

Many developers in Melbourne and Sydney already rely on this kind of lightweight tooling, particularly because their companies often host infrastructure inside the country to meet local data sovereignty expectations and align with the Australian Privacy Principles. A short script that runs from cron, exports a PNG, and emails a summary can replace a heavy monitoring subscription. The rest of this guide walks through the moving pieces in order: reading logs, counting codes, drawing a chart, and keeping the whole pipeline fast enough for production use.

The value of status code analytics

Every HTTP response carries a three-digit number that tells a story. A 200 means the request was served cleanly, a 301 confirms a permanent redirect, a 403 signals an access control rule firing, and a 500 points to a problem on the server side. Aggregating those numbers across an hour, a day, or a month turns individual requests into trend lines that engineers can act on.

For Australian operators, the same data feeds compliance reporting. The Australian Cyber Security Centre regularly publishes guidance on detecting anomalous web traffic, and a sudden surge in 4xx codes from a single subnet is often the first indicator of credential stuffing or scraping attempts. Treating the log files as a primary signal source rather than an archive is what separates reactive teams from proactive ones.

Reading Apache and Nginx log files

Apache writes combined log entries that look like 127.0.0.1 - - [10/Oct/2024:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 2326. Nginx produces a near-identical line by default. Python's built-in open() is enough to start, but reading line by line keeps the memory footprint low even when the log has rotated to several gigabytes. Compressed .gz files from logrotate can be handled with gzip.open() so the script can be run against yesterday's archive without manual extraction.

A generator function that yields one parsed record at a time is a clean way to keep the consumer free of buffering concerns. That approach also makes the pipeline testable, since each parsed record is just a plain dictionary with predictable keys that downstream stages can rely on.

Crafting the right regular expression

The status code sits in a fixed position in both Apache and Nginx formats, between the closing quote of the request line and the response size. Pulling it out by position is brittle because custom log formats rearrange the fields. A regular expression that captures only the status code is more portable and easier to maintain as the underlying server configuration changes.

This is exactly the kind of pattern-heavy work that benefits from a dedicated debugger. Earlier work on Kodos explored similar challenges, and the write-up on testing-kodos-with-thousands-of-regex-patterns-to-find-performance-bottlenecks walks through how pattern performance changes once the input scales. A single capture group around \s(\d{3})\s is usually enough, but it pays to test against log samples drawn from production rather than synthetic fixtures.

Building a counter with collections

Once a generator is yielding status code strings, a collections.Counter can accumulate the totals in a single line. Counter is implemented in C under the hood, which makes it faster than a plain dict with manual increment logic. After a pass through the file, calling most_common() returns the codes sorted by frequency, and converting the result to a dictionary gives a stable shape that can be passed into the visualisation layer.

For long-running aggregations, it sometimes makes sense to persist intermediate counts to a JSON or SQLite file so that the chart can be built from the previous total plus today's delta. That pattern is especially useful for high-traffic properties in Brisbane or Perth, where log volumes peak during business hours and a partial restart should not lose history.

Plotting the results in a bar chart

Matplotlib remains the most reliable choice for static server-side charts, and Seaborn adds a more polished default style with a single import. A horizontal bar chart works best when more than a handful of status codes are present, because long legends on a vertical chart quickly eat horizontal space. Colour coding helps separate success codes from redirects and errors, and labelling each bar with the actual count avoids the need for a separate data table.

Saving the figure to a .png with savefig() keeps the output compatible with email attachments, Slack uploads, or static dashboards hosted on a CDN. Australian teams shipping to Atlassian-style tools can drop the file into a Confluence page or attach it to a Jira ticket for review.

Speeding up large-scale parsing

Pure Python is plenty fast for sites serving a few million requests per day, but anything heavier should reach for concurrency. The concurrent.futures module provides a thread pool that overlaps reading and parsing nicely because the bottleneck is usually disk I/O rather than CPU. For truly large backlogs, the built-in re module can be swapped for the third-party regex library, which compiles patterns more efficiently and supports a wider feature set.

Profiling with cProfile before optimising prevents wasted effort. In practice, most slowdowns come from repeated calls to re.match on the same pattern, and compiling the pattern once at module level is the single highest-impact change available.

Putting the script to work

A finished script typically exposes a small command-line interface using argparse, takes a log path and an output path, and exits with a non-zero code when the 5xx count exceeds a configured threshold. Wrapping that in a systemd timer or a Kubernetes CronJob makes the chart generation part of the regular operational rhythm. Many Australian engineering teams publish the resulting PNG to an internal dashboard hosted on a Sydney-region cloud bucket, satisfying residency expectations under the local data handling rules.

A weekly review of the chart is often enough to catch broken redirects, misconfigured caches, and bot traffic spikes before they affect users. Combined with alerting on unusual 4xx bursts, the same script quietly becomes a piece of the broader security posture without ever needing a dedicated SaaS subscription.

Habits that keep the pipeline healthy

• Compile every regular expression once at module scope rather than inside the parsing loop.
• Stream the log file line by line to avoid loading gigabytes into memory.
• Persist running totals to disk so a restart does not erase historical counts.
• Test pattern changes against a representative sample drawn from real production traffic.
• Set a sensible threshold for 5xx responses and exit non-zero when it is breached.