Advertisement
July 17, 2011
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Kodos — Python Regex Debugger
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
DenyHosts — SSH Attack Blocker
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Scratchy — Apache Log Analyzer
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Canyonero — Project Management
Open Source Projects by Phil Schwartz

Building a Simple Statistics API with Python BaseHTTPServer

When developers maintain open-source tools, exposing project statistics through a programmatic interface can transform how users and contributors interact with the codebase. A lightweight HTTP endpoint that returns download counts, commit history, or active contributor data opens doors for dashboards, badge generators, and automated monitoring. Python's standard library includes the BaseHTTPServer module, which provides just enough functionality to construct a minimal yet functional API without external dependencies. For solo developers and small teams running their own infrastructure, this approach avoids the overhead of Flask or FastAPI while remaining fully transparent.

Many Australian developers operate from home offices in suburban Brisbane, Melbourne, or Perth, often relying on the National Broadband Network for connectivity. The simplicity of BaseHTTPServer fits well in these contexts, where a single VPS or even a Raspberry Pi behind an NBN connection can host a working statistics endpoint. The module ships with Python itself, which means no pip installs, no virtual environment wrangling, and no supply-chain surprises. This article walks through constructing a small API that serves JSON-formatted metrics for a project portfolio.

Laying the Groundwork with BaseHTTPRequestHandler

The heart of the module is the BaseHTTPRequestHandler class, which an application subclasses to define behaviour for incoming requests. A typical implementation overrides the do_GET method, inspects the request path, and writes a response with the appropriate status code and headers. For a statistics API, the handler can map URL paths such as /stats/downloads or /stats/commits to specific data sources. Because the class runs on a per-request basis, there is no need to manage connection pools or thread pools unless traffic demands it.

Threading can be added with the ThreadingHTTPServer wrapper when concurrent requests become common. For a portfolio site that might receive bursts of traffic after a project gets shared on a Sydney-based mailing list or a Melbourne developer forum, threading prevents slow responses. The handler should set Content-Type to application/json before writing any body, ensuring clients correctly parse the payload. A simple try/except block around the response writing protects the server from crashing when a client disconnects mid-stream.

Returning JSON from Project Data Sources

Project statistics can come from many places: a SQLite database tracking download events, a CSV file updated by a cron job, or live calls to Git hosting APIs. The statistics endpoint should aggregate these sources and return them as a structured JSON document. A dictionary containing keys like total_downloads, unique_visitors, last_release_date, and active_contributors provides enough context for most consumers. Python's json module handles serialisation, with json.dumps converting the dictionary to a string and json.loads parsing any incoming query parameters.

Including a timestamp field formatted in ISO 8601 helps clients cache responses intelligently. Australian servers benefit from using Australia/Sydney or Australia/Perth as the timezone identifier, particularly when coordinating with overseas collaborators. Returning consistent field names across endpoints reduces friction for anyone writing client libraries. It also helps to round numerical values to a reasonable precision, since download counters rarely need more than four significant figures.

Securing the Endpoint with Basic Authentication

Exposing internal metrics without any access control invites abuse. Even a simple API benefits from HTTP Basic Authentication, where the handler inspects the Authorization header and validates credentials against an expected pair. Storing the password as a hashed string in a configuration file avoids embedding secrets in source code, which is especially important when the repository is public on a platform like GitHub. For higher assurance, environment variables or a secrets manager on a hosted VPS provides better isolation.

Rate limiting prevents a single client from monopolising the server. A straightforward in-memory dictionary mapping IP addresses to request timestamps works for modest traffic. Australian organisations operating under the Privacy Act 1988 should also be mindful of which statistics they expose, particularly if the data could be traced back to identifiable individuals. Aggregating data so that no single user's behaviour is reconstructable strikes a reasonable balance between utility and privacy.

Deploying Behind a Reverse Proxy on Linux

While BaseHTTPServer can listen directly on port 8000, production deployments typically place it behind nginx or Apache. The reverse proxy handles TLS termination, static file serving, and request buffering. On a typical Ubuntu 22.04 instance provisioned through a local Australian provider or a cloud region in Sydney, the setup involves configuring nginx to forward /api/ requests to the Python backend. A systemd unit file keeps the server running across reboots, and journalctl provides straightforward log inspection.

The server should bind to 127.0.0.1 rather than all interfaces when a reverse proxy fronts it. This prevents direct external access to the raw Python process. Adding a small health-check endpoint at /api/health returning {"status": "ok"} makes monitoring easier, whether through a cron-driven curl from a developer workstation in Adelaide or a dedicated uptime service. Logs written to a dedicated file with rotation managed by logrotate keeps disk usage predictable.

Monitoring Usage and Logging Requests

Understanding how the API gets used informs future development. The handler can append a line to a log file for each request, capturing the timestamp, IP address, path, and status code. Parsing these logs with simple shell scripts or Python's built-in csv module surfaces trends over time. Australian developers contributing to projects listed on open-source aggregators often want to know which regions generate the most traffic, and IP geolocation helps answer that without collecting personal information.

Log analysis can also reveal bot activity. Patterns such as a single address requesting hundreds of endpoints per minute suggest scanning behaviour. Blocking offending addresses at the firewall level protects the backend from unnecessary load. The Australian Cyber Security Centre publishes guidance through the Essential Eight framework, which recommends application-level logging as part of a broader defence strategy. Following these recommendations aligns the small API with mainstream security practice without requiring enterprise tooling.

Practical Recommendations for Maintainers