Using Python’s itertools for efficient batch processing

Large datasets rarely need to be loaded into memory all at once. Whether the source is an Apache log, a CSV export, a database cursor, or a stream of API responses, processing records incrementally keeps memory use predictable and makes failures easier to manage.

Python’s itertools module is well suited to this style of work. Its functions produce lazy iterators, allowing a program to read, group, filter, and submit records as they arrive. This approach is useful for Linux utilities, scheduled maintenance jobs, analytics scripts, and open-source tools that need to run reliably on modest hardware.

A batch is simply a small collection of records handled as one unit. The right batch size depends on the work being performed, the available memory, database limits, network latency, and the cost of retrying a failed operation. A 1,000-row database write may be sensible, while a remote API might work better with groups of 50.

For Australian teams, operational details can shape the design. A job running in a Sydney cloud region may have different latency from a service in Perth, while regional offices may deal with less consistent NBN connections. Time zones, privacy obligations, and cloud egress charges also deserve attention before a high-volume pipeline goes into production.

Start with lazy data pipelines

The central idea is to keep each stage lazy. Instead of creating several large intermediate lists, connect generators and iterator functions so that records move through the pipeline one at a time. filterfalse(), map(), islice(), chain(), and takewhile() can all contribute to this pattern.

from itertools import islice

def batches(iterable, size):
    iterator = iter(iterable)
    while batch := list(islice(iterator, size)):
        yield batch

def valid_lines(lines):
    return (
        line.strip()
        for line in lines
        if line.strip() and not line.startswith("#")
    )

for group in batches(valid_lines(log_file), 500):
    process(group)

Only the current batch is held as a list. The input remains lazy, so a multi-gigabyte file does not become a multi-gigabyte Python object. The same design works with a database cursor or a generator that fetches pages from an HTTP service.

On Python 3.12 and later, itertools.batched() provides a concise standard implementation:

from itertools import batched

for group in batched(records, 500):
    process(group)

The final group may contain fewer records than the requested size. That is normally the desired behaviour, although an application that requires complete groups can check len(group) before submitting the work.

Combine batching with selective processing

Batching is more effective when unwanted records are removed before they enter a group. Filtering early reduces memory use, database traffic, and the amount of work performed by downstream functions.

from itertools import batched, compress

def usable(entry):
    return entry["status"] == 200 and entry["path"] != "/health"

selected = filter(usable, source_records)

for group in batched(selected, 250):
    write_rows(group)

compress() is useful when a separate iterator of Boolean selectors determines which records should continue. starmap() can expand tuples into function arguments, while chain.from_iterable() can flatten pages returned by a paginated reader without building a large nested list.

Grouping requires care. itertools.groupby() groups adjacent records, rather than finding every matching record in the complete dataset. The input must generally be sorted by the same key, or the result may contain several groups for one category. For a streaming job, that distinction can prevent subtle reporting errors.

A practical rule is to separate pure iterator transformations from side effects. Parsing, validation, and normalisation can remain easy to test. Database writes, file updates, and network requests should happen at a clear boundary where retries and failure handling are visible.

Match batch size to the workload

There is no universal ideal batch size. Small groups reduce the amount of work lost when a failure occurs, but they can increase transaction overhead. Large groups improve throughput until memory pressure, lock duration, request limits, or timeout risk begins to dominate.

For database writes, measure executemany() or bulk insert performance at several sizes. A local PostgreSQL instance might handle 1,000 rows comfortably, while a hosted service with strict request limits may perform better with 100. Keep the setting configurable rather than burying it in the processing function.

Backpressure matters when the producer is faster than the consumer. A simple iterator loop naturally pauses while process(group) is running. If a queue or thread pool is added, cap its pending work so the application does not replace one large list with an equally large backlog of futures.

A failed batch should be identifiable and repeatable. Store a source offset, filename and line range, event ID, or another stable key. If an operation is retried, design it to be idempotent where possible, using unique constraints or upserts to avoid duplicate records.

Add observability for real operations

Batch jobs become much easier to support when they report useful progress. Track records read, records accepted, records written, elapsed time, batch duration, and failures. Logging every individual record can overwhelm the output, so emit summaries every few batches or at a timed interval.

Australian deployments often need consistent timestamp handling across states and sites. Store event times in UTC, then convert to AEST, AEDT, or another local display zone when producing reports. This avoids confusing daylight-saving changes between Melbourne, Sydney, and Brisbane, while Perth operators can still receive correctly localised dashboards.

Privacy should influence what gets logged. A pipeline handling customer data under the Australian Privacy Act should avoid writing full email addresses, authentication tokens, or raw request payloads to debug logs. Hashes, redacted identifiers, and aggregate counts usually provide enough diagnostic value.

The design of Scratchy’s original log analyser illustrates why focused command-line tools can be valuable. A log-processing utility can stream input, extract only the fields it needs, and produce actionable summaries without requiring a large analytics platform.

Batch design checks

A small test dataset should cover an empty input, a single partial batch, malformed records, a failure during the middle batch, and a source that raises an exception while being consumed. These cases expose assumptions about iterator exhaustion and retry behaviour.

Benchmark with representative data rather than a tiny sample. Include realistic record sizes, network conditions, database indexes, and the amount of filtering expected in production. For an Australian business, test from the intended hosting region and account for the latency between a Sydney service and users or systems in other states.

Before running at scale

• Keep input and transformation stages lazy
• Make batch size configurable
• Record a stable checkpoint for retries
• Measure memory, throughput, and failure rates

Useful safeguards

• Reject malformed records before bulk writes
• Apply timeouts to external requests
• Limit queued work to provide backpressure
• Use UTC for stored timestamps

With these practices, itertools becomes more than a collection of convenient functions. It provides the foundation for streaming ETL jobs, log analysis, import utilities, and scheduled Linux services that process large datasets with controlled memory use and predictable recovery.