Counting source lines in Python across many programming languages

Software developers often want to know how much code lives in a project. Lines of code is not a perfect measure of productivity, yet it remains a useful signal for estimating effort, comparing module sizes, and tracking growth over time. Building a source code line counter that works across multiple languages adds a layer of complexity, since each language has its own comment syntax, block delimiters, and conventions for documenting code.

Most off-the-shelf counters either treat every line the same way or rely on brittle heuristics. A Python-based approach can be precise because the standard library includes everything needed to read files, walk trees, and parse structured data. For developers in Sydney, Melbourne, or Brisbane managing mixed-code repositories, a small reliable tool often beats pulling in another heavy dependency just to produce a few numbers.

This article walks through building such a counter from scratch in Python. We'll explore directory traversal, language detection, comment stripping, and output formats. The techniques here sit comfortably next to other maintenance utilities documented on this site, including recent notes on the DenyHosts modular refactor covering a similar transformation for a long-running Python project.

Why count lines of code in heterogeneous projects

A figure showing thousands of physical lines versus logical lines per language gives a quick snapshot of where the heavy code lives. In Australian software houses, ranging from Atlassian in Sydney to Canva and smaller studios across Perth, project leads often share these metrics during planning sessions to argue for refactoring time or to flag modules drifting toward unwieldy size.

Beyond internal reporting, line counts help with audit work. Government departments in Canberra sometimes request approximate code volumes for procurement reporting, and an accurate per-language breakdown makes those answers far easier. The same numbers assist open-source maintainers who need to credit contributors fairly across polyglot codebases spanning Go services, Python utilities, and front-end TypeScript.

Designing the project layout

A clean module layout keeps the counter maintainable as language support grows. Start with a package called loc-counter containing submodules for detection, parsing, and reporting. Place language definitions in a JSON or YAML file rather than hard-coding them inside Python, so adding a new language does not require touching the parser logic directly.

Keep the CLI entry point in a thin wrapper that imports from the package. This separation echoes patterns where splitting responsibilities made future changes far less risky, especially when the tool runs unattended for years at a time. A tests directory at the same level holds unit tests for each language's comment rules and edge cases such as string literals containing comment markers.

Identifying languages from extensions and shebangs

Detection starts with file extensions. A small dictionary mapping .py, .rb, .js, .go, .rs, .java, and .c to language identifiers covers the bulk of typical projects. For files lacking an extension, peek at the first line: a #!/usr/bin/env python3 shebang signals a Python script, while #!/bin/bash indicates shell.

Store language metadata in a structured file so non-developers can update it. Each entry contains the display name, the line comment marker, an optional block comment opener and closer, and any file extensions associated with it. This scheme avoids hard-coding behaviour throughout the codebase and keeps the tool predictable when running on developer machines in Adelaide or Hobart.

Stripping comments and blank lines correctly

Per-language comment awareness is the heart of a credible counter. A naïve count inflates totals whenever a chunk of the file contains only documentation. Python uses # for line comments, Lua uses --, SQL uses --, and C-family languages support both // line comments and /* ... */ blocks.

Be careful with strings. A line like print("contains a # but is real code") should not be stripped of anything after the hash symbol. Process the file as a sequence of tokens rather than regex at the line level when languages include inline comments inside multiline strings. For block comments, track a flag that toggles when an opener appears and remains active until the matching closer is encountered, handling nested forms where the language supports them.

Walking directory trees efficiently

Use os.walk or pathlib.Path.rglob to enumerate files, skipping common build and dependency directories such as node_modules, .git, dist, __pycache__, and vendor. A configurable ignore list prevents the counter from inflating totals with generated or vendored code that nobody actually wrote.

For large repositories, reading files with pathlib and decoding with UTF-8 while tolerating errors keeps traversal robust against files produced on systems with different locale settings. Australian developers occasionally inherit codebases from overseas contributors, so resilience to mixed encodings saves a tedious debugging session during the arvo when a missing BOM throws off naive readers.

Reporting results in useful formats

A well-structured report turns raw counts into something stakeholders can act on. The default CSV output maps each path and language combination to physical lines, logical lines, and comment lines. A JSON variant supports downstream tooling, while a pretty-printed terminal table works for quick checks at the barbie after the build server finishes.

Optional flags can filter by language, sort results by total lines, or group by directory. Developers running audits often want a per-author breakdown, which pairs well with git blame to attribute code to the people who actually wrote it. Keeping the report formats pluggable means the core counter stays focused while consumers choose what they need.

Testing and benchmarking the tool

Unit tests cover tricky cases: file with no extension but a shebang, mixed line endings from Windows commits, block comments containing nested markers, and UTF-8 BOMs. A fixture directory of small samples exercises the detection and stripping paths without slowing the suite.

Benchmark large repositories to confirm traversal scales. Reading a few hundred thousand files should stay well under a minute on modest hardware. If profiles point at comment parsing as the hot spot, a precompiled token table keeps the inner loop fast. The counter is small enough to maintain alone, which suits a side project shared with the open-source community.