Python

Packaging Is Where Python Deployments Fail First

Python code is portable. Its dependencies are not. pip installs a prebuilt wheel when one matches the target platform and falls back to compiling from source when none does, and the build then needs a toolchain and header files that a slim base image deliberately does not carry. Wheel filenames spell the constraint out: a file tagged cp312-cp312-manylinux_2_28_x86_64 will not install on 3.11, on musl, or on arm64.

Two mismatches account for most of it. manylinux wheels are compiled against glibc, so an Alpine image matches nothing and rebuilds every C extension from source; the musllinux tag exists but coverage is thinner, and moving to a Debian slim base is usually less work than maintaining a build stage. The other is architecture. An image built on an Apple Silicon laptop is arm64 by default and will not run on an x86_64 host unless the build sets the target platform on purpose.

The failures sort cleanly by their error text, which is worth knowing before anyone starts changing pins at random.

• "error: command 'gcc' failed" or "fatal error: Python.h: No such file or directory" during pip install: the package is being compiled rather than installed from a wheel.
• "exec format error" when the container starts, or a container that runs but at a fraction of expected speed under emulation: the image architecture does not match the host.
• pip reports that it is looking at multiple versions of a package and that this could take a while, then sits for several minutes: the resolver is backtracking through old releases because two requirements have no overlapping compatible version. Read the conflict it eventually prints instead of loosening pins one at a time.
• "numpy.dtype size changed, may indicate binary incompatibility", or a message that a module compiled with NumPy 1.x cannot run under 2.x: the extension was built against a different NumPy C ABI, which changed at 2.0. Rebuild the extension against the installed version, or cap NumPy below 2 until you can.
• "error: externally-managed-environment": PEP 668, a distribution-managed interpreter refusing installs into the system site-packages. Create a virtual environment; --break-system-packages is not the fix, it is the thing PEP 668 was written to prevent.

The Version You Can Actually Target

CPython ships a feature release each October and supports it for roughly five years, about two of bug fixes and three of security patches only. 3.8 reached end of life in October 2024 and 3.9 followed a year later, so an application still pinned to either is running an interpreter that receives nothing at all, including for vulnerabilities in the standard library.

What blocks an upgrade is rarely the language. It is the slowest compiled dependency: until a package publishes wheels for the new interpreter, moving means building it yourself or waiting for the maintainer. Standard library removals are the other trap. 3.12 dropped distutils, so build scripts that import it fail unless setuptools supplies the shim, and several long-deprecated modules went with it.

Pin the minor version everywhere it is recorded: the image tag as python:3.12-slim rather than python:3 or python:latest, requires-python in the project metadata, and the same version in CI. A floating tag means the interpreter can change underneath a build that nobody touched, and the resulting failure looks like a code regression rather than a base image update.

One Blocking Call Stalls the Whole Event Loop

Async Python gets its concurrency from a single thread running an event loop, so one blocking call inside a coroutine stops every other request in that process for as long as it takes. The signature is distinctive: latency rises across all endpoints at once, including ones that do no work, rather than on the slow path alone. Timing the handler in isolation shows nothing, because the time is spent waiting in the queue rather than in the function.

The usual causes are a synchronous database driver, an HTTP client such as requests instead of an async one, plain file reads, and CPU-heavy serialization. Either move the call off the loop with asyncio.to_thread, or do not declare the handler async in the first place: in FastAPI a path operation written with def runs in a worker threadpool, which is the right answer for blocking code and far cheaper than rewriting it. Django's ORM is synchronous underneath and raises SynchronousOnlyOperation when called directly from async context, so ORM work there belongs behind sync_to_async or in an ordinary sync view.

Worker counts and pool sizes multiply, and the arithmetic is easy to skip. Each gunicorn or uvicorn worker is a separate process holding its own connection pool, so eight workers with a pool of ten open eighty database connections when saturated, against a PostgreSQL default max_connections of one hundred. Size the pool from the worker count, or put a pooler such as PgBouncer in front and let the application ask for more than the database would otherwise allow.

Background Jobs Run at Least Once, Not Exactly Once

Task queues deliver at least once by design. A worker that completes the work and then dies before acknowledging leaves the message on the broker, and it is handed to another worker. Any task with an effect outside the process, sending mail, calling a payment API, writing a file, therefore needs a key it can check before acting, or the retry that was meant to add reliability sends the message twice.

Two conventions remove most of the remaining debugging. Pass identifiers rather than objects: a serialized model instance is a snapshot of a row that may have changed by the time a worker picks it up, and a primary key forces a fresh read. And run exactly one scheduler. A beat process started in two containers produces two of every periodic task, which stays invisible until something that must run once a night runs twice.

Long tasks interact badly with broker defaults. Redis and SQS style transports redeliver a message whose visibility timeout expires while the task is still running, so a job that legitimately takes an hour ends up running alongside a second copy of itself. Check the actual default for the transport you use rather than assuming it exceeds your worst case, and prefer splitting the work into pieces that finish quickly over raising the timeout indefinitely.

When the Data Outgrows pandas

pandas holds the entire frame in memory, and peak usage during a merge, sort or groupby is a multiple of the input rather than equal to it. Text is the worst case: read without dtypes specified, a string column becomes Python objects with per-value overhead, so a CSV of a few gigabytes can need several times that in RAM before any computation starts. Declaring dtypes on read, using the categorical type for low-cardinality strings, and storing Parquet instead of CSV so only the needed columns are loaded will often keep a job inside the machine it already has.

Past that point the answer is usually not a larger instance. DuckDB runs SQL directly against Parquet files and reads only the columns and row groups a query touches, with no load step at all. Polars builds a lazy plan and can stream a dataset larger than memory. And when the data already lives in PostgreSQL, the aggregation generally belongs in the query rather than in a Python process that pulls every row across the network in order to count them.