Most products will use or build a job queue at some point. It might be a managed service such as Amazon SQS, a dedicated system such as RabbitMQ, or a queue built on PostgreSQL.
The gist is the same: some work takes a long or unpredictable amount of time, can fail halfway through, must run on another machine, or is expensive enough to blow your latency SLA—the response time your application has promised to meet. Whether you are building for the web, mobile, or anything else, users expect a snappy experience. A job queue lets the application hand slow or unreliable work to a system that tracks it until it succeeds or needs attention.
Index
- A basic job queue
- What could go wrong?
- Keeping the queue reliable
- Magical, but not psychic
- Why use Postkit?
A basic job queue
A queue does not do the work itself. That would be convenient. You still need a computer—a worker—to pick up each job and run it.
Suppose server A receives a request to register a new user. We need to confirm that the user controls the email address or phone number they provided. This should be non-blocking: server A should not sit and wait because we now have to ask somebody else’s computer to do some work. To send a one-time password (OTP) over WhatsApp, we ask Meta; for email, we call an email provider; for SMS, it might be Twilio. Somebody else’s computer can be slow, unavailable, or simply say no.
Instead of making server A wait, it puts a message on the queue asking for an OTP. Another server picks up the message and does the work. Sometimes that server’s entire job is picking work from the queue. Unsurprisingly, we call it a worker.
We will use Postkit for the examples. The code is deliberately simplified: assume queue is a configured Postkit client and database and otp_provider belong to our application. We will add the plumbing when it has something useful to teach us.
Server A handles the user request:
def register_user(request):
user = database.create_user(request.email)
queue.push("send_otp", {"user_id": str(user.id)})
return {"status": "verification_pending"}, 202
push adds a job named send_otp to the queue. The second argument is its payload: the information the worker will need. We only need the user ID. Server B can look up everything else when it gets the job.
While verification is pending, the frontend displays an input for the OTP. It does not need to wait for the provider before showing the next step.
Server B runs a worker:
def worker():
while True:
job = queue.pull("send_otp")
if job is None:
wait_for_work()
continue
user = database.get_user(job["payload"]["user_id"])
otp_provider.send(user)
pull asks for the next send_otp job. If it gets one, server B reads the payload, finds the user and sends the OTP. The job also has an ID, which server B will use to report back. No job? Wait a bit and ask again.
This is how queues work on a good day. In a large system, though, most days are pretty shit.
What could go wrong?
Any line can fail, usually the one that looked too obvious to fail. Here is how this one falls over.
A worker loses power before finishing
Server B could lose power after picking up the job but before sending the OTP. Picking up work is not proof that it finished, so the queue needs an update from the worker.
It is the same as telling your manager when you finish a task. Without that update, the work may be reassigned while your manager wonders whether you did anything at all.
Once the OTP is sent, the worker acknowledges the job:
queue.ack(job["id"])
ack is short for acknowledge. Postkit can now delete the job or archive it as completed. We will return to what happens when the acknowledgement never arrives.
The OTP provider is temporarily unavailable
Server B is fine, but the provider may be unavailable. The job cannot finish right now. Instead of acknowledging it, the worker calls nack, short for negative acknowledgement:
try:
otp_provider.send(user)
except TemporaryError as error:
queue.nack(job["id"], error=str(error))
Postkit returns the job to the queue with exponential backoff. In plain English: wait a little before trying again, then wait longer after each failure. This gives the provider time to recover instead of making a bad day worse with thousands of immediate retries.
The phone number is not registered on WhatsApp
Some failures are permanent. Retrying will not magically make the phone number appear on WhatsApp. The worker marks the job as failed:
try:
otp_provider.send(user)
except PermanentError as error:
queue.fail(job["id"], error=str(error))
Postkit moves the job to a dead-letter queue, usually shortened to DLQ. This is where jobs go when the system has given up trying. We will not go deeply into it here. Imagine somebody complains that they never received an OTP: Postkit kept the failed job and its error, so we can investigate.
A worker dies after sending the OTP
Imagine Meta sends the OTP, but server B loses power before calling ack. Postkit receives no confirmation, so the job eventually becomes available again and the user may receive the same OTP twice.
That sounds like a bug, but the alternative is worse: the queue quietly loses the job. Postkit chooses to risk doing the work twice. This is called at-least-once delivery. A job may be handed to a worker more than once, so the worker has to cope with repeats.
That is the basic loop: pull a job, try it, then acknowledge it, retry it, or give up. It becomes more interesting once several computers are doing this at the same time and any one of them can vanish halfway through.
Keeping the queue reliable
Our app grows, so we add servers C and D. They run the same worker code as B and pull from the same queue. We can now do three times as much work and have three times as many computers to break.
The examples so far left out some plumbing. We need it now. Each worker gets a worker_id: the name it gives Postkit when claiming a job, such as server-b, server-c, or server-d.
Two workers ask for the same job
Servers C and D ask for work at exactly the same time. Without coordination, both could receive the same OTP job.
Postkit selects and claims a job inside one database transaction—a group of operations that all succeed together or do not happen at all. While one server claims the job, PostgreSQL locks that row. The other server skips it and takes another job, or goes back to waiting. All of that happens behind an ordinary pull call:
WORKER_ID = "server-c"
job = queue.pull("send_otp", worker_id=WORKER_ID)
database.connection.commit()
The commit makes the claim stick. Postkit records WORKER_ID as the owner. The worker uses the same ID when it completes, retries, or permanently fails the job:
queue.ack(job["id"], worker_id=WORKER_ID)
database.connection.commit()
The earlier examples left out the worker ID and commits so we could see the basic loop without drowning it in plumbing. Real code should pass the same ID to ack, nack, and fail, then commit the change.
A worker disappears after claiming a job
A drone strike damages the data centre hosting server C. If C had claimed a job, it can no longer finish or acknowledge it.
When Postkit hands out a job, it marks the job as running and puts a deadline on the claim. This is the visibility timeout. Other workers cannot see the job before the deadline. After it, we assume the worker is not coming back any time soon.
Postkit’s queue is SQL, not a little program sitting in the background with a stopwatch. One of our workers, or a scheduled process, must call tick_timeouts every so often. A tick finds expired claims and puts those jobs back on the queue:
queue.tick_timeouts()
database.connection.commit()
Server D can now claim the abandoned job. It is fine to have more than one process doing the ticking; PostgreSQL stops them from rescuing the same job at once.
A job may also be perfectly healthy and merely slow. A video transcode, for example, could outlive its visibility timeout. The worker can extend the deadline while it is still making progress:
from datetime import timedelta
queue.extend_visibility(job["id"], extension=timedelta(minutes=5))
database.connection.commit()
A stale worker tries to finish the job
The power cut was simple: server C was plainly gone. A broken network is more annoying because C can carry on working. Imagine C sends the OTP but loses its connection to PostgreSQL. Its visibility timeout expires, tick_timeouts puts the job back, and D claims it. Then C reconnects and tries to call ack.
Postkit refuses and ack returns False. C claimed the job as server-c, but it now belongs to server-d. C did the work; it no longer owns the paperwork.
The server dies between saving the user and queueing the OTP
Server A saves the user, then loses power before queueing the OTP. The user now exists, but the job does not. They may wait forever for a message that was never scheduled.
Postkit’s queue lives in PostgreSQL, so saving the user and queueing the OTP can share a transaction:
from postkit.queue import QueueClient
def register_user(request):
with database.transaction() as cursor:
queue = QueueClient(cursor, namespace="my-app")
user = database.create_user(cursor, request.email)
queue.push("send_otp", {"user_id": str(user.id)})
return {"status": "verification_pending"}, 202
cursor is what we use to issue SQL inside the transaction. The namespace keeps this application’s jobs apart from anything else using the same Postkit installation. PostgreSQL saves the user and queues the OTP, or does neither. There is no awkward half-registered user waiting for a job that does not exist.
The job keeps failing
Meta is down. Then the power goes out. When it comes back, DNS has decided to take the afternoon off. At some point, retrying stops being resilience and starts becoming a hobby.
Every time a worker pulls a job, Postkit increments its attempt count. Jobs also have a maximum number of attempts. If a worker calls nack after that maximum is reached, Postkit stops retrying and moves the job to the DLQ, where we can investigate why this particular OTP appears to be cursed.
Magical, but not psychic
Postkit can track work, rescue abandoned jobs and retry failures. It cannot tell whether your code is correct, stop someone from requesting 10,000 OTPs, or keep PostgreSQL alive. Those are still your problems. Sorry.
Why use Postkit?
You can build a job queue yourself. It usually starts with a jobs table and a SELECT statement. Then real users arrive, computers fall over, and you slowly rebuild everything in this article.
Postkit packages those boring but important parts into a queue that runs inside PostgreSQL: safe claims, acknowledgements, retries, visibility timeouts, dead letters and transactional job creation. Your application can save its own data and queue work in the same transaction, without operating Redis, RabbitMQ or another managed service alongside a database you already have.
Postkit is open source, the queue module can be installed on its own, and the Python SDK is deliberately small. You still write the worker and decide what the job should do. Postkit deals with the bookkeeping when a server falls over at precisely the wrong moment. The installation instructions are in the repository.