RQ is a surprisingly simple Python library for queuing jobs and processing them in the background, and it’s built on top of Redis, which is basically a super-fast in-memory data structure store that can also persist data.
Let’s see it in action. Imagine you have a Flask app that needs to send out a lot of emails. Sending emails synchronously can block your web requests, making your app feel sluggish. With RQ, you can offload this work.
Here’s a basic app.py for a Flask app:
from flask import Flask, request, jsonify
from redis import Redis
from rq import Queue
from your_tasks import send_email # Assuming you have a tasks.py file
app = Flask(__name__)
# Connect to Redis
redis_conn = Redis(host='localhost', port=6379, db=0)
q = Queue(connection=redis_conn)
@app.route('/send-email', methods=['POST'])
def trigger_email_send():
data = request.get_json()
recipient = data.get('recipient')
subject = data.get('subject')
body = data.get('body')
if not recipient or not subject or not body:
return jsonify({"error": "Missing recipient, subject, or body"}), 400
# Enqueue the job
job = q.enqueue(send_email, recipient, subject, body)
return jsonify({"message": "Email job enqueued", "job_id": job.id}), 202
if __name__ == '__main__':
app.run(debug=True)
And here’s a simple your_tasks.py with the send_email function:
import time
def send_email(recipient, subject, body):
print(f"Sending email to: {recipient}")
print(f"Subject: {subject}")
print(f"Body: {body}")
# Simulate a time-consuming email sending process
time.sleep(10)
print(f"Email successfully sent to {recipient}")
return f"Email sent to {recipient}"
To run this, you’ll need Redis installed and running (e.g., redis-server in your terminal). Then, you need to start an RQ worker:
rq worker
Now, if you send a POST request to http://127.0.0.1:5000/send-email with a JSON body like:
{
"recipient": "test@example.com",
"subject": "Hello from RQ",
"body": "This is the body of the email."
}
Your Flask app will immediately return a 202 Accepted with a job_id. Meanwhile, the rq worker process will pick up the send_email job and execute it in the background. You’ll see the print statements appear in the worker’s terminal.
The core problem RQ solves is the disconnect between immediate user feedback and potentially slow, background operations. By offloading tasks like sending emails, processing images, or running complex calculations to a separate worker process, your web server can respond quickly, providing a better user experience. RQ handles the serialization of your task function and its arguments, pushing them onto a Redis list (acting as a queue). The worker process then pops these tasks off the list, deserializes them, and executes the function.
The exact levers you control are primarily within your task functions and how you configure RQ and Redis. For your task functions, you can add retries, specify a timeout, and even set a delay before a job starts.
Here’s how you might set a retry mechanism for a task:
from rq import Queue
from redis import Redis
from your_tasks import send_email
redis_conn = Redis(host='localhost', port=6379, db=0)
q = Queue(connection=redis_conn)
# Enqueue with retries and a timeout
job = q.enqueue(send_email, recipient, subject, body,
retries=3, # Try up to 3 times if it fails
retry_delay=60, # Wait 60 seconds between retries
job_timeout=300) # Fail the job if it runs longer than 300 seconds
This retry_delay is crucial for transient network issues or temporary service unavailability. The worker will attempt to re-run the send_email function up to three times, waiting a full minute between each attempt, before giving up and marking the job as failed.
One thing most people don’t know is how easily RQ integrates with Flask-RQ2, which provides decorators and convenient ways to manage queues and jobs directly within your Flask application context, often simplifying the boilerplate code for connecting to Redis and creating queues. For instance, you can define your queues in your Flask config.py and access them via current_app.extensions['rq'].get_queue('default').
The next concept you’ll likely encounter is how to monitor your RQ queues and workers, and how to handle failed jobs.