How to Set Up and Use Celery in Django? (Ubuntu & Windows)

Asynchronous task queues are essential when dealing with time-consuming operations like sending emails, generating reports, or processing files. Celery is the go-to task queue for Python developers. In this guide, we'll walk through how to set up Celery with Django on both Ubuntu and Windows environments.

 

---

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation but supports scheduling as well. Celery requires a message broker to handle task messages—Redis and RabbitMQ are common choices.

 

---

Requirements

• Python 3.x
• Django
• Redis or RabbitMQ
• Celery
   

 

---

Step-by-Step Setup

1. Install Required Packages

Install Celery and Redis packages in your virtual environment:

After installation, create a file called celery.py in your project root directory, and a tasks.py file in your app directory for defining tasks.

Your project structure will look like this:

Configuration

celery.py (in the same directory as settings.py):

 

__init__.py (same directory as settings.py):

 

 

settings.py

 

 

tasks.py (inside your app):

 

 

On Ubuntu

1. Install and run Redis:

 

 

2. Run the Celery worker:

3. Test your Celery worker in the Django shell:
    

 

On Windows

Although Linux is the preferred OS for Celery, you can still run it on Windows for development.

Step 1: Install Required Packages

 

Step 2: Install and Run Redis

Option 1: Memurai (recommended)

• Download: https://www.memurai.com
   
• It works like Redis on Windows.
   

Option 2: Redis for Windows (community-maintained)

• Download: https://github.com/microsoftarchive/redis/releases
   
• Extract and run redis-server.exe.
   

Test Redis with:

(Should return PONG)

Step 3: Run Celery on Windows

Celery uses the prefork pool by default, which does not work on Windows. Use the solo pool instead.

In celery.py, add:

In CMD or PowerShell:

In PowerShell Core or Git Bash:

Test the Celery Task

 

 

Troubleshooting Tips

• Redis Not Running: Ensure Redis is active (sudo systemctl status redis on Ubuntu or check Memurai/Redis service on Windows).
   
• Task Not Executing: Verify the Celery worker is running and check logs for errors.
   
• Windows Pool Issues: Always use --pool=solo on Windows to avoid prefork errors.
   
• Broker Connection Errors: Confirm the CELERY_BROKER_URL matches your Redis server's address and port.
   

Best Practices

• Use a Virtual Environment: Keep dependencies isolated to avoid conflicts.
   
• Monitor Workers: Use tools like Flower (pip install flower, then celery -A myproject flower) to monitor Celery tasks.
   
• Production Setup: For production, use a robust broker like RabbitMQ, configure multiple workers, and deploy Celery with a process manager like Supervisor (Ubuntu) or NSSM (Windows).
   
• Logging: Configure detailed logging in settings.py to debug issues effectively.
   

Conclusion

Celery is a powerful tool for handling asynchronous tasks in Django, making your application more efficient and user-friendly. By following this guide, you can set up Celery with Redis on both Ubuntu and Windows, create and test tasks, and even schedule periodic jobs. Whether you're developing locally or preparing for production, Celery's flexibility ensures your Django app can handle heavy workloads with ease.

For further reading, check the Celery documentation or Django Celery Beat.

Happy task queuing!