Set up Celery production for Django project
#experience#celery#django#production
Văn Hiếu Lê

Văn Hiếu Lê @heterl0

About: Hi, I'm Hieu, Front-End Developer at Kdigi Corporation, Vietnam.

Joined:
Oct 22, 2024

Set up Celery production for Django project

Publish Date: Apr 5
1 0

🚀 Background

Celery is a cron job service commonly used in Django projects. In my case, I used Celery to reset user streaks at midnight (00:00 AM) for my application.

Set up the Celery for django project

In development, I had to run three terminal windows:

  • One for the Django server: python manage.py runserver

  • One for the Celery worker: celery -A proj worker

  • One for the Celery beat: celery -A proj beat

That setup was a bit complex. So in this blog, I’ll share how I set up Celery as a production-ready service using systemd.

🛠️ Approach

When I first moved my app to production, I used Gunicorn to serve Django, but forgot about Celery. As a result, scheduled tasks didn’t run because both Celery Worker and Celery Beat need to run in parallel.

✅ Option 1: Using tmux

Initially, I used tmux:

  1. SSH into the server.

  2. Start a tmux session and split the window with Ctrl + b → %.

  3. Run the worker and beat processes in separate panes.

Even after closing SSH, the processes stayed alive (confirmed using htop). This works, but it’s not ideal for long-term use.

✅ Option 2: Using systemd Services

When I got a new VPS, I wanted a better solution. After some research (and help from ChatGPT 😄), I found a reliable approach using systemd services to run Celery in the background.

Reference: Django with Celery in Production

⚙️ 1. Create the Celery Worker Service

Create a new service file:

sudo nano /etc/systemd/system/celery.service
Enter fullscreen mode Exit fullscreen mode

Paste the following (replace user, paths, and [celery_app] accordingly):

[Unit]
Description=Celery Worker Service
After=network.target

[Service]
Type=simple
User=your_username
Group=your_username
WorkingDirectory=/home/your_username/your_project
ExecStart=/home/your_username/.local/share/virtualenvs/your-venv/bin/celery -A [celery_app] worker --loglevel=INFO
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

👉 Replace:

  • your_username with your actual Linux username.

  • [celery_app] with the value from your celery.py file.

Then run:

sudo systemctl daemon-reload
sudo systemctl enable celery
sudo systemctl start celery
Enter fullscreen mode Exit fullscreen mode

🕒 2. Create the Celery Beat Service

Now create the Beat service:

sudo nano /etc/systemd/system/celery-beat.service
Enter fullscreen mode Exit fullscreen mode

Paste the following:

[Unit]
Description=Celery Beat Service
After=network.target

[Service]
Type=simple
User=your_username
Group=your_username
WorkingDirectory=/home/your_username/your_project
ExecStart=/home/your_username/.local/share/virtualenvs/your-venv/bin/celery -A [celery_app] beat --loglevel=INFO
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Then run:

sudo systemctl daemon-reload
sudo systemctl enable celery-beat
sudo systemctl start celery-beat
Enter fullscreen mode Exit fullscreen mode

✅ Check status:

sudo systemctl status celery-beat
Enter fullscreen mode Exit fullscreen mode

If you see active (running), it means everything is set up correctly.

🧾 Conclusion

That’s how I set up Celery and Celery Beat in production using systemd. It’s a clean, reliable, and maintainable way to manage background tasks in Django.

📚 References:

Thanks for reading! 🙏 Hope this helps you in your deployment journey.

Comments 0 total

    Add comment