Scheduling Python Scripts: Cron, Celery & Cloud Options

Scheduling Python Scripts: Run Automation on Your Schedule

Automation is only useful if it runs reliably. This guide covers all the ways to schedule Python scripts—from simple cron jobs to enterprise-grade task queues and cloud functions.

Choosing the Right Scheduling Method

Option 1: Cron (Linux/Mac)

The simplest scheduling option for Unix systems.

Basic Syntax

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 6)
# │ │ │ │ │
# * * * * * command

Common Examples

# Every day at 6 AM
0 6 * * * /usr/bin/python3 /path/to/script.py

# Every hour
0 * * * * /usr/bin/python3 /path/to/script.py

# Every Monday at 9 AM
0 9 * * 1 /usr/bin/python3 /path/to/script.py

# First day of month at midnight
0 0 1 * * /usr/bin/python3 /path/to/script.py

# Every 15 minutes
*/15 * * * * /usr/bin/python3 /path/to/script.py

Setting Up Cron

# Edit crontab
crontab -e

# Add your job
0 6 * * * cd /path/to/project && /usr/bin/python3 script.py >> /var/log/myscript.log 2>&1

# View current crontab
crontab -l

Option 2: Windows Task Scheduler

For Windows environments:

1. Open Task Scheduler
2. Create Basic Task
3. Set trigger (daily, weekly, etc.)
4. Action: Start a program
5. Program: python.exe path
6. Arguments: script.py path

Option 3: APScheduler (In-App)

Schedule tasks within your Python application:

from apscheduler.schedulers.background import BackgroundScheduler

def my_job():
    print("Running scheduled task...")

scheduler = BackgroundScheduler()

# Every day at 6 AM
scheduler.add_job(my_job, 'cron', hour=6)

# Every 30 minutes
scheduler.add_job(my_job, 'interval', minutes=30)

# One-time execution
scheduler.add_job(my_job, 'date', run_date='2024-12-31 23:59:59')

scheduler.start()

Option 4: Celery (Distributed Tasks)

For complex, scalable task processing:

# tasks.py
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def process_report():
    # Your task logic
    generate_daily_report()

# Schedule with Celery Beat
app.conf.beat_schedule = {
    'daily-report': {
        'task': 'tasks.process_report',
        'schedule': crontab(hour=6, minute=0),
    },
}

Running Celery

# Start worker
celery -A tasks worker --loglevel=info

# Start beat scheduler
celery -A tasks beat --loglevel=info

Option 5: Cloud Functions

AWS Lambda + CloudWatch Events

# lambda_function.py
def lambda_handler(event, context):
    # Your scheduled task
    process_daily_data()
    return {'statusCode': 200}

Schedule via CloudWatch Events rule.

Google Cloud Functions + Cloud Scheduler

# main.py
def scheduled_task(request):
    process_daily_data()
    return 'OK'

Create Cloud Scheduler job targeting your function.

Best Practices

Logging

import logging

logging.basicConfig(
    filename='/var/log/myapp.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

def my_scheduled_job():
    logging.info("Job started")
    try:
        do_work()
        logging.info("Job completed successfully")
    except Exception as e:
        logging.error(f"Job failed: {e}")

Error Handling

def robust_scheduled_job():
    try:
        main_task()
    except Exception as e:
        send_alert(f"Scheduled job failed: {e}")
        raise

Monitoring

• Log all runs with timestamps
• Set up alerts for failures
• Monitor execution duration
• Track success/failure rates

Troubleshooting Common Issues

• Environment variables: Cron has minimal environment
• Working directory: Always use absolute paths
• Permissions: Ensure script is executable
• Dependencies: Use full Python path or virtualenv

Need Scheduled Automation Help?

Our Python automation team sets up reliable scheduled tasks for businesses, from simple scripts to enterprise task queues.

Contact us for scheduling automation help.