Celeryd - send an email on log level error and above

Celeryd - send an email on log level error and above

We have django configured to send us emails on any error or above which is
triggered. This is done using the standard LOGGING configuration in
Django. I want this same behavior in celery. I have it working to send me
emails on Exceptions ([CELERY_SEND_TASK_ERROR_EMAILS][2]), but I want an
email on any defined level - coincidentally error and above.
For example in any django files we can do this.
log = logging.getLogger(__name__)
log.error("Oh No!")
And voilà it will send us an email assuming the following is setup in
settings.
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'standard': {
'format': "[%(asctime)s] %(levelname)s
[%(name)s.%(funcName)s:%(lineno)d] %(message)s",
'datefmt': "%d/%b/%Y %H:%M:%S"
},
},
'handlers': {
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'include_html': True,
},
},
'loggers': {
'': {
'handlers': ['logfile', 'mail_admins'],
'level': os.environ.get('DEBUG_LEVEL', 'ERROR'),
},
}
}
And for clarity I am calling celeryd like so.
../bin/python manage.py celeryd --settings=DJANGO_SETTINGS_MODULE \
--broker=amqp://RABBITMQ_USER:RABBITMQ_PASSWORD@localhost:5672/axis \
--beat --events --pidfile=/home/var/celeryd.pid \
--logfile=/home/logs/celeryd.log \
--loglevel=WARNING > /dev/null 2>&1
And the necessary celery related settings.
CELERY_SEND_EVENTS = True
CELERY_TRACK_STARTED = True
CELERYD_CONCURRENCY = 2
CELERYD_TASK_TIME_LIMIT = 60 * 60 * 2 # Kill anything longer than 2 hours
CELERY_TASK_RESULT_EXPIRES = 60 * 60 * 2
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_SEND_TASK_ERROR_EMAILS = True
CELERY_MAX_PREPARING = 600
And finally a basic test case which I feel should send two emails. One for
the error and the other for the exception..
from django.contrib.auth.models import User
import celery
from celery.schedules import crontab
from celery.utils.serialization import UnpickleableExceptionWrapper
from celery.utils.log import get_task_logger
log = get_task_logger(__name__)
@periodic_task(run_every=datetime.timedelta(minutes=5))
def noise_test(**kwargs):
log.info("Info NOISE TEST")
log.warning("Warning NOISE TEST")
log.error("Error NOISE TEST")
try:
User.objects.get(id=9999999)
except Exception as err:
from celery import current_app
err.args = list(err.args) + ["Crap"]
raise UnpickleableExceptionWrapper(
err.__class__.__module__, err.__class__.__name__, err.args)