Add inbound mail handling

Add normalized event, signal, and webhooks for inbound mail. Closes #43 Closes #86
2025-12-20 03:41:05 -05:00 · 2018-02-02 10:38:53 -08:00
parent c924c9ec03
commit b57eb94f64
35 changed files with 2968 additions and 130 deletions
--- a/docs/esps/index.rst
+++ b/docs/esps/index.rst
@@ -48,6 +48,10 @@ Email Service Provider                        |Mailgun|   |Mailjet|   |Mandrill|
 ---------------------------------------------------------------------------------------------------------------------
 :attr:`~AnymailMessage.anymail_status`        Yes         Yes         Yes         Yes         Yes         Yes
 |AnymailTrackingEvent| from webhooks          Yes         Yes         Yes         Yes         Yes         Yes
+
+.. rubric:: :ref:`Inbound handling <inbound>`
+---------------------------------------------------------------------------------------------------------------------
+|AnymailInboundEvent| from webhooks           Yes         Yes         Yes         Yes         Yes         Yes
 ============================================  ==========  ==========  ==========  ==========  ==========  ===========


@@ -63,6 +67,7 @@ meaningless. (And even specific features don't matter if you don't plan to use t
 .. |SendGrid| replace:: :ref:`sendgrid-backend`
 .. |SparkPost| replace:: :ref:`sparkpost-backend`
 .. |AnymailTrackingEvent| replace:: :class:`~anymail.signals.AnymailTrackingEvent`
+.. |AnymailInboundEvent| replace:: :class:`~anymail.signals.AnymailInboundEvent`


 Other ESPs
--- a/docs/esps/mailgun.rst
+++ b/docs/esps/mailgun.rst
@@ -215,3 +215,36 @@ a Django :class:`~django.http.QueryDict` object of `Mailgun event fields`_.

 .. _Mailgun dashboard: https://mailgun.com/app/dashboard
 .. _Mailgun event fields: https://documentation.mailgun.com/user_manual.html#webhooks
+
+
+.. _mailgun-inbound:
+
+Inbound webhook
+---------------
+
+If you want to receive email from Mailgun through Anymail's normalized :ref:`inbound <inbound>`
+handling, follow Mailgun's `Receiving, Storing and Fowarding Messages`_ guide to set up
+an inbound route that forwards to Anymail's inbound webhook. (You can configure routes
+using Mailgun's API, or simply using the "Routes" tab in your `Mailgun dashboard`_.)
+
+The *action* for your route will be either:
+
+   :samp:`forward("https://{random}:{random}@{yoursite.example.com}/anymail/mailgun/inbound/")`
+   :samp:`forward("https://{random}:{random}@{yoursite.example.com}/anymail/mailgun/inbound_mime/")`
+
+     * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
+     * *yoursite.example.com* is your Django site
+
+Anymail accepts either of Mailgun's "fully-parsed" (.../inbound/) and "raw MIME" (.../inbound_mime/)
+formats; the URL tells Mailgun which you want. Because Anymail handles parsing and normalizing the data,
+both are equally easy to use. The raw MIME option will give the most accurate representation of *any*
+received email (including complex forms like multi-message mailing list digests). The fully-parsed option
+*may* use less memory while processing messages with many large attachments.
+
+If you want to use Anymail's normalized :attr:`~anymail.inbound.AnymailInboundMessage.spam_detected` and
+:attr:`~anymail.inbound.AnymailInboundMessage.spam_score` attributes, you'll need to set your Mailgun
+domain's inbound spam filter to "Deliver spam, but add X-Mailgun-SFlag and X-Mailgun-SScore headers"
+(in the `Mailgun dashboard`_ on the "Domains" tab).
+
+.. _Receiving, Storing and Fowarding Messages:
+   https://documentation.mailgun.com/en/latest/user_manual.html#receiving-forwarding-and-storing-messages
--- a/docs/esps/mailjet.rst
+++ b/docs/esps/mailjet.rst
@@ -249,3 +249,26 @@ for each event in the batch.)

 .. _Event tracking (triggers): https://app.mailjet.com/account/triggers
 .. _Mailjet event: https://dev.mailjet.com/guides/#events
+
+
+.. _mailjet-inbound:
+
+Inbound webhook
+---------------
+
+If you want to receive email from Mailjet through Anymail's normalized :ref:`inbound <inbound>`
+handling, follow Mailjet's `Parse API inbound emails`_ guide to set up Anymail's inbound webhook.
+
+The parseroute Url parameter will be:
+
+   :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/mailjet/inbound/`
+
+     * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
+     * *yoursite.example.com* is your Django site
+
+Once you've done Mailjet's "basic setup" to configure the Parse API webhook, you can skip
+ahead to the "use your own domain" section of their guide. (Anymail normalizes the inbound
+event for you, so you won't need to worry about Mailjet's event and attachment formats.)
+
+.. _Parse API inbound emails:
+    https://dev.mailjet.com/guides/#parse-api-inbound-emails
--- a/docs/esps/mandrill.rst
+++ b/docs/esps/mandrill.rst
@@ -185,27 +185,31 @@ See the `Mandrill's template docs`_ for more information.


 .. _mandrill-webhooks:
+.. _mandrill-inbound:

-Status tracking webhooks
------------------------
+Status tracking and inbound webhooks
+------------------------------------

-If you are using Anymail's normalized :ref:`status tracking <event-tracking>`,
-setting up Anymail's webhook URL requires deploying your Django project twice:
+If you are using Anymail's normalized :ref:`status tracking <event-tracking>`
+and/or :ref:`inbound <inbound>` handling, setting up Anymail's webhook URL
+requires deploying your Django project twice:

 1. First, follow the instructions to
-   :ref:`configure Anymail's webhooks <webhooks-configuration>`. You *must*
-   deploy before adding the webhook URL to Mandrill, because it will attempt
+   :ref:`configure Anymail's webhooks <webhooks-configuration>`. You *must deploy*
+   before adding the webhook URL to Mandrill, because Mandrill will attempt
   to verify the URL against your production server.

-   Follow `Mandrill's instructions`_ to add Anymail's webhook URL in their settings:
+   Once you've deployed, then set Anymail's webhook URL in Mandrill, following their
+   instructions for `tracking event webhooks`_ (be sure to check the boxes for the
+   events you want to receive) and/or `inbound route webhooks`_.
+   In either case, the webhook url is:

-      :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/mandrill/tracking/`
+      :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/mandrill/`

        * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
        * *yoursite.example.com* is your Django site
-
-   Be sure to check the boxes in the Mandrill settings for the event types you want to receive.
-   The same Anymail tracking URL can handle all Mandrill "message" and "change" events.
+        * (Note: Unlike Anymail's other supported ESPs, the Mandrill webhook uses this
+          single url for both tracking and inbound events.)

 2. Mandrill will provide you a "webhook authentication key" once it verifies the URL
   is working. Add this to your Django project's Anymail settings under
@@ -226,7 +230,7 @@ else fails, you can set Anymail's :setting:`MANDRILL_WEBHOOK_URL <ANYMAIL_MANDRI
 to the same public webhook URL you gave Mandrill.

 Mandrill will report these Anymail :attr:`~anymail.signals.AnymailTrackingEvent.event_type`\s:
-sent, rejected, deferred, bounced, opened, clicked, complained, unsubscribed. Mandrill does
+sent, rejected, deferred, bounced, opened, clicked, complained, unsubscribed, inbound. Mandrill does
 not support delivered events. Mandrill "whitelist" and "blacklist" change events will show up
 as Anymail's unknown event_type.

@@ -235,8 +239,18 @@ a `dict` of Mandrill event fields, for a single event. (Although Mandrill calls
 webhooks with batches of events, Anymail will invoke your signal receiver separately
 for each event in the batch.)

-.. _Mandrill's instructions:
+.. _tracking event webhooks:
    https://mandrill.zendesk.com/hc/en-us/articles/205583217-Introduction-to-Webhooks
+.. _inbound route webhooks:
+    https://mandrill.zendesk.com/hc/en-us/articles/205583197-Inbound-Email-Processing-Overview
+
+
+.. versionchanged:: 1.3
+    Earlier Anymail releases used :samp:`.../anymail/mandrill/{tracking}/` as the tracking
+    webhook url. With the addition of inbound handling, Anymail has dropped "tracking"
+    from the recommended url for new installations. But the older url is still
+    supported. Existing installations can continue to use it---and can even install it
+    on a Mandrill *inbound* route to avoid issuing a new webhook key.


 .. _migrating-from-djrill:
@@ -298,8 +312,15 @@ Changes to settings
  Use :setting:`ANYMAIL_MANDRILL_WEBHOOK_KEY` instead.

 ``DJRILL_WEBHOOK_URL``
-  Use :setting:`ANYMAIL_MANDRILL_WEBHOOK_URL`, or eliminate if
-  your Django server is not behind a proxy that changes hostnames.
+  Often no longer required: Anymail can normally use Django's
+  :meth:`HttpRequest.build_absolute_uri <django.http.HttpRequest.build_absolute_uri>`
+  to figure out the complete webhook url that Mandrill called.
+
+  If you are experiencing webhook authorization errors, the best solution is to adjust
+  your Django :setting:`SECURE_PROXY_SSL_HEADER`, :setting:`USE_X_FORWARDED_HOST`, and/or
+  :setting:`USE_X_FORWARDED_PORT` settings to work with your proxy server.
+  If that's not possible, you can set :setting:`ANYMAIL_MANDRILL_WEBHOOK_URL` to explicitly
+  declare the webhook url.


 Changes to EmailMessage attributes
@@ -393,14 +414,17 @@ parameters is that most logging and analytics systems are aware of the
 need to keep auth secret.)

 Anymail replaces `djrill.signals.webhook_event` with
-`anymail.signals.tracking` for delivery tracking events.
-(It does not currently handle inbound message webhooks.)
+`anymail.signals.tracking` for delivery tracking events,
+and `anymail.signals.inbound` for inbound events.
 Anymail parses and normalizes
-the event data passed to the signal receiver: see :ref:`event-tracking`.
+the event data passed to the signal receiver: see :ref:`event-tracking`
+and :ref:`inbound`.

 The equivalent of Djrill's ``data`` parameter is available
 to your signal receiver as
 :attr:`event.esp_event <anymail.signals.AnymailTrackingEvent.esp_event>`,
 and for most events, the equivalent of Djrill's ``event_type`` parameter
 is `event.esp_event['event']`. But consider working with Anymail's
-normalized :class:`~anymail.signals.AnymailTrackingEvent` instead.
+normalized :class:`~anymail.signals.AnymailTrackingEvent` and
+:class:`~anymail.signals.AnymailInboundEvent` instead for easy portability
+to other ESPs.
--- a/docs/esps/postmark.rst
+++ b/docs/esps/postmark.rst
@@ -201,3 +201,26 @@ a `dict` of Postmark `delivery <http://developer.postmarkapp.com/developer-deliv
 or `open <http://developer.postmarkapp.com/developer-open-webhook.html>`_ webhook data.

 .. _Postmark account settings: https://account.postmarkapp.com/servers
+
+
+.. _postmark-inbound:
+
+Inbound webhook
+---------------
+
+If you want to receive email from Postmark through Anymail's normalized :ref:`inbound <inbound>`
+handling, follow Postmark's `Inbound Processing`_ guide to configure
+an inbound server pointing to Anymail's inbound webhook.
+
+The InboundHookUrl setting will be:
+
+   :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/postmark/inbound/`
+
+     * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
+     * *yoursite.example.com* is your Django site
+
+Anymail handles the "parse an email" part of Postmark's instructions for you, but you'll
+likely want to work through the other sections to set up a custom inbound domain, and
+perhaps configure inbound spam blocking.
+
+.. _Inbound Processing: https://postmarkapp.com/developer/user-guide/inbound
--- a/docs/esps/sendgrid.rst
+++ b/docs/esps/sendgrid.rst
@@ -302,6 +302,37 @@ for each event in the batch.)
 .. _Sendgrid event: https://sendgrid.com/docs/API_Reference/Webhooks/event.html


+.. _sendgrid-inbound:
+
+Inbound webhook
+---------------
+
+If you want to receive email from SendGrid through Anymail's normalized :ref:`inbound <inbound>`
+handling, follow SendGrid's `Inbound Parse Webhook`_ guide to set up
+Anymail's inbound webhook.
+
+The Destination URL setting will be:
+
+   :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/sendgrid/inbound/`
+
+     * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
+     * *yoursite.example.com* is your Django site
+
+Be sure the URL has a trailing slash. (SendGrid's inbound processing won't follow Django's
+:setting:`APPEND_SLASH` redirect.)
+
+If you want to use Anymail's normalized :attr:`~anymail.inbound.AnymailInboundMessage.spam_detected` and
+:attr:`~anymail.inbound.AnymailInboundMessage.spam_score` attributes, be sure to enable the "Check
+incoming emails for spam" checkbox.
+
+You have a choice for SendGrid's "POST the raw, full MIME message" checkbox. Anymail will handle
+either option (and you can change it at any time). Enabling raw MIME will give the most accurate
+representation of *any* received email (including complex forms like multi-message mailing list
+digests). But disabling it *may* use less memory while processing messages with many large attachments.
+
+.. _Inbound Parse Webhook:
+   https://sendgrid.com/docs/Classroom/Basics/Inbound_Parse_Webhook/setting_up_the_inbound_parse_webhook.html
+

 .. _sendgrid-v3-upgrade:

--- a/docs/esps/sparkpost.rst
+++ b/docs/esps/sparkpost.rst
@@ -220,3 +220,23 @@ The esp_event is the raw, `wrapped json event structure`_ as provided by SparkPo
    https://support.sparkpost.com/customer/portal/articles/1976204-webhook-event-reference
 .. _wrapped json event structure:
    https://support.sparkpost.com/customer/en/portal/articles/2311698-comparing-webhook-and-message-event-data
+
+
+.. _sparkpost-inbound:
+
+Inbound webhook
+---------------
+
+If you want to receive email from SparkPost through Anymail's normalized :ref:`inbound <inbound>`
+handling, follow SparkPost's `Enabling Inbound Email Relaying`_ guide to set up
+Anymail's inbound webhook.
+
+The target parameter for the Relay Webhook will be:
+
+   :samp:`https://{random}:{random}@{yoursite.example.com}/anymail/sparkpost/inbound/`
+
+     * *random:random* is an :setting:`ANYMAIL_WEBHOOK_AUTHORIZATION` shared secret
+     * *yoursite.example.com* is your Django site
+
+.. _Enabling Inbound Email Relaying:
+   https://www.sparkpost.com/docs/tech-resources/inbound-email-relay-webhook/