In Amazon SES webhook views (tracking and inbound):
- Close boto3 clients after use. (Not strictly required, but doesn't hurt.
Amazon SES backend was already doing this.)
- Break out some webhook functionality to simplify subclassing.
(E.g., to handle S3 object encryption through outside tooling, as
AWS hasn't released a Python version of their S3 encryption client.)
- Fix signature checking to avoid false validation errors
on webhook payloads including `/` (including all "clicked"
and most "opened" events). And in general, avoid depending
on specific details of Unisender Go's JSON serialization.
(Fixes #398.)
- Handle "use single event" webhook option (which has a different
payload format).
- Verify basic auth when Anymail's WEBHOOK_SECRET is used.
(This is optional for Unisender Go, since payloads are signed,
but it needs to be checked when enabled.)
- Treat "soft_bounced" events as "deferred" rather than "bounced",
since they will be retried later.
- Update validation error to reference Project ID if the webhook
is configured for a specific project.
- Expose Unisender Go's delivery_status code and unsubscribe form
comment as Anymail's normalized event.description.
- Update webhook tests based on actual payloads and add several
missing tests.
- Update docs to clarify webhook use with Unisender Go projects.
Add support for Brevo's new "Complained," "Error" and
"Loaded by proxy" events in Brevo tracking webhook.
Closes#385.
---------
Co-authored-by: Mike Edmunds <medmunds@gmail.com>
(See previous commit.)
- Maintain deprecated compatibility
versions on the old names/URLs.
(Split into separate commit
to make renamed files more
obvious.)
- Replace "SendinBlue" with "Brevo"
throughout the code.
- Maintain deprecated compatibility
versions on the old names/URLs.
(Split into separate commit
to make renamed files more
obvious.)
- Update docs to reflect change,
provide migration advice.
- Update integration workflow.
Replace generic `combine` with
specific `merge_dicts_deep`,
`merge_dicts_shallow`,
`merge_dicts_one_level` or
`concat_lists`, depending on
appropriate behavior for each
message attribute.
Fixes merging global `SEND_DEFAULTS`
with message `esp_extra` for ESP APIs
that use nested payload structures.
And clarifies intent for other properties.
- Support Postmark's RawEmail option;
recommend it in docs
- Handle Bcc when provided by Postmark
- Obtain `envelope_sender` from Return-Path info
Postmark now adds, rather than parsing Received-SPF
Related:
- Add `AnymailInboundMessage.bcc` convenience prop
- Test against full Postmark "check" inbound payloads
(which don't match their docs or real inbound payloads)
- Don't warn about receiving "check" payload
Switch to pyproject.toml packaging, using hatchling.
- Replace all uses of setup.py with updated equivalent
- BREAKING: Change extra name `amazon_ses` to
`amazon-ses`, to comply with Python packaging
name normalization
- Use hatch custom build hook to freeze version number
in readme (previously custom setup.py code)
- Move separate requirements for dev, docs, tests
into their own requirements.txt files
- Fix AnymailImproperlyInstalled to correctly refer
to package extra name
- Update testing documentation
- Update docs readme rendering to match PyPI
(and avoid setup.py)
- In tox tests, use isolated builds and update pip
- Remove AUTHORS.txt (it just referred to GitHub)
Postmark's "test" button in their inbound settings
posts data with attachments that don't match their docs or
actual inbound behavior. Accept that and issue a warning.
Closes#304
Workaround for Django multipart/form-data limitation
where certain attachment filenames cause fields to be dropped
or to end up in request.POST rather than request.FILES.
Handle the MultiValueDictKeyError in inbound webhooks when
this has occurred. Also update docs to recommend avoiding
the problem by using Mailgun and SendGrid's "raw MIME" options.
Also handle reported cases of empty, duplicate keys in Mailgun's
content-id-map.
Fixes#272
Allow inbound and tracking webhooks using SNS topics from any AWS region.
The topic subscription must be confirmed in the topic's own region (not
the boto3 default), determined by examing the topic's ARN.
Fixes#235
Minimum supported versions are now Django 2.0, Python 3.5.
This touches a lot of code, to:
* Remove obsolete portability code and workarounds
(six, backports of email parsers, test utils, etc.)
* Use Python 3 syntax (class defs, raise ... from, etc.)
* Correct inheritance for mixin classes
* Fix outdated docs content and links
* Suppress Python 3 "unclosed SSLSocket" ResourceWarnings
that are beyond our control (in integration tests due to boto3,
python-sparkpost)
Fix a crash or text-mangling issue when an inbound message
uses a charset other than utf-8 for its text or html body,
and SendGrid's "post raw" inbound parse option is *not*
enabled.
Update docs to recommend "post raw" option.
Fixes#187
Additional changes related to SendinBlue improvements in #158:
* Support multiple tags in webhooks (closes#162)
* Remove additional outdated template code in backend
* Update integration tests
* Update docs and changelog; note breaking changes as discussed in #161
Map Mailgun severity: temporary failure event to Anymail "deferred" event, to distinguish it from severity: permanent failures which will show up as Anymail "bounced".
Also remap Mailgun reason: generic failure to Anymail "other" reject reason (rather than "bounced").
Closes#130
SendGrid does not always correctly provide the sent Message-ID header value
to a tracking webhook's smtp-id field, making it unreliable to use for Anymail's
`message_id`.
Instead, generate a UUID `message_id` for Anymail tracking, and pass it from
send to webhooks in SendGrid custom args as anymail_id.
Webhooks will fall back to smtp-id for compatibility with previously-sent
messages that didn't have an anymail_id custom arg.
Fixes#108
Delay raising AnymailImproperlyInstalled from webhooks.amazon_ses
until an SES webhook view is instantiated. Allows anymail.urls
to import webhooks.amazon_ses without error.
Fixes#103
Simplify Postmark tracking webhook code by using new "RecordType"
field introduced with Postmark "modular webhooks". (Rather than
looking for fields that are probably only in certain events.)
Also issue configuration error on inbound url installed as tracking
webhook (and vice versa).
Drop support for the WEBHOOK_AUTHORIZATION setting deprecated in v1.4.
Only the WEBHOOK_SECRET replacement is allowed now.
Most Django management commands will now issue a system check error
if the old name is still used in settings.py
This fixes a low severity security issue affecting Anymail v0.2--v1.3.
Django error reporting includes the value of your Anymail
WEBHOOK_AUTHORIZATION setting. In a properly-configured deployment,
this should not be cause for concern. But if you have somehow exposed
your Django error reports (e.g., by mis-deploying with DEBUG=True or by
sending error reports through insecure channels), anyone who gains
access to those reports could discover your webhook shared secret. An
attacker could use this to post fabricated or malicious Anymail
tracking/inbound events to your app, if you are using those Anymail
features.
The fix renames Anymail's webhook shared secret setting so that
Django's error reporting mechanism will [sanitize][0] it.
If you are using Anymail's event tracking and/or inbound webhooks, you
should upgrade to this release and change "WEBHOOK_AUTHORIZATION" to
"WEBHOOK_SECRET" in the ANYMAIL section of your settings.py. You may
also want to [rotate the shared secret][1] value, particularly if you
have ever exposed your Django error reports to untrusted individuals.
If you are only using Anymail's EmailBackends for sending email and
have not set up Anymail's webhooks, this issue does not affect you.
The old WEBHOOK_AUTHORIZATION setting is still allowed in this release,
but will issue a system-check warning when running most Django
management commands. It will be removed completely in a near-future
release, as a breaking change.
Thanks to Charlie DeTar (@yourcelf) for responsibly reporting this
security issue through private channels.
[0]: https://docs.djangoproject.com/en/stable/ref/settings/#debug
[1]: https://anymail.readthedocs.io/en/1.4/tips/securing_webhooks/#use-a-shared-authorization-secret
Anymail's webhook validation was vulnerable to a timing attack.
An attacker could have used this to recover your WEBHOOK_AUTHORIZATION
shared secret, potentially allowing them to post fabricated or malicious
email tracking events to your app.
There have not been any reports of attempted exploit in the wild. (The
vulnerability was discovered through code review.) Attempts would be
visible in http logs as a very large number of 400 responses on
Anymail's webhook urls, or in Python error monitoring as a very large
number of AnymailWebhookValidationFailure exceptions.
If you are using Anymail's webhooks, you should upgrade to this release.
In addition, you may want to rotate to a new WEBHOOK_AUTHORIZATION
secret ([docs](http://anymail.readthedocs.io/en/stable/tips/securing_webhooks/#use-a-shared-authorization-secret)),
particularly if your logs indicate attempted exploit.
Mailgun merges user-variables (metadata) into the webhook post data
interspersed with the actual event params. This can lead to ambiguity
interpreting post data.
To extract metadata from an event, Anymail had been attempting to avoid
that ambiguity by instead using X-Mailgun-Variables fields found in the
event's message-headers param. But message-headers isn't included in
some tracking events (opened, clicked, unsubscribed), resulting in
empty metadata for those events. (#76)
Also, conflicting metadata keys could confuse Anymail's Mailgun event
parsing, leading to unexpected values in the normalized event. (#77)
This commit:
* Cleans up Anymail's tracking webhook to be explicit about which
multi-value params it uses, avoiding conflicts with metadata keys.
Fixes#77.
* Extracts metadata from post params for opened, clicked and
unsubscribed events. All unknown event params are assumed to be
metadata. Fixes#76.
* Documents a few metadata key names where it's impossible (or likely
to be unreliable) for Anymail to extract metadata from the post data.
For reference, the order of params in the Mailgun's post data *appears*
to be (from live testing):
* For the timestamp, token and signature params, any user-variable with
the same name appears *before* the corresponding event data.
* For all other params, any user-variable with the same name as a
Mailgun event param appears *after* the Mailgun data.
