Work around Python 2 email.parser.Parser bug handling RFC5322 folded
headers. Fixes problems where long headers in inbound mail (e.g.,
Subject) get truncated or have unexpected spaces.
This change also updates AnymailInboundMessage.parse_raw_mime to use
the improved "default" email.policy on Python 3 (rather than the
default "compat32" policy). This likely fixes several other parsing
bugs that will still affect code running on Python 2.
Improves inbound parsing for all ESPs that provide raw MIME email.
(Mailgun, Mandrill, SendGrid, SparkPost)
* Set up tox for testing supported Django/Python combinations
* Also include tox env for checking and building docs
* Use tox-travis for Travis CI integration
* Add tests against Django master
* Document building docs and running tests with tox
Using undeliverable @example.com recipient addresses leads some ESPs
to flag the Anymail test accounts. Switch all live integration tests
to mailinator.com recipients (unless they were already using the ESP's
own "test sink" addresses).
* Don't send *quite* so many emails during live integration tests.
(Our test account is throttled to 40/hour.)
* Relax message_id check in integration tests. SendinBlue appears
to use both @smtp-relay.mail.fr and @smtp-relay.sendinblue.com
Message-IDs.
* Note requirement for HTML message body in docs.
Drop support for the WEBHOOK_AUTHORIZATION setting deprecated in v1.4.
Only the WEBHOOK_SECRET replacement is allowed now.
Most Django management commands will now issue a system check error
if the old name is still used in settings.py
Django's SMTP EmailBackend allows spoofing the To header by setting
`message.extra_headers["To"]`` different from `message.to`.
No current Anymail ESP supports this. Treat extra_headers["To"] as
an unsupported ESP feature, to flag attempts to use it.
Also document Anymail's special header handling that replicates
Django's SMTP EmailBackend behavior.
New EmailMessage attribute `envelope_sender` controls ESP's sender,
sending domain, or return path where supported:
* Mailgun: overrides SENDER_DOMAIN on individual message
(domain portion only)
* Mailjet: becomes `Sender` API param
* Mandrill: becomes `return_path_domain` API param
(domain portion only)
* SparkPost: becomes `return_path` API param
* Other ESPs: not believed to be supported
Also support undocumented Django SMTP backend behavior, where envelope
sender is given by `message.from_email` when
`message.extra_headers["From"]` is set. Fixes#91.
Django allows setting the reply address with either message.reply_to
or message.extra_headers["Reply-To"]. If both are supplied, the extra
headers version takes precedence. (See EmailMessage.message().)
Several Anymail backends had duplicate logic to handle conflicting
properties. Move that logic into the base Payload.
(Also prepares for common handling of extra_headers['From'], later.)
Related changes:
* Use CaseInsensitiveDict for processing extra_headers.
This is potentially a breaking change, but any code that was trying
to send multiple headers differing only in case was likely already
broken. (Email header field names are case-insensitive, per RFC-822.)
* Handle CaseInsensitiveDict in RequestsPayload.serialize_json().
(Several backends had duplicate code for handling this, too.)
* Fixes SparkPost backend, which had been incorrectly treating
message.reply_to and message.extra_headers['Reply-To'] differently.
Add support for sending transactional email through SendinBlue. (Thanks to @RignonNoel.)
Partially implements #84. (Tracking webhooks will be a separate PR. SendinBlue doesn't support inbound handling.)
This fixes a low severity security issue affecting Anymail v0.2--v1.3.
Django error reporting includes the value of your Anymail
WEBHOOK_AUTHORIZATION setting. In a properly-configured deployment,
this should not be cause for concern. But if you have somehow exposed
your Django error reports (e.g., by mis-deploying with DEBUG=True or by
sending error reports through insecure channels), anyone who gains
access to those reports could discover your webhook shared secret. An
attacker could use this to post fabricated or malicious Anymail
tracking/inbound events to your app, if you are using those Anymail
features.
The fix renames Anymail's webhook shared secret setting so that
Django's error reporting mechanism will [sanitize][0] it.
If you are using Anymail's event tracking and/or inbound webhooks, you
should upgrade to this release and change "WEBHOOK_AUTHORIZATION" to
"WEBHOOK_SECRET" in the ANYMAIL section of your settings.py. You may
also want to [rotate the shared secret][1] value, particularly if you
have ever exposed your Django error reports to untrusted individuals.
If you are only using Anymail's EmailBackends for sending email and
have not set up Anymail's webhooks, this issue does not affect you.
The old WEBHOOK_AUTHORIZATION setting is still allowed in this release,
but will issue a system-check warning when running most Django
management commands. It will be removed completely in a near-future
release, as a breaking change.
Thanks to Charlie DeTar (@yourcelf) for responsibly reporting this
security issue through private channels.
[0]: https://docs.djangoproject.com/en/stable/ref/settings/#debug
[1]: https://anymail.readthedocs.io/en/1.4/tips/securing_webhooks/#use-a-shared-authorization-secret
* Un-hardcode status message_id in test backend
For the test EmailBackend, get message ID's based on array position in
`mail.outbox`, so that tests can predict the message ID.
* Add a console backend for use in development
Adds an EmailBackend derived from both Anymail's test backend and
Django's console backend, to provide anymail statuses and signal
handling while printing messages to the console. For use during
development on localhost.
Closes#87
Use a default timeout of 30 seconds for all requests, and add a
REQUESTS_TIMEOUT Anymail setting to override.
(I'm making a judgement call that this is not a breaking change in the
real world, and not bumping the major version. Theoretically, it could
affect you if your network somehow takes >30s to connect to your ESP,
but eventually succeeds. If so, set REQUESTS_TIMEOUT to None to restore
the earlier behavior.)
Fixes#80.
get_api_call_arg had incorrectly returned None if a kwarg was passed
to the mocked function with a False-y value (e.g., [] or {})
get_api_call_json had only considered data param, ignoring json param
requests added a while back
Within an EmailAddress (previously ParsedEmail object), properties
now match Python 3.6 email.headerregistry.Address naming:
* .email --> .addr_spec
* .name --> .display_name
* .localpart --> .username
(Completes work started in 386668908423d1d4eade90cf7a21a546a1e96514;
this updates remaining uses of old names and removes them.)
Mailgun merges user-variables (metadata) into the webhook post data
interspersed with the actual event params. This can lead to ambiguity
interpreting post data.
To extract metadata from an event, Anymail had been attempting to avoid
that ambiguity by instead using X-Mailgun-Variables fields found in the
event's message-headers param. But message-headers isn't included in
some tracking events (opened, clicked, unsubscribed), resulting in
empty metadata for those events. (#76)
Also, conflicting metadata keys could confuse Anymail's Mailgun event
parsing, leading to unexpected values in the normalized event. (#77)
This commit:
* Cleans up Anymail's tracking webhook to be explicit about which
multi-value params it uses, avoiding conflicts with metadata keys.
Fixes#77.
* Extracts metadata from post params for opened, clicked and
unsubscribed events. All unknown event params are assumed to be
metadata. Fixes#76.
* Documents a few metadata key names where it's impossible (or likely
to be unreliable) for Anymail to extract metadata from the post data.
For reference, the order of params in the Mailgun's post data *appears*
to be (from live testing):
* For the timestamp, token and signature params, any user-variable with
the same name appears *before* the corresponding event data.
* For all other params, any user-variable with the same name as a
Mailgun event param appears *after* the Mailgun data.
Update internal-use ParsedEmail to be more like Python 3.6+
email.headerregistry.Address, and remove "internal use only"
recommendation.
(Prep for exposing inbound email headers in a convenient form.
Old names remain temporarily available for internal use;
should clean up at some point.)
SendGrid requires extra headers and metadata values be strings.
Anymail has always coerced int and float; this treats Python 2's
`long` integer type the same.
Fixes#74
* Change Anymail's test EmailBackend to collect sent messages in
django.core.mail.outbox, same as Django's own locmem EmailBackend.
(So Django's test runner will automatically clear accumulated mail
between test cases.)
* Rename EmailMessage `test_response` attr to `anymail_test_response`
to avoid conflicts, and record merged ESP send params in
new `anymail_send_params` attr.
* Add docs
Closes#36.
[RFC-5322 allows](https://tools.ietf.org/html/rfc5322#section-3.6.2)
multiple addresses in the From header.
Django's SMTP backend supports this, as a single comma-separated
string (*not* a list of strings like the recipient params):
from_email='one@example.com, two@example.com'
to=['one@example.com', 'two@example.com']
Both Mailgun and SparkPost support multiple From addresses
(and Postmark accepts them, though truncates to the first one
on their end). For compatibility with Django -- and because
Anymail attempts to support all ESP features -- Anymail now
allows multiple From addresses, too, for ESPs that support it.
Note: as a practical matter, deliverability with multiple
From addresses is pretty bad. (Google outright rejects them.)
This change also reworks Anymail's internal ParsedEmail object,
and approach to parsing addresses, for better consistency with
Django's SMTP backend and improved error messaging.
In particular, Django (and now Anymail) allows multiple email
addresses in a single recipient string:
to=['one@example.com', 'two@example.com, three@example.com']
len(to) == 2 # but there will be three recipients
Fixes#60
Issue a better error message if message.reply_to
is set to a single string.
(Would also like to do this for to, cc, and bcc,
but Django core EmailMessage.recipients is called
and stumbles over thoses cases before Anymail's
backend gets involved.)
Fixes#57
Track change to Mailgun's events API, which
no longer includes message recipients.
(Only affected check for successful send
in the integration tests; Anymail doesn't
use the events API outside test code.)
Fixes#58
Only real problem is in json serialization tests:
Python 3.6 [changed][1] the json serialization
error message to use the object's class name
rather than its repr. E.g.:
"Decimal('19.99') is not JSON serializable"
becomes:
"Object of type 'Decimal' is not JSON serializable"
Update tests that looked for specific serialization
error message to just look for the word "Decimal"
instead. (Works with all Python versions.)
[1]: https://bugs.python.org/issue26623
* **Future breaking change:**
Rename all Anymail backends to just `EmailBackend`,
matching Django's naming convention.
(E.g., switch to "anymail.backends.mailgun.EmailBackend"
rather than "anymail.backends.mailgun.MailgunBackend".)
The old names still work, but will issue a DeprecationWarning
and will be removed in some future release.
(Apologies for this change; the old naming convention was
a holdover from Djrill, and I wanted consistency with
other Django EmailBackends before hitting 1.0.)
Fixes#49.
Mandrill's webhook signature calculation uses the
*exact url* Mandrill is posting to. If HTTP basic
auth is also used, that auth is included in the url.
Anymail was using Django's request.build_absolute_uri,
which doesn't include HTTP basic auth. Anymail now
includes the auth in the calculation, if it was present
in the request.
This should eliminate the need to use the
ANYMAIL_MANDRILL_WEBHOOK_URL override,
if Django's SECURE_PROXY_SSL_HEADER and
USE_X_FORWARDED_HOST (and/or
USE_X_FORWARDED_PROTO) settings are correct
for your server.
(The calculated url is now also included in
the validation failure error message, to aid
debugging.)
Fixes#48
SendGrid: update to v3 send API
**SendGrid:** **[possibly-breaking]** Update SendGrid backend to newer Web API v3. This should be a transparent change for most projects. Exceptions: if you use SendGrid username/password auth, esp_extra with "x-smtpapi", or multiple Reply-To addresses, please review the [porting notes](http://anymail.readthedocs.io/en/latest/esps/sendgrid/#sendgrid-v3-upgrade).
Closes#28
In BasePayload, ensure any Django ugettext_lazy
(or similar) are converted to real strings before
handing off to ESP code. This resolves problems where
calling code expects it can use lazy strings "anywhere",
but non-Django code (requests, ESP packages) don't
always handle them correctly.
* Add utils helpers for lazy objects (is_lazy, force_non_lazy*)
* Add lazy object handling to utils.Attachment
* Add lazy object handling converters to BasePayload attr
processing where appropriate. (This ends up varying by
the expected attribute type.)
Fixes#34.
Anymail was requiring Mandrill's webhook authentication key for the initial webhook url validation request from Mandrill, but Mandrill doesn't issue the key until that validation request succeeds.
* Defer complaining about missing Mandrill webhook key until actual event post.
* Document the double-deploy process required to set up Mandrill webhooks.
Fixes#46.
Add support for Postmark's recently-released [delivery tracking webhook] to Anymail's normailized status event handling. The existing Anymail tracking webhook URL can be copied to "Delivery webhook" in your Postmark outbound server settings.
Closes#45.
