Amazon SES Undetermined Bounce Handling – Production Use Case

Amazon SES Bounce

Amazon SES Bounce

Using Amazon SES is really cheap for sending your marketing newsletters. While using Amazon SES we need to make sure we are sending emails to only valid email ids. Bounced or invalid emails should not be retry email sending. Amazon SES can penalize by blocking your account if you are sending marketing newsletters to invalid email ids. Also it can badly affect your SES email delivery reputation. If you are looking for a tool to improve your Amazon SES spam reputation, below article will help you:

If you are a beginner with Amazon Web Services, you can get started with below articles:

Amazon SES is very strict regarding hard bounces. This is also called as permanent bounces. Permanent bounce response should be marked as hard bounced and should not be retry sending any email. As per Amazon SES best practices, your bounce rate should be less than 5%. However we faced issue with one particular Amazon SES bounce response for a significant number of users. If you are not aware, for each email sent through Amazon SES, response is available in SQS if configured. Below types of SQS responses are possible.

  • Success Email Delivery
  • Email Bounced
  • Email Marked as Spam

Above responses can have different sub types. Amazon SES all bounce types and sub types can be referred here. I am going to discuss how we managed to solve a hard bounce issue without affecting our Amazon SES reputation.

Amazon SES Undetermined Bounce Issue?

Amazon SES Undetermined Bounce Type & Sub Type

Amazon SES Undetermined Bounce Type & Sub Type

Because of this issue we were not able to send emails to a significant number of users. Below are series of events happened which alerted us about this issue.

  • Amazon SES send response in SQS through SNS for every email which is sent.
  • We were processing the response in an asynchronous process. This was done by feedback daemon. Feedback daemon mark emails valid, bounced or complain based on this response.
  • One particular day internal user complained that he is not receiving any email.
  • After checking the logs we found that he was marked hard bounced. That’s why he was not receiving emails. Digging deeper we found that his last sent email response was:
    • bounceType => “undetermined
    • bounceSubType => “undetermined
  • We mark users with bounce status undetermined as hard bounced and don’t send them any email in future.
  • We also found that there are more similar users. Around 1200 of which were marked undetermined hard bounced. A lot of them were internal users. Most of them were valid emails. This was really strange. This means we might be missing real users from receiving our marketing newsletter.

Amazon SES Undetermined Bounce Problem

A significant number of emails were marked Amazon SES undetermined bounce status. But we found that lot of them are valid email ids. I also asked about this on reddit and got helpful feedbacks.

Reddit Question about Amazon SES Undetermined Bounce Status

Reddit Question about Amazon SES Undetermined Bounce Status

Current Undetermined Bounce Logic

Amazon SES response status “undetermined” was marked hard bounced in our system. No further mail is retried to hardbounce users.

Amazon SES Undetermined Sample JSON Response

Understanding Undetermined Bounce Issue

Below question came to our mind:

  • Why ‘undetermined‘ response is received even for valid emails? Instead success “delivery” response should come.
  • Should bounce response with ‘undetermined‘ status be marked as hardbounce? Is the logic correct?
  • Can we white list these users and retry sending them email? While doing that we needed to make sure that our overall bounce status doesn’t go above threshold(5%).
  • Can we treat them as softbounce? Our current softbounce logic is we retry sending mail to these users incrementally(based on date time). e.g. after 1st soft bounce user will be sent email after 10 days. After 2nd soft bounce we send after 20 days.

We also discussed with Amazon SES support team asking above questions with a sample message id. Key learnings after the discussion:

  • Amazon SES does its best to determine the reason of a bounce. This allows customer better understand why a message couldn’t be delivered or why it generated a bounce. In some cases, the nature of the bounce cannot be determined depending on the message content, or how specific data may have been encoded.
  • The primary reason for customer to get a bounce is usually invalid addresses. The second most common is usually an out-of-the-office (OOTO) message is sent to a person out of the office via Amazon SES.
    • The message is delivered to the mailbox.
    • The mail server handling the mailbox replies to the message using the “Return-Path” in a separate email.
    • Usually, the mailbox owner may have customized the notification.
    • The message is received by Amazon SES.
    • Amazon SES informs the customer of the received notification. As you can see above, the message is still accepted. But because an automated agent replied back to Amazon SES, this is why you have received the notification even for a valid email address.
    • As to why the notification is marked as “undetermined” is probably in relation of the message content.

Bellow is sample email metadata and headers.

    • In this case, the message “Out of Office” is  mixed with the original subject. But Amazon SES couldn’t assert with precision the nature of this message. This resulted in the “undetermined” status.
  • We had verified our domain and also enabled the Amazon SES notification via SNS which is great. But we had turned off the “forwarding” for all our  notification. This “Forwarding” feature allows you to receive the content of the notifications. It’s not so useful to handle bounces and complaints automatically. It’s really useful should we wish to manually review the content of a notification.
    • This way, if a notification is marked as “undetermined”, we could pull back a notification from the sender’s mailbox and further analyze the reason.
    • This would then help us determine if you should reach this mailbox again.
    • However we were not in mood to handle these users manually by enabling email forwarding since that will be too pain.

Key Learnings

  • As per response from Amazon SES support team, looks like we can retry sending email to users having “undetermined” status.
  • When a bounce occurs and is marked as “undetermined”, it will not count towards your overall bounce rate. It should be safe enough for us to reach this email address again.
  • We discussed to change our logic and treat this users as soft bounce instead of hard bounce. Before coming to any conclusion and treat Amazon SES undetermined bounce as softbounce we wanted to verify if our approach is correct.

Step1 – Solving the Amazon SES Undetermined Bounce Issue

In the first iteration of solving this issue, we followed below steps:

  • We enabled 200 users and made them normal users. We white listed them for our next campaign.
  • This means marketing newsletter will be sent to these users.
  • Based on these users Amazon SES response, we will decide next step.
  • We are not changing any logic. We will change undetermined bounce logic only after white listing all above users and monitoring their bounce results.

Step1 – Monitoring SQS Response Results

After sending these users newsletter and monitoring their Amazon SES response. Below are status and sub status of SQS responses after sending 200 undetermined status emails.

STATUSSUB STATUSCOUNT
Successfully DeliveredGeneral150
Hard bouncedGeneral4
UndeterminedUndetermined38
Soft bouncedGeneral8

As you can see, only 2%(4) users were marked hard bounce. Again 38 were undetermined. But these 38 doesn’t count in Amazon SES bounce rate. This was the first positive sign towards our end goal.

Step2 – Solving the Amazon Undetermined Bounce Issue

In the next iteration, we followed below steps:

  • Enabled next set of 500 undetermined hardbounce users. We made them normal users.
  • Sent them marketing newsletter.
  • Decided to change Amazon SES undetermined bounce logic after monitoring step2 SQS responses.

Step2 – Monitoring SQS Response Results

Again number of hard bounced users count were very less ~= 2%. After looking at this data we decided to change our Amazon SES undetermined logic.

Conclusion – Changing the Logic

After analyzing above production data and recommendation from Amazon SES support team, we changed our logic to below:

  • In the new logic Amazon SES undetermined bounce SQS response users will be treated as “softbounce” .
  • As explained above, softbounce users will be resent emails as per logic described above.
  • This was a significant logic change since we are very strict towards our Amazon SES hard bounce rate. We monitor them on daily basis. However we are extremely happy that we analyzed production data and took the right step.

Hopefully this use case will help you analyze your email sending activity and take proper decisions. You can read and learn more about AWS services by clicking below link:

Thank you for reading my article. If you face any problem or having any doubts, let me know in comments below. If you like my article please like our Facebook page and also follow us on Twitter so that you get regular updates. For regular updates you can also subscribe to hackpundit.com with your email.