Email Delays
Incident Report for SendGrid

Due to a backend service going down on 11/3/2015, the Marketing Campaigns frontend went down at 7:02am MT. Mail flow in both mail processing data centers was delayed beginning at 7:40am MT, and mail for Marketing Campaigns were effectively paused starting at 8:30am MT. By 8:31am MT, the Marketing Campaigns frontend was back up, Marketing Campaigns mail started sending again and transactional mail started processing again.

New transactional mail was no longer delayed starting at 10:28am for all customers, but 5 million messages took until 8am MT on 11/4/2015 to finish sending due to a spike of traffic in our West Coast data center.

SendGrid is working on rebalancing our capacity to help prevent future incidents like this.

Posted over 1 year ago. Nov 04, 2015 - 09:36 PDT

Resolved
Our Ops team has confirmed that all the delays have been resolved. No mail was lost during this time. We apologize for any inconvenience this has caused. If you have any questions or concerns, please reach out to our support team at support@sendgrid.com
Posted over 1 year ago. Nov 03, 2015 - 22:19 PDT
Update
We're continuing to monitor the one stubborn server, which currently has 2.8M messages.

To clarify, all new mail is flowing as normal; these 2.8M are left form the morning delay.
Posted over 1 year ago. Nov 03, 2015 - 14:40 PDT
Update
The vast majority of the delayed mail has completely dequeued, and all incoming mail is being processed as normal.
One of our servers still has a queue built up, and we're working to get that flowing faster.
Posted over 1 year ago. Nov 03, 2015 - 11:24 PDT
Monitoring
Mail flow is rapidly dequeueing. Delayed mail is being processed, and new incoming mail should not be delayed.
We're still monitoring, and will resolve once queues are maintaining normal.
Posted over 1 year ago. Nov 03, 2015 - 09:53 PDT
Identified
We've identified and corrected the issue.
Mail is currently dequeueing, so delays are still processing.
We will update once the queues are back to normal.
Posted over 1 year ago. Nov 03, 2015 - 08:33 PDT
Investigating
We are currently investigating reports of email delays affecting some customers. We have our Operations team looking at this with the highest priority, and we will update shortly with more information on a resolution.
Posted over 1 year ago. Nov 03, 2015 - 06:47 PDT