Email Delays
Incident
Report for SendGrid
Postmortem
Due to a backend service going down on 11/3/2015, the Marketing Campaigns frontend went down at 7:02am MT. Mail flow in both mail processing data centers was delayed beginning at 7:40am MT, and mail for Marketing Campaigns were effectively paused starting at 8:30am MT. By 8:31am MT, the Marketing Campaigns frontend was back up, Marketing Campaigns mail started sending again and transactional mail started processing again.
New transactional mail was no longer delayed starting at 10:28am for all customers, but 5 million messages took until 8am MT on 11/4/2015 to finish sending due to a spike of traffic in our West Coast data center.
SendGrid is working on rebalancing our capacity to help prevent future incidents like this.
Resolved
Our Ops team has confirmed that all the delays have been resolved. No mail was lost during this time. We apologize for any inconvenience this has caused. If you have any questions or concerns, please reach out to our support team at support@sendgrid.com
Update
We're continuing to monitor the one stubborn server, which currently has 2.8M messages.
To clarify, all new mail is flowing as normal; these 2.8M are left form the morning delay.
To clarify, all new mail is flowing as normal; these 2.8M are left form the morning delay.
Update
The vast majority of the delayed mail has completely dequeued, and all incoming mail is being processed as normal.
One of our servers still has a queue built up, and we're working to get that flowing faster.
One of our servers still has a queue built up, and we're working to get that flowing faster.
Monitoring
Mail flow is rapidly dequeueing. Delayed mail is being processed, and new incoming mail should not be delayed.
We're still monitoring, and will resolve once queues are maintaining normal.
We're still monitoring, and will resolve once queues are maintaining normal.
Identified
We've identified and corrected the issue.
Mail is currently dequeueing, so delays are still processing.
We will update once the queues are back to normal.
Mail is currently dequeueing, so delays are still processing.
We will update once the queues are back to normal.
Investigating
We are currently investigating reports of email delays affecting some customers. We have our Operations team looking at this with the highest priority, and we will update shortly with more information on a resolution.