Follow-up: the impact was wider than we reported at the time.
Between 16:41 and 17:05 UTC, roughly 9,200 transactional emails were accepted by our API and then held before delivery, by up to 25 minutes. This came from a database issue caused by the action we took to clear the broadcast backlog. All of those emails were delivered and none were lost. Some customers also saw errors loading email details or lists during the same minutes.
Resolved
Follow-up: the impact was wider than we reported at the time.
Between 16:41 and 17:05 UTC, roughly 9,200 transactional emails were accepted by our API and then held before delivery, by up to 25 minutes. This came from a database issue caused by the action we took to clear the broadcast backlog. All of those emails were delivered and none were lost. Some customers also saw errors loading email details or lists during the same minutes.
Resolved
Broadcast sending is fully recovered and the backlog has cleared.
All broadcasts delayed during this incident have been sent or are sending normally. No emails were lost and no action is required.
Monitoring
Broadcast sending has recovered and the queued backlog is draining.
Broadcasts that were delayed are sending now. No emails were lost and no action is required: affected broadcasts continue automatically and do not need to be resent.
We are monitoring until the backlog is fully clear.
Identified
We are investigating delays in broadcast sending. Broadcasts started after 15:05 UTC may stay in a queued state and make little or no progress.
No emails have been lost. Affected broadcasts remain queued and will continue sending once the backlog clears.
We have identified the cause and are working on a mitigation. Next update in 30 minutes.
