On January 8, we discovered that we had failed to deliver multiple voicemail messages from the prior few weeks. We resent these messages on January 10. Here's what happened, and why it won't happen again.
To improve the reliability and speed of our voicemail and transcription delivery, we have been working on a new collection and delivery system. Before we had the entire voicemail system set up and running, we accidentally deployed the recording component to one of our servers on December 17th. That server thus stopped delivering voicemail on December 17th and didn't start until January 8th. No messages were lost, but they were not delivered in the prompt manner that you expect.
We monitor voicemail delivery in several ways. One of those is to watch for a count of messages that are awaiting delivery. This monitoring for the new recording was not in place yet, because the new recording method was not supposed to be running yet. Another way we monitor is to look at the number of voicemails we deliver daily. A large change in that number could be an indication of a problem. Because this issue coincided with Christmas and an overall reduced calling volume, there was a natural dip in the total number of voicemails sent. The messages affected by this issue were hidden by this dip.
During the window where messages were being recorded and not delivered on this server, just under 4% of all calls landed on this server. Not all calls result in a voicemail, so a very small percentage of all voicemail messages were delayed.
The missing components of the new voicemail system are now fully deployed, and we are rolling out the recording component to a small set of servers later this week and closely monitoring delivery performance. Over the following days, we will upgrade the entire set of servers.