Denali Service Announcement February 17, 2021

Good morning, Denali users. Our development team has identified an issue that is preventing users from connecting to the system. Some outbound calls may also have issues connecting. All hands are on deck to investigate. Updates will be posted here as they become available. 

UPDATE 9:50am Pacific Time: The initial issue has passed. Users should be able to connect again. Development is still all hands on deck to confirm system stability and analyze the cause. 

ROOT CAUSE ANALYSIS
At 9:20am PST, Dialsource engineering was notified by its internal monitoring systems of a high rate of failures for outbound calls. Upon investigation, it was discovered that a core telephony routing server was partially failing call routes at the database level. While these components are redundant, the nature of this failure was only partial and was not detected by the failover mechanisms. At 9:42am PST, the error was resolved and calls continued to complete as normal. Dialsource Engineering will be pushing a fix out this evening to address the underlying database issue and to improve the systems ability to detect this and similar failures such that it can automatically failover to backup equipment.