Invoca Follower Database Issue - 6/11/24
Incident Report for Invoca
Resolved
Incident Timeframe: 18:30 ET (15:30 PT) June 11, 2024 - 10:20 ET (07:20 PT) June 12, 2024

Summary:
At approximately 6:30 pm ET (3:30 pm PT) on June 11th, a common follower database (a read-only copy of our database) was terminated. As a result, we needed to stand up a replacement follower database. This process took an extended amount of time (approx. 9 hours) and during this timeframe, there was an impact on a handful of our services which can be found below.

Impact:
For the incident timeframe above, a subset of Call Event Worker (CEW) messages failed. A CEW is a service that connects to databases to process events and messages from phone calls. The primary feature that was impacted by this was pre-transfer webhooks. However, there was a nominal impact on additional services.

Full List of Services That Experienced Degraded Performance:
- Pre-transfer Webhooks
- Real-time Webhooks
- Obtaining Demographic Data (ECP)
- Bundled Transfer Destination

Furthermore, the ability to identify if a call was a repeat caller was also impacted. If a call used real-time or pre-transfer webhooks to determine if the number was a repeat caller, the above failures would have affected our ability to determine if the call was a repeat.

Lastly, there was an impact on the processing of a subset of new configuration requests sent to our Network Integrations API (NIAPI)
Posted Jun 11, 2024 - 15:30 PDT