We currently face an partial outage of all our services

Incident Report for Magicline

Postmortem

Incident Summary

On 4 September 2025, at approximately 13:30 CEST, we detected failures in one of our services, followed shortly by similar issues in others. The incident was caused by an error during a software update that created inconsistencies between the application and the database. As a result, newly deployed services were unable to start, and existing ones could not process requests.

Impact on Users

All customers and all services where effected.

Our Response

13:09 CEST the faulty deployment has been started
13:30 CEST first alert was triggered due to a high number of 500 error codes in the device-manager-api
13:35 CEST investigating of the issue was started, a meeting with all involved has been created
13:40 CEST issue was identified
14:00 CEST fix was applied and incident status was changed to monitoring
14:36 CEST the incident was resolved

Resolution

The incident was resolved by fixing the database migration and running it again.

Lessons Learned

The incident was related to a human error during the review process. To prevent recurrence, engineers will receive additional training on proper database migration procedures, and the importance of thorough 4-eye checks will be re-emphasized.

Posted Sep 09, 2025 - 14:42 CEST

Resolved

From 2025-09-04 13:25 CEST to 2025-09-04 14:00 CEST we faced a partial outage of all our systems.
Our Engineering team has resolved the main issue, but will continue on implementing further measures, to prevent that such incidents are happening again.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Posted Sep 04, 2025 - 14:36 CEST

Update

We are continuing to monitor for any further issues.

Posted Sep 04, 2025 - 14:04 CEST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 04, 2025 - 14:02 CEST

Identified

The issue has been identified and a fix is being implemented.

Posted Sep 04, 2025 - 13:55 CEST

Update

We are experiencing unexpected service downtime affecting a portion of our platform users. Immediate efforts are underway to diagnose and resolve the issue as quickly as possible. We will keep you updated as we progress.

Posted Sep 04, 2025 - 13:51 CEST

Update

We are continuing to investigate this issue.

Posted Sep 04, 2025 - 13:47 CEST

Investigating

We are currently investigating this issue.

Posted Sep 04, 2025 - 13:44 CEST

This incident affected: Web Application, Device Manager API, Open API, and MySports API.