Encoding failures on AWS
Affected
Major outage from 3:45 AM to 4:53 AM
Major outage from 3:45 AM to 4:53 AM
Major outage from 3:45 AM to 4:53 AM
- Update · UTC
We have identified the root cause of the encoding failures: a misconfiguration in our S3 bucket.
The S3 configuration has been corrected, and encoding jobs on AWS are now recovering. We are seeing encoding tasks completing successfully again.
We are actively monitoring the system to confirm full recovery. - Resolved · UTC
Root Cause Analysis: Encoding Failures on AWS
Summary
On September 4, 2025, between 06:31 AM and 08:10 AM CEST, encoding jobs running on AWS failed due to an issue accessing our S3 storage.
Root Cause
The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage.
Impact
Only encoding jobs running on AWS were affected.
Encodings on other cloud providers and all other Bitmovin services were not impacted.
Resolution
The configuration was corrected at 08:10 AM CEST, restoring access to S3. Encoding operations on AWS recovered immediately and have been stable since.
Preventive Measures
To prevent this from happening again, we are:
Updating our key rotation procedure to ensure keys are not deleted prematurely.
Automating the key rotation process to reduce the chance of operator error.
- Investigating · UTC
We are currently experiencing failures with encodings on AWS.
Our engineering team is currently investigating the issue