Partial Outage — Container Registry

Incident Report for BoxExchanger

Postmortem

Summary

During routine firewall maintenance, our public GitLab Runner temporarily lost the ability to push images to the Container Registry (rg.boxexchanger.net).
The registry itself remained fully operational, but push operations from the runner consistently failed with timeout errors (context deadline exceeded).

Impact

  • Public CI/CD pipelines were unable to complete Docker image builds requiring a push to the registry.
  • No impact to end-user services or internal operations outside CI/CD.
  • Pull operations and direct registry access remained unaffected.

Root Cause

While updating the list of allowed IPs in the firewall, an incorrect rule was applied.
As a result, the public GitLab Runner’s outgoing traffic was unintentionally blocked from reaching the registry endpoint.
The registry service was functioning normally — only connectivity from the runner was affected.

Resolution

The firewall configuration was corrected and the relevant IP ranges were re-added to the allowlist.
After applying the fix, all push operations from the public runner resumed normal operation.

Posted Nov 28, 2025 - 09:48 UTC

Resolved

The issue was caused by a firewall configuration error.
The configuration has now been fixed, and the registry is fully operational.
Posted Nov 28, 2025 - 09:45 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 28, 2025 - 09:32 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Nov 28, 2025 - 09:28 UTC

Investigating

We are currently experiencing intermittent issues with our Container Registry (rg.boxexchanger.net), some push operations may fail with errors.
Our team is investigating the cause and working on restoring full functionality.
Posted Nov 28, 2025 - 09:21 UTC
This incident affected: Git system (Container & package registry).