Resolved -
Between 15:05 UTC and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Customers experienced intermittent "logged out" behavior as erroneous 401 responses triggered repeated authentication flows in app integrations. Affected requests also experienced approximately 800ms of additional latency.
A memcached proxy service rollout to our internal API infrastructure caused our authentication service to pick up an incorrect memcached host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached to use the correct host.
To prevent similar issues in the future, we plan to migrate our authentication system to the new memcached infrastructure to improve resilience and strengthen overall reliability posture.
Jun 10, 16:39 UTC
Monitoring -
The degradation has been mitigated. We are monitoring to ensure stability.
Jun 10, 16:37 UTC
Update -
The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.
Jun 10, 16:36 UTC
Update -
We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Erroneous 401 responses are causing app integrations to trigger authentication flows. We have identified a problematic component in our infrastructure and are working to mitigate.
Jun 10, 16:21 UTC
Update -
The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.
Jun 10, 15:46 UTC
Update -
We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Further updates will be provided as we work to mitigate.
Jun 10, 15:46 UTC
Update -
We are investigating issues related to sporadic authentication failures impacting approximately 15% of API traffic. We will continue to investigate and provide updates.
Jun 10, 15:27 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Jun 10, 15:27 UTC
Update -
API Requests is experiencing degraded availability. We are continuing to investigate.
Jun 10, 15:23 UTC
Investigating -
We are investigating reports of impacted performance for some GitHub services.
Jun 10, 15:20 UTC