We're investigating an outage of our Ceph storage system.
-
We're investigating an outage of our Ceph storage system. We'll hit you up once we know more.
Currently, most Codeberg services are not available.
-
We're investigating an outage of our Ceph storage system. We'll hit you up once we know more.
Currently, most Codeberg services are not available.
We are slowly bringing services up and monitoring Ceph very closely. We're still continuing our investigation, we don't have a clear root cause yet.
-
We are slowly bringing services up and monitoring Ceph very closely. We're still continuing our investigation, we don't have a clear root cause yet.
Services are operational again and performance has recovered.
We conclude that the problem originated from three factors: We deleted a lot of large abusive content recently. Automated snapshot trimming started today and deleted more data than usual. This in turn resulted in a high I/O load on our last remaining HDD that consequently brought down a host and some Ceph services (MDS), bringing performance to a near-halt for a while.
-
Services are operational again and performance has recovered.
We conclude that the problem originated from three factors: We deleted a lot of large abusive content recently. Automated snapshot trimming started today and deleted more data than usual. This in turn resulted in a high I/O load on our last remaining HDD that consequently brought down a host and some Ceph services (MDS), bringing performance to a near-halt for a while.
@codebergstatus For the past hour or so I've been getting
! server error: 500when replying to an issue of a repo. Any lingering issues? The status page reports occasional short interruptions. -
@codebergstatus For the past hour or so I've been getting
! server error: 500when replying to an issue of a repo. Any lingering issues? The status page reports occasional short interruptions.@codebergstatus I was eventually able to post with no errors.