Front End webserver down

Incident Report for ICEcorp

Resolved

This incident has been resolved.
Posted Apr 18, 2025 - 08:18 CEST

Update

The backup is completed. We are checking the systems to see if everything has recovered successfully.
Posted Apr 18, 2025 - 07:44 CEST

Update

The new server is up and running. We are restoring backups and bringing all websites back up one by one.
Posted Apr 17, 2025 - 23:39 CEST

Update

A few systemfiles of the front-end server got corrupted. We were only able to repair a part of the system. We are creating a new server and recovering backups this night to avoid problems in the near future. There will be no loss of data, the data is secured on separate servers with duplicates in another datacenter.
Posted Apr 17, 2025 - 21:44 CEST

Update

The applications on the tablets are live. Icecloud.be is still down but we are keeping this state until the ice cream vans end their day shift. We'll continue working after 22:00.
Posted Apr 17, 2025 - 19:36 CEST

Monitoring

In some regions the apps are live again.
Posted Apr 17, 2025 - 19:23 CEST

Investigating

The front end webserver is down.
Posted Apr 17, 2025 - 19:17 CEST
This incident affected: ICEcloud (ICEcloud).