On Sunday (15th Jan) evening we had issues with our timetabling service and it was unavailable. The UoD IT team investigated the issue and full service was restored at 7.00 a.m. this morning. Apologies for any inconvenience this may have caused.
Incident review outcome
A full incident review took place on Monday and a summary of the outcomes of this discussion is given below.
Each time a customer uses the online timetable service a record of the events that take place during their session is logged in a file alongside that of everyone else. Due to the higher than normal volume of activity, the size of this logfile exceeded the total amount of space allowed for it on the University’s network infrastructure, which led to loss of service. The IT Team were unable to respond pro-actively in this instance because 100% space capacity was reached quickly, which meant monitoring measures proved ineffective.
What was the root cause?
Previous investigations carried out by the vendor (Scientia) in response to similar disruptive events suggest that repeated denial-of-service attacks may be responsible for these periods of online timetable unavailability. If this is a cyber-attack case, it is likely the service was flooded with illegitimate requests that overloaded the system, preventing customers from making their own legitimate requests to access timetables.
What are we doing now?
The team will continue to investigate this incident to identify any potential improvements that may prevent it from happening again. This review has highlighted a need to re-check current monitoring practices, the online timetabling set-up, as well as the way we communicate incidents like this to customers.
Did you notice this issue?
In this instance, a much lower number of calls were received from customers to the IT Service Desk than usual. We would like to take this opportunity to encourage everybody to report any IT issues as and when they come across them by getting in touch with Help4U at their earliest convenience. This will ensure as quick a resolution as possible is reached.