[MacTUG] SharePoint Service incident summary

[Natasha Jennings]

The SharePoint Service offered by IST experienced some difficulties earlier this week. SharePoint was unavailable to most campus areas from late Wednesday, January 28 to Thursday, January 29 at 4:00 p.m.

What happened? SharePoint stores its information in a database structure called a 'Content Database'. Because SharePoint is used extensively across campus, and stores a large number of documents, this information is spread across a number of content databases. During the evening hours of Wednesday, January 28,  most of the content databases became damaged. The damage seemed to cascade across the content databases; the end-result was the majority of SharePoint sites were unavailable for access or use.

How was the issue fixed? The immediate goal was to get SharePoint back to a useful state, with usable data. Our first step was to restore individual sites to the last available good backup (Wednesday, January 27 after 8 p.m.). These restore attempts had mixed success. It was determined that the damaged content databases needed to be fully restored to the Wednesday, January 27 backups that ran after 12 p.m. (noon). Because of the large amount of data, the complete restore of the damaged content databases took just under three hours to complete.

Parts of the restore have continued today as some individual sites were still experiencing some problems.

Future steps: The SharePoint Services Team is investigating the specific cause of the corruption. SharePoint is a service many units rely on for business processes. IST is aware of the inconvenience this service outage has caused. We have identified issues with the current network storage location, and, are working on creating an alternative network storage infrastructure for SharePoint data, which will help protect the data, and prevent a reoccurrence of this type of an outage. This will involve moving the content databases to a different storage location. Detailed communications about this work will be shared next week.

Feedback and questions:  If you have any concerns or questions about this incident, please contact the IST SharePoint Service Coordinator, Stephen Markan, at smarkan at uwaterloo.ca<mailto:smarkan at uwaterloo.ca>

Recipients of this list: isthd, ist-staff, admin-support, faccus, mactug, ctsc, uwweb, ucist, Sharepoint-Alerts


Natasha Jennings
Communications Officer
Information Systems & Technology
University of Waterloo
519-888-4567 ext. 47951
[university-of-waterloo-logo-esig]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.uwaterloo.ca/pipermail/mactug/attachments/20150130/4639a09b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 5807 bytes
Desc: image003.png
URL: <http://lists.uwaterloo.ca/pipermail/mactug/attachments/20150130/4639a09b/attachment.png>