[Faccus] NETWORK ALERT - 2013-10-30 - SERVICE DEGRADATION remains

bruce at uwaterloo.ca bruce at uwaterloo.ca
Wed Oct 30 12:41:45 EDT 2013



Description:     SERVICE DEGRADATION remains

Date: (YYYY-MM-DD)     2013-10-30

Start Time:              7:30am

End Time:		

Impact:           

Resolution:     

Submitted By: bruce at uwaterloo.ca

Comment:        
Some major services remain impacted, staff continue to investigate.  

Some additional technical information on the issue is as follows:

We use a pair of network load balancers operating in a redundant failover mode,  to provide load balancing services in the machine rooms.  We use the vendor procedure to upgrade them, that is to upgrade the passive node first, perform a manual failover from the active to the passive, then upgrade the former active node.   This is the standard procedure we have applied several times since the load balancers were installed in 2008.

After this mornings upgrade, our Microsoft Exchange service did not work.  After some debugging, we reverted the load balancers to the same version they had been running previously, and later performed a manual failover so we were running on the same load balancer that had been active previously.  We also opened a support case with the load balancer vendor.

Unfortunately multiple services, including Exchange, the home page, authentication, and others, were still down, despite restoring the load balancers to yesterdays state.

Staff in IST have been identifying some workarounds, taking some servers out from behind the load balancer, to restore partial service.   We still do not know why the problems are happening, and some services remain offline.     All IST staff members involved in the load balancer operation and management of the affected services are working constantly on this issue.

I will share more information on the issue as it becomes available.  

Unfortunately some key services (e.g.jobmine) remain down, and others remain degraded (e.g.Exchange).


Notice Submitted:    Wed Oct 30 12:41:45 EDT 2013

Follow us on twitter https://twitter.com/UWNetworkAlert

Note:   If you have any questions or concerns please contact the IST Help Desk at ext: 84357 or helpdesk at uwaterloo.ca



More information about the Faccus mailing list