[Faccus] [ist-itms] [UW-RT #437717] SERVICE Update - 2015-07-10 VMware cluster - Esx node failure

Jason Gorrie via RT rt at rt.uwaterloo.ca
Fri Jul 10 21:56:51 EDT 2015


Please note, the following people received this message and will receive all replies to this message: 

Requestors: jbgorrie(jbgorrie at uwaterloo.ca)
CCs: acohelp at watarts.uwaterloo.ca(acohelp at watarts.uwaterloo.ca), admin-support at lists.uwaterloo.ca(admin-support at lists.uwaterloo.ca), admin at pdeng.uwaterloo.ca(admin at pdeng.uwaterloo.ca), arbhagat(arbhagat at uwaterloo.ca), cjaray(cjaray at uwaterloo.ca), cmseitz(cathy.seitz at uwaterloo.ca), cnsc at lists.uwaterloo.ca(cnsc at lists.uwaterloo.ca), cscf-tg-networks at cs.uwaterloo.ca(cscf-tg-networks at cs.uwaterloo.ca), cscfhelp(cscfhelp at uwaterloo.ca), ctsc at lists.uwaterloo.ca(ctsc at lists.uwaterloo.ca), dcithelpdesk(dcithelpdesk at uwaterloo.ca), es-servicealerts at lists.uwaterloo.ca(es-servicealerts at lists.uwaterloo.ca), esag at engmail.uwaterloo.ca(esag at engmail.uwaterloo.ca), faccus at lists.uwaterloo.ca(faccus at lists.uwaterloo.ca), ist-itms at lists.uwaterloo.ca(ist-itms at lists.uwaterloo.ca), ist-mgmt at lists.uwaterloo.ca(ist-mgmt at lists.uwaterloo.ca), ist-sas at lists.uwaterloo.ca(ist-sas at lists.uwaterloo.ca), ist-staff at lists.uwaterloo.ca(ist-staff at lists.uwaterloo.ca), ist-tis at lists.uwaterloo.ca(ist-tis at lists.uwaterloo.!
 ca), ist-windows at lists.uwaterloo.ca(ist-windows at lists.uwaterloo.ca), ist-workstations at lists.uwaterloo.ca(ist-workstations at lists.uwaterloo.ca), isthd at lists.uwaterloo.ca(isthd at lists.uwaterloo.ca), jobmine(jobmine at uwaterloo.ca), joe.radman(joe.radman at uwaterloo.ca), jweigel(jweigel at uwaterloo.ca), kevin.kennedy at family-medicine.ca(kevin.kennedy at family-medicine.ca), kjjack(kjjack at uwaterloo.ca), learnhelp(learnhelp at uwaterloo.ca), mfcfhelp(mfcfhelp at uwaterloo.ca), noc(noc at uwaterloo.ca), reshelp(reshelp at uwaterloo.ca), s5bailey(s5bailey at uwaterloo.ca), sbradley(sbradley at uwaterloo.ca), shad.lusted(shad.lusted at uwaterloo.ca), telephoneadmin(telephoneadmin at uwaterloo.ca), tkanerva(tkanerva at uwaterloo.ca), uw.network at rumours.uwaterloo.ca(uw.network at rumours.uwaterloo.ca), watcard(watcard at uwaterloo.ca), wcarroll(wcarroll at uwaterloo.ca), wnag at engmail.uwaterloo.ca(wnag at engmail.uwaterloo.ca)
CCs: 



Sorry missed this email- this outage should not have directly impacted
SQL.  the storage and servers for MS SQL were not involved.

There is a chance that an application server was impacted.  As noted all
vms are back in operation.

--
Jason

On 15-07-10 07:59 PM, Koorus Bookan via RT wrote:
> Please note, the following people received this message and will receive all replies to this message: 
> 
> Requestors: jbgorrie(jbgorrie at uwaterloo.ca)
> CCs: acohelp at watarts.uwaterloo.ca(acohelp at watarts.uwaterloo.ca), admin-support at lists.uwaterloo.ca(admin-support at lists.uwaterloo.ca), admin at pdeng.uwaterloo.ca(admin at pdeng.uwaterloo.ca), arbhagat(arbhagat at uwaterloo.ca), cjaray(cjaray at uwaterloo.ca), cmseitz(cathy.seitz at uwaterloo.ca), cnsc at lists.uwaterloo.ca(cnsc at lists.uwaterloo.ca), cscf-tg-networks at cs.uwaterloo.ca(cscf-tg-networks at cs.uwaterloo.ca), cscfhelp(cscfhelp at uwaterloo.ca), ctsc at lists.uwaterloo.ca(ctsc at lists.uwaterloo.ca), dcithelpdesk(dcithelpdesk at uwaterloo.ca), es-servicealerts at lists.uwaterloo.ca(es-servicealerts at lists.uwaterloo.ca), esag at engmail.uwaterloo.ca(esag at engmail.uwaterloo.ca), faccus at lists.uwaterloo.ca(faccus at lists.uwaterloo.ca), ist-itms at lists.uwaterloo.ca(ist-itms at lists.uwaterloo.ca), ist-mgmt at lists.uwaterloo.ca(ist-mgmt at lists.uwaterloo.ca), ist-sas at lists.uwaterloo.ca(ist-sas at lists.uwaterloo.ca), ist-staff at lists.uwaterloo.ca(ist-staff at lists.uwaterloo.ca), ist-tis at lists.uwaterloo.ca(ist-tis at lists.uwaterlo!
 o.!
>  ca), ist-windows at lists.uwaterloo.ca(ist-windows at lists.uwaterloo.ca), ist-workstations at lists.uwaterloo.ca(ist-workstations at lists.uwaterloo.ca), isthd at lists.uwaterloo.ca(isthd at lists.uwaterloo.ca), jobmine(jobmine at uwaterloo.ca), joe.radman(joe.radman at uwaterloo.ca), jweigel(jweigel at uwaterloo.ca), kevin.kennedy at family-medicine.ca(kevin.kennedy at family-medicine.ca), kjjack(kjjack at uwaterloo.ca), learnhelp(learnhelp at uwaterloo.ca), mfcfhelp(mfcfhelp at uwaterloo.ca), noc(noc at uwaterloo.ca), reshelp(reshelp at uwaterloo.ca), s5bailey(s5bailey at uwaterloo.ca), sbradley(sbradley at uwaterloo.ca), shad.lusted(shad.lusted at uwaterloo.ca), telephoneadmin(telephoneadmin at uwaterloo.ca), tkanerva(tkanerva at uwaterloo.ca), uw.network at rumours.uwaterloo.ca(uw.network at rumours.uwaterloo.ca), watcard(watcard at uwaterloo.ca), wcarroll(wcarroll at uwaterloo.ca), wnag at engmail.uwaterloo.ca(wnag at engmail.uwaterloo.ca)
> CCs: 
> 
> 
> 
> Will this impact sql cluster, will have to warn people. ..
> 
> On July 10, 2015 7:00:18 PM EDT, Jason Gorrie via RT <rt at rt.uwaterloo.ca> wrote:
>>
>> Fri Jul 10 19:00:17 2015: Request 437717 was acted upon.
>> Transaction: Ticket created by jbgorrie
>>       Queue: IST-LEARNsupp
>> Subject: SERVICE Update - 2015-07-10 VMware cluster - Esx node failure
>>       Owner: Nobody
>>  Requestors: jbgorrie at uwaterloo.ca
>>      Status: new
>> Ticket <URL: https://rt.uwaterloo.ca/Ticket/Display.html?id=437717 >
>>
>>
>>
>>
>>
>> Description:     Esx node failure
>>
>> Date: (YYYY-MM-DD)     2015-07-10
>>
>> Start Time:              16:50
>>
>> End Time:	17:30	
>>
>> Impact:           65 machines
>>
>> Resolution:     
>>
>> Submitted By: jbgorrie at uwaterloo.ca
>>
>> Comment:        One of the six nodes providing service in mc had
>> hardware issues causing vmware to panic.  VMware ha restarted all
>> machines (but one) on other modes and at this time they are running. 
>> The node itself is down running diagnostics trying to isolate/reproduce
>> the issue.
>> The one machine not automatically restarted was set to only run on the
>> machine and it is now up as well.
>>
>> At this time staff are looking at affected services and checking if
>> they recovered as they should.
>>
>> Notice Submitted:    Fri Jul 10 18:59:48 EDT 2015
>>
>> Follow us on twitter https://twitter.com/UWNetworkAlert
>>
>> Note:   If you have any questions or concerns please contact the IST
>> Service Desk at ext: 44357 or helpdesk at uwaterloo.ca
>>
>>
>>
>>
>> _______________________________________________
>> ist-itms mailing list
>> ist-itms at lists.uwaterloo.ca
>> https://lists.uwaterloo.ca/mailman/listinfo/ist-itms
> 
> 
> <URL: https://rt.uwaterloo.ca/Ticket/Display.html?id=437717 >
> 


-- 
Jason Gorrie - IST Servers & Storage
jbgorrie at uwaterloo.ca
Work Hours: M-F: 07:00-15:00 Eastern


<URL: https://rt.uwaterloo.ca/Ticket/Display.html?id=437717 >


More information about the Faccus mailing list