[Faccus] Update 10:00pm: Status of LEARN - site unavailable; problem identified

chappell at uwaterloo.ca chappell at uwaterloo.ca
Thu Jan 31 22:20:32 EST 2013


Hello everyone,
Sorry for the long lapse, but there were no definitive answers or time
lines to report. We have been in touch with many of you as we focused on
getting messages to the Deans to send to their professors (sent to Deans
at the end of the day and won't be forwarded until morning in some cases).
Also, we were providing information for a message to students (see posted
on the LEARN Help site, thanks to Daspina:
https://uwaterloo.ca/learn-help/news/message-provost-regarding-learn )

As of a 9:22pm update, D2L has confirmed that they know the solution to
the problem, and it involves moving our data (and that for about 25% of
their customers) so it is not behind a device that is causing the
problems. They are working towards that now. In parallel, they are
calculating estimates for how long the solution will take, and will send
that information when it is available. They have said they will also
update the LEARN redirect site with information on progress, presumably
once they have established the expected time to completion. Otherwise, the
next update is at midnight.

You may have noted already the article in the Record.
http://www.therecord.com/news/business/article/878997--desire2learn-s-cloud-data-centre-issues-have-uw-wlu-offline

Thank you, everyone, for your support and patience in this, and for all
that you are doing to try to support faculty and students during this very
frustrating time.
Andrea

> Hello,
> There has been no significant news on a recovery time. This update is to
> let you know there is a memo going to the Deans, meant as a message to
> send to all instructors. In it will be a pointer to a support site,
> showing instructors how to generate a mailing list to contact their
> students by email. Instructions are also available for how to give access
> to large files, rather than sending them through email.
> https://uwaterloo.ca/learn-help/while-learn-down
>
> News on LEARN is at the same site.
> https://uwaterloo.ca/learn-help/news
>
> We're not sure when the message will be sent to instructors, but this is a
> heads up of the support they may be looking for.
>
> More news as we get it.
> Andrea
>
>
>> [Thu., Jan. 31, 10:45am]
>> Hello,
>> The latest news indicates that other clients are also experiencing the
>> slowness, and that specific clients (and we don't yet know if we are one
>> of them) will have data migrated to a different storage device. Later I
>> will send an update from a phone call Dave Wallace and I had with some
>> D2L
>> executives. We will have short calls with the Deans over lunch hour to
>> discuss the situation.
>> Andrea
>>
>>> [Thu., Jan. 31, 8:55am]
>>> Hello again,
>>> We reported to D2L the HTTP 500 errors and also that our system was
>>> very
>> slow. They know there is a problem and were working on it. We agreed it
>> was better to have the system unavailable than run with those problems.
>> We
>>> do not have a recovery time as yet, sorry.
>>> Andrea
>>>
>>>> [Thu., Jan. 31, D2L 7:45am update]
>>>> Hello,
>>>> Desire2Learn made our system available again and said we can "assume
>>>> that
>>>> all work has been completed and that we have finished our verification
>> testing of your site". However, the support team is seeing errors (HTTP
>> 500) and has had reports of problems logging in by students and faculty
>> members. The system is very slow. We will be talking to D2L about this
>> and
>>>> we will keep you updated as soon as we know more.
>>>> Andrea
>>>>> [Thu., Jan. 31, D2L 4:25am update]
>>>>> Hello,
>>>>> D2L sent an update at midnight, indicating that the process they had
>> anticipated finishing by then was 98% done, and that once it
>> completed,
>>>>> they would start verification.
>>>>> At 4:25am the update indicates a 2-3 hour process needed. It is a bit
>> confusing as to what this process is, except that it involves
>> migration
>>>>> of
>>>>> our data and files to the new configuration. They will provide a
>>>>> further
>>>>> status update upon completion, which could be 6:30 - 7:30am based on
>> their
>>>>> estimate.
>>>>> Andrea
>>>>>> [Wed. Jan. 30, 10:00pm update]
>>>>>> Hello,
>>>>>> The latest message from D2L, at 10:12pm, says they still expect to
>>>>>> have
>>>>>> the recovery by midnight. They have mentioned the need for
>>>>>> verification
>>>>>> but have not indicated how long that might take. They have said
>>>>>> they'll
>>>>>> take off the "maintenance" message when the recovery and
>>>>>> verification
>>>>>> is
>>>>>> complete. I assume that means they will remove the redirect then.
>> Questions are arising individually about what will happen with
>> assignments
>>>>>> and tests. I think instructors are altering due dates for
>> assignments,
>>>>>> but
>>>>>> mid-terms are likely more difficult. Hopefully instructors will post
>> announcements in their courses to indicate any changes, so student
>> see
>>>>>> those as the system is back up.
>>>>>> Next update is at midnight and I may not be awake for that!
>>>>>> Andrea
>>>>>>> [Wed. Jan. 30, 5:45pm update]
>>>>>>> Hello,
>>>>>>> The changes anticipated to be complete by 6:00pm are not complete.
>>>>>>> The
>>>>>> revised finish time is estimated for between 10:00pm and midnight,
>> followed by a verification process.
>>>>>>> The Waterloo LEARN system remains inaccessible during this time.
>>>>>>> Access
>>>>>> to
>>>>>>> the barely responsive system created many more problems for our
>>>>>>> faculty
>>>>>> and students (long wait times to access the system, if at all,
>>>>>>> interruptions to most course functions making those functions
>>>>>>> unusable,
>>>>>> inability to access files, etc.).
>>>>>>> We expect the next update at about 10pm. At this time the system
>>>>>>> has
>>>>>> for
>>>>>>> 27 hours. We will be seeking detailed explanations from D2L, as
>>>>>>> well
>>>>>>> as
>>>>>> their actions to prevent this from happening again.
>>>>>>> Andrea
>>>>>>>> [Wed. Jan. 30, 3:30pm update]
>>>>>>>> Hello,
>>>>>>>> The message from D2L remains the same, that the expected up-time
>>>>>>>> is
>>>>>> 6pm.
>>>>>>> Thanks to Pat Lafranier for enlisting the help of Jonathan Woodcock
>>>>>>> to
>>>>>>> put
>>>>>>>> for now a "Help with LEARN" button on the pathways pages on the
>>>>>> Waterloo
>>>>>>> website, to point to our LEARN update page, maintained by Daspina
>> Fefekos.
>>>>>>>> https://uwaterloo.ca/learn-help/
>>>>>>>> Our next update from D2L is expected at 6pm and I will send a
>>>>>>>> message
>>>>>>> again then.
>>>>>>>> Andrea
>>>>>>>>> [Wed. Jan. 30, 12:30pm update]
>>>>>>>>> (Apologies if you receive a second copy of this message. There
>>>>>> appeared
>>>>>>>> to
>>>>>>>>> be a problem when I sent it the first time, and I want to make
>> sure
>>>>>>>>> it
>>>>>>>> gets out.)
>>>>>>>>> Hello all,
>>>>>>>>> I'm afraid the news is not good. D2L reports that they have
>>>>>>>>> received
>>>>>>> new
>>>>>>>> devices at the hosted site, to fix the problem. There were to be
>>>>>>> configuration changes expected to finish at noon which have not
>>>>>>>> completed.
>>>>>>>>> They expect it to be completed by 6pm ET, which seems to imply
>> that
>>>>>>>>> is
>>>>>>>> when we will be back up. Our next update from D2L will be at
>> 3:00pm.
>>>>>>> Unless we hear anything more before then, that will be the time of
>>>>>>> our
>>>>>> next update.
>>>>>>>>> Andrea
>>>>>>>>>> Hello everyone,
>>>>>>>>>> We asked D2L to redirect our site to a page that says the system
>>>>>>>>>> is
>>>>>>>>> unavailable. We have not had an update since the message at
>> 8:07am,
>>>>>>>>> so
>>>>>>>> still anticipate the system to be back by noon. We have asked that
>>>>>>>> the
>>>>>>> redirect page provide updated information, but cannot guarantee it.
>>>>>>>>>> We will let you know as more information comes in.
>>>>>>>>>> Andrea
>>>>>>>>>>> Hello,
>>>>>>>>>>> We have heard from Desire2Learn (8:07am) and have done some
>>>>>>>>>>> testing
>>>>>>> of
>>>>>>>>>> our
>>>>>>>>>>> own. According to D2L, they have "partially implemented" their
>>>>>>>>> solution.
>>>>>>>>>> They caution that some sites may still see impacts. They
>>>>>>>>>> "anticipate
>>>>>>>>> steady improvement to normal performance" by noon today.
>>>>>>>>>>> From our tests, Waterloo LEARN seems to be very slow for login
>>>>>>>>>>> and
>>>>>>> for
>>>>>>>>>> accessing files. (At about 7am it was performing well, but is
>>>>>>>>>> not
>>>>>>> now.)
>>>>>>>>>>> We will keep you apprised as we hear more. Our sincere
>>>>>>>>>>> apologies
>> for
>>>>>>>>>> this
>>>>>>>>>>> significant interruption. We will be pursuing detailed
>>>>>>>>>>> explanations
>>>>>>>>> from
>>>>>>>>>> D2L.
>>>>>>>>>>> Regards,
>>>>>>>>>>> Andrea
>>>
>>>
>>
>>
>>
>>
>
>



More information about the Faccus mailing list