At 7:13am on Tuesday (2/4/20) we experienced a hardware failure with the on-campus mail server which is impacting a number of faculty and staff. The process to repair is underway but is taking a lot of time to repair and maintain data. Updates will be listed below...
Update as of 7:30am 2/13/20
We are about 95% complete on mailbox restores. We have had to redo a few that did not upload correctly but will be continuing to work on these today. IT Staff are reviewing the specialty mail boxes that have been used for various functions. We believe we have already repaired the most important ones but we continue to work through them.
If you are having issues still please contact the Help desk and let us know just the same.
Update as of 5pm 2/11/20
We are about 75% complete on mailbox restores and hope to finish by tomorrow. If you hear of a colleague who cannot access e-mail at all please have them contact us either at 830-372-6000 or ishelp@tlu.edu from a personal e-mail.
Update as of 5pm 2/10/20
We show about 60 mailboxes have been fully restored with about 120 more to go.
Update as of 7:45am 2/10/20
Just wanted to let everyone know that we have made progress from my last note. Actual mailbox restores (for those who were impacted) have begun. It is a manual process unfortunately but it is working and should recover everyone to the point of the failure last week. We anticipate it could take 2-3 days to complete everybody but once your mailbox is restored you will see the old messages then.
As a reminder, all faculty/staff should have email access to send and receive through web mail http://outlook.office.com If not please let us know. This includes retirees. We did miss a few people last week but have since fixed those. Desktop outlook access may require a visit by a help desk technician if it does not come up like normal.
Update as of 10pm 2/9/20
Mailbox restores have begun! Unfortunately it is a slow process because of the data migration to the cloud. We anticipate completion in the next 2-3 days.
Update as of 4:00pm 2/8/20
The disk repair has completed! So far so good (other than the dealy of course). Next step is to export from old server and import into Microsoft cloud. That will start in earnest tomorrow so staff could take a break.
Update as of 3:00pm 2/7/20
Based on what we have seen today, we are estimating that the disk repair of the Mail server will complete late tonight. Next steps will be to ensure integrity of the data (Daniel will do tomorrow morning) and then the process of exporting mailboxes from the server and importing that into Office 365. Rob will start that process once Daniel finishes his portion. We hope the impacted people will start seeing their data late Sunday or early Monday. Microsoft controls the import process into cloud mailboxes so it might be a couple of overnight exercises. We will know more late Saturday and early Sunday once we are at that stage.
Update as of 8:30pm 2/6/20
- Retiree accounts are now operational
- Special function accounts for imagenow are working
- Data restore still progressing
Update as on 1:30pm 2/6/20
The recovery process continues slowly unfortunately and will likely go into the weekend. The reason is that the hardware where the mail files are stored has 2 problems. In addition to the hard drive that failed, we are having trouble with the controller (this is the path that the computer uses to communicate to the hard drives) as well. In other words it is running in a degraded fashion. The repair process continues to so show progress so we are cautiously optimistic but the reality is it should have finished by now and it has not. We know that these files are important to everyone so we are going to do everything we can to recover them. In the meantime:
- All TLU faculty/staff should be functional with e-mail (albeit many of you without old e-mails still) on the Microsoft Office 365 cloud now. This was our intent and direction all along anyway so we decided to press on (all our students and about 100+ colleagues had been moved previously). The plan is that once the old data is accessible, IT staff would restore them into everyone’s mailboxes.
- If a faculty member or staff member is not able to connect to e-mail let us know at 830-372-6000 (please leave a message if we can’t answer), or ishelp@tlu.edu (we have that address working again) or let your building deputy know (we will touch base with them).
- We are also looking to get people back on outlook (if they were using it previously). That will likely require a visit by one of our Help Desk folks just so we can help with re attaching any archive e-mails you might have had stored. Archived e-mails would not be impacted since they were not on the server.
- While we are “baby-sitting” the restore process, we are focusing time on several of the service e-mail accounts so that they are operational in short order
- Our retirees are currently down but we will be creating mailboxes for them later today.
Update as of 7:30am 2/6/20
Waiting on staff to review statistics from overnight processing etc. Will be sending out an e-mail to faculty/staff later this morning with more.
Update as of 4pm 2/5/20
- Scan to e-mail is now working again
- Regarding the restore process, we have decided to try a different tactic since the current one was not making the progress that we expected (we expected slow but not this slow). We will post an update once we have had an opportunity to observe how it is working
Update as of 11:30am 2/5/20
The restore process continues to run very slow which likely means many hours before real progress can be seen. All active faculty and staff who were impacted should have a functioning mailbox in the Microsoft cloud. For now the best way to connect is by using webmail or https://outlook.office.com - sign in with your tlu email address and your regular tlu password. Note that old e-mails will not be present until the restore completes but you can send and receive e-mails.
Update as of 9:45 am 2/5/20
IT emphasis right now is to get everyone back sending/receiving e-mail. We are working on getting special usage accounts (that had not been moved prior) back up and operational.
We will be looking at instructions for those who use outlook to connect to their own mailboxes. It is supposed to automatically configure but that has been hit-or-miss.
Update as of 6am 2/5/20
Data restore of old mail data continues but very slow. We will try and determine a better estimate later this morning. Here is latest status:
- Student e-mail - working as normal
- Faculty/staff previously moved to Office 365 cloud - working as normal
- Remaining faculty/staff - new mailboxes created in Office 365 cloud so mail can be received and sent (old mail still being restored). Connect to: https://outlook.office.com and use your TLU email address as the username and your regular password.
- Retirees - not working
- Scan to e-mail from copier - not working but scan to folder is working
- Sending e-mail to students from e-racer/e-learning or MyTLU faculty tab - working
Update as of 8pm 2/4/20
IT staff have created mailboxes on Office 365 for faculty and staff who had not been previously moved. These are empty but can send and receive e-mail now. Once the on campus server is rebuilt we will look to restore older mail to them. The easiest way to access is to use https://outlook.office.com or the webmail link in the quick link section of MyTLU (to the left of the screen). Sign in with your tlu e-mail address as the username and your regular TLU password
We will have instructions on accessing with outlook from your desktop tomorrow.
Update as of 6pm 2/4/20
To expedite the process, IT staff will be creating mailboxes on Office 365 for all faculty/staff that don't already have them. Those will be created over night. They will be empty until the restore completes but this way people will be able to send and receive e-mail.
Restore is continuing to work but is very slow due to the size of data.
Update as of 5pm 2/4/20
- e-mail from MyTLU courses and faculty course control to students is now working again
- Everything else same status
Update as of 2pm 2/4/20
- Students are not impacted if they use http://offfice.com for web mail or use their phone or mobile device
- faculty/staff with mailboxes in the Office 365 cloud can access through outlook like normal and use http://office.com for webmail
- Http://mail.tlu.edu is not functioning
- faculty/staff not on Office 365 are currently down
- e-mail from MyTLU classes is currently down but we hope to have that up by 5pm today
- Retiree e-mails are down
- Scan to e-mail from copier is down - scan to folder is working
TLU IT