[Micronet] Merits of effective disaster recovery - or Emory Univ wipes all Windows systems by accident

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] Merits of effective disaster recovery - or Emory Univ wipes all Windows systems by accident

jon kuroda-2
Sometimes, the disaster in "disaster recovery scenario" is not some
external event like a hurricane or earthquake, but an "own goal", a
disaster perpetrated by one's own systems.

Short Version:

Windows 7 image deployed to all of Emory University's managed
windows systems - laptops, desktops and servers. Including the
Microsoft System Center Configuration Manager system that itself
was used to deploy the images.

Recovery is now in its 2nd day.  I guess it could have been worse.

--Jon

http://it.emory.edu/windows7-incident/

"A Windows 7 deployment image was accidently sent to all Windows machines,
including laptops, desktops, and even servers. This image started with
a repartition / reformat set of tasks.

As soon as the accident was discovered, the SCCM server was powered off
– however, by that time, the SCCM server itself had been repartitioned
and reformatted.

Restoration of servers began immediately but the process took far longer
than expected – we have been using consultants to help validate the
health of the SCCM servers and that work only completed last night.

So, we were without our preferred methods for deploying images to
desktops/laptops all yesterday and relied on older methods – USB +
Ghost, LANDesk (we still had our old LANDesk server) + PXE. These methods
required a lot of manual work plus our success was uneven with them.

Today we are pausing – briefly this morning – to see if we can now
use our preferred method – SCCM. This will allow us to have a one-touch
method for restoring desktops/laptops to a production ready state."


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Merits of effective disaster recovery - or Emory Univ wipes all Windows systems by accident

Philip Weekly
Wow,   what a mess-up that was. I guess it only shows that things could always be worse. My job now seems easier after reading this.  

Thanks for sharing!

Philip


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of jon kuroda
Sent: Friday, May 16, 2014 3:08 PM
To: [hidden email]
Subject: [Micronet] Merits of effective disaster recovery - or Emory Univ wipes all Windows systems by accident

Sometimes, the disaster in "disaster recovery scenario" is not some external event like a hurricane or earthquake, but an "own goal", a disaster perpetrated by one's own systems.

Short Version:

Windows 7 image deployed to all of Emory University's managed windows systems - laptops, desktops and servers. Including the Microsoft System Center Configuration Manager system that itself was used to deploy the images.

Recovery is now in its 2nd day.  I guess it could have been worse.

--Jon

http://it.emory.edu/windows7-incident/

"A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks.

As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

Restoration of servers began immediately but the process took far longer than expected – we have been using consultants to help validate the health of the SCCM servers and that work only completed last night.

So, we were without our preferred methods for deploying images to desktops/laptops all yesterday and relied on older methods – USB + Ghost, LANDesk (we still had our old LANDesk server) + PXE. These methods required a lot of manual work plus our success was uneven with them.

Today we are pausing – briefly this morning – to see if we can now use our preferred method – SCCM. This will allow us to have a one-touch method for restoring desktops/laptops to a production ready state."


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Merits of effective disaster recovery - or Emory Univ wipes all Windows systems by accident

Ian Crew
In reply to this post by jon kuroda-2

Ouch. I so empathize with what they must be going through right now. 

Ian

_____________
Sent from my phone. Please excuse the brevity and typos.

On May 16, 2014, at 3:07 PM, jon kuroda <[hidden email]> wrote:

Sometimes, the disaster in "disaster recovery scenario" is not some
external event like a hurricane or earthquake, but an "own goal", a
disaster perpetrated by one's own systems.

Short Version:

Windows 7 image deployed to all of Emory University's managed
windows systems - laptops, desktops and servers. Including the
Microsoft System Center Configuration Manager system that itself
was used to deploy the images.

Recovery is now in its 2nd day.  I guess it could have been worse.

--Jon

http://it.emory.edu/windows7-incident/

"A Windows 7 deployment image was accidently sent to all Windows machines,
including laptops, desktops, and even servers. This image started with
a repartition / reformat set of tasks.

As soon as the accident was discovered, the SCCM server was powered off
– however, by that time, the SCCM server itself had been repartitioned
and reformatted.

Restoration of servers began immediately but the process took far longer
than expected – we have been using consultants to help validate the
health of the SCCM servers and that work only completed last night.

So, we were without our preferred methods for deploying images to
desktops/laptops all yesterday and relied on older methods – USB +
Ghost, LANDesk (we still had our old LANDesk server) + PXE. These methods
required a lot of manual work plus our success was uneven with them.

Today we are pausing – briefly this morning – to see if we can now
use our preferred method – SCCM. This will allow us to have a one-touch
method for restoring desktops/laptops to a production ready state."



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.