[Micronet] Emergency options: twitter feeds

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] Emergency options: twitter feeds

Phyllis Orrick
I endorse Christopher's comments and appreciation for the IT teams that are working to resolve this.

At our unit, we are taking this as a learning moment, as Christopher suggests, by compiling an internally sharable list of all staff's alternate email addresses and encouraging staff who don't already have ones that they're comfortable sharing to establish one.

I'd like to hear thoughts about another emergency suggestion, made by the instructor at a use of social media workshop hosted recently at Haas: that the university establish a list of Twitter feeds to distribute information in the event that email won't suffice. This is already being done at major institutions, and no doubt someone here at Cal is hard at work on this. As the instructor pointed out, Twitter is often available when other services aren't and is an excellent broadcast tool. We are exploring setting up one for this purpose at our unit.

If anyone has info on Cal's efforts in this area, it would be a great time to share it. 

Phyllis

On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
Shel,
I appreciate all the efforts that you and the staff are going through to
bring Calmail back on line.

I take this as a wake up call for disaster recovery plans for my own cluster.

Under what circumstances do we fall back on a different set of Calmail
hardware?

Presumably there are disaster recovery plans that have us host our email
system elsewhere.

Are those plans publicly available for CalMail?  If the building that
contains Calmail is destroyed, what happens?  Is there some sort of
service level agreement and if a certain level of service is not met, then
we move our mail system to somewhere else (UC Davis?)?

At this point, it seems that moving Calmail to another location is not
necessary, but at what point would Calmail be moved?

These are all good questions and answers for a lessons learned document
that I hope is forthcoming.  I'd be happy with an email message to
Micronet that addresses these topics.

I briefly searched for information about Calmail's disaster recovery plans
but did not find anything.  It would be OE to determine what sort of
disaster recovery information should be made available and to have sites
like Calmail and the Hub provide this information.

I, for one, see this is an excellent moment to update my list of
alternative email addresses for my frequent campus contacts.

I'm also going to take this opportunity to check my back up system and
push forward on replacing some old hardware.

Many thanks again to everyone for all their efforts over the weekend.  I'm
sure some long, anguished hours have been put in.

_Christopher

Christopher Brooks, PMP                       University of California
CHESS Executive Director                      US Mail: 337 Cory Hall
Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
ph: <a href="tel:510.643.9841" value="+15106439841">510.643.9841                                (Office: 545Q Cory)
home: (F-Tu) <a href="tel:707.665.0131" value="+17076650131">707.665.0131 cell: <a href="tel:707.332.0670" value="+17073320670">707.332.0670

Shel writes:

> I very much appreciate that it is unacceptable to have a core campus
> system like CalMail offline for any length of time.  Today`s continued
> problems with CalMail have been particularly difficult for all involved.
> We work hard to design and operate systems that can handle the needs of
> our community and when we fail to meet that standard we bring in outside
> experts to help us improve.  We have added additional outside experts from
> other campuses, vendors and new team members from the Berkeley technical
> community as we work through this crisis. This is our highest priority and
> will remain so until we have the environment fully stabilized.  The
> following message provides current information.  You can also check
> http://systemstatus.berkeley.edu for the latest status.
>
> Regards,
> Shel Waggener
> Associate Vice Chancellor and CIO
>
> CURRENT STATUS SUMMARY
> 1) The CalMail system is available only through web clients at
> http://calmail.berkeley.edu. All messages are being sent and received but
> can only be accessed via webmail.  Webmail sessions may be slower than
> normal due to volume.
> 2) Students are strongly encouraged to forward their email to alternate
> email accounts. Instructions for how to do that can be found on the
> Calmail site (http://calmail.berkeley.edu, Manage Account option).
>
> DETAILS:
>
> CalMail has been substantially impacted during the last 36 hours after the
> successful recovery from database corruption this weekend.  The load on
> the system has remained extremely high as the millions of backlogged
> messages are delivered.  The load situation worsened considerably Monday
> morning as tens of thousands of campus community members returned from the
> holidays and connected for the first time, pushing the load above
> operating limits.  Normal processes that usually run in the background
> unnoticed, including copying data off of a failed hard drive, effectively
> shut down the system for many people.  Attempts to keep email moving
> during the day Monday were minimally successful.  Monday night during off
> hours work was undertaken to accelerate the repair of the failed disk and
> prepare for anticipated continued high load through this week.  On Tuesday
> morning, the load exceeded even the unusually high levels experienced on
> Monday and ultimately caused the entire system to become unusable.
> Unfortunately the root cause of this problem - insufficient capacity on
> the legacy CalMail environment - cannot be resolved safely without the new
> storage array that is not expected to be available until the weekend in
> spite of overnight delivery of key components and around the clock work of
> staff and vendors. We recognize how critical email is, this week in
> particular, to our ability to perform our work and have taken the
> following immediate actions to assist in lowering the load to allow email
> to continue to flow.
> 1)While faculty and staff email must remain on a university provisioned
> email service, students with external email accounts are encouraged to
> forward their CalMail messages there.  Doing so will lower both email
> volume and login attempts and thereby reduce load on the Calmail system.
> 2)Access to email from anything but the CalMail web clients (available at
> http:calmail.berkeley.edu) has been disabled. This dramatically reduces
> the number of simultaneous connections from cell phones, iPads, and
> clients such as Outlook and MacMail, which are often configured to
> maintain persistent connections and in doing so place extremely heavy load
> on the CalMail platform.
> 3)Moving some users to other campus email services temporarily to further
> reduce load.
>
> While these are drastic actions, none were undertaken lightly but were
> done after extensive consultation with technical experts from campus
> central and departmental staff as well as vendors we have enlisted to work
> on the problem.
> Once the new storage system is installed, configured and tested, we will
> begin the migration process from the legacy storage system to one more
> than double in size and expected to handle at least several times our
> current load.  That process itself will put substantial load on the legacy
> storage environment as data is copied so we are planning on doing this
> work off hours.
>
> We will continue to provide updates and information about the latest plans
> via Calmessages as well as posting the information here:
> http://ist.berkeley.edu/ciocalmailupdates
> We will also continue to provide regular updates via
> http://systemstatus.berkeley.edu
>
>




-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.



--
Phyllis Orrick
Communications Director
Safe Transportation Research and Education Center (SafeTREC)
University of California Transportation Center (UCTC)
Institute for Urban and Regional Development (IURD)
2614 Dwight Way
UC Berkeley
Berkeley CA 94720-7374
510-643-1779
@transsafe
@californiaUTC

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

Ian Crew

Me too--I really appreciate the efforts all the folks who are working so hard to get this resolved.  I'm sure we all know the nasty feeling in the pit of our stomachs when a service we're responsible for isn't working the way it should.  So, to all the people working on CalMail, my sympathies and thoughts are with you!

Phyllis:  It sounds like what you're describing is the WarnMe service--see http://warnme.berkeley.edu and http://emergency.berkeley.edu.  If the folks in your unit haven't signed up, this might be a good opportunity to get them to do so.

Cheers,

Ian

On Wed, 30 Nov 2011 07:25:15 -0800, Phyllis Orrick <[hidden email]> wrote:

I endorse Christopher's comments and appreciation for the IT teams that are working to resolve this.

At our unit, we are taking this as a learning moment, as Christopher suggests, by compiling an internally sharable list of all staff's alternate email addresses and encouraging staff who don't already have ones that they're comfortable sharing to establish one.
I'd like to hear thoughts about another emergency suggestion, made by the instructor at a use of social media workshop hosted recently at Haas: that the university establish a list of Twitter feeds to distribute information in the event that email won't suffice. This is already being done at major institutions, and no doubt someone here at Cal is hard at work on this. As the instructor pointed out, Twitter is often available when other services aren't and is an excellent broadcast tool. We are exploring setting up one for this purpose at our unit.
If anyone has info on Cal's efforts in this area, it would be a great time to share it. 
Phyllis

On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
Shel,
I appreciate all the efforts that you and the staff are going through to
bring Calmail back on line.

I take this as a wake up call for disaster recovery plans for my own cluster.

Under what circumstances do we fall back on a different set of Calmail
hardware?

Presumably there are disaster recovery plans that have us host our email
system elsewhere.

Are those plans publicly available for CalMail?  If the building that
contains Calmail is destroyed, what happens?  Is there some sort of
service level agreement and if a certain level of service is not met, then
we move our mail system to somewhere else (UC Davis?)?

At this point, it seems that moving Calmail to another location is not
necessary, but at what point would Calmail be moved?

These are all good questions and answers for a lessons learned document
that I hope is forthcoming.  I'd be happy with an email message to
Micronet that addresses these topics.

I briefly searched for information about Calmail's disaster recovery plans
but did not find anything.  It would be OE to determine what sort of
disaster recovery information should be made available and to have sites
like Calmail and the Hub provide this information.

I, for one, see this is an excellent moment to update my list of
alternative email addresses for my frequent campus contacts.

I'm also going to take this opportunity to check my back up system and
push forward on replacing some old hardware.

Many thanks again to everyone for all their efforts over the weekend.  I'm
sure some long, anguished hours have been put in.

_Christopher

Christopher Brooks, PMP                       University of California
CHESS Executive Director                      US Mail: 337 Cory Hall
Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
ph: 510.643.9841                                (Office: 545Q Cory)
home: (F-Tu) 707.665.0131 cell: 707.332.0670

Shel writes:

> I very much appreciate that it is unacceptable to have a core campus
> system like CalMail offline for any length of time.  Today`s continued
> problems with CalMail have been particularly difficult for all involved.
> We work hard to design and operate systems that can handle the needs of
> our community and when we fail to meet that standard we bring in outside
> experts to help us improve.  We have added additional outside experts from
> other campuses, vendors and new team members from the Berkeley technical
> community as we work through this crisis. This is our highest priority and
> will remain so until we have the environment fully stabilized.  The
> following message provides current information.  You can also check
> http://systemstatus.berkeley.edu for the latest status.
>
> Regards,
> Shel Waggener
> Associate Vice Chancellor and CIO
>
> CURRENT STATUS SUMMARY
> 1) The CalMail system is available only through web clients at
> http://calmail.berkeley.edu. All messages are being sent and received but
> can only be accessed via webmail.  Webmail sessions may be slower than
> normal due to volume.
> 2) Students are strongly encouraged to forward their email to alternate
> email accounts. Instructions for how to do that can be found on the
> Calmail site (http://calmail.berkeley.edu, Manage Account option).
>
> DETAILS:
>
> CalMail has been substantially impacted during the last 36 hours after the
> successful recovery from database corruption this weekend.  The load on
> the system has remained extremely high as the millions of backlogged
> messages are delivered.  The load situation worsened considerably Monday
> morning as tens of thousands of campus community members returned from the
> holidays and connected for the first time, pushing the load above
> operating limits.  Normal processes that usually run in the background
> unnoticed, including copying data off of a failed hard drive, effectively
> shut down the system for many people.  Attempts to keep email moving
> during the day Monday were minimally successful.  Monday night during off
> hours work was undertaken to accelerate the repair of the failed disk and
> prepare for anticipated continued high load through this week.  On Tuesday
> morning, the load exceeded even the unusually high levels experienced on
> Monday and ultimately caused the entire system to become unusable.
> Unfortunately the root cause of this problem - insufficient capacity on
> the legacy CalMail environment - cannot be resolved safely without the new
> storage array that is not expected to be available until the weekend in
> spite of overnight delivery of key components and around the clock work of
> staff and vendors. We recognize how critical email is, this week in
> particular, to our ability to perform our work and have taken the
> following immediate actions to assist in lowering the load to allow email
> to continue to flow.
> 1)While faculty and staff email must remain on a university provisioned
> email service, students with external email accounts are encouraged to
> forward their CalMail messages there.  Doing so will lower both email
> volume and login attempts and thereby reduce load on the Calmail system.
> 2)Access to email from anything but the CalMail web clients (available at
> http:calmail.berkeley.edu) has been disabled. This dramatically reduces
> the number of simultaneous connections from cell phones, iPads, and
> clients such as Outlook and MacMail, which are often configured to
> maintain persistent connections and in doing so place extremely heavy load
> on the CalMail platform.
> 3)Moving some users to other campus email services temporarily to further
> reduce load.
>
> While these are drastic actions, none were undertaken lightly but were
> done after extensive consultation with technical experts from campus
> central and departmental staff as well as vendors we have enlisted to work
> on the problem.
> Once the new storage system is installed, configured and tested, we will
> begin the migration process from the legacy storage system to one more
> than double in size and expected to handle at least several times our
> current load.  That process itself will put substantial load on the legacy
> storage environment as data is copied so we are planning on doing this
> work off hours.
>
> We will continue to provide updates and information about the latest plans
> via Calmessages as well as posting the information here:
> http://ist.berkeley.edu/ciocalmailupdates
> We will also continue to provide regular updates via
> http://systemstatus.berkeley.edu
>
>




-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

--
Phyllis Orrick
Communications Director
Safe Transportation Research and Education Center (SafeTREC)
University of California Transportation Center (UCTC)
Institute for Urban and Regional Development (IURD)
2614 Dwight Way
UC Berkeley
Berkeley CA 94720-7374
510-643-1779
@transsafe
@californiaUTC

 


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

Beth Muramoto
In reply to this post by Phyllis Orrick
Phyllis,

I join the voices of endorsements, appreciation and praise for all of the
tireless efforts by all of the IT personnel involved in this horrendous
process. I want them to know how much I appreciate and laud their efforts.

In regards to other ways to notify when email is not an option, I agree
that something has to be in place other than email to globally inform
people of changes such as that only the CalMail web page is available for
active email correspondence. I'm facing having to call many people to tell
them about that and to have them proliferate the information to faculty
and students who are not in my immediate communications pipeline. It's not
as simple a task as one might think it is.

I know that people are asking "what the heck did we do without email?" and
wondering how we can become so paralyzed or complain that we have to use a
phone to call people about these kinds of global problems, but the fact of
the matter is, email is THAT vital now and we can't realistically turn
back the clock. I admit to being just as dependent on it as a
communications conduit as anyone else so when it goes down
catastrophically as it has, some kind of alternate form of global
communications HAS to be put in place because in a true life and death
emergency, this kind of paralysis can't happen. WARNME is fine, but I
wonder how many people know about it or have even signed up. Also, it begs
the question, would you use WARNME for a situation like this? To tell
people globally to use CalMail on the web because they can't/won't get
their email from their client software or mobile devices? Is that a good
way to use it?

Beth







> I endorse Christopher's comments and appreciation for the IT teams that
> are
> working to resolve this.
>
> At our unit, we are taking this as a learning moment, as Christopher
> suggests, by compiling an internally sharable list of all staff's
> alternate
> email addresses and encouraging staff who don't already have ones that
> they're comfortable sharing to establish one.
>
> I'd like to hear thoughts about another emergency suggestion, made by the
> instructor at a use of social media workshop hosted recently at Haas: that
> the university establish a list of Twitter feeds to distribute information
> in the event that email won't suffice. This is already being done at major
> institutions, and no doubt someone here at Cal is hard at work on this. As
> the instructor pointed out, Twitter is often available when other services
> aren't and is an excellent broadcast tool. We are exploring setting up one
> for this purpose at our unit.
>
> If anyone has info on Cal's efforts in this area, it would be a great time
> to share it.
>
> Phyllis
>
> On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
>
>> Shel,
>> I appreciate all the efforts that you and the staff are going through to
>> bring Calmail back on line.
>>
>> I take this as a wake up call for disaster recovery plans for my own
>> cluster.
>>
>> Under what circumstances do we fall back on a different set of Calmail
>> hardware?
>>
>> Presumably there are disaster recovery plans that have us host our email
>> system elsewhere.
>>
>> Are those plans publicly available for CalMail?  If the building that
>> contains Calmail is destroyed, what happens?  Is there some sort of
>> service level agreement and if a certain level of service is not met,
>> then
>> we move our mail system to somewhere else (UC Davis?)?
>>
>> At this point, it seems that moving Calmail to another location is not
>> necessary, but at what point would Calmail be moved?
>>
>> These are all good questions and answers for a lessons learned document
>> that I hope is forthcoming.  I'd be happy with an email message to
>> Micronet that addresses these topics.
>>
>> I briefly searched for information about Calmail's disaster recovery
>> plans
>> but did not find anything.  It would be OE to determine what sort of
>> disaster recovery information should be made available and to have sites
>> like Calmail and the Hub provide this information.
>>
>> I, for one, see this is an excellent moment to update my list of
>> alternative email addresses for my frequent campus contacts.
>>
>> I'm also going to take this opportunity to check my back up system and
>> push forward on replacing some old hardware.
>>
>> Many thanks again to everyone for all their efforts over the weekend.
>> I'm
>> sure some long, anguished hours have been put in.
>>
>> _Christopher
>>
>> Christopher Brooks, PMP                       University of California
>> CHESS Executive Director                      US Mail: 337 Cory Hall
>> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
>> ph: 510.643.9841                                (Office: 545Q Cory)
>> home: (F-Tu) 707.665.0131 cell: 707.332.0670
>>
>> Shel writes:
>>
>> > I very much appreciate that it is unacceptable to have a core campus
>> > system like CalMail offline for any length of time.  Today`s continued
>> > problems with CalMail have been particularly difficult for all
>> involved.
>> > We work hard to design and operate systems that can handle the needs
>> of
>> > our community and when we fail to meet that standard we bring in
>> outside
>> > experts to help us improve.  We have added additional outside experts
>> from
>> > other campuses, vendors and new team members from the Berkeley
>> technical
>> > community as we work through this crisis. This is our highest priority
>> and
>> > will remain so until we have the environment fully stabilized.  The
>> > following message provides current information.  You can also check
>> > http://systemstatus.berkeley.edu for the latest status.
>> >
>> > Regards,
>> > Shel Waggener
>> > Associate Vice Chancellor and CIO
>> >
>> > CURRENT STATUS SUMMARY
>> > 1) The CalMail system is available only through web clients at
>> > http://calmail.berkeley.edu. All messages are being sent and received
>> but
>> > can only be accessed via webmail.  Webmail sessions may be slower than
>> > normal due to volume.
>> > 2) Students are strongly encouraged to forward their email to
>> alternate
>> > email accounts. Instructions for how to do that can be found on the
>> > Calmail site (http://calmail.berkeley.edu, Manage Account option).
>> >
>> > DETAILS:
>> >
>> > CalMail has been substantially impacted during the last 36 hours after
>> the
>> > successful recovery from database corruption this weekend.  The load
>> on
>> > the system has remained extremely high as the millions of backlogged
>> > messages are delivered.  The load situation worsened considerably
>> Monday
>> > morning as tens of thousands of campus community members returned from
>> the
>> > holidays and connected for the first time, pushing the load above
>> > operating limits.  Normal processes that usually run in the background
>> > unnoticed, including copying data off of a failed hard drive,
>> effectively
>> > shut down the system for many people.  Attempts to keep email moving
>> > during the day Monday were minimally successful.  Monday night during
>> off
>> > hours work was undertaken to accelerate the repair of the failed disk
>> and
>> > prepare for anticipated continued high load through this week.  On
>> Tuesday
>> > morning, the load exceeded even the unusually high levels experienced
>> on
>> > Monday and ultimately caused the entire system to become unusable.
>> > Unfortunately the root cause of this problem - insufficient capacity
>> on
>> > the legacy CalMail environment - cannot be resolved safely without the
>> new
>> > storage array that is not expected to be available until the weekend
>> in
>> > spite of overnight delivery of key components and around the clock
>> work
>> of
>> > staff and vendors. We recognize how critical email is, this week in
>> > particular, to our ability to perform our work and have taken the
>> > following immediate actions to assist in lowering the load to allow
>> email
>> > to continue to flow.
>> > 1)While faculty and staff email must remain on a university
>> provisioned
>> > email service, students with external email accounts are encouraged to
>> > forward their CalMail messages there.  Doing so will lower both email
>> > volume and login attempts and thereby reduce load on the Calmail
>> system.
>> > 2)Access to email from anything but the CalMail web clients (available
>> at
>> > http:calmail.berkeley.edu) has been disabled. This dramatically
>> reduces
>> > the number of simultaneous connections from cell phones, iPads, and
>> > clients such as Outlook and MacMail, which are often configured to
>> > maintain persistent connections and in doing so place extremely heavy
>> load
>> > on the CalMail platform.
>> > 3)Moving some users to other campus email services temporarily to
>> further
>> > reduce load.
>> >
>> > While these are drastic actions, none were undertaken lightly but were
>> > done after extensive consultation with technical experts from campus
>> > central and departmental staff as well as vendors we have enlisted to
>> work
>> > on the problem.
>> > Once the new storage system is installed, configured and tested, we
>> will
>> > begin the migration process from the legacy storage system to one more
>> > than double in size and expected to handle at least several times our
>> > current load.  That process itself will put substantial load on the
>> legacy
>> > storage environment as data is copied so we are planning on doing this
>> > work off hours.
>> >
>> > We will continue to provide updates and information about the latest
>> plans
>> > via Calmessages as well as posting the information here:
>> > http://ist.berkeley.edu/ciocalmailupdates
>> > We will also continue to provide regular updates via
>> > http://systemstatus.berkeley.edu
>> >
>> >
>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> The following was automatically added to this message by the list
>> server:
>>
>> To learn more about Micronet, including how to subscribe to or
>> unsubscribe
>> from its mailing list and how to find out about upcoming meetings,
>> please
>> visit the Micronet Web site:
>>
>> http://micronet.berkeley.edu
>>
>> Messages you send to this mailing list are public and world-viewable,
>> and
>> the list's archives can be browsed and searched on the Internet.  This
>> means these messages can be viewed by (among others) your bosses,
>> prospective employers, and people who have known you in the past.
>>
>
>
>
> --
> Phyllis Orrick
> Communications Director
> Safe Transportation Research and Education
> Center<http://www.safetrec.berkeley.edu>(SafeTREC)
> University of California Transportation Center <http://www.uctc.net>
> (UCTC)
> Institute for Urban and Regional Development
> <http://www.iurd.berkeley.edu>(IURD)
> 2614 Dwight Way
> UC Berkeley
> Berkeley CA 94720-7374
> 510-643-1779
> @transsafe <https://twitter.com/#!/transsafe>
> @californiaUTC <https://twitter.com/#!/CaliforniaUTC>
> @IURDBerkeley <https://twitter.com/#!/IURDBerkeley>
>
> -------------------------------------------------------------------------
> The following was automatically added to this message by the list server:
>
> To learn more about Micronet, including how to subscribe to or unsubscribe
> from its mailing list and how to find out about upcoming meetings, please
> visit the Micronet Web site:
>
> http://micronet.berkeley.edu
>
> Messages you send to this mailing list are public and world-viewable, and
> the list's archives can be browsed and searched on the Internet.  This
> means these messages can be viewed by (among others) your bosses,
> prospective employers, and people who have known you in the past.
>



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

billallison
In reply to this post by Phyllis Orrick
Just a quick note that the Calmail team has been operating a Twitter
account at   @UCBCalmail

We've been using it as a two-way communications vehicle.

> I endorse Christopher's comments and appreciation for the IT teams that
> are
> working to resolve this.
>
> At our unit, we are taking this as a learning moment, as Christopher
> suggests, by compiling an internally sharable list of all staff's
> alternate
> email addresses and encouraging staff who don't already have ones that
> they're comfortable sharing to establish one.
>
> I'd like to hear thoughts about another emergency suggestion, made by the
> instructor at a use of social media workshop hosted recently at Haas: that
> the university establish a list of Twitter feeds to distribute information
> in the event that email won't suffice. This is already being done at major
> institutions, and no doubt someone here at Cal is hard at work on this. As
> the instructor pointed out, Twitter is often available when other services
> aren't and is an excellent broadcast tool. We are exploring setting up one
> for this purpose at our unit.
>
> If anyone has info on Cal's efforts in this area, it would be a great time
> to share it.
>
> Phyllis
>
> On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
>
>> Shel,
>> I appreciate all the efforts that you and the staff are going through to
>> bring Calmail back on line.
>>
>> I take this as a wake up call for disaster recovery plans for my own
>> cluster.
>>
>> Under what circumstances do we fall back on a different set of Calmail
>> hardware?
>>
>> Presumably there are disaster recovery plans that have us host our email
>> system elsewhere.
>>
>> Are those plans publicly available for CalMail?  If the building that
>> contains Calmail is destroyed, what happens?  Is there some sort of
>> service level agreement and if a certain level of service is not met,
>> then
>> we move our mail system to somewhere else (UC Davis?)?
>>
>> At this point, it seems that moving Calmail to another location is not
>> necessary, but at what point would Calmail be moved?
>>
>> These are all good questions and answers for a lessons learned document
>> that I hope is forthcoming.  I'd be happy with an email message to
>> Micronet that addresses these topics.
>>
>> I briefly searched for information about Calmail's disaster recovery
>> plans
>> but did not find anything.  It would be OE to determine what sort of
>> disaster recovery information should be made available and to have sites
>> like Calmail and the Hub provide this information.
>>
>> I, for one, see this is an excellent moment to update my list of
>> alternative email addresses for my frequent campus contacts.
>>
>> I'm also going to take this opportunity to check my back up system and
>> push forward on replacing some old hardware.
>>
>> Many thanks again to everyone for all their efforts over the weekend.
>> I'm
>> sure some long, anguished hours have been put in.
>>
>> _Christopher
>>
>> Christopher Brooks, PMP                       University of California
>> CHESS Executive Director                      US Mail: 337 Cory Hall
>> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
>> ph: 510.643.9841                                (Office: 545Q Cory)
>> home: (F-Tu) 707.665.0131 cell: 707.332.0670
>>
>> Shel writes:
>>
>> > I very much appreciate that it is unacceptable to have a core campus
>> > system like CalMail offline for any length of time.  Today`s continued
>> > problems with CalMail have been particularly difficult for all
>> involved.
>> > We work hard to design and operate systems that can handle the needs
>> of
>> > our community and when we fail to meet that standard we bring in
>> outside
>> > experts to help us improve.  We have added additional outside experts
>> from
>> > other campuses, vendors and new team members from the Berkeley
>> technical
>> > community as we work through this crisis. This is our highest priority
>> and
>> > will remain so until we have the environment fully stabilized.  The
>> > following message provides current information.  You can also check
>> > http://systemstatus.berkeley.edu for the latest status.
>> >
>> > Regards,
>> > Shel Waggener
>> > Associate Vice Chancellor and CIO
>> >
>> > CURRENT STATUS SUMMARY
>> > 1) The CalMail system is available only through web clients at
>> > http://calmail.berkeley.edu. All messages are being sent and received
>> but
>> > can only be accessed via webmail.  Webmail sessions may be slower than
>> > normal due to volume.
>> > 2) Students are strongly encouraged to forward their email to
>> alternate
>> > email accounts. Instructions for how to do that can be found on the
>> > Calmail site (http://calmail.berkeley.edu, Manage Account option).
>> >
>> > DETAILS:
>> >
>> > CalMail has been substantially impacted during the last 36 hours after
>> the
>> > successful recovery from database corruption this weekend.  The load
>> on
>> > the system has remained extremely high as the millions of backlogged
>> > messages are delivered.  The load situation worsened considerably
>> Monday
>> > morning as tens of thousands of campus community members returned from
>> the
>> > holidays and connected for the first time, pushing the load above
>> > operating limits.  Normal processes that usually run in the background
>> > unnoticed, including copying data off of a failed hard drive,
>> effectively
>> > shut down the system for many people.  Attempts to keep email moving
>> > during the day Monday were minimally successful.  Monday night during
>> off
>> > hours work was undertaken to accelerate the repair of the failed disk
>> and
>> > prepare for anticipated continued high load through this week.  On
>> Tuesday
>> > morning, the load exceeded even the unusually high levels experienced
>> on
>> > Monday and ultimately caused the entire system to become unusable.
>> > Unfortunately the root cause of this problem - insufficient capacity
>> on
>> > the legacy CalMail environment - cannot be resolved safely without the
>> new
>> > storage array that is not expected to be available until the weekend
>> in
>> > spite of overnight delivery of key components and around the clock
>> work
>> of
>> > staff and vendors. We recognize how critical email is, this week in
>> > particular, to our ability to perform our work and have taken the
>> > following immediate actions to assist in lowering the load to allow
>> email
>> > to continue to flow.
>> > 1)While faculty and staff email must remain on a university
>> provisioned
>> > email service, students with external email accounts are encouraged to
>> > forward their CalMail messages there.  Doing so will lower both email
>> > volume and login attempts and thereby reduce load on the Calmail
>> system.
>> > 2)Access to email from anything but the CalMail web clients (available
>> at
>> > http:calmail.berkeley.edu) has been disabled. This dramatically
>> reduces
>> > the number of simultaneous connections from cell phones, iPads, and
>> > clients such as Outlook and MacMail, which are often configured to
>> > maintain persistent connections and in doing so place extremely heavy
>> load
>> > on the CalMail platform.
>> > 3)Moving some users to other campus email services temporarily to
>> further
>> > reduce load.
>> >
>> > While these are drastic actions, none were undertaken lightly but were
>> > done after extensive consultation with technical experts from campus
>> > central and departmental staff as well as vendors we have enlisted to
>> work
>> > on the problem.
>> > Once the new storage system is installed, configured and tested, we
>> will
>> > begin the migration process from the legacy storage system to one more
>> > than double in size and expected to handle at least several times our
>> > current load.  That process itself will put substantial load on the
>> legacy
>> > storage environment as data is copied so we are planning on doing this
>> > work off hours.
>> >
>> > We will continue to provide updates and information about the latest
>> plans
>> > via Calmessages as well as posting the information here:
>> > http://ist.berkeley.edu/ciocalmailupdates
>> > We will also continue to provide regular updates via
>> > http://systemstatus.berkeley.edu
>> >
>> >
>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> The following was automatically added to this message by the list
>> server:
>>
>> To learn more about Micronet, including how to subscribe to or
>> unsubscribe
>> from its mailing list and how to find out about upcoming meetings,
>> please
>> visit the Micronet Web site:
>>
>> http://micronet.berkeley.edu
>>
>> Messages you send to this mailing list are public and world-viewable,
>> and
>> the list's archives can be browsed and searched on the Internet.  This
>> means these messages can be viewed by (among others) your bosses,
>> prospective employers, and people who have known you in the past.
>>
>
>
>
> --
> Phyllis Orrick
> Communications Director
> Safe Transportation Research and Education
> Center<http://www.safetrec.berkeley.edu>(SafeTREC)
> University of California Transportation Center <http://www.uctc.net>
> (UCTC)
> Institute for Urban and Regional Development
> <http://www.iurd.berkeley.edu>(IURD)
> 2614 Dwight Way
> UC Berkeley
> Berkeley CA 94720-7374
> 510-643-1779
> @transsafe <https://twitter.com/#!/transsafe>
> @californiaUTC <https://twitter.com/#!/CaliforniaUTC>
> @IURDBerkeley <https://twitter.com/#!/IURDBerkeley>
>
> -------------------------------------------------------------------------
> The following was automatically added to this message by the list server:
>
> To learn more about Micronet, including how to subscribe to or unsubscribe
> from its mailing list and how to find out about upcoming meetings, please
> visit the Micronet Web site:
>
> http://micronet.berkeley.edu
>
> Messages you send to this mailing list are public and world-viewable, and
> the list's archives can be browsed and searched on the Internet.  This
> means these messages can be viewed by (among others) your bosses,
> prospective employers, and people who have known you in the past.
>



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

Beth Muramoto
Bill,

Thanks for telling us about this. I'm now following it, but I have to be
honest, until you mentioned it, I had no idea that CalMail had a twitter
account so that's great to know as a twitter user. Was there an
announcement about it that I missed?

Thanks again.

Beth

> Just a quick note that the Calmail team has been operating a Twitter
> account at   @UCBCalmail
>
> We've been using it as a two-way communications vehicle.
>
>> I endorse Christopher's comments and appreciation for the IT teams that
>> are
>> working to resolve this.
>>
>> At our unit, we are taking this as a learning moment, as Christopher
>> suggests, by compiling an internally sharable list of all staff's
>> alternate
>> email addresses and encouraging staff who don't already have ones that
>> they're comfortable sharing to establish one.
>>
>> I'd like to hear thoughts about another emergency suggestion, made by
>> the
>> instructor at a use of social media workshop hosted recently at Haas:
>> that
>> the university establish a list of Twitter feeds to distribute
>> information
>> in the event that email won't suffice. This is already being done at
>> major
>> institutions, and no doubt someone here at Cal is hard at work on this.
>> As
>> the instructor pointed out, Twitter is often available when other
>> services
>> aren't and is an excellent broadcast tool. We are exploring setting up
>> one
>> for this purpose at our unit.
>>
>> If anyone has info on Cal's efforts in this area, it would be a great
>> time
>> to share it.
>>
>> Phyllis
>>
>> On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
>>
>>> Shel,
>>> I appreciate all the efforts that you and the staff are going through
>>> to
>>> bring Calmail back on line.
>>>
>>> I take this as a wake up call for disaster recovery plans for my own
>>> cluster.
>>>
>>> Under what circumstances do we fall back on a different set of Calmail
>>> hardware?
>>>
>>> Presumably there are disaster recovery plans that have us host our
>>> email
>>> system elsewhere.
>>>
>>> Are those plans publicly available for CalMail?  If the building that
>>> contains Calmail is destroyed, what happens?  Is there some sort of
>>> service level agreement and if a certain level of service is not met,
>>> then
>>> we move our mail system to somewhere else (UC Davis?)?
>>>
>>> At this point, it seems that moving Calmail to another location is not
>>> necessary, but at what point would Calmail be moved?
>>>
>>> These are all good questions and answers for a lessons learned document
>>> that I hope is forthcoming.  I'd be happy with an email message to
>>> Micronet that addresses these topics.
>>>
>>> I briefly searched for information about Calmail's disaster recovery
>>> plans
>>> but did not find anything.  It would be OE to determine what sort of
>>> disaster recovery information should be made available and to have
>>> sites
>>> like Calmail and the Hub provide this information.
>>>
>>> I, for one, see this is an excellent moment to update my list of
>>> alternative email addresses for my frequent campus contacts.
>>>
>>> I'm also going to take this opportunity to check my back up system and
>>> push forward on replacing some old hardware.
>>>
>>> Many thanks again to everyone for all their efforts over the weekend.
>>> I'm
>>> sure some long, anguished hours have been put in.
>>>
>>> _Christopher
>>>
>>> Christopher Brooks, PMP                       University of California
>>> CHESS Executive Director                      US Mail: 337 Cory Hall
>>> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
>>> ph: 510.643.9841                                (Office: 545Q Cory)
>>> home: (F-Tu) 707.665.0131 cell: 707.332.0670
>>>
>>> Shel writes:
>>>
>>> > I very much appreciate that it is unacceptable to have a core campus
>>> > system like CalMail offline for any length of time.  Today`s
>>> continued
>>> > problems with CalMail have been particularly difficult for all
>>> involved.
>>> > We work hard to design and operate systems that can handle the needs
>>> of
>>> > our community and when we fail to meet that standard we bring in
>>> outside
>>> > experts to help us improve.  We have added additional outside experts
>>> from
>>> > other campuses, vendors and new team members from the Berkeley
>>> technical
>>> > community as we work through this crisis. This is our highest
>>> priority
>>> and
>>> > will remain so until we have the environment fully stabilized.  The
>>> > following message provides current information.  You can also check
>>> > http://systemstatus.berkeley.edu for the latest status.
>>> >
>>> > Regards,
>>> > Shel Waggener
>>> > Associate Vice Chancellor and CIO
>>> >
>>> > CURRENT STATUS SUMMARY
>>> > 1) The CalMail system is available only through web clients at
>>> > http://calmail.berkeley.edu. All messages are being sent and received
>>> but
>>> > can only be accessed via webmail.  Webmail sessions may be slower
>>> than
>>> > normal due to volume.
>>> > 2) Students are strongly encouraged to forward their email to
>>> alternate
>>> > email accounts. Instructions for how to do that can be found on the
>>> > Calmail site (http://calmail.berkeley.edu, Manage Account option).
>>> >
>>> > DETAILS:
>>> >
>>> > CalMail has been substantially impacted during the last 36 hours
>>> after
>>> the
>>> > successful recovery from database corruption this weekend.  The load
>>> on
>>> > the system has remained extremely high as the millions of backlogged
>>> > messages are delivered.  The load situation worsened considerably
>>> Monday
>>> > morning as tens of thousands of campus community members returned
>>> from
>>> the
>>> > holidays and connected for the first time, pushing the load above
>>> > operating limits.  Normal processes that usually run in the
>>> background
>>> > unnoticed, including copying data off of a failed hard drive,
>>> effectively
>>> > shut down the system for many people.  Attempts to keep email moving
>>> > during the day Monday were minimally successful.  Monday night during
>>> off
>>> > hours work was undertaken to accelerate the repair of the failed disk
>>> and
>>> > prepare for anticipated continued high load through this week.  On
>>> Tuesday
>>> > morning, the load exceeded even the unusually high levels experienced
>>> on
>>> > Monday and ultimately caused the entire system to become unusable.
>>> > Unfortunately the root cause of this problem - insufficient capacity
>>> on
>>> > the legacy CalMail environment - cannot be resolved safely without
>>> the
>>> new
>>> > storage array that is not expected to be available until the weekend
>>> in
>>> > spite of overnight delivery of key components and around the clock
>>> work
>>> of
>>> > staff and vendors. We recognize how critical email is, this week in
>>> > particular, to our ability to perform our work and have taken the
>>> > following immediate actions to assist in lowering the load to allow
>>> email
>>> > to continue to flow.
>>> > 1)While faculty and staff email must remain on a university
>>> provisioned
>>> > email service, students with external email accounts are encouraged
>>> to
>>> > forward their CalMail messages there.  Doing so will lower both email
>>> > volume and login attempts and thereby reduce load on the Calmail
>>> system.
>>> > 2)Access to email from anything but the CalMail web clients
>>> (available
>>> at
>>> > http:calmail.berkeley.edu) has been disabled. This dramatically
>>> reduces
>>> > the number of simultaneous connections from cell phones, iPads, and
>>> > clients such as Outlook and MacMail, which are often configured to
>>> > maintain persistent connections and in doing so place extremely heavy
>>> load
>>> > on the CalMail platform.
>>> > 3)Moving some users to other campus email services temporarily to
>>> further
>>> > reduce load.
>>> >
>>> > While these are drastic actions, none were undertaken lightly but
>>> were
>>> > done after extensive consultation with technical experts from campus
>>> > central and departmental staff as well as vendors we have enlisted to
>>> work
>>> > on the problem.
>>> > Once the new storage system is installed, configured and tested, we
>>> will
>>> > begin the migration process from the legacy storage system to one
>>> more
>>> > than double in size and expected to handle at least several times our
>>> > current load.  That process itself will put substantial load on the
>>> legacy
>>> > storage environment as data is copied so we are planning on doing
>>> this
>>> > work off hours.
>>> >
>>> > We will continue to provide updates and information about the latest
>>> plans
>>> > via Calmessages as well as posting the information here:
>>> > http://ist.berkeley.edu/ciocalmailupdates
>>> > We will also continue to provide regular updates via
>>> > http://systemstatus.berkeley.edu
>>> >
>>> >
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------------
>>> The following was automatically added to this message by the list
>>> server:
>>>
>>> To learn more about Micronet, including how to subscribe to or
>>> unsubscribe
>>> from its mailing list and how to find out about upcoming meetings,
>>> please
>>> visit the Micronet Web site:
>>>
>>> http://micronet.berkeley.edu
>>>
>>> Messages you send to this mailing list are public and world-viewable,
>>> and
>>> the list's archives can be browsed and searched on the Internet.  This
>>> means these messages can be viewed by (among others) your bosses,
>>> prospective employers, and people who have known you in the past.
>>>
>>
>>
>>
>> --
>> Phyllis Orrick
>> Communications Director
>> Safe Transportation Research and Education
>> Center<http://www.safetrec.berkeley.edu>(SafeTREC)
>> University of California Transportation Center <http://www.uctc.net>
>> (UCTC)
>> Institute for Urban and Regional Development
>> <http://www.iurd.berkeley.edu>(IURD)
>> 2614 Dwight Way
>> UC Berkeley
>> Berkeley CA 94720-7374
>> 510-643-1779
>> @transsafe <https://twitter.com/#!/transsafe>
>> @californiaUTC <https://twitter.com/#!/CaliforniaUTC>
>> @IURDBerkeley <https://twitter.com/#!/IURDBerkeley>
>>
>> -------------------------------------------------------------------------
>> The following was automatically added to this message by the list
>> server:
>>
>> To learn more about Micronet, including how to subscribe to or
>> unsubscribe
>> from its mailing list and how to find out about upcoming meetings,
>> please
>> visit the Micronet Web site:
>>
>> http://micronet.berkeley.edu
>>
>> Messages you send to this mailing list are public and world-viewable,
>> and
>> the list's archives can be browsed and searched on the Internet.  This
>> means these messages can be viewed by (among others) your bosses,
>> prospective employers, and people who have known you in the past.
>>
>
>
>
>
> -------------------------------------------------------------------------
> The following was automatically added to this message by the list server:
>
> To learn more about Micronet, including how to subscribe to or unsubscribe
> from its mailing list and how to find out about upcoming meetings, please
> visit the Micronet Web site:
>
> http://micronet.berkeley.edu
>
> Messages you send to this mailing list are public and world-viewable, and
> the list's archives can be browsed and searched on the Internet.  This
> means these messages can be viewed by (among others) your bosses,
> prospective employers, and people who have known you in the past.
>



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

Graham Patterson
In reply to this post by Beth Muramoto

We use our internal staff web page for service notices. It does mean one
of our team has to keep an eye on the available information streams. The
important thing is to filter the information to suit the audience.
ucbstystems.org is useful to me, but 90% of the content is of no interest
to the regular user in any one department.

I am surprised that more people have not been using the phone 8-)




On Wed, November 30, 2011 8:28 am, [hidden email] wrote:

> Phyllis,
>
> I join the voices of endorsements, appreciation and praise for all of the
> tireless efforts by all of the IT personnel involved in this horrendous
> process. I want them to know how much I appreciate and laud their efforts.
>
> In regards to other ways to notify when email is not an option, I agree
> that something has to be in place other than email to globally inform
> people of changes such as that only the CalMail web page is available for
> active email correspondence. I'm facing having to call many people to tell
> them about that and to have them proliferate the information to faculty
> and students who are not in my immediate communications pipeline. It's not
> as simple a task as one might think it is.
>
> I know that people are asking "what the heck did we do without email?" and
> wondering how we can become so paralyzed or complain that we have to use a
> phone to call people about these kinds of global problems, but the fact of
> the matter is, email is THAT vital now and we can't realistically turn
> back the clock. I admit to being just as dependent on it as a
> communications conduit as anyone else so when it goes down
> catastrophically as it has, some kind of alternate form of global
> communications HAS to be put in place because in a true life and death
> emergency, this kind of paralysis can't happen. WARNME is fine, but I
> wonder how many people know about it or have even signed up. Also, it begs
> the question, would you use WARNME for a situation like this? To tell
> people globally to use CalMail on the web because they can't/won't get
> their email from their client software or mobile devices? Is that a good
> way to use it?
>
> Beth
>
>
>
>
>
>
>
>> I endorse Christopher's comments and appreciation for the IT teams that
>> are
>> working to resolve this.
>>
>> At our unit, we are taking this as a learning moment, as Christopher
>> suggests, by compiling an internally sharable list of all staff's
>> alternate
>> email addresses and encouraging staff who don't already have ones that
>> they're comfortable sharing to establish one.
>>
>> I'd like to hear thoughts about another emergency suggestion, made by
>> the
>> instructor at a use of social media workshop hosted recently at Haas:
>> that
>> the university establish a list of Twitter feeds to distribute
>> information
>> in the event that email won't suffice. This is already being done at
>> major
>> institutions, and no doubt someone here at Cal is hard at work on this.
>> As
>> the instructor pointed out, Twitter is often available when other
>> services
>> aren't and is an excellent broadcast tool. We are exploring setting up
>> one
>> for this purpose at our unit.
>>
>> If anyone has info on Cal's efforts in this area, it would be a great
>> time
>> to share it.
>>
>> Phyllis
>>
>> On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
>>
>>> Shel,
>>> I appreciate all the efforts that you and the staff are going through
>>> to
>>> bring Calmail back on line.
>>>
>>> I take this as a wake up call for disaster recovery plans for my own
>>> cluster.
>>>
>>> Under what circumstances do we fall back on a different set of Calmail
>>> hardware?
>>>
>>> Presumably there are disaster recovery plans that have us host our
>>> email
>>> system elsewhere.
>>>
>>> Are those plans publicly available for CalMail?  If the building that
>>> contains Calmail is destroyed, what happens?  Is there some sort of
>>> service level agreement and if a certain level of service is not met,
>>> then
>>> we move our mail system to somewhere else (UC Davis?)?
>>>
>>> At this point, it seems that moving Calmail to another location is not
>>> necessary, but at what point would Calmail be moved?
>>>
>>> These are all good questions and answers for a lessons learned document
>>> that I hope is forthcoming.  I'd be happy with an email message to
>>> Micronet that addresses these topics.
>>>
>>> I briefly searched for information about Calmail's disaster recovery
>>> plans
>>> but did not find anything.  It would be OE to determine what sort of
>>> disaster recovery information should be made available and to have
>>> sites
>>> like Calmail and the Hub provide this information.
>>>
>>> I, for one, see this is an excellent moment to update my list of
>>> alternative email addresses for my frequent campus contacts.
>>>
>>> I'm also going to take this opportunity to check my back up system and
>>> push forward on replacing some old hardware.
>>>
>>> Many thanks again to everyone for all their efforts over the weekend.
>>> I'm
>>> sure some long, anguished hours have been put in.
>>>
>>> _Christopher
>>>
>>> Christopher Brooks, PMP                       University of California
>>> CHESS Executive Director                      US Mail: 337 Cory Hall
>>> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
>>> ph: 510.643.9841                                (Office: 545Q Cory)
>>> home: (F-Tu) 707.665.0131 cell: 707.332.0670
>>>
>>> Shel writes:
>>>
>>> > I very much appreciate that it is unacceptable to have a core campus
>>> > system like CalMail offline for any length of time.  Today`s
>>> continued
>>> > problems with CalMail have been particularly difficult for all
>>> involved.
>>> > We work hard to design and operate systems that can handle the needs
>>> of
>>> > our community and when we fail to meet that standard we bring in
>>> outside
>>> > experts to help us improve.  We have added additional outside experts
>>> from
>>> > other campuses, vendors and new team members from the Berkeley
>>> technical
>>> > community as we work through this crisis. This is our highest
>>> priority
>>> and
>>> > will remain so until we have the environment fully stabilized.  The
>>> > following message provides current information.  You can also check
>>> > http://systemstatus.berkeley.edu for the latest status.
>>> >
>>> > Regards,
>>> > Shel Waggener
>>> > Associate Vice Chancellor and CIO
>>> >
>>> > CURRENT STATUS SUMMARY
>>> > 1) The CalMail system is available only through web clients at
>>> > http://calmail.berkeley.edu. All messages are being sent and received
>>> but
>>> > can only be accessed via webmail.  Webmail sessions may be slower
>>> than
>>> > normal due to volume.
>>> > 2) Students are strongly encouraged to forward their email to
>>> alternate
>>> > email accounts. Instructions for how to do that can be found on the
>>> > Calmail site (http://calmail.berkeley.edu, Manage Account option).
>>> >
>>> > DETAILS:
>>> >
>>> > CalMail has been substantially impacted during the last 36 hours
>>> after
>>> the
>>> > successful recovery from database corruption this weekend.  The load
>>> on
>>> > the system has remained extremely high as the millions of backlogged
>>> > messages are delivered.  The load situation worsened considerably
>>> Monday
>>> > morning as tens of thousands of campus community members returned
>>> from
>>> the
>>> > holidays and connected for the first time, pushing the load above
>>> > operating limits.  Normal processes that usually run in the
>>> background
>>> > unnoticed, including copying data off of a failed hard drive,
>>> effectively
>>> > shut down the system for many people.  Attempts to keep email moving
>>> > during the day Monday were minimally successful.  Monday night during
>>> off
>>> > hours work was undertaken to accelerate the repair of the failed disk
>>> and
>>> > prepare for anticipated continued high load through this week.  On
>>> Tuesday
>>> > morning, the load exceeded even the unusually high levels experienced
>>> on
>>> > Monday and ultimately caused the entire system to become unusable.
>>> > Unfortunately the root cause of this problem - insufficient capacity
>>> on
>>> > the legacy CalMail environment - cannot be resolved safely without
>>> the
>>> new
>>> > storage array that is not expected to be available until the weekend
>>> in
>>> > spite of overnight delivery of key components and around the clock
>>> work
>>> of
>>> > staff and vendors. We recognize how critical email is, this week in
>>> > particular, to our ability to perform our work and have taken the
>>> > following immediate actions to assist in lowering the load to allow
>>> email
>>> > to continue to flow.
>>> > 1)While faculty and staff email must remain on a university
>>> provisioned
>>> > email service, students with external email accounts are encouraged
>>> to
>>> > forward their CalMail messages there.  Doing so will lower both email
>>> > volume and login attempts and thereby reduce load on the Calmail
>>> system.
>>> > 2)Access to email from anything but the CalMail web clients
>>> (available
>>> at
>>> > http:calmail.berkeley.edu) has been disabled. This dramatically
>>> reduces
>>> > the number of simultaneous connections from cell phones, iPads, and
>>> > clients such as Outlook and MacMail, which are often configured to
>>> > maintain persistent connections and in doing so place extremely heavy
>>> load
>>> > on the CalMail platform.
>>> > 3)Moving some users to other campus email services temporarily to
>>> further
>>> > reduce load.
>>> >
>>> > While these are drastic actions, none were undertaken lightly but
>>> were
>>> > done after extensive consultation with technical experts from campus
>>> > central and departmental staff as well as vendors we have enlisted to
>>> work
>>> > on the problem.
>>> > Once the new storage system is installed, configured and tested, we
>>> will
>>> > begin the migration process from the legacy storage system to one
>>> more
>>> > than double in size and expected to handle at least several times our
>>> > current load.  That process itself will put substantial load on the
>>> legacy
>>> > storage environment as data is copied so we are planning on doing
>>> this
>>> > work off hours.
>>> >
>>> > We will continue to provide updates and information about the latest
>>> plans
>>> > via Calmessages as well as posting the information here:
>>> > http://ist.berkeley.edu/ciocalmailupdates
>>> > We will also continue to provide regular updates via
>>> > http://systemstatus.berkeley.edu
>>> >
>>> >
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------------
>>> The following was automatically added to this message by the list
>>> server:
>>>
>>> To learn more about Micronet, including how to subscribe to or
>>> unsubscribe
>>> from its mailing list and how to find out about upcoming meetings,
>>> please
>>> visit the Micronet Web site:
>>>
>>> http://micronet.berkeley.edu
>>>
>>> Messages you send to this mailing list are public and world-viewable,
>>> and
>>> the list's archives can be browsed and searched on the Internet.  This
>>> means these messages can be viewed by (among others) your bosses,
>>> prospective employers, and people who have known you in the past.
>>>
>>
>>
>>
>> --
>> Phyllis Orrick
>> Communications Director
>> Safe Transportation Research and Education
>> Center<http://www.safetrec.berkeley.edu>(SafeTREC)
>> University of California Transportation Center <http://www.uctc.net>
>> (UCTC)
>> Institute for Urban and Regional Development
>> <http://www.iurd.berkeley.edu>(IURD)
>> 2614 Dwight Way
>> UC Berkeley
>> Berkeley CA 94720-7374
>> 510-643-1779
>> @transsafe <https://twitter.com/#!/transsafe>
>> @californiaUTC <https://twitter.com/#!/CaliforniaUTC>
>> @IURDBerkeley <https://twitter.com/#!/IURDBerkeley>
>>
>> -------------------------------------------------------------------------
>> The following was automatically added to this message by the list
>> server:
>>
>> To learn more about Micronet, including how to subscribe to or
>> unsubscribe
>> from its mailing list and how to find out about upcoming meetings,
>> please
>> visit the Micronet Web site:
>>
>> http://micronet.berkeley.edu
>>
>> Messages you send to this mailing list are public and world-viewable,
>> and
>> the list's archives can be browsed and searched on the Internet.  This
>> means these messages can be viewed by (among others) your bosses,
>> prospective employers, and people who have known you in the past.
>>
>
>
>
>
> -------------------------------------------------------------------------
> The following was automatically added to this message by the list server:
>
> To learn more about Micronet, including how to subscribe to or unsubscribe
> from its mailing list and how to find out about upcoming meetings, please
> visit the Micronet Web site:
>
> http://micronet.berkeley.edu
>
> Messages you send to this mailing list are public and world-viewable, and
> the list's archives can be browsed and searched on the Internet.  This
> means these messages can be viewed by (among others) your bosses,
> prospective employers, and people who have known you in the past.
>


--
Graham Patterson, System Administrator
Lawrence Hall of Science, UC Berkeley



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

billallison
In reply to this post by Beth Muramoto
not a big one, but email was down then.  It was noted on
systemstatus.berkeley.edu I think

> Bill,
>
> Thanks for telling us about this. I'm now following it, but I have to be
> honest, until you mentioned it, I had no idea that CalMail had a twitter
> account so that's great to know as a twitter user. Was there an
> announcement about it that I missed?
>
> Thanks again.
>
> Beth
>
>> Just a quick note that the Calmail team has been operating a Twitter
>> account at   @UCBCalmail
>>
>> We've been using it as a two-way communications vehicle.
>>
>>> I endorse Christopher's comments and appreciation for the IT teams that
>>> are
>>> working to resolve this.
>>>
>>> At our unit, we are taking this as a learning moment, as Christopher
>>> suggests, by compiling an internally sharable list of all staff's
>>> alternate
>>> email addresses and encouraging staff who don't already have ones that
>>> they're comfortable sharing to establish one.
>>>
>>> I'd like to hear thoughts about another emergency suggestion, made by
>>> the
>>> instructor at a use of social media workshop hosted recently at Haas:
>>> that
>>> the university establish a list of Twitter feeds to distribute
>>> information
>>> in the event that email won't suffice. This is already being done at
>>> major
>>> institutions, and no doubt someone here at Cal is hard at work on this.
>>> As
>>> the instructor pointed out, Twitter is often available when other
>>> services
>>> aren't and is an excellent broadcast tool. We are exploring setting up
>>> one
>>> for this purpose at our unit.
>>>
>>> If anyone has info on Cal's efforts in this area, it would be a great
>>> time
>>> to share it.
>>>
>>> Phyllis
>>>
>>> On Wed, Nov 30, 2011 at 4:30 AM, <[hidden email]> wrote:
>>>
>>>> Shel,
>>>> I appreciate all the efforts that you and the staff are going through
>>>> to
>>>> bring Calmail back on line.
>>>>
>>>> I take this as a wake up call for disaster recovery plans for my own
>>>> cluster.
>>>>
>>>> Under what circumstances do we fall back on a different set of Calmail
>>>> hardware?
>>>>
>>>> Presumably there are disaster recovery plans that have us host our
>>>> email
>>>> system elsewhere.
>>>>
>>>> Are those plans publicly available for CalMail?  If the building that
>>>> contains Calmail is destroyed, what happens?  Is there some sort of
>>>> service level agreement and if a certain level of service is not met,
>>>> then
>>>> we move our mail system to somewhere else (UC Davis?)?
>>>>
>>>> At this point, it seems that moving Calmail to another location is not
>>>> necessary, but at what point would Calmail be moved?
>>>>
>>>> These are all good questions and answers for a lessons learned
>>>> document
>>>> that I hope is forthcoming.  I'd be happy with an email message to
>>>> Micronet that addresses these topics.
>>>>
>>>> I briefly searched for information about Calmail's disaster recovery
>>>> plans
>>>> but did not find anything.  It would be OE to determine what sort of
>>>> disaster recovery information should be made available and to have
>>>> sites
>>>> like Calmail and the Hub provide this information.
>>>>
>>>> I, for one, see this is an excellent moment to update my list of
>>>> alternative email addresses for my frequent campus contacts.
>>>>
>>>> I'm also going to take this opportunity to check my back up system and
>>>> push forward on replacing some old hardware.
>>>>
>>>> Many thanks again to everyone for all their efforts over the weekend.
>>>> I'm
>>>> sure some long, anguished hours have been put in.
>>>>
>>>> _Christopher
>>>>
>>>> Christopher Brooks, PMP                       University of California
>>>> CHESS Executive Director                      US Mail: 337 Cory Hall
>>>> Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA 94720-1774
>>>> ph: 510.643.9841                                (Office: 545Q Cory)
>>>> home: (F-Tu) 707.665.0131 cell: 707.332.0670
>>>>
>>>> Shel writes:
>>>>
>>>> > I very much appreciate that it is unacceptable to have a core campus
>>>> > system like CalMail offline for any length of time.  Today`s
>>>> continued
>>>> > problems with CalMail have been particularly difficult for all
>>>> involved.
>>>> > We work hard to design and operate systems that can handle the needs
>>>> of
>>>> > our community and when we fail to meet that standard we bring in
>>>> outside
>>>> > experts to help us improve.  We have added additional outside
>>>> experts
>>>> from
>>>> > other campuses, vendors and new team members from the Berkeley
>>>> technical
>>>> > community as we work through this crisis. This is our highest
>>>> priority
>>>> and
>>>> > will remain so until we have the environment fully stabilized.  The
>>>> > following message provides current information.  You can also check
>>>> > http://systemstatus.berkeley.edu for the latest status.
>>>> >
>>>> > Regards,
>>>> > Shel Waggener
>>>> > Associate Vice Chancellor and CIO
>>>> >
>>>> > CURRENT STATUS SUMMARY
>>>> > 1) The CalMail system is available only through web clients at
>>>> > http://calmail.berkeley.edu. All messages are being sent and
>>>> received
>>>> but
>>>> > can only be accessed via webmail.  Webmail sessions may be slower
>>>> than
>>>> > normal due to volume.
>>>> > 2) Students are strongly encouraged to forward their email to
>>>> alternate
>>>> > email accounts. Instructions for how to do that can be found on the
>>>> > Calmail site (http://calmail.berkeley.edu, Manage Account option).
>>>> >
>>>> > DETAILS:
>>>> >
>>>> > CalMail has been substantially impacted during the last 36 hours
>>>> after
>>>> the
>>>> > successful recovery from database corruption this weekend.  The load
>>>> on
>>>> > the system has remained extremely high as the millions of backlogged
>>>> > messages are delivered.  The load situation worsened considerably
>>>> Monday
>>>> > morning as tens of thousands of campus community members returned
>>>> from
>>>> the
>>>> > holidays and connected for the first time, pushing the load above
>>>> > operating limits.  Normal processes that usually run in the
>>>> background
>>>> > unnoticed, including copying data off of a failed hard drive,
>>>> effectively
>>>> > shut down the system for many people.  Attempts to keep email moving
>>>> > during the day Monday were minimally successful.  Monday night
>>>> during
>>>> off
>>>> > hours work was undertaken to accelerate the repair of the failed
>>>> disk
>>>> and
>>>> > prepare for anticipated continued high load through this week.  On
>>>> Tuesday
>>>> > morning, the load exceeded even the unusually high levels
>>>> experienced
>>>> on
>>>> > Monday and ultimately caused the entire system to become unusable.
>>>> > Unfortunately the root cause of this problem - insufficient capacity
>>>> on
>>>> > the legacy CalMail environment - cannot be resolved safely without
>>>> the
>>>> new
>>>> > storage array that is not expected to be available until the weekend
>>>> in
>>>> > spite of overnight delivery of key components and around the clock
>>>> work
>>>> of
>>>> > staff and vendors. We recognize how critical email is, this week in
>>>> > particular, to our ability to perform our work and have taken the
>>>> > following immediate actions to assist in lowering the load to allow
>>>> email
>>>> > to continue to flow.
>>>> > 1)While faculty and staff email must remain on a university
>>>> provisioned
>>>> > email service, students with external email accounts are encouraged
>>>> to
>>>> > forward their CalMail messages there.  Doing so will lower both
>>>> email
>>>> > volume and login attempts and thereby reduce load on the Calmail
>>>> system.
>>>> > 2)Access to email from anything but the CalMail web clients
>>>> (available
>>>> at
>>>> > http:calmail.berkeley.edu) has been disabled. This dramatically
>>>> reduces
>>>> > the number of simultaneous connections from cell phones, iPads, and
>>>> > clients such as Outlook and MacMail, which are often configured to
>>>> > maintain persistent connections and in doing so place extremely
>>>> heavy
>>>> load
>>>> > on the CalMail platform.
>>>> > 3)Moving some users to other campus email services temporarily to
>>>> further
>>>> > reduce load.
>>>> >
>>>> > While these are drastic actions, none were undertaken lightly but
>>>> were
>>>> > done after extensive consultation with technical experts from campus
>>>> > central and departmental staff as well as vendors we have enlisted
>>>> to
>>>> work
>>>> > on the problem.
>>>> > Once the new storage system is installed, configured and tested, we
>>>> will
>>>> > begin the migration process from the legacy storage system to one
>>>> more
>>>> > than double in size and expected to handle at least several times
>>>> our
>>>> > current load.  That process itself will put substantial load on the
>>>> legacy
>>>> > storage environment as data is copied so we are planning on doing
>>>> this
>>>> > work off hours.
>>>> >
>>>> > We will continue to provide updates and information about the latest
>>>> plans
>>>> > via Calmessages as well as posting the information here:
>>>> > http://ist.berkeley.edu/ciocalmailupdates
>>>> > We will also continue to provide regular updates via
>>>> > http://systemstatus.berkeley.edu
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------
>>>> The following was automatically added to this message by the list
>>>> server:
>>>>
>>>> To learn more about Micronet, including how to subscribe to or
>>>> unsubscribe
>>>> from its mailing list and how to find out about upcoming meetings,
>>>> please
>>>> visit the Micronet Web site:
>>>>
>>>> http://micronet.berkeley.edu
>>>>
>>>> Messages you send to this mailing list are public and world-viewable,
>>>> and
>>>> the list's archives can be browsed and searched on the Internet.  This
>>>> means these messages can be viewed by (among others) your bosses,
>>>> prospective employers, and people who have known you in the past.
>>>>
>>>
>>>
>>>
>>> --
>>> Phyllis Orrick
>>> Communications Director
>>> Safe Transportation Research and Education
>>> Center<http://www.safetrec.berkeley.edu>(SafeTREC)
>>> University of California Transportation Center <http://www.uctc.net>
>>> (UCTC)
>>> Institute for Urban and Regional Development
>>> <http://www.iurd.berkeley.edu>(IURD)
>>> 2614 Dwight Way
>>> UC Berkeley
>>> Berkeley CA 94720-7374
>>> 510-643-1779
>>> @transsafe <https://twitter.com/#!/transsafe>
>>> @californiaUTC <https://twitter.com/#!/CaliforniaUTC>
>>> @IURDBerkeley <https://twitter.com/#!/IURDBerkeley>
>>>
>>> -------------------------------------------------------------------------
>>> The following was automatically added to this message by the list
>>> server:
>>>
>>> To learn more about Micronet, including how to subscribe to or
>>> unsubscribe
>>> from its mailing list and how to find out about upcoming meetings,
>>> please
>>> visit the Micronet Web site:
>>>
>>> http://micronet.berkeley.edu
>>>
>>> Messages you send to this mailing list are public and world-viewable,
>>> and
>>> the list's archives can be browsed and searched on the Internet.  This
>>> means these messages can be viewed by (among others) your bosses,
>>> prospective employers, and people who have known you in the past.
>>>
>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> The following was automatically added to this message by the list
>> server:
>>
>> To learn more about Micronet, including how to subscribe to or
>> unsubscribe
>> from its mailing list and how to find out about upcoming meetings,
>> please
>> visit the Micronet Web site:
>>
>> http://micronet.berkeley.edu
>>
>> Messages you send to this mailing list are public and world-viewable,
>> and
>> the list's archives can be browsed and searched on the Internet.  This
>> means these messages can be viewed by (among others) your bosses,
>> prospective employers, and people who have known you in the past.
>>
>
>
>



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Emergency options: twitter feeds

Greg Merritt *

Yes, in fact -- it was indeed posted on the UCB system status page.

By the way, for departments that have been following the guidance of our
Office of Continuity Planning, all of these "Plans B" will have been
entered and maintained in the campus-supported tool designed for just this:

http://ocp.berkeley.edu/plan/tool

Not that I can claim that my department is up to snuff, but the resource
is there; folks should probably not try to reinvent the wheel, and just use
this tool.

-Greg


On Wed, 30 Nov 2011 08:51:46 -0800, "Bill Allison" <[hidden email]>
wrote:
> not a big one, but email was down then.  It was noted on
> systemstatus.berkeley.edu I think
>
>> Bill,
>>
>> Thanks for telling us about this. I'm now following it, but I have to
be
>> honest, until you mentioned it, I had no idea that CalMail had a
twitter
>> account so that's great to know as a twitter user. Was there an
>> announcement about it that I missed?
>>
>> Thanks again.
>>
>> Beth


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.