[Micronet] July 15th Service Interruption for Off Campus Locations

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] July 15th Service Interruption for Off Campus Locations

Isaac Orr
Folks,

Back on Monday July 15th we had a network service interruption that
impacted multiple off campus locations.  Though the network
interruption itself was brief, the number of locations involved and
problems triggered by the interruption caused significant impact.  The
Campus Shared Service center on 4th St was one of the sites affected,
and this caused problems for voice service to that location.

At the time I promised to follow up with more details of the outage as
they were available.  The locations involved are all connected via our
new metro ethernet network that is provided by AT&T.  All of the sites
that are off campus reach campus via one of several 10 Gb/s links to
AT&T's network.  It appears that the problem impacted all of the sites
on one of those 10 Gb/s links.

When the outage occurred we asked AT&T for information about the cause
of the problem.  Initially we did not receive many useful details, and
we escalated this within AT&T's service organization.  Unfortunately
by the time appropriate resources were applied to the problem by AT&T
(several days later), the logs that could have given further details
of the problem were no longer available to AT&T technicians.  As a
result, the official answer from AT&T is that the cause of the service
interruption is "undetermined".

In the aftermath of this we have discussed with AT&T how we can ensure
that when an outage occurs the right level of attention is given to
determining the cause.  I've made sure that everyone in the network
group is aware of the process to use when dealing with AT&T to make
sure that they are fully aware of outages like this and their impact
to the University.

Regards

iso

--
Isaac Simon Orr
Manager, Network Operations and Services
IST Telecommunications, UC Berkeley
P: +1 510 643 9837 C: +1 510 517 9408 E: [hidden email]

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] July 15th Service Interruption for Off Campus Locations

Harriet Johnson
Issac:

Thank you!  One of the bigger issues was that we have been told to contact
Shared Services to report a network outage and with the internet down and
their phones down that was not possible.  We had no way to tell if this
was just our offices or all offices and no way to report it.  I ended up
calling Jay Bryon at his desk which was totally inappropriate but it was
the only number I had.

Please provide the name and number of who we should contact at CNS in the
event of a network outage that cannot be reported to CSS.

Thank you
Harriet

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Isaac Orr
Sent: Friday, August 02, 2013 9:07 AM
To: group Micronet-UCB user microcomputer support
Cc: Josie Galvan
Subject: [Micronet] July 15th Service Interruption for Off Campus
Locations

Folks,

Back on Monday July 15th we had a network service interruption that
impacted multiple off campus locations.  Though the network interruption
itself was brief, the number of locations involved and problems triggered
by the interruption caused significant impact.  The Campus Shared Service
center on 4th St was one of the sites affected, and this caused problems
for voice service to that location.

At the time I promised to follow up with more details of the outage as
they were available.  The locations involved are all connected via our new
metro ethernet network that is provided by AT&T.  All of the sites that
are off campus reach campus via one of several 10 Gb/s links to AT&T's
network.  It appears that the problem impacted all of the sites on one of
those 10 Gb/s links.

When the outage occurred we asked AT&T for information about the cause of
the problem.  Initially we did not receive many useful details, and we
escalated this within AT&T's service organization.  Unfortunately by the
time appropriate resources were applied to the problem by AT&T (several
days later), the logs that could have given further details of the problem
were no longer available to AT&T technicians.  As a result, the official
answer from AT&T is that the cause of the service interruption is
"undetermined".

In the aftermath of this we have discussed with AT&T how we can ensure
that when an outage occurs the right level of attention is given to
determining the cause.  I've made sure that everyone in the network group
is aware of the process to use when dealing with AT&T to make sure that
they are fully aware of outages like this and their impact to the
University.

Regards

iso

--
Isaac Simon Orr
Manager, Network Operations and Services IST Telecommunications, UC
Berkeley
P: +1 510 643 9837 C: +1 510 517 9408 E: [hidden email]


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe
from its mailing list and how to find out about upcoming meetings, please
visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and
the list's archives can be browsed and searched on the Internet.  This
means these messages can be viewed by (among others) your bosses,
prospective employers, and people who have known you in the past.

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.