Networking-Forums.com

General Category => Forum Lobby => Topic started by: deanwebb on June 10, 2015, 04:18:01 AM

Title: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on June 10, 2015, 04:18:01 AM
 :problem?:

Yay I have the on-call phone and I can confirm that the phone works just fine, even at 4AM...

:glitch:
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: FilipiNomad on June 10, 2015, 05:14:33 AM
LOL I feel your pain.  What is your rotation like?
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: SimonV on June 10, 2015, 06:17:50 AM
I've been on Standby for almost a year now, one week every month. The first few months I had calls regularly but for the last six months I haven't received any.
Lots of stuff was off-shored though, so maybe they just don't know the number at the new helpdesk. But I'm not complaining  8)
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: Netwörkheäd on June 10, 2015, 08:32:15 AM
Quote from: FilipiNomad on June 10, 2015, 05:14:33 AM
LOL I feel your pain.  What is your rotation like?
One week every 7 weeks. We used to have a follow the sun model where we would have weeknights off, but now we're back to a "just guys in the main US location" model.

Thankfully, WAN provider knew how to issue a shut / no shut power combo and the link is back up. Which is good because the backup link doesn't work right now.
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: Reggle on June 11, 2015, 04:19:43 PM
I had once every 8 weeks. Now I'm off the hook. Don't miss it...
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on June 12, 2015, 10:43:38 AM
Another major WAN outage... looks like the cause of this one as well as Wednesday's was a *fiber cut due to construction*.

:rage:

The on-call phone is currently going through massive gyrations from all the Orion text updates flowing in from the Orion monitor that lost connectivity to all its nodes... and just now had that connectivity restored...

:kiwf:

The WAN is back up, though, so it should be a few hours yet before the backhoe operator finds the fiber and cuts it again. I will be able to get lunch.
:joy:
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: SimonV on June 12, 2015, 02:00:54 PM
How critical is that site? Not fully diverse when it comes to WAN connections? For production or critical sites, we always insist on having physically redundant paths just for that reason. Those construction guys eh  :twisted:

This one guy that broke a fiber, he was gonna fix it with this

(http://www.gwsupplies.co.uk/product_images/1201601852.jpg)

:awesome:
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on June 12, 2015, 02:36:51 PM
It's only the main data center for the Americas. That's all. We only lose one-third of our entire operation when it goes down.

Thing is, this thread is also connected to the "WAN Provider Experiences" thread... we have WAN provider issues AND construction issues at the same time... yeesh.
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on July 09, 2015, 11:35:25 PM
New fun... I'm already up to do a presentation for APAC, so I might as well stay up for when Europe comes online so I can talk to a team there to resolve an issue with our certs that's been killing our wireless 802.1X with mobile devices.

Sometimes "follow the sun" support hours happen to you, even though you're not following the sun yourself. I'm going to be soooooo messed up in about 12 hours, when I finish my last call for the day. :partay:
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: SimonV on July 10, 2015, 02:03:38 AM
Yes, Europe just got out of bed. How can we help?
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: mmcgurty on July 10, 2015, 07:22:12 AM
I do on-call every 4 weeks for 1 week and every 4 weeks for 1 week as a backup on-call.  My last rotation just about killed me due to working on other projects in addition to on-call problems.  I have been at the same place since 2008 and it was probably the worst on-call rotation I have done since I started in my early days and knew nothing of the environment.  I couldn't hand off the on-call phone quickly enough on the rotation day.  I was really glad to go to sleep and not get bothered.  We added a new guy about a month ago, I can't wait until we add him to the on-call and we go to a 5 week rotation.
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on July 10, 2015, 08:26:20 AM
Quote from: SimonV on July 10, 2015, 02:03:38 AM
Yes, Europe just got out of bed. How can we help?

Can you destroy all iPhones? Then they'd all go back to using laptops, like proper IT professionals.
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: that1guy15 on July 10, 2015, 09:04:49 AM
Currently once every 8 weeks for me. Its nice here as help-desk and desktop support field most issues so its usually pretty quiet. I feel for those desktop guys though.

My only problem with is though is the on-call schedule starts on Friday. Why Friday...
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: deanwebb on July 10, 2015, 09:47:15 AM
Our oncall goes from Tuesday to Tuesday, which means I have to rearrange my life for oncall, since I work from home most Tuesdays.
Title: Re: Sky Is Dark, Network Dark, Phone is Lit Up
Post by: Nerm on July 10, 2015, 12:14:07 PM
My on-call schedule kinda goes to one of those "problems of a company of a certain size". Being a small company I am the on-call rotation lol. We do have one other guy that takes emergency calls so I guess I can say I am 1/2 the rotation. :partay: