looks like our main server that runs pretty much every single little thing this company does took a big, fat shit just now. we could be totally fucked.
hooooboy! what a day!
- Thread starter avi
- Start date
xfer
I JERK OFF TO ARCTOPUS
FuSoYa
Lunarian
crazy, ours did that friday afternoon, everything gone (backed up though, i think)
this week sucks, + moving
this week sucks, + moving
ENRYU_T52
INRI
ID^^^^^$$$$10#####$$$$$T(*(*(*($
MY SYMPATH1ES T0 TH# P00R 0L L4SS. MAY SH3 RES7 IN PIECE.
1F'N Y0U NEED FUR7HER ASSIST. PLEAS3 DI4L +0.1.999.328.7448
x (.)_(.)>>>>>>>>>>> (|)(|)(|) >>>>>>>>>>>>>>>>>.>.>.>
MY SYMPATH1ES T0 TH# P00R 0L L4SS. MAY SH3 RES7 IN PIECE.
1F'N Y0U NEED FUR7HER ASSIST. PLEAS3 DI4L +0.1.999.328.7448
x (.)_(.)>>>>>>>>>>> (|)(|)(|) >>>>>>>>>>>>>>>>>.>.>.>
ENRYU_T52
INRI
....___..._........___....~~
Y0U KN0WC0ME2TH1NK0F1TT, 1M F33L1NG 4 B1T P00P3D MYS3LF.
"ZCK. ZGGH." TH4T'S H0W Y0U M4MM4LS SAY 1T?
x ...____....~~.~~. \
Y0U KN0WC0ME2TH1NK0F1TT, 1M F33L1NG 4 B1T P00P3D MYS3LF.
"ZCK. ZGGH." TH4T'S H0W Y0U M4MM4LS SAY 1T?
x ...____....~~.~~. \
xfer
I JERK OFF TO ARCTOPUS
mindspell
vvv Jake's ass vvv
What kind OS are you running? I would never run something vital to a company on Windows unles it is in a cluster of some sort.
I almost positive the box was running redhat. I'm pretty sure the only windows server stuff we do is for stuff running actual windows media streams.
mindspell
vvv Jake's ass vvv
RedHat is a crappy distro out of the box. You have to do some major tweaking to have it run the way you need. Default installation is even worse than a vanilla windows install... I am sure this has been done though if it doesn't crash that much... Ever done a Risk analysis in your shop? How's the Business recovery/contingency plan?
I was told it was the box itself, not the system that died.
I'd like to say there's plans for all that stuff in case something dies, but knowing the way things go around here, I can't be sure. too many of the engineers need to talk shit to death before anything actually gets done. i just want to scream "FUCKING DO IT ALREADY" and strangle them. seriously - during a mass of live streaming events today, I had to practically yell at my frickin' boss to help me. anytime I said something he'd babble "so you want me to so-and-so...?" jeez! don't question it, just get it done. I ended up just rushing by him to do it myself.
I'd like to say there's plans for all that stuff in case something dies, but knowing the way things go around here, I can't be sure. too many of the engineers need to talk shit to death before anything actually gets done. i just want to scream "FUCKING DO IT ALREADY" and strangle them. seriously - during a mass of live streaming events today, I had to practically yell at my frickin' boss to help me. anytime I said something he'd babble "so you want me to so-and-so...?" jeez! don't question it, just get it done. I ended up just rushing by him to do it myself.
lizard
Member
FalseTodd
Skirt Wizard
mindspell
vvv Jake's ass vvv
I see that everybody's company is well versed in business continuity planning. It might be a good idea to hire a consultant to look at that. Just a thought.
full description of the problem from the head of IT:
The primary data store, the storage array which supports all the production requirements, experienced a fatal hardware failure at approximately 12:04 PM. It attempted to re-boot itself, which attempt failed. That left the web site completely out of service. We went to the data center and activated the "hot spare" filer, which assumed the processing load and brought the site back to life. No data was lost, and processing was restored at approximately 1:30PM. The bulk of the elapsed time was the hour that it takes to drive to Sunnyvale, where the data center is located.
The primary data store, the storage array which supports all the production requirements, experienced a fatal hardware failure at approximately 12:04 PM. It attempted to re-boot itself, which attempt failed. That left the web site completely out of service. We went to the data center and activated the "hot spare" filer, which assumed the processing load and brought the site back to life. No data was lost, and processing was restored at approximately 1:30PM. The bulk of the elapsed time was the hour that it takes to drive to Sunnyvale, where the data center is located.
mindspell
vvv Jake's ass vvv
Well that's not much of an explanation. All we know is that hardware in the storage array failed. What was it, hard disk failing, card, cable, what? It would have been a good idea to have the "hot spare" online instead of having to drive there to actually put it in.
don't lecture me, it's not my deal. my understanding is that the box itself failed b/c all they did was switch the drives to the spare and turn it on. I believe the matter of having the spare online et al. is primarily a budget constraint.
mindspell
vvv Jake's ass vvv
I am not lecturing you, I am commenting on their work, not yours. Also, you deal with this crap everyday and have to answer to clients you should be entitled to some explanation that is a little more thorough than :" hugh! the box failed." which is exactly what you had there....
well, there was a bit more to the note, but I'm not posting it. I guess there was also a cable length issue that complicated activation of the spare, too. anyhoo, it's fixed - that's what matters. and also, this msg went out to everyone so it had to be dumbed down to a salesperson level.
mindspell
vvv Jake's ass vvv
Similar threads
