I received a PM from Marko (Mark) at Brightstar (our hosts). His apologies were not required as when kit fails it fails. I thought you would appreciate seeing the sequence of events and how well looked after we are (Mark has left us his mobile number and office number in case we have problems):
QuoteHi Blueball
Please accept my appologies for yesterdays network failure.
It had been a very trying day to say the least.
A core peice of our networking infrastructure started to fail at about 4.00am.
I worked with Cisco TAC support to resolve the problem and at 7.00am Cisco pronounced the router dead.
As we have a Cisco Gold Smartnet support contract we were able to have a replacement router dispatched.
The replacement router arrived with us at 10.30am with a Cisco engineer who was able to rebuild the system and get us operational by 11.00am.
The peice of kit that failed costs approximately £32,000 and has 2 PSU's and 2 Processing Engines. As Cisco said, "it simply is not supposed to fail". For both processing engines to fail it is virtually unheard of.
I feel that we have had the next 10 years worth of faults and problems all at once. Hopefully we should be good for a while.
Please pass the on my comments to the rest of the tribe.
I beleive that the packet loss issue that we were seeing may have been caused by this fault and I am hoping that this replacement router will solve this.
Please advise?
Regards
Mark
I think that speaks for itself and needs no further comment from me :clap:
Yeah, They certainly seem to be on the case and are a nice bunch to work with. Just being kept in the loop is nice :)
£32,000 :blink:
PEN
Seems like they've got something under control. I don't hold something like this against them as it could happen to anyone. If it clears up the other issues thats great too.
Thats nice of him to get back to us. I hope they test it to see what happened to the router.
Quote from: GhostMjr;220293Thats nice of him to get back to us. I hope they test it to see what happened to the router.
Cisco will I am sure.
TL.
Quote from: Penfold;220251£32,000 :blink:
They can cost pretty much. That is not even close to the most expensive ones. A large router can cost around £100,000 and then customers normally buy maintenance/replace/fix contracts of different levels on top of that. A top level support contract can cost over 10,000/year.
It is interesting that it failed totally so fast. Normally that level of router have redundancy (like this one had). It can be monitored and get a warning when one thing brakes before the whole system fails. They are often modular (you insert different kind of cards into a rack and they are connected) so often a fault is in one card and that can be changed fast.
Aquilifer You are quite right.
We have had some feed back from Cisco regarding the failure.
It would appear that the failure with one of the processing engines was so severe that it actually damaged the back plane of the chassis (6509 chassis for those of you interested) and the other processing engine. (SUP 720 3BXL II)
I personally thought that the chassis was totally passive, however cisco said that this is largely the case but not completely true. there are some active electronics involved.
The router started showing significant signs of mis-behaving at about 4.00am but I think I may have pushed it over the edge when I powered it completely off for a few mins and then back on. At that point it was never the same again.
Cisco were convinced that it was a software / configuration issue and beleived that it was totally possible to get it up running on the second processing engine but our efforts proved fruitless and the unit was pronounced dead at about 7.00am.
Well moving forward we are fully operational again and I / We are very grateful for your support.
My appologies for any inconvenience.
As said before Mark, we know good service when we see it. Keep up the good work :clap:
im sure we all can say that there is no need to apologise and thanks for getting us up again so quick, carry on with the excelant work:yahoo::yahoo::yahoo:
@M_a_r_k_o
Thanks for the more detailed story. That sure was interesting reading (for me at least :) ). That was quite new to me too that it is possible to get that kind of...'chain effect' (or how to call it), sometimes.
Anyway thanks for the good service. :)
Interesting info Marko, thanks for that. I've never heard of any slot devices breaking in such a way that they actually cause damage to the backplane o.0 A power surge of some kind maybe?