Watching a large UPS go from full to empty

[Titantornado]

Really not much to discuss here, but it was an interesting observation.

Last week, down at work, at a telephone company I work for, we experienced a grid power failure in the wee hours of the morning.  Normally, no big deal, as the 500KVA Caterpillar genny picks up the load in about 5 ~ 7 seconds . . . . . except this time . . . . . the first time in 10 years.  It faulted out on a false over-temperature alarm and wouldn't start. So we have a few relatively large UPSs. An 80KVA for the networking room and local workstations, and a 160KVA for the main data center. The 80 is loaded fairly light at 25KVA, but the 160 is carrying around 95 ~ 100KVA.

So, everything was being carried on the UPSs, and alarm calls went out.  Problem is, anyone who has the pleasure of responding to these emergencies (myself included) lives 45+ minutes away, (I'm a solid hour) and the main UPS gave up the ghost at 26 minutes and the entire data center crashed.   In typical fashion, grid power was restored just under eight minutes later.  The smaller UPS never lost total power.  What was interesting was how hot those batteries got after going through a full rundown and subsequent rapid recharging. (1.5 hrs) Put them A/C units into overdrive! Quite a sight.

That little 34 minute loss of power caused about 6 hours of down time for many systems, and who knows how much money lost.  I'm still getting repercussions from that mess a week later, and a CEO riding my butt as to why he has spent "hundreds of thousands so this doesn't happen." Once he cooled down a bit, (nearly as hot as them batteries) I explained it's only a machine, and mechanical failures do happen.  If he truely wants a fail-proof system, then he needs to invest in more than a single generator.  Surprisingly, that seemed to penetrate his cranium and backed off.

Ah yes, work can be so much fun.

[ghurd]

"seemed to penetrate his cranium".

Or his wallet?  ;-)

Makes me wonder if a manual $500 B&S/Robin/Honda gas 110V gen, or 2, would have charged enough to cover 8 minutes of run time, if it had over 34 minutes to do it?

Draining at 2C, it sure would have helped the AH.  Peukert effect, etc.

G-

[vawtman]

I understand what you went though Rod

 Im resposible for 2 gens one for police and another for fire.

 We had constant intermitant outages last month and lost it.WEIRD only light freezing drizzle but strong winds.

 Took a drive out of town to see what could be causing this and couldnt beleive what i saw.Transmision lines feeding the city were swirling so badly i thought for sure it could be days.Yet the power guys were attempting repairs(lots of guts)

 After all the maintenance and test runs things can still go wrong.My gens are wore out just from that since it was probably 6 yrs prior to the last outage but performed well.

 Does yours use a block heater?Seems to be hard on the coolant thus maybe the temp fault.

 When things work nobody cares but when they fail watch out{i hate that part}

[Titantornado]

> Makes me wonder if a manual $500 B&S/Robin/Honda gas

> 110V gen, or 2, would have charged enough to cover 8

> minutes of run time, if it had over 34 minutes to do

> it? Draining at 2C, it sure would have helped the AH.  

> Peukert effect, etc.

>

I doubt it Ghurd.  Problem is, eight minutes just happened to be the case this time around, but it could have just as easily been far longer.  Plus we're talking three phase 208/480V bank supplying some 90,000 watts. (with the batteries giving up probably significantly more with conversion losses) I'm actually surprised it hung in as long as it did. (batteries are 2 1/2 years old)

> Does yours use a block heater? Seems to be hard on

> the coolant thus maybe the temp fault.

>

Yes, it does.  It's a constant 125~145F degrees so it's ready to go in a moments notice.  The diesel tech found a corroded connector for the temperature sender that he believed may have generated the false alarm.  I had him replace the entire sensor, mostly as a symbolic measure to appease the "Powers-that-be", as I doubt there is anything wrong with it.  I also ordered a full load test to be performed to prove the coolant system is up to it's job.  Again, symbolic, but looks good in the eyes of the CEO.  I never seen a load bank trailer, and can't wait to see how 400 Kw gets dumped.  Not to mention hearing the growl of the Cat engine at full bore for the first time ever!!  LOL

> When things work nobody cares but when they fail

> watch out{i hate that part}

>

I hear ya Vawtman.  My job is to assure everything runs smoothly and without problems.  Unfortunately, nobody really sees me doing things in the background and assumes I don't do much.  Makes it tough to be considered for raises.  Then an incident like this, and it doesn't help that image either.  It's a loose-loose situation.

[wdyasq]

Some years ago I suggested a back-up to a system on the Gulf Coast for a 'data center'. They had a (as I remember) a Cummings 800kW unit to power a whole office building and the data center's servers. IIRC, the servers used about 15% of the total load. My suggestion was to have the second generator a 'minimal' generator capable of handling the server loads and be at least 25% loaded but less than 75% load.

Well Katrina came. The Cummings cranked and worked overtime at a low power level. I have been told a new secondary system will now be installed similar to what I suggested. The 800kW Cummings can then be torn down, 'cleaned up' and rebuilt from the abuse of running two months under 'minimal load' without oil change. I am also told a secondary system would have paid for itself in fuel savings over the time the system was required.

There were other 'improvements' too. On this building fuel was delivered by barrel and a freight elevator. There is now a fuel line, receiving tank and pump at street level. One of the 'executives' had to help handle the 250 gallons a day of fuel the generator was eating. At full power I think it took near 50 gallons an hour.

The good thing about 'standby' stuff is it is seldom needed. The bad thing about standby stuff is it is seldom needed and usually ignored and neglected until a drastic event occurs.

Ron

[Bruce S]

Ron;

 I hear you on that. 50 gals/hour? And I was worried about our drinking 6gal/hr.

For the telephone system that went down every 8 hours (due in large part that their phone guys forgot about refueling the propane gen :-( ,) I now have them switching out the T1 for an OC3 unit. This gets us on the fibre and now have it located in out Hi-CAP room where it can run off our gen.

The CAT is do for a tear down and fuel is do for complete cleaning.? seems when the last re-fuel ( read last ice storm came through ) they didn't put the anti-gel in the fuel now it's just a mess. I offered to lend them some bio-diesel from my car , but they just gave me a nasty look.

OK by me their's the ones who has to clean it out. WE have a maintence contract:-)

Most people here just say hi when they see me, when to phones went out, I was either their best friend or worst enemy:!

AND ohh did the VPs and up visit me on a regular basis those days. Those of course who could get here:--)

Never have had a problem with genny, but inline UPS to handle power unitl genny comes up to full speed, once saw a ground fault and shut down. Installer had connected to genny phased ground, instead of provided ground ( he said it was closer!!) caused tons phones calls and lost man-hours-- meaning money$$$.

Bruce S

 

[Mary B]

Casino I used to work for had 2 large 12 cylinder cat diesel generators. I was in the generator room one day when we switched over during a power failure. Loud doesn't begin to describe the racket. We lost power about 20 times a year so the units got a workout fairly often. 1600 slot machines (average current draw was 4 amps per machine) and a 600 room hotel use a lot of power.

[BigBreaker]

My parent's residential NG genset automatically starts-up and self tests once a month.  It reports any errors (light goes on, etc...) and tops up the battery so that you can get it fixed ahead of time.  Dead batteries are a frequent cause of non-starting gens.

[Ungrounded Lightning Rod]

Casino I used to work for had 2 large 12 cylinder cat diesel generators. I was in the generator room one day when we switched over during a power failure. Loud doesn't begin to describe the racket. We lost power about 20 times a year so the units got a workout fairly often.

That brings back memories.

Back around the turn of the millennium, when we were building our Nevada place, we would sometimes take a room in the casino/lodge a few miles away.  One time there was a nasty storm that took down the power.  "Honk!  GrrrrVROOMMMMMM!" said the diesel genny as it kicked in, bringing the lights back on just a few seconds later.

[dalibor]

from time to time in my company those things happen, but not so often and it makes us happy.

i am working on maintaining rectifiers, inverters, converters and back up batteries in various sizes and capacities. not so long ago i was also included in maintaining electrical part of diesel back systems for equipment.

one thing i have learned for the last 13 years in all that - things happen. thats why we are testing battery capacities all the time, starting and testing diesel electrical generators. and again things can go wrong. most of the time problem with generators are starting batteries - the strength of electrostarters are around 4,5 KW .

here in Serbia problems with electrical company are more often - the grid needs to be replaced on many places, and people are shutting power down for safety while working.

sometimes this comes to my mind - older equipment was less demanding when talking about sensors, old things were somehow rigid compared to new devices.

i am happy that all went well finally in your company. in some other meaning, it is good to make "upper levels" from company people to see that somewhere lower someone is trying to do its best. a bit of shaking can make " company blood" to move faster :-) .

And yes, work can be so much fun. 100 % true.

[ULR]

(Ungrounded Lightning Rod posting as ULR again because the BBS software thinks I post too much...)

My job is to assure everything runs smoothly and without problems.  Unfortunately, nobody really sees me doing things in the background and assumes I don't do much.  Makes it tough to be considered for raises.

Start doing a monthly test on the genny.  Then you'll get noticed.  B-)

You'll also find out if the genny needs maintenance, BEFORE the power failure.

[Titantornado]

Actually, it does get run every Friday, but I don't normally through the load over.  Just fire it up for an hour and check for errors. We also have a quarterly maintenance contract to assure tip top condition, but sometimes the unavoidable does occur.

[tecker]

I've seen this happen at Grocery stores offten . The hambuger that goes to the shelf is  so red it looks like candy .

[TomW]

ULR;

Sorry this catches you but without it I wake up some mornings with dozens of spam to kill. I think if you actually just wait awhile it will reset unless you keep trying then it locks you out for progressively longer. Blame the Chinese, about 99.99% of the posting SPAM is from them.

I will go look at your acct in another tab and unlock it if it still is locked. Nope not locked so it must have expired.

Again, sorry for the hassle but it saves me a lot of work sometimes. At least you know a workaround.

Cheers.

TomW

[RP]

"I never seen a load bank trailer, and can't wait to see how 400 Kw gets dumped"

PLEASE try to get a picture of that!  :-)

[Ungrounded Lightning Rod]

Actually, it does get run every Friday, but I don't normally through the load over.

Then take off the muffler.  THEN they'll notice the tests.  B-)