November 15th, 2009 - Natura Ex Machina

I was reading my friend Kirk's blog yesterday (Twin Cities Naturalist), and he was describing how they believe a bird dropping bread crumbs into an opening in the $10 billion Large Hadron Collider is what caused it to overheat, causing a multi-million dollar stoppage and much down time.
Isn't it amazing how as scientists we can be smart enough to build an extremely complex machine such as this, but as humans we somehow lack the common sense to prevent what could be a serious and expensive problem by simply foreseeing a vulnerability?
It brings to mind an episode out of my time as a maintenance technician at a large 24 hour photolab.
I was off-shift, but the lab was coming off of a long overnight shift of photo-processing, and trying to finish up their work right around dawn.
One technician was on staff at the time, along with the maintenance supervisor. Please realize that a photolab is a noisy, active, smelly environment, with buzzers and alarms going off at regular and irregular intervals, much machine noise, people talking, and often the radio playing over the PA system.
People are in and out of darkrooms, going through light-tight doors, and working behind curtains. Techs often have their heads stuck into large noisy machines, and everyone is focused on their assigned tasks at hand, or zoning off into the repetition that is their assigned task at hand.
Photolabs require uninterrupted power. If that power goes out during printing, it sucks, you have to reprint, and it becomes an expense. However, it's not the end of the world.

On the other hand, if the power goes out during film processing, it is nearly the end of the world. Film that belongs to a lot of people who care about it very much becomes irreparably destroyed.
The customer service department of the lab likes you even less than usual, if that is even possible.
The lab uses large machines that need to be kept at a precisely consistent temperature and if they vary from that temperature, they take time and hence expense to readjust.
This in principle is very similar to a Large Hadron Collider.
Little or no output can be achieved during this downtime.
Also, almost all of the machines in a photolab use a lot of gears, belts, and pulleys to drive or pull film or paper through them, so if you shut off the power while they are in the middle of what they are doing, they tend to keep doing it as long as inertia will allow, and then begin to do it in a way in which they were not designed, with much munching, crunching, swearing, and groaning.
So loss of power is a bad thing.
That is why facilities such as this have a backup generator. If the power goes out, or actually even thinks about going out, there are complex circuits which sense the drop in source power, and within milliseconds switchover to batteries and/or generator power if necessary.
Of course, this doesn't happen very often. In the U.S. it happens very seldom. Power in most parts of the United States is amazingly consistent compared to some other countries and is normally pretty predictable considering it's demand. That is why we don't often check the backup generator. Hey, it'll work, it's almost new, it never gets used!
One day, on a nice sunny fall morning in Minnesota, a squirrel was busily storing it's nuts in a far corner of the roof on top of the photolab building.

As it ran along the wires and jumped from place to place it happened to jump to place it had never jumped before. As it's front feet touched one point of contact on the sub-stage transformer and it's back feet touched another at the same time, the moisture in it's little mammalian body provided the equivelant of a little mammalian connecting wire which short-circuited the transformer providing power to the building, thus heating up the fluids in it's little mammalian body with much alacrity. However, not with enough alacrity to avoid placing a large load on the transformer, which caused it to overheat, explode, and reduce power output to the building to approaching near zero in a matter of milliseconds.
Thankfully, the complex circuits attached to the backup generator decided that the voltage coming into them was too low, and started up the generator. In a matter of seconds the power was back up to normal levels and the large machines didn't change their temperature due to having their large tanks of fluids already warmed up, or notice much of a change in their transport systems, due to inertia of their moving parts.
Objects in motion tend to maintain that motion, while objects at rest tend to stay at rest. These objects had been in motion all night.
Inside the lab, the tech was busy working on some piece of equipment, and the maintenance supervisor was having a cup of coffee and a smoke in the breakroom. (Yes, there was a time when you could have a smoke in the breakroom.)
Neither noticed the pop of the transformer, or the alarm way back in the corner of the building that said that the generator was now powering the building.
The big machines rolled on.

However, the small machines that had their own little computers controlling them began to hiccup. A few milliseconds is a long time to a computer. Some computers have batteries built into them that can keep them going for 'a while' and some do not. Those that did not began doing strange things. Well, strange from the perspective of what they normally did in a photolab, probably not so strange from the perspective of a computer that had it's power shut off and then turned back on for a second.
So suddenly the tech had a few more things to fix and called the maintenance supervisor who stubbed out his cigarette and downed his coffee. They both began to fix the machines weren't working right, as they always did when things stopped working, and the big machines rolled on.
Outside, a puff of smoke that smelled like burnt fur and insulation dissipated into the air at dawn in the big city, and no one noticed. Nearby, the generator that the photolab designers created to insure that their livelihood wouldn't be interrupted puttered on, sucking up diesel fuel from a tank attached to it though it's fuel pump. God was in his heaven.
Inside, the tech and the maintenance supervisor focused on fixing their machines' problems, and cursed the world for presenting them with so many things to do when they just wanted to go home and sleep.
After about twenty minutes, they had almost all the problems fixed (well, you never have all your problems fixed) and were breathing a sigh of collective relief.
Just as the maintenance supervisor was about to light another cigarette and bemoan his fate with the technician, the fuel pump on the generator was sucking up the last drops of the two gallons of diesel fuel that had been the only two gallons of diesel fuel ever put in the one hundred gallon tank for the generator, because why would anyone spend money on a lot of diesel fuel to sit in a generator that was rarely used...?

My question is not of whether it was the squirrel's fate that made it step where he had never stepped before, nor if it is man's fate to have the folly to build insurance into something and not adequately use it, but rather, did anyone ever put a cover over the contacts of the sub-stage transformer?