Major UA Systemwide Computer Failure!

SFOFlyer · Wed Jun 20, 2007 10:53 pm

I am sitting on UA842 (B772) right now at ORD in the 9R Pad with about 8 other UA Mainline planes. There seems to be some sort of major computer failure with UA's Load Planning at ORD, where we don't have our Weight and Balance info. We have been sitting here since around 8:00am and I believe that we were the first one to experience this problem. I am not sure if this is system wide or not.

It seems to only be affecting UA's Mainline, as I can hear (on Channel 9), UAX Planes departing.

Not sure what type of impact this is causing on UA today.

[Edited 2007-06-20 16:22:24]

SFOFlyer · Wed Jun 20, 2007 11:01 pm

Just heard it was systemwide computer failure at UA, per our captain!

The 9R Pad (Holding Area) is now full of UA Mainline planes and I have not hear of any UA plane movements at ORD. Sounds like they are holding the planes at the gates now.

Damn!

Bicoastal · Wed Jun 20, 2007 11:05 pm

The computer system that's down is called Unimatic. They can't finalize their loads until it's up. And the ramp can't load airplanes without proper load/balance information. Could be a horrible day for United if the system doesn't come back up soon.

SFOFlyer · Wed Jun 20, 2007 11:26 pm

Anyone from UA have any sort of update? Our plane has been sitting out here for an hour and half.

At least they have the movies going and serving drinks. Not too bad yet... Wink

[Edited 2007-06-20 16:32:08]

bcoz · Wed Jun 20, 2007 11:30 pm

Quoting SFOFlyer (Reply 3):
Not too bad yet...

Ruh-rho! Looks like another long day at the old airfield! I imagine this can't help the cleaning up of the leftovers from yesterday's backlog from weather!

bcoz

SFOFlyer · Thu Jun 21, 2007 12:13 am

Was just told by the captain of our plane that the reboot of the computer system worked and we should have our numbers any minute. I'll keep my fingers crossed...

BTW - I am very impressed by the way that the crew has handled this situation (at least on my plane). The captain walked around (twice) to everyone and chatted and gave constant updates. The flight attendents were also very accomodating and pleasant. Maybe the have learned from the past few experiences!

LAXintl · Thu Jun 21, 2007 12:13 am

Unimatic is back up, and everyone should be leaving shortly.

Should make an interesting and messy rest of the day however as no UA flights managed to depart for near 2 hours!.

Boston92 · Thu Jun 21, 2007 12:20 am

Quoting Laxintl (Reply 6):
Should make an interesting and messy rest of the day however as no UA flights managed to depart for near 2 hours!.

Every hour will affect about 1 day... UA has the knowledge and resources to this mess cleaned up in no time.

SFOFlyer · Thu Jun 21, 2007 12:20 am

We just got our numbers and clearance. Gotta shut down the laptop and fasten my seatbelt! Wink

If only we had taken off 2 minutes before we would have been in the air when the computer system failed. Oh well...

Headed home at last!

airtran737 · Thu Jun 21, 2007 12:34 am

http://cbs3.com/topstories/topstories_story_171105938.html

atlaaron · Thu Jun 21, 2007 12:35 am

Why in the heck do the airlines not have failover systems? There should me a mirror image running of each one of these programs in the event of mishaps like this. My company needs to go in there and sell them some stuff!

ANCFlyer · Thu Jun 21, 2007 12:38 am

Quoting Boston92 (Reply 7):
Every hour will affect about 1 day...

Well, by Friday morning, it'll be - without any further issues - back to normal. And that's good, I'm on UA's 0945 ORD-ANC that morning . . . .

I feel for the UA folks . . . hang in there ladies and gents.

mymiles2go · Thu Jun 21, 2007 12:46 am

Quoting ATLAaron (Reply 10):
Why in the heck do the airlines not have failover systems? There should me a mirror image running of each one of these programs in the event of mishaps like this. My company needs to go in there and sell them some stuff!

I'm guessing they do have that. Likely something else (such as a configuration change that didn't act like exepcted) that caused it. Failover systems are great, but if the two systems are part of the larger whole, and the whole has a problem - than secondary systems become just as useless as the primary (regardless of geopgraphy).

AirSpare · Thu Jun 21, 2007 12:46 am

Sounds like Cisco, Nortel or Sun will be lnocking on some doors selling Business Continuity Plans and systems. The 2 hour outage would have probably paid for 25% of a robust network.

Quoting ATLAaron (Reply 10):
My company

Ramper: "Chief, can I charge my phone here?".
Chief: "Yea, unplug the coffee pot over there".
Ramper: "Here?".
Chief: "Yea".
Chief: "Doh!!".

What a crap feeling, causing an outage.

ckfred · Thu Jun 21, 2007 1:17 am

According to the Chicago Tribune, the system was down from 8am to 10am, Central Time. A UA spokesman said that a back-up system is up and running, so they are now trying to get planes in the air. It will be interesting to see how long it takes for UA to get back on schedule. With all mainline flights grounded, I bet it will take until very late tonight to get everything caught up. There will also be a lot of misconnects with UAX, which will be a pain to clean up.

davescj · Thu Jun 21, 2007 1:23 am

On the other hand, if ALL mainline UA is 2 hours behind......fewer missed connections, no? Because, literally, everyone will be 2 hrs behind.....and they can slowly move toward "normal" by delaying some flights, rebooking etc, no?

RDUDDJI · Thu Jun 21, 2007 1:23 am

Unimatic (United's Flight Operations system) was down for about two hours. The cause of the outage is under investigation. The average delay per flight during the affected period was about 65 minutes. Schedule reductions will be likely as a means of recovery throughout the day today.

I believe UA did away with manual load planners a few years ago, but Unimatic does far more than just load planning. It also does fuel messaging, ACARS messaging, and I believe they still use it as the backbone for some crew scheduling apps.

Thankfully, Unimatic doesn't go down often (but causes chaos when it does). It really is one of the best LP systems I've seen.

mymiles2go · Thu Jun 21, 2007 1:59 am

Quoting Davescj (Reply 15):
On the other hand, if ALL mainline UA is 2 hours behind......fewer missed connections, no? Because, literally, everyone will be 2 hrs behind.....and they can slowly move toward "normal" by delaying some flights, rebooking etc, no?

No, because of connections to UAX (which run a ton of United taged flights) as well as Star Alliance flights.

ikramerica · Thu Jun 21, 2007 2:13 am

Quoting Mymiles2go (Reply 17):
No, because of connections to UAX (which run a ton of United taged flights) as well as Star Alliance flights.

And not all planes are behind 2 hours. Only planes scheduled to depart in that 2 hour window were delayed. Some as little as 10 minutes, others as much as 2 hours. The average was stated at 65 minutes.

In general, a one hour delay for a bunch of flights is no worse than ORD being shut for 1 hour for a snow event. And since it's not weather, once back up, it's full bore, not extra spacing, so they can recover.

bnatraveler · Thu Jun 21, 2007 2:20 am

UA 743 ATL-DEN was delayed almost 3 hours due to this outage.

bcoz · Thu Jun 21, 2007 2:41 am

Mother Tribune here in Chicago has a pretty misleading subhed on its story about the outage on its website.

"All incoming and outgoing United Airlines flights at Chicago's O'Hare Airport are backed up by one to two hours, the city says."

Of course, the general public thinks UAX flights are United flights, and it appears that only mainline flights have been impacted.

Also...I hate how the media around here always runs to the Department of Aviation, and not the airline, for a quote on delays.... And they are always generalized quotes that you good and well know don't apply to every situation.

Just a pet peeve of mine!

And here last week I was applauding the Tribune's coverage!

bcoz

ikramerica · Thu Jun 21, 2007 2:44 am

Quoting Bcoz (Reply 20):
I hate how the media around here always runs to the Department of Aviation, and not the airline, for a quote on delays...

Well, Chicago is the land of the muckrakers...

Why go to a source that can shed light on an issue when you can find someone to generalize?

JER757 · Thu Jun 21, 2007 3:09 am

Quoting RDUDDJI (Reply 16):
I believe UA did away with manual load planners a few years ago

Why's that? Manual loadplans/sheets are pretty easy once you know what you're doing and only take a day or two training to get up to speed. I'm sure they would have had other resources available to them, if they didn't they are far too reliant on the single system. Are there no hard copies of crewing rosters, or stand alone PC's for SITA messaging for example?

Although there would have still been delays, at least some aircraft would have got off the ground and the whole operation would have been easier to recover.

GoAllegheny · Thu Jun 21, 2007 4:57 am

Quoting Bcoz (Reply 20):
Mother Tribune here in Chicago has a pretty misleading subhed on its story about the outage on its website.

Agreed that the story should make clear that the flights are mainline only, but 99.95% of the public doesn't understand that other companies own the regionals, so clarity in reporting might actually cause more confusion.

Quoting Bcoz (Reply 20):
Also...I hate how the media around here always runs to the Department of Aviation, and not the airline, for a quote on delays.... And they are always generalized quotes that you good and well know don't apply to every situation.

A United spokesperson (Robin Urbanski) is the first person quoted in the Trib article, so not sure about your criticism here.

Also, re outsourcing, the Trib article reports that "Urbanski said that some information technology work formerly done at United had been outsourced to a vendor, but could not say whether the outsourced work was involved in the computer failure." Does anyone know if that would matter in this case? I suspect that outsourcing or not, the real solution is investing in stronger redundancy.

MEACEDAR · Thu Jun 21, 2007 5:25 am

I was at LAX in the morning to go on UA272 to MCO. My flight got canceled. Now, they might either put me on the night flight the sameday or the same flight on 2morrow.

S*!t happens!

bcoz · Thu Jun 21, 2007 5:29 am

Quoting GoAllegheny (Reply 23):
A United spokesperson (Robin Urbanski) is the first person quoted in the Trib article, so not sure about your criticism here.

I think my criticism was a venting less directed at this specific story and more toward the way that the Chicagoland media behave in general when it comes to delay issues at O'Hare or Midway. It seems that Wendy Abrams is always quoted and its always a generic statement regarding delays that don't take into account the 1,001 variables that will go into whether or not a specific flight is delayed.

Frankly, Urbanski offers the solidest quote of them all regarding delays when she says "Go to our website and check!'

Quoting GoAllegheny (Reply 23):

Agreed that the story should make clear that the flights are mainline only, but 99.95% of the public doesn't understand that other companies own the regionals, so clarity in reporting might actually cause more confusion.

They've changed the hed now, but as I recall it, they could have avoided any sort of confusion by changing "all" to "some."

I used to be a reporter and, of course, I loved covering aviation issues (that was at COU ). So, I get peeved really easily when it comes to aviation reporting Smile

.

bcoz

hiflyer · Thu Jun 21, 2007 5:41 am

Quoting RDUDDJI (Reply 16):
I believe UA did away with manual load planners a few years ago

Yes...I also heard that load planning was taken from the stations and centralized several years ago as a cost cutting measure during the chapter 11 proceedings. With the loss of Unimatic sounds like UA lost the ability to send any sita traffic internally as well as all load plan, fuel sheets, and flight paperwork. Monday morning quarterbacking would say perhaps too many eggs in one basket without a quick functioning backup basket....and IMHO that is a fair assumption looking from the outside without knowing any real internal details. Two hours for a very large carrier to be grounded, in this day age of advanced computers and systems, might be considered a little long,

Tango-Bravo · Thu Jun 21, 2007 5:45 am

Quoting GoAllegheny (Reply 23):
Agreed that the story should make clear that the flights are mainline only, but 99.95% of the public doesn't understand that other companies own the regionals, so clarity in reporting might actually cause more confusion.

The legacies have brought this misunderstanding on themselves. Ever notice how United, for example, will boast about "United announces 30 new routes" while glossing over the fact that 29 of those routes are operated by Mesa, Skywest and/or whomever? All of the emphasis is on new "United" service. On the other hand, when serious service issues effecting their UAX surrogates arise, United will do everything possible to minimize and even disavow their connection with their UAX contractors.

skyrat · Thu Jun 21, 2007 10:09 am

At least UAX was not really affected. UAX makes up a big part of the operation in ORD and that would have been really bad if both UA & UAX were grounded.

Boston92 · Thu Jun 21, 2007 10:52 am

Quoting Skyrat (Reply 28):
At least UAX was not really affected. UAX makes up a big part of the operation in ORD and that would have been really bad if both UA & UAX were grounded.

UAX may not have been directly affected by the glitch, but where do you think all the UAX pax come from?

RDUDDJI · Thu Jun 21, 2007 11:12 am

Quoting Hiflyer (Reply 26):
Yes...I also heard that load planning was taken from the stations and centralized several years ago as a cost cutting measure during the chapter 11 proceedings. With the loss of Unimatic sounds like UA lost the ability to send any sita traffic internally as well as all load plan, fuel sheets, and flight paperwork. Monday morning quarterbacking would say perhaps too many eggs in one basket without a quick functioning backup basket....and IMHO that is a fair assumption looking from the outside without knowing any real internal details. Two hours for a very large carrier to be grounded, in this day age of advanced computers and systems, might be considered a little long,

Two hours IS very long. However, Unimatic controls many flight functions besides Load Planning. Maintenance releases, FIDS operations and many other things are also all rolled up. United's system (under normal ops) saves a ton of paper over other systems...but alas, you are relying on a computer. Unimatic occasionally will go out, but I haven't seen an unplanned outage this long in a very long time. Usually outages are only for 5-10 minutes. It will be interesting to find out what caused it.

For UA to revert to manual load planning would require significant work effort. Stations would once again have to be certified in LP and the FAA would have to sign off on the program (again). At some point it becomes more efficient to make sure Unimatic doesn't fail...

ANCFlyer · Thu Jun 21, 2007 11:58 pm

How is the system this morning? Curiosity . . . as I'm flying UA tomorrow?

RDUDDJI · Fri Jun 22, 2007 12:33 am

Quoting ANCFlyer (Reply 31):
How is the system this morning? Curiosity . . . as I'm flying UA tomorrow?

Everything appears back to normal. Morning departures are mostly on-time.

B6sFinest · Fri Jun 22, 2007 12:59 am

Seems funy though, that there isnt alot of bashing like B6 got a couple of months ago on the media outlets... Go figure....

RDUDDJI · Fri Jun 22, 2007 1:22 am

Quoting B6sFinest (Reply 33):
Seems funy though, that there isnt alot of bashing like B6 got a couple of months ago on the media outlets... Go figure....

Go to http://news.google.com Search "United computer outage" There's hundreds of articles from all over the World. And I wouldn't nearly equate a 2 hour computer outage the same as a snowstorm that incapacitated B6 for many days.

Tango-Bravo · Fri Jun 22, 2007 1:29 am

Quoting B6sFinest (Reply 33):
Seems funy though, that there isnt alot of bashing like B6 got a couple of months ago on the media outlets... Go figure....

...I would "go figure" that there is little comparison between a number of flights being delayed for several hours in one day and a systemwide service meltdown and flight cancellations lasting the better part of one week after operations could have and should have been back to normal.

RDUDDJI · Fri Jun 22, 2007 3:22 am

For those who care, turns out the root cause was human error. They were running some tests (the hardware that supports Unimatic was recently upgraded). Someone erred during testing and caused not only the production system, but the back up system to fail simultaneously. Doh!

blueflyer · Fri Jun 22, 2007 3:47 am

Quoting AirSpare (Reply 13):
Sounds like Cisco, Nortel or Sun will be lnocking on some doors selling Business Continuity Plans and systems. The 2 hour outage would have probably paid for 25% of a robust network.

No, they won't. A gazillion switches from Cisco, Nortel's best engineers and a sea of Sun servers would not have prevented this problem. Sadly, there isn't a technological answer to everything, especially to...

Quoting RDUDDJI (Reply 36):
human error

The only technological answer they could come up with, if possible (and that is a very big if), would be to program both the backup and main systems to reject simultaneous operations-interrupting commands. These types of fail-safe controls are very hard to implement, because the definition of an operation-interrupting command can be very broad and indeed variable (eg: loading a new config may not interrupt operations unless the new config doesn't actually work...).

Quoting RDUDDJI (Reply 30):
Two hours IS very long

Two hours to restart a big iron and restore it to its last-known-good configuration and process the backlog in the queues before it can actually go back to live is actually not bad at all... We're not exactly talking about rebooting your average Starbucks' Pentium server here.

MEACEDAR · Fri Jun 22, 2007 10:26 am

To add to reply 20:

After talking to a bunch of people in LAX that worked for UA, it turns out that the night flight for June 20 was booked at the morning flight for June 21 was also booked. They put me on UA 276 that leaves at 10:20 PM on June 21. This is the first time that UA made me made.

Oh well, things happen all the time. Technology can help and the same time destroy.

By the way, the night flight for June 20 was delayed about 3 hours. I feel kind of lucky for not being on that flight.

DL WIDGET HEAD · Fri Jun 22, 2007 10:42 am

Quoting B6sFinest (Reply 33):
Seems funy though, that there isnt alot of bashing like B6 got a couple of months ago on the media outlets... Go figure....

Not even close to a similar situation. UA's was in essence a 2 hour computer outage which of course had ramifications throughout their system but they were basically back to normal by the next morning. On average 65min delays and possibly a few hundred flights cancelled. B6 suffered a complete and devastating meltdown that lasted a week or more and cancelled well over 1000 flights with untold number of horrendously long delays. No need for the media to beat UA up over this like they did to B6.

OPNLguy · Fri Jun 22, 2007 10:59 am

Quoting RDUDDJI (Reply 36):
For those who care, turns out the root cause was human error. They were running some tests (the hardware that supports Unimatic was recently upgraded). Someone erred during testing and caused not only the production system, but the back up system to fail simultaneously. Doh!

Sounds kinda similar to the comm failure at ZLA 2-3 years ago. IIRC, it was the Voice Switching Control System (VSCS), and someone forgot to do some computer-related stuff to it (deleting old files?) and it crashed. (Not the more recent comm and radar failure from the nearby car accident that knocked down a power pole and put ZLA on back-up power, which failed after about an hour.)

Who is online

Popular Searches On Airliners.net