Moderators: jsumali2, richierich, ua900, PanAm_DC10, hOMSaR

 
Scarebus34
Topic Author
Posts: 734
Joined: Tue Feb 12, 2019 10:54 pm

Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:16 pm

Southwest is facing its second system-wide IT outage in as many days. All flights currently stopped. No resolution time given.
 
User avatar
lightsaber
Moderator
Posts: 22715
Joined: Wed Jan 19, 2005 10:55 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:19 pm

Flights resumed:
https://www.msn.com/en-us/travel/news/s ... NewsSearch

Interesting that weather information stopped them.

Lightsaber
 
Scarebus34
Topic Author
Posts: 734
Joined: Tue Feb 12, 2019 10:54 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:21 pm

lightsaber wrote:
Flights resumed:
https://www.msn.com/en-us/travel/news/s ... NewsSearch

Interesting that weather information stopped them.

Lightsaber

That article is referencing their outage last night. They have another one occurring at this moment.

https://twitter.com/southwestair/status ... 06182?s=21
 
jholio
Posts: 41
Joined: Mon Aug 19, 2019 10:23 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:28 pm

lightsaber wrote:
Flights resumed:
https://www.msn.com/en-us/travel/news/s ... NewsSearch

Interesting that weather information stopped them.

Lightsaber


As the other poster said, this is new and current. I was listening to KDAL tower around 12:45p central time and the controller mentioned Southwest was in a ground stop, so it was a great time to fly into the airfield. There were WN flights already airborne still arriving, but none leaving.
Last edited by jholio on Tue Jun 15, 2021 6:32 pm, edited 3 times in total.
 
av8tiongeek
Posts: 165
Joined: Wed Dec 09, 2020 10:23 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:28 pm

You get a Yeti. You get a Yeti. You all get a Yeti.
 
orlandocfi
Posts: 141
Joined: Mon Nov 23, 2020 12:53 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:38 pm

Kinda sucks that these outages are overshadowing the celebratory mood of anniversary week.
 
DEN1895
Posts: 321
Joined: Wed Dec 12, 2018 5:21 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:46 pm

Currently 22 aircraft on the ground in DEN with no available gates, nearly every deice pad full.
 
QXAS
Posts: 383
Joined: Thu Nov 19, 2015 5:26 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 6:57 pm

Southwest flights appear to be pushing back at PHX. Although they could be empty jets moving to stands to open up gates for arriving flights.

Edit: WN4165 just departed for ONT, 3 hours late.
 
av8tiongeek
Posts: 165
Joined: Wed Dec 09, 2020 10:23 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:00 pm

Some systems have returned to normal.
 
User avatar
dabpit
Posts: 969
Joined: Tue May 01, 2012 10:19 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:07 pm

Has it been reported which system crashed this time? Surprised they haven’t had more issues with all the patch work that has been done with all their systems.
 
aviationjunky
Posts: 430
Joined: Fri Mar 25, 2016 10:27 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:14 pm

WN3256 just departed LAS for ELP, almost 2 hours late.
 
Sooner787
Posts: 2860
Joined: Thu Jul 18, 2013 1:44 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:18 pm

SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL
 
AWACSooner
Posts: 2638
Joined: Tue Jan 22, 2008 12:35 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:24 pm

Sooner787 wrote:

I think the 50% off fare sale may have crashed their system LOL

I know I took advantage of that this morning.

Still...WTH, WN? Unless it's a third-party hack, having your system crash twice in less than 24 hours shows how fragile and vulnerable your IT system is.
 
Eiszeit
Posts: 81
Joined: Tue Jul 09, 2019 8:50 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:32 pm

Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL


That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:
 
catiii
Posts: 3889
Joined: Mon Mar 31, 2008 1:18 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:43 pm

AWACSooner wrote:
Sooner787 wrote:

I think the 50% off fare sale may have crashed their system LOL

I know I took advantage of that this morning.

Still...WTH, WN? Unless it's a third-party hack, having your system crash twice in less than 24 hours shows how fragile and vulnerable your IT system is.


It wasn't their system yesterday. It was their weather provider's system, the failure of which would not allow them to dispatch.
 
MO11
Posts: 1846
Joined: Sun Jan 08, 2017 5:07 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:45 pm

AWACSooner wrote:
Sooner787 wrote:

I think the 50% off fare sale may have crashed their system LOL

I know I took advantage of that this morning.

Still...WTH, WN? Unless it's a third-party hack, having your system crash twice in less than 24 hours shows how fragile and vulnerable your IT system is.


Different systems. Yesterday was the weather vendor which also affected AS and DL.
 
AWACSooner
Posts: 2638
Joined: Tue Jan 22, 2008 12:35 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 7:58 pm

I stand corrected...but still concerning.
 
User avatar
lightsaber
Moderator
Posts: 22715
Joined: Wed Jan 19, 2005 10:55 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:04 pm

jholio wrote:
lightsaber wrote:
Flights resumed:
https://www.msn.com/en-us/travel/news/s ... NewsSearch

Interesting that weather information stopped them.

Lightsaber


As the other poster said, this is new and current. I was listening to KDAL tower around 12:45p central time and the controller mentioned Southwest was in a ground stop, so it was a great time to fly into the airfield. There were WN flights already airborne still arriving, but none leaving.

Well, I picked the wrong week to quit sniffing glue!

https://youtu.be/KAccN64YmCs

Not good news.

What level of operation are they up to?

Lightsaber
 
Sooner787
Posts: 2860
Joined: Thu Jul 18, 2013 1:44 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:07 pm

Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL



SWABIZ update 1506 cdt .... appears to be back online. I was just able to complete
a booking. Now let's hope WN get their ops back to normal soon.
 
User avatar
vatveng
Posts: 1267
Joined: Wed Jan 07, 2004 11:49 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:10 pm

Eiszeit wrote:
Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL


That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.
 
32andBelow
Posts: 5645
Joined: Mon Sep 03, 2012 2:54 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:25 pm

Eiszeit wrote:
Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL


That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:

Everything in an airline is connected to the res system. You have to be able to pull flight manifests to dispatch a flight. The red system IS critical.

Not saying that is what happened but it’s all certainly connected.
 
dtdude
Posts: 14
Joined: Sun Apr 10, 2016 5:06 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:41 pm

vatveng wrote:
Eiszeit wrote:
Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL


That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.
 
WN732
Posts: 1021
Joined: Sat Feb 26, 2011 12:49 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 8:49 pm

dtdude wrote:
vatveng wrote:
Eiszeit wrote:

That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.


From an IT perspective, I generally am really interested in what the innards of the WN network are and what kind of failover and backup systems they have. They just had a job posting for a Veeam specialist the other day, and other jobs for Office 365 Admin, etc. None of that is cheap, and it shows that at least outside of the reservations system (which has seen some recent major overhaul), their IT has kept up with the times.
 
dtdude
Posts: 14
Joined: Sun Apr 10, 2016 5:06 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:05 pm

WN732 wrote:
dtdude wrote:
vatveng wrote:

As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.


From an IT perspective, I generally am really interested in what the innards of the WN network are and what kind of failover and backup systems they have. They just had a job posting for a Veeam specialist the other day, and other jobs for Office 365 Admin, etc. None of that is cheap, and it shows that at least outside of the reservations system (which has seen some recent major overhaul), their IT has kept up with the times.


Travelers are reporting that NOTHING works. Not even internal on-net telephone calls in Denver (Cisco CUCM, I believe). That really makes me think it's more along the lines of an AD/DNS/DHCP failure, or an issue at a datacenter (although you would think they'd have a local domain controller and a CUCM subscriber node). Could sit and speculate all day long, but definitely sounds network related and not specific to any 1 system.

But yes, I am very curious about the inner-workings.
 
User avatar
lightsaber
Moderator
Posts: 22715
Joined: Wed Jan 19, 2005 10:55 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:19 pm

2nd glitch making financial news just within last hour:
https://www.msn.com/en-us/travel/news/s ... NewsSearch

Southwest Airlines said it was working Tuesday to restore normal operations after a technology-related issue interfered with flights for the second straight day.

By midafternoon, the nation’s fourth-largest airline had canceled about 500 flights and delayed nearly 1,300 others, according to tracking service FlightAware. The combination of cancellations and delays affected about half of Southwest’s planned flights for the day.


Oops.
What more can be said. Without knowing the inner workings... If due to obsolete systems, that could be a huge job ahead.

Lightsaber
 
chrisp390
Posts: 729
Joined: Fri May 16, 2014 6:37 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:36 pm

How likely is it that this is a foreign hacking and extortion operation similar to the Colonial Pipeline hack and others recently?
 
Speedy752
Posts: 115
Joined: Fri Dec 11, 2020 4:13 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:41 pm

Cubsrule wrote:
MO11 wrote:
AWACSooner wrote:
I know I took advantage of that this morning.

Still...WTH, WN? Unless it's a third-party hack, having your system crash twice in less than 24 hours shows how fragile and vulnerable your IT system is.


Different systems. Yesterday was the weather vendor which also affected AS and DL.


There were more issues than that across a number of carriers yesterday, but I don't know if they were related. DL had some sort of booking engine burp last night that resulted in no Main Cabin fares being displayed across some large number of city pairs. I can't believe that's related to weather but maybe there was something else on the back end that was bigger than just WN?


This was a system wide glitch? I was checking out Hawaii fares and thought maybe it was just demand into all city pairs
 
bnatraveler
Posts: 402
Joined: Tue Dec 02, 2003 10:10 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:42 pm

dtdude wrote:
vatveng wrote:
Eiszeit wrote:

That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.


Southwest actually swapped AWAY from their Sabre-hosted SAAS system (which has heritage back to Braniff's Cowboy system) to Amadeus. Amadeus is a solid solution and if they were down, you'd see lots of airlines impacted. They also moved away from a very old (mainframe hosted) accounting system to SAP which is also hosted by a third party. Southwest also has spend a huge sum of money to build an alternate/backup data center for their operations. Amadeus isn't hosted in-house, nor is SAP, but they rely on traffic paths that go through their data centers.

The brittle part of the airline, as I understand it, is their dispatch system that was written in house in the late 90s/early 2000s. I think the system is called fetch or rover or something like that. Obviously, it has been substantially updated over time, but I understand that it needs wholesale replacement. Southwest is a much more complex airline now than it was then.

Also, I understand that their passenger revenue accounting system needs work in order to accommodate passenger fare/service purchase transactions in currencies other than USD. Not sure where that stands on the list of replacement priorities.
 
dtdude
Posts: 14
Joined: Sun Apr 10, 2016 5:06 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 9:57 pm

bnatraveler wrote:
dtdude wrote:
vatveng wrote:

As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.


Southwest actually swapped AWAY from their Sabre-hosted SAAS system (which has heritage back to Braniff's Cowboy system) to Amadeus. Amadeus is a solid solution and if they were down, you'd see lots of airlines impacted. They also moved away from a very old (mainframe hosted) accounting system to SAP which is also hosted by a third party. Southwest also has spend a huge sum of money to build an alternate/backup data center for their operations. Amadeus isn't hosted in-house, nor is SAP, but they rely on traffic paths that go through their data centers.

The brittle part of the airline, as I understand it, is their dispatch system that was written in house in the late 90s/early 2000s. I think the system is called fetch or rover or something like that. Obviously, it has been substantially updated over time, but I understand that it needs wholesale replacement. Southwest is a much more complex airline now than it was then.

Also, I understand that their passenger revenue accounting system needs work in order to accommodate passenger fare/service purchase transactions in currencies other than USD. Not sure where that stands on the list of replacement priorities.


Ah, thanks for the correction. I had their new content deals with Sabre in mind. I forgot that they moved to Amadeus.
 
UPNYGuy
Posts: 344
Joined: Thu Jan 06, 2011 4:14 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 10:45 pm

AWACSooner wrote:
Sooner787 wrote:

I think the 50% off fare sale may have crashed their system LOL

I know I took advantage of that this morning.


Same here. I re-rated 2 flights I have in biz select and got back roughly $1500
 
Cubsrule
Posts: 15270
Joined: Sat May 15, 2004 12:13 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 10:57 pm

Speedy752 wrote:
Cubsrule wrote:
MO11 wrote:

Different systems. Yesterday was the weather vendor which also affected AS and DL.


There were more issues than that across a number of carriers yesterday, but I don't know if they were related. DL had some sort of booking engine burp last night that resulted in no Main Cabin fares being displayed across some large number of city pairs. I can't believe that's related to weather but maybe there was something else on the back end that was bigger than just WN?


This was a system wide glitch? I was checking out Hawaii fares and thought maybe it was just demand into all city pairs


I don’t know if it was systemwide but I can say with certainty that it was not just demand related and seemed to be resolved this morning.
 
FlyingElvii
Posts: 1605
Joined: Wed Dec 27, 2017 10:53 pm

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 11:35 pm

dtdude wrote:
vatveng wrote:
Eiszeit wrote:

That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.

Sabre....
A system still based on the late 70’s Cray.
 
User avatar
SuseJ772
Posts: 1063
Joined: Wed Aug 10, 2005 11:13 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 11:51 pm

vatveng wrote:
Eiszeit wrote:
Sooner787 wrote:
SWABIZ is offline right now ( 1417 cdt ) and we have couple travelers in middle of their trips to DAL.

I think the 50% off fare sale may have crashed their system LOL


That would be crazy, there is no way operation critical systems have any connection to a sales system... But sometimes the crazy stuff is not as crazy as we think. :?:


As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


As someone who led just that (in non-aviation), I would agree. It also doesn't have to be as expensive as everyone always thinks it is, but few, very few in technology actually care about saving money so it ends up costing 5x as much and takes 2x as long.
 
User avatar
SuseJ772
Posts: 1063
Joined: Wed Aug 10, 2005 11:13 am

Re: Southwest faces another IT outage, all flights stopped.

Tue Jun 15, 2021 11:54 pm

bnatraveler wrote:
dtdude wrote:
vatveng wrote:

As someone who maintains a (non-aviation) cobbled-together multi-vendor legacy I.T. system that I neither designed nor have the authority to replace, I can say with confidence that systems that have nothing to do with each other and shouldn't have any communication do sometimes cause each other to crash. Yes, it is crazy. And infuriating, and frustrating. A joke I tell at work is that these systems are so bizarrely interconnected in ways they shouldn't be that sometimes when you flip a light switch in a storage closet, the fire hydrant out front explodes.

Really the only way to fix it is to replace everything, all at once. Even if you have access to enough capital to do that, it's incredibly disruptive and the replacement comes with training and teething issues of its own. Often companies decide to stick with the devil you know. But eventually you get to a point where the legacy systems cause more PR disasters than the shareholders will tolerate. Then, and only then, will companies like Southwest replace their legacy systems.


Southwest's systems are fairly new, IIRC. They switched away from an older proprietary reservations system to Sabre not long ago. I would image this would require significant hardware investment.

But...who knows. This could be a shoddy implementation of Sabre. If could be failed hardware. It could be a WAN connectivity issue (which if done right should NEVER be an issue). Could be lots of things.


Southwest actually swapped AWAY from their Sabre-hosted SAAS system (which has heritage back to Braniff's Cowboy system) to Amadeus. Amadeus is a solid solution and if they were down, you'd see lots of airlines impacted. They also moved away from a very old (mainframe hosted) accounting system to SAP which is also hosted by a third party. Southwest also has spend a huge sum of money to build an alternate/backup data center for their operations. Amadeus isn't hosted in-house, nor is SAP, but they rely on traffic paths that go through their data centers.

The brittle part of the airline, as I understand it, is their dispatch system that was written in house in the late 90s/early 2000s. I think the system is called fetch or rover or something like that. Obviously, it has been substantially updated over time, but I understand that it needs wholesale replacement. Southwest is a much more complex airline now than it was then.

Also, I understand that their passenger revenue accounting system needs work in order to accommodate passenger fare/service purchase transactions in currencies other than USD. Not sure where that stands on the list of replacement priorities.


SAP is rough. I know it is the "gold standard" but it always, always, always fails to meet expectations. Also, SAP = Sudden Additional People. You IT costs go through the roof and it makes it harder to approve future capital projects because the first one (SAP) usually goes so poorly.

Southwest, if you need need your in-house dispatch re-written, I'd love to take on that project and give me a call {wink}
 
alasizon
Posts: 3044
Joined: Sat Apr 28, 2007 8:57 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 12:03 am

My understanding is their whole domain was down so not really an issue with a particular program but rather the most modern portion of their system.
 
dtdude
Posts: 14
Joined: Sun Apr 10, 2016 5:06 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 12:06 am

alasizon wrote:
My understanding is their whole domain was down so not really an issue with a particular program but rather the most modern portion of their system.


I guess someone applied those patches to their domain controllers without testing first ha!
 
CALMSP
Posts: 3675
Joined: Wed Aug 13, 2003 3:18 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 12:45 am

was flying through DEN on my way to GJT and every gate on C was full of WN and then I counted 27 just on the west side of the airport waiting for gates. What a mess.
 
smokeybandit
Posts: 1574
Joined: Mon Mar 17, 2014 3:24 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 1:03 am

alasizon wrote:
My understanding is their whole domain was down so not really an issue with a particular program but rather the most modern portion of their system.


"Oh we don't need this DNS entry, let me clean things up"
 
User avatar
lightsaber
Moderator
Posts: 22715
Joined: Wed Jan 19, 2005 10:55 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 1:58 am

CALMSP wrote:
was flying through DEN on my way to GJT and every gate on C was full of WN and then I counted 27 just on the west side of the airport waiting for gates. What a mess.

Photos Please.

This is interesting...

Lightsaber
 
nine4nine
Posts: 800
Joined: Tue Mar 28, 2017 3:44 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 2:42 am

I landed at SNA a little after 5pm today and counted 11 WN tails. Never seen that many at once at SNA.
 
DEN1895
Posts: 321
Joined: Wed Dec 12, 2018 5:21 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 2:56 am

lightsaber wrote:
CALMSP wrote:
was flying through DEN on my way to GJT and every gate on C was full of WN and then I counted 27 just on the west side of the airport waiting for gates. What a mess.

Photos Please.

This is interesting...

Lightsaber


While not as exciting, here is the ground radar from 1230ish today. All 24 gates are full and the 4 main deice pads are full with a total of 22 aircraft waiting for gates, around this time is when the ground stop went into affect.
Image
 
slcdeltarumd11
Posts: 5207
Joined: Fri Jan 09, 2004 7:30 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 3:09 am

CALMSP wrote:
was flying through DEN on my way to GJT and every gate on C was full of WN and then I counted 27 just on the west side of the airport waiting for gates. What a mess.


Not ok. Even with all the IT meltdowns at WN that seems unacceptabe. Clearly WN was a mess and that IT meltdown drained alot of other resources for that happen.
 
User avatar
BMWdrvr75
Posts: 97
Joined: Wed Jul 08, 2015 6:23 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 5:08 am

what do y’all expect from an airline that can’t even do red eyes and their stewardesses still serve off trays??? Just saying!
 
User avatar
ADent
Posts: 1210
Joined: Fri Dec 22, 2006 12:11 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 5:30 am

Some local press coverage:

Tonight:
Child Hospitalized For Dehydration After Long Delays On Southwest Airlines Flight
https://denver.cbslocal.com/2021/06/15/ ... pitalized/


Mid Day:
Southwest Airlines delays, cancellations at Denver International Airport caused by network issues
Southwest has 88 cancellations today at DIA
https://www.denverpost.com/2021/06/15/s ... l-airport/



Yesterday:
Dozens Of Flights Delayed Or Canceled At DIA After Southwest Airlines Computer Failure
https://denver.cbslocal.com/2021/06/15/ ... r-failure/
 
kwp302
Posts: 175
Joined: Thu Feb 28, 2019 4:59 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 5:46 am

WN4030 just now departing DEN for SDF. Set to arrive at 3:56 AM EDT…over 4 hours late.

http://flightaware.com/live/flight/SWA4 ... /KDEN/KSDF
 
32andBelow
Posts: 5645
Joined: Mon Sep 03, 2012 2:54 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 6:47 am

BMWdrvr75 wrote:
what do y’all expect from an airline that can’t even do red eyes and their stewardesses still serve off trays??? Just saying!

Trays is way better. Doesn’t block the whole aisle for an hour
 
AA100
Posts: 134
Joined: Sun Nov 22, 2015 10:57 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 7:14 am

Each to their own... he has a point... we are talking about IT systems, and it is fairly well known that red-eyes are limited by technology. They only recently were able to fly internationally.

I find the trays a bit odd, but it does seem to work albeit with a lot of running/shuffling from the poor crew. I think they use each drink slot for a seat number so it allows them to hustle drinks out. One person takes orders, one person makes drinks, one person serves them etc. I was impressed the entire 737 I was on from LAS to SAN offered a snack and a drink. So however their system works.. it works!
 
slcdeltarumd11
Posts: 5207
Joined: Fri Jan 09, 2004 7:30 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 8:18 am

ADent wrote:
Some local press coverage:

Tonight:
Child Hospitalized For Dehydration After Long Delays On Southwest Airlines Flight
https://denver.cbslocal.com/2021/06/15/ ... pitalized/


Mid Day:
Southwest Airlines delays, cancellations at Denver International Airport caused by network issues
Southwest has 88 cancellations today at DIA
https://www.denverpost.com/2021/06/15/s ... l-airport/



Yesterday:
Dozens Of Flights Delayed Or Canceled At DIA After Southwest Airlines Computer Failure
https://denver.cbslocal.com/2021/06/15/ ... r-failure/


This wasn't an IT failure this became an airline failure. How could they allow this to get so bad? Southwest needs to examine it's emergency plans to get passengers off planes and get gate space open. They just failed running an airline allowing all those planes to continue to land in Denver when they couldnt get the people off the planes already or get gates open. They didn't need an it system to move empty planes out of gates or to get people off those planes. Huge Fail by southwest airlines, let's hope they can make it thru this week. What a mess, again for southwest airlines.
 
slcdeltarumd11
Posts: 5207
Joined: Fri Jan 09, 2004 7:30 am

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 8:28 am

https://www.transportation.gov/individu ... mac-delays

Shouldn't the tarmac delays fine southwest airlines alot of money? Reading the various Denver media articles It looks like southwsst broke pretty much event tarmac rule having people sit there while they were totally out of control
 
Silver1SWA
Posts: 4770
Joined: Wed Mar 10, 2004 6:11 pm

Re: Southwest faces another IT outage, all flights stopped.

Wed Jun 16, 2021 12:10 pm

slcdeltarumd11 wrote:
CALMSP wrote:
was flying through DEN on my way to GJT and every gate on C was full of WN and then I counted 27 just on the west side of the airport waiting for gates. What a mess.


Not ok. Even with all the IT meltdowns at WN that seems unacceptabe. Clearly WN was a mess and that IT meltdown drained alot of other resources for that happen.


It caused a ground stop. The planes waiting were already enroute when the system went down and with the arrival rate of a station the size of DEN things get backed up very quickly.

Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos