Moderators: jsumali2, richierich, ua900, PanAm_DC10, hOMSaR

 
stlgph
Topic Author
Posts: 11229
Joined: Tue Oct 12, 2004 4:19 pm

Delta Air Lines system outage August 8th, 2016

Mon Aug 08, 2016 8:06 am

Seeing on Twitter, lots of folks having problems checking in for Delta flights all around the world.

Flight delays ensuing, too.

Per Delta, looks to be a computer issue.


http://www.breakingnews.com/


Stay tuned.
Last edited by SQ22 on Mon Aug 08, 2016 5:08 pm, edited 1 time in total.
Reason: Thread title was misleading
if assumptions could fly, airliners.net would be the world's busiest airport
 
bhmdiversion
Posts: 285
Joined: Mon Dec 29, 2008 2:41 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:00 am

Deltanet and DCI pipeline are down...
 
stlgph
Topic Author
Posts: 11229
Joined: Tue Oct 12, 2004 4:19 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:17 am

Flights grounded.
Not a good start to the day.
if assumptions could fly, airliners.net would be the world's busiest airport
 
User avatar
Acey
Posts: 1125
Joined: Wed Jun 27, 2007 2:06 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:31 am

This includes OO, CP, G7 and everybody else, right? Or just mainline?
If a man hasn't discovered something that he will die for, he isn't fit to live. -- Martin Luther King, Jr.
 
fxra
Posts: 600
Joined: Fri Jul 16, 1999 1:03 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:44 am

Acey wrote:
This includes OO, CP, G7 and everybody else, right? Or just mainline?


Looks like all

ATCSCC ADVZY 024 DCC 08/08/2016 GROUND STOP
EVENT TIME: 08/0930 - 08/1030
DESTINATION AIRPORT: ALL
FACILITIES INCLUDED: ALL
GROUND STOP
PERIOD: 08/0930Z - 08/1030Z
REASON: AIRLINE REQUEST DUE TO COMPUTER
FAILURE
REMARKS: DAL AND DAL SUBS ONLY
EFFECTIVE TIME: 08/0937 - 08/1100
Visualize Whirled Peas
 
bgm
Posts: 2511
Joined: Fri Sep 11, 2009 9:37 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:46 am

Oops, there goes those on-time stats for August. ;)
 
User avatar
Acey
Posts: 1125
Joined: Wed Jun 27, 2007 2:06 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 9:50 am

Rest in peace, sweet prince. #deltageddon
If a man hasn't discovered something that he will die for, he isn't fit to live. -- Martin Luther King, Jr.
 
User avatar
tlecam
Posts: 1500
Joined: Tue Jul 23, 2013 1:38 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:07 am

I don't even know what to do. I have an 8AM departure out of LGA. Do I even bother going? No word on delays yet.
BOS-LGA-JFK | A:319/20/21, 332/3, 346 || B:717, 735, 737, 738, 739, 752, 753, 762, 763, 764, 787, 772, 744 || MD80, MD90
 
User avatar
Acey
Posts: 1125
Joined: Wed Jun 27, 2007 2:06 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:10 am

tlecam wrote:
I don't even know what to do. I have an 8AM departure out of LGA. Do I even bother going? No word on delays yet.


DL says to "check the status of the flight before heading to the airport."

In other words, DL has nothing of value to tell you at this point.
If a man hasn't discovered something that he will die for, he isn't fit to live. -- Martin Luther King, Jr.
 
User avatar
OA940
Posts: 1990
Joined: Fri May 20, 2016 6:18 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:29 am

I heard on the news it might have been a cyberattack / hack. Any possibility of that?
A350/CSeries = bae
 
User avatar
Acey
Posts: 1125
Joined: Wed Jun 27, 2007 2:06 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:37 am

OA940 wrote:
I heard on the news it might have been a cyberattack / hack. Any possibility of that?

Of course it's possible, but nothing has been confirmed yet. If that were the case, DL probably wouldn't say till they had it back up and reasonably secure.
If a man hasn't discovered something that he will die for, he isn't fit to live. -- Martin Luther King, Jr.
 
b747400erf
Posts: 3172
Joined: Wed Jun 19, 2013 4:33 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:38 am

Another router-mageddon?
 
Corpsnerd09
Posts: 587
Joined: Tue Mar 30, 2004 2:05 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:40 am

bgm wrote:
Oops, there goes those on-time stats for August. ;)


Plus completion factor and complaints numbers...

a few weeks ago people were complaining about Southwest and sizing them up to DL on computer programming, goes to show it can happen to anyone.

It's still only 640am, Hopefully it picks back up soon before the morning rush.
 
toltommy
Posts: 2799
Joined: Tue Dec 09, 2003 9:04 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:41 am

tlecam wrote:
I don't even know what to do. I have an 8AM departure out of LGA. Do I even bother going? No word on delays yet.


It's probably not going anywhere. But it appears Delta can't post delays right now. A quick check of flights statuses on Delta.com shows everything "on time". Gonna be a rough day.
A300/A310/A319/A320/A321/A332/A333 / 707/712/727/732/733/734/735/738/739/752/753
/762/763/764/772/788/789/DC8/DC9-10/30/40/50/MD81/83/87/88/90/L1011-/250/500/CRJ200/440 /700/900/EMB135/140/145/170/175/190/328Jet/F70/SF3/BE1/J31
 
wedgetail737
Posts: 5287
Joined: Mon Aug 04, 2003 8:44 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:58 am

Sounds like other airlines will be bolstering their pax loads a bit today and in the near-future. Here in SEA, maybe AS will win some additional pax at DL's expense.
 
akelley728
Posts: 2062
Joined: Tue Dec 07, 1999 12:35 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 10:59 am

I am at ORF and was supposed to be on the 6:45am to ATL. My boss was supposed to be on the 7:22 out of JAX to ATL. We were both trying to connect to the 9:21 to SAN. We are both stuck and nothing is flying. They said don't worry about connections since nothing is flying. They just told us they would give us an update at 7:30am Eastern Time.
 
UAL777UK
Posts: 2369
Joined: Sun Nov 20, 2005 1:16 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:04 am

Hi the news over the pond as well.

http://www.bbc.co.uk/news/world-us-canada-37007908

Hopefully problem will be resolved soon.
 
washingtonflyer
Posts: 1609
Joined: Sun Sep 08, 2013 9:45 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:08 am

Wonder if DL is reaccommodating on other carriers, or keeping folks in place. Bet they may be hoping for that ticket arrangement with AA today...
 
washingtonflyer
Posts: 1609
Joined: Sun Sep 08, 2013 9:45 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:08 am

 
User avatar
airportugal310
Posts: 3674
Joined: Sun Apr 11, 2004 12:49 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:12 am

Yikes...for a sober dose of reality, filter out DL on FR24 and that will show you this is not pretty. At all
“They bought their tickets, they knew what they were getting into. I say, let 'em crash.”
 
WaywardMemphian
Posts: 1508
Joined: Sat Dec 13, 2014 9:05 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:12 am

How long before we see the letter from the pilots demanding the CEO to step down
 
a380787
Posts: 4573
Joined: Tue Jul 09, 2013 4:38 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:14 am

washingtonflyer wrote:
Wonder if DL is reaccommodating on other carriers, or keeping folks in place. Bet they may be hoping for that ticket arrangement with AA today...


Given the route network differences and interline arrangements, UA might come out as the biggest beneficiary from any IRROPS REACCOM, if DL even have any call center agents to handle those calls.

I've always found their interline cutting to be short sighted at best. They're treating it too much like a P&L item when it's supposed to be a rainy day insurance policy. On the flip side, it's an excellent excuse to use when PRASM numbers come in less than ideal.
 
User avatar
readytotaxi
Posts: 7534
Joined: Mon Dec 11, 2006 2:09 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:20 am

At Heathrow T4 they are handing out bottles of water, people waiting over two half hours, they have been warned that some flights will be cancelled.
Virgin Atlantic has asked customers, or be it a bit late, to check-in extra early as they desk share with Delta.
Wouldn't want to be in that Terminal today. :o
you don't get a second chance to make a first impression!
Growing older, but not up.
 
AWACSooner
Posts: 2561
Joined: Tue Jan 22, 2008 12:35 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:24 am

WN and DL within a few weeks time? Maybe it's just me, but I highly doubt this is purely coincidence.
 
dirk88
Posts: 67
Joined: Mon Jun 27, 2016 2:10 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:25 am

Seems like some flights are departing now?

https://www.flightradar24.com/DAL179/a9adf7b
 
User avatar
tlecam
Posts: 1500
Joined: Tue Jul 23, 2013 1:38 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:30 am

I think I'm just going to wait it out and rebook to tomorrow AM.
BOS-LGA-JFK | A:319/20/21, 332/3, 346 || B:717, 735, 737, 738, 739, 752, 753, 762, 763, 764, 787, 772, 744 || MD80, MD90
 
coolian2
Posts: 2483
Joined: Sun Oct 22, 2006 3:34 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:31 am

AWACSooner wrote:
WN and DL within a few weeks time? Maybe it's just me, but I highly doubt this is purely coincidence.

I'd bet my house it's a coincidence.
Q300/ATR72-600/737-200/-300/-400/-700/-800/A320/767-200/-300/757-200/777-300ER/
747-200/-300/-400/ER/A340-300/A380-800/MD-83/-88/CRJ-700/-900
 
Kilopond
Posts: 563
Joined: Wed Jun 15, 2016 10:08 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:42 am

Three DAL flights just left Amsterdam for JFK, PDX and SEA.
 
toltommy
Posts: 2799
Joined: Tue Dec 09, 2003 9:04 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:42 am

washingtonflyer wrote:
Wonder if DL is reaccommodating on other carriers, or keeping folks in place. Bet they may be hoping for that ticket arrangement with AA today...


Probably not, at least not yet. If everything is down, they can't rebook or issue a FIM.
A300/A310/A319/A320/A321/A332/A333 / 707/712/727/732/733/734/735/738/739/752/753
/762/763/764/772/788/789/DC8/DC9-10/30/40/50/MD81/83/87/88/90/L1011-/250/500/CRJ200/440 /700/900/EMB135/140/145/170/175/190/328Jet/F70/SF3/BE1/J31
 
b747400erf
Posts: 3172
Joined: Wed Jun 19, 2013 4:33 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:43 am

AWACSooner wrote:
WN and DL within a few weeks time? Maybe it's just me, but I highly doubt this is purely coincidence.

That makes no sense.
 
User avatar
readytotaxi
Posts: 7534
Joined: Mon Dec 11, 2006 2:09 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:43 am

dirk88 wrote:
Seems like some flights are departing now?

https://www.flightradar24.com/DAL179/a9adf7b

Just did a quick sweep of the Atlantic, there is only one other flight in the air, DL83 from CDG which left at 0900.
The four delayed flights at LHR T4, 2 are now boarding and the others show gate open.
Chin up people.
you don't get a second chance to make a first impression!
Growing older, but not up.
 
Curiousflyer
Posts: 587
Joined: Sun Oct 22, 2006 3:19 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:46 am

I was looking forward to an empty flight tomorrow... Now it will be full with the people who were stranded today.

Is the situation getting better?
 
User avatar
Polot
Posts: 10910
Joined: Thu Jul 28, 2011 3:01 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:46 am

Priority is to probably get those intercontinental flights out asap, to minimize issues for the returns/ next international flights they need to take. Harder to hit the reset button on the long haul network vs domestic.
 
UAL777UK
Posts: 2369
Joined: Sun Nov 20, 2005 1:16 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 11:48 am

readytotaxi wrote:
At Heathrow T4 they are handing out bottles of water, people waiting over two half hours, they have been warned that some flights will be cancelled.
Virgin Atlantic has asked customers, or be it a bit late, to check-in extra early as they desk share with Delta.
Wouldn't want to be in that Terminal today. :o


They hardly desk share. They are in two different terminals!. VS is in T3 and DL in T4.

The biggest issue will be getting passenger from T4 over to T3 in central if they want to go on VS, which is a pain at the best of times!
 
downdata
Posts: 589
Joined: Wed Feb 17, 2016 2:38 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:04 pm

Holy shit, another? its like 911 everyday in the united states of south north america.
 
User avatar
moo
Posts: 5005
Joined: Sun May 13, 2007 2:27 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:18 pm

Apparently Delta are blaming this on a power outage in Atlanta...

Sorry Delta, the blame lies with your IT team, because they failed. Power outages shouldnt take down your service, why wasnt this tested for?

Why can Netflix get High Availability and service redundancy right and airlines cannot? Netflix even have a tool which randomly kills servers, data centers and availability regions (sets of data centers) in production, and treats any interruption of service that results as a bug which needs to be fixed ASAP. The goal is such that the world would have to end before Netflix stops working. Its a sad state of affairs

Someone pass this link on to Delta - https://github.com/Netflix/SimianArmy
 
Bongodog1964
Posts: 3542
Joined: Wed Oct 18, 2006 6:29 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:20 pm

UAL777UK wrote:
readytotaxi wrote:
At Heathrow T4 they are handing out bottles of water, people waiting over two half hours, they have been warned that some flights will be cancelled.
Virgin Atlantic has asked customers, or be it a bit late, to check-in extra early as they desk share with Delta.
Wouldn't want to be in that Terminal today. :o


They hardly desk share. They are in two different terminals!. VS is in T3 and DL in T4.

The biggest issue will be getting passenger from T4 over to T3 in central if they want to go on VS, which is a pain at the best of times!


As they code share, the check in queue at both terminals will have a mixture of DL and VS passengers, I guess the DL passengers will not be able to check in for VS flights, so it will take extra time at the desk to handle them. It may well be that the only way they can travel is to purchase a new ticket from VS and claim back from DL when their system is back up.
 
User avatar
DIRECTFLT
Posts: 2335
Joined: Sat Jan 02, 2010 3:00 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:37 pm

moo wrote:
Apparently Delta are blaming this on a power outage in Atlanta...

Sorry Delta, the blame lies with your IT team, because they failed. Power outages shouldn't take down your service, why wasn't this tested for?

Why can Netflix get High Availability and service redundancy right and airlines cannot? Netflix even have a tool which randomly kills servers, data centers and availability regions (sets of data centers) in production, and treats any interruption of service that results as a bug which needs to be fixed ASAP. The goal is such that the world would have to end before Netflix stops working. Its a sad state of affairs

Someone pass this link on to Delta - https://github.com/Netflix/SimianArmy


I don't know if Delta would care to say, but I also wonder if their IT was also pushing through any kind of internal programming "update" overnight, that caused this? But, maybe they really did have a power outage that wasn't planned for, but should have?? Just amazing that they didn't think this through. I guess the FAA will need to come in with some regulations to try and prevent this from happening with other carriers.
Smoothest Ride so far ~ AA A300B4-600R ~~ Favorite Aviation Author ~ Robert J. Serling
 
lavalampluva
Posts: 1433
Joined: Tue Jun 24, 2014 7:33 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:39 pm

moo wrote:
Apparently Delta are blaming this on a power outage in Atlanta...

Sorry Delta, the blame lies with your IT team, because they failed. Power outages shouldnt take down your service, why wasnt this tested for?

Why can Netflix get High Availability and service redundancy right and airlines cannot? Netflix even have a tool which randomly kills servers, data centers and availability regions (sets of data centers) in production, and treats any interruption of service that results as a bug which needs to be fixed ASAP. The goal is such that the world would have to end before Netflix stops working. Its a sad state of affairs

Someone pass this link on to Delta - https://github.com/Netflix/SimianArmy



You would think there would be more redundancy to insure situations like this don't happen. It's going to be a LONG day at DL.
Remind me to send a thank you note to Mr. Boeing.
 
coolian2
Posts: 2483
Joined: Sun Oct 22, 2006 3:34 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:42 pm

It doesn't matter how much redundancy you build in - sometimes it's going to hell and you're along for the ride.
Q300/ATR72-600/737-200/-300/-400/-700/-800/A320/767-200/-300/757-200/777-300ER/
747-200/-300/-400/ER/A340-300/A380-800/MD-83/-88/CRJ-700/-900
 
TheAccountant
Posts: 47
Joined: Wed Oct 14, 2015 10:41 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:44 pm

Seems like things are back up and running but priority is being given to the Intl flights out of Europe and Asia. They are starting to depart en masse right now.

If this is really under control in the next hour or so, it will be very bad, no doubt....but not quite as catastrophic as it could have been
 
DfwRevolution
Posts: 9306
Joined: Sat Jan 09, 2010 7:31 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 12:52 pm

moo wrote:
Apparently Delta are blaming this on a power outage in Atlanta...

Sorry Delta, the blame lies with your IT team, because they failed. Power outages shouldnt take down your service, why wasnt this tested for?

Why can Netflix get High Availability and service redundancy right and airlines cannot? Netflix even have a tool which randomly kills servers, data centers and availability regions (sets of data centers) in production, and treats any interruption of service that results as a bug which needs to be fixed ASAP. The goal is such that the world would have to end before Netflix stops working. Its a sad state of affairs

Someone pass this link on to Delta - https://github.com/Netflix/SimianArmy


Data centers are pretty much the sole and exclusive capital investment for Netflix. Delta has to operate those plus another mission-critical capital asset: airplanes. I am neither trivializing what Netflix does or excusing Delta's system crash, but the two are really not comparable operations.

As I said last week during the WN fiasco, most enterprises that have existed for more than 20 years are a mix-mash of legacy systems, modern systems, and all sorts of custom interconnections. This results in network complexity that makes it extremely challenging to avoid random, unforeseen flukes that are nearly impossible to test and debug to perfection. That's a challenge Netflix hasn't yet faced in their far shorter existence.
I have a three post per topic limit. You're welcome to have the last word.
 
aaexecplat
Posts: 511
Joined: Tue Sep 29, 2009 2:49 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:01 pm

DfwRevolution wrote:
Data centers are pretty much the sole and exclusive capital investment for Netflix. Delta has to operate those plus another mission-critical capital asset: airplanes. I am neither trivializing what Netflix does or excusing Delta's system crash, but the two are really not comparable operations.

As I said last week during the WN fiasco, most enterprises that have existed for more than 20 years are a mix-mash of legacy systems, modern systems, and all sorts of custom interconnections. This results in network complexity that makes it extremely challenging to avoid random, unforeseen flukes that are nearly impossible to test and debug to perfection. That's a challenge Netflix hasn't yet faced in their far shorter existence.


While their systems are undoubtedly super complex, there is ZERO excuse for not having ANY redundancy to prevent full global outage due to a power outage, bad router, or anything localized like that. One can create redundancy either of two ways. Locally or via the cloud. Both options are the cost of doing business these days, especially for companies whose POS (point of sale) is online. That DL would not have that line up is absurd by any standard.
 
User avatar
enilria
Posts: 10310
Joined: Fri Feb 22, 2008 7:15 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:01 pm

coolian2 wrote:
AWACSooner wrote:
WN and DL within a few weeks time? Maybe it's just me, but I highly doubt this is purely coincidence.

I'd bet my house it's a coincidence.
lavalampluva wrote:
moo wrote:
Apparently Delta are blaming this on a power outage in Atlanta...

Sorry Delta, the blame lies with your IT team, because they failed. Power outages shouldnt take down your service, why wasnt this tested for?

Why can Netflix get High Availability and service redundancy right and airlines cannot? Netflix even have a tool which randomly kills servers, data centers and availability regions (sets of data centers) in production, and treats any interruption of service that results as a bug which needs to be fixed ASAP. The goal is such that the world would have to end before Netflix stops working. Its a sad state of affairs

Someone pass this link on to Delta - https://github.com/Netflix/SimianArmy


You would think there would be more redundancy to insure situations like this don't happen. It's going to be a LONG day at DL.
Acey wrote:
OA940 wrote:
I heard on the news it might have been a cyberattack / hack. Any possibility of that?

Of course it's possible, but nothing has been confirmed yet. If that were the case, DL probably wouldn't say till they had it back up and reasonably secure.

So, even US Airways had a cold site in the 1990s. Does Delta not have a cold site? I find that hard to believe. Does Southwest not have a cold site? They should be able to just flip over. A cold site is specifically for things like power or equipment failures.

Second question...Has any airline *EVER* admitted to a hack that took down systems EVER EVER? I think the answer is NO. The systems the airlines use are archaic and have what I have to assume are crap security. I know from talking to airline web site people that they have DDoS attempts routinely against the booking platform which I think happens to pretty much all websites and there is even a visual DDoS security monitoring site where you can see who is being attacked at any given point. None of this has ever succeeded? I find it hard to believe.

I think the reason airlines don't talk about hackers is that unlike J C Penney or Buy.com if the public loses confidence in an airline's IT it could really hurt their business for safety reasons.

This implied there is an unspoken problem...https://www.washingtonpost.com/news/the ... ree-miles/

Just yesterday it was revealed that airline systems are so bad that boarding passes can be easily self-created. http://fortune.com/2016/08/07/hacker-fa ... ng-passes/
 
bmacleod
Posts: 2990
Joined: Sat Aug 25, 2001 3:10 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:11 pm

Does Delta not have a cold site?


Could you define cold site?
"What good are wings without the courage to fly?" - Atticus
 
DfwRevolution
Posts: 9306
Joined: Sat Jan 09, 2010 7:31 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:12 pm

aaexecplat wrote:
DfwRevolution wrote:
Data centers are pretty much the sole and exclusive capital investment for Netflix. Delta has to operate those plus another mission-critical capital asset: airplanes. I am neither trivializing what Netflix does or excusing Delta's system crash, but the two are really not comparable operations.

As I said last week during the WN fiasco, most enterprises that have existed for more than 20 years are a mix-mash of legacy systems, modern systems, and all sorts of custom interconnections. This results in network complexity that makes it extremely challenging to avoid random, unforeseen flukes that are nearly impossible to test and debug to perfection. That's a challenge Netflix hasn't yet faced in their far shorter existence.


While their systems are undoubtedly super complex, there is ZERO excuse for not having ANY redundancy to prevent full global outage due to a power outage, bad router, or anything localized like that. One can create redundancy either of two ways. Locally or via the cloud. Both options are the cost of doing business these days, especially for companies whose POS (point of sale) is online. That DL would not have that line up is absurd by any standard.


Who says Delta has no redundancy in their systems? Redundant back-ups can fail, too. Rarely are back-up systems totally independent, so it's possible for the same software bug to exist on a production system and offsite/cloud back-up.

WN had a redundant back-up to their system, but the failed router kept reporting that it was healthy so operations never switched to the back-up. Life has a way of throwing unforeseen and untested conditions at you.

But agree with Enrilla, strange they don't have a dark site up yet. That is poor planning.
I have a three post per topic limit. You're welcome to have the last word.
 
simairlinenet
Posts: 817
Joined: Sat Oct 29, 2005 2:24 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:13 pm

enilria wrote:
Just yesterday it was revealed that airline systems are so bad that boarding passes can be easily self-created. http://fortune.com/2016/08/07/hacker-fa ... ng-passes/

First time this happened was ten years ago: https://www.wired.com/2006/10/make_your_own_f/
 
lavalampluva
Posts: 1433
Joined: Tue Jun 24, 2014 7:33 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:14 pm

Looks like Delta Connection flights are moving,
Remind me to send a thank you note to Mr. Boeing.
 
bobnwa
Posts: 4514
Joined: Fri Dec 22, 2000 12:10 am

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:15 pm

AWACSooner wrote:
WN and DL within a few weeks time? Maybe it's just me, but I highly doubt this is purely coincidence.

Yep Delta just confirmed its computor system was getting erroneous messages from a grassy knoll in Dallas
 
User avatar
enilria
Posts: 10310
Joined: Fri Feb 22, 2008 7:15 pm

Re: Big Problems at Big Delta

Mon Aug 08, 2016 1:20 pm

simairlinenet wrote:
First time this happened was ten years ago: https://www.wired.com/2006/10/make_your_own_f/
bmacleod wrote:
Does Delta not have a cold site?


Could you define cold site?

So, basically whenever any transaction occurs in the RES system the data splits to two destinations. One to the "Hot Site" and another to the "Cold Site". It's sort of the mega version of a RAID drive. The cold site always has identical data to the hot site. You literally flip a switch and after a few minutes the cold site becomes the hot site and operates basically without interruption. During the night when load is light you recover the broken site from the backup site and flip back once the problems are solved.

https://en.wikipedia.org/wiki/Backup_site
Last edited by enilria on Mon Aug 08, 2016 1:20 pm, edited 1 time in total.

Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos