Moderators: richierich, ua900, PanAm_DC10, hOMSaR

  • 1
  • 4
  • 5
  • 6
  • 7
  • 8
 
GRIVely
Posts: 143
Joined: Wed Dec 13, 2006 5:46 am

RE: Airbus A400 Crashes In Spain

Wed Jun 10, 2015 11:44 pm

Tricky stuff software. One of the biggest challenges in aerospace software, well, all software really, is the difficulty of understanding how the various modules interact with sensors and other modules in a bewildering variety of possible states. With your "desired operation" flow chart in hand you begin assembling your library objects and then linking the elements together. Need a temperature input for a decision tree--reach out for a routine that pulses a thermocouple and returns the temperature, integrate that value into your tree and invoke the rule that somebody wrote for that condition. All these dependencies, codependencies, independent threads, trees, assembly kernels and everything swirl around in a major cloud of brownian uncertainty.

I visited the Philips TV factory in Eindhoven a few years ago and their vice-president for production was going crazy because the software development for their new remote control logic was way over budget and behind time. He said that he couldn't believe 30% of the cost of their new flatscreen TVs was going to be software. It had something like 125,000 lines of code. Think how much more complicated the control, sensor and logic applications are in a modern spacecraft or airplane. Oh, and if some human being didn't think of some obscure, once in ten thousand flight hours, or more, condition that could happen and then got supervisor permission to spend the time and money to successfully write and test the equivalent of a rule for what to do in that exact case your software literally can't respond. Computers don't think their way through a situation. If someone didn't write the rules for that specific case there is no solution from the software.

When I was in spacecraft payload management we met every morning to review the things that were wrong with the systems and decided what steps to take to keep it alive and working longer. Any spacecraft accumulates dozens, or even hundreds of problems over its service life. Major and minor hardware faults, failed components, dead solar panel motors, horizon sensor failures, reaction wheels, pumps, whatever. But fixing most of the problems involve reworking some part of the software load to work around the problems. This is a continuing challenge and a never-ending battle to find solutions. You carefully analyze the problem, painstakingly write a recommended solution, test, rewrite, retest and finally certify the fix. Format the code, upload the new software to the bird. Have a copy sent back down with the health and safety feed to compare the bits to make sure it was received properly. Then you cross your fingers and load the software into the correct register and execute. Most of the time your fix ends up solving that particular problem and you go on to deal with other DRs waiting in the queue.

But all of us in the biz have had the odd occasion where the fix is loaded and we are hovering over the payload management terminal watching the telemetry as the new software is integrated into mission Magement modules and then a whole cascade of errors floods down. If you are lucky, it wasn't mission critical and you can try again. Sometimes it is a critical element and you just trashed an X-Million spacecraft and cost the operator, in the of a Comsat, hundreds of millions in revenue, plus the replacement spacecraft build and launch costs. As the mission director you get to meet a lot of new friends who want to know why you and your team thought that would work and what the hell were you thinking?

I had an error, painful but not loss of mission, that the post failure analysis suggested was caused by a write error induced by a stray cosmic ray. Others have eerily like the supposed A400 problem. Failure to clear a register or write the correct fill. Human beings and cosmic rays are so unreliable.

Why this lengthy screed? Software, at our present state of tools, understanding and ability sucks. Despite our very best efforts errors will occur and lives, airplanes and spacecraft will continue to be lost to those software errors. Pilots are better trained, engines are more reliable (except for the software portion) and everything else improved as well but there will still be those randomly-generated software induced losses.

What's the biggest difference between spacecraft and airplanes? I have never heard of a near-instantaneous, on-orbit loss of a spacecraft. Thanks to Dr. Newton, while they may be broken, spacecraft tend to stay in orbit and we have at least hours, and usually days or even months, to troubleshoot and recover from software problems. Not so with aircraft.
 
hivue
Posts: 2078
Joined: Tue Feb 26, 2013 2:26 am

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 12:37 am

Quoting Okie (Reply 352):
Here is a little more insight.

From the computerworld article:
"The $22.5 billion plane has four engines and each engine is run by a separate computer, called an Electronic Control Unit (ECU)."

I knew the A400 was expensive, but jeez...

Zeke, cut me some slack for quoting from the article, but I just couldn't resist.  
Quoting B8887 (Reply 348):
a theory that files known as "torque calibration parameters" had been accidentally deleted during a software installation process ahead of the plane's first flight."

If this is actually true, I wonder why #4 was apparently unaffected?

[Edited 2015-06-10 17:44:28]
"You're sitting. In a chair. In the SKY!!" ~ Louis C.K.
 
chuchoteur
Posts: 610
Joined: Tue Sep 12, 2006 9:17 pm

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 8:02 am

Quoting hivue (Reply 354):
If this is actually true, I wonder why #4 was apparently unaffected?

From what I understand, #4 ECU had an installation issue and was replaced pre-flight with an ECU that had a prior software load on it.
 
User avatar
moo
Posts: 4980
Joined: Sun May 13, 2007 2:27 am

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 8:07 am

Quoting nomadd22 (Reply 349):
It's a little hard to believe that the system could appear to boot normally with vital files missing.

Being a software developer, its not hard to believe at all - it probably resets back to defaults which aren't calibrated.
 
ZaphodHarkonnen
Posts: 1038
Joined: Sun Jan 04, 2015 10:20 am

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 11:33 am

Yup, as another dev this is very plausible. If you don't do it right or the installer is not right you could easily wipe that bit of memory.

However I would have expected them to make sure such areas of storage do not change between updates or only in expected ways.

Software is hard.
 
User avatar
Revelation
Posts: 24584
Joined: Wed Feb 09, 2005 9:37 pm

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 12:44 pm

Quoting B8887 (Reply 348):
http://www.bbc.com/news/technology-33078767
Quoting Okie (Reply 352):
http://www.computerworld.com/article...ed-to-fatal-a400m-plane-crash.html

Thanks for posting these. They corroborate some earlier suggestions above that the issue could have been with configuration parameters (i.e. data) instead of the software's logic (i.e. code). Both articles cite the Reuters article at http://www.reuters.com/article/2015/...irbus-a400m-idUSKBN0OP2AS20150609. One part I found interesting:

Quote:

Without the vital data parameters, information from the engines is effectively meaningless to the computers controlling them. The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage.

This is what the computers apparently did on the doomed flight, just as they were designed to do.

"Nobody imagined a problem like this could happen to three engines," a person familiar with the 12-year-old project said.

So very unfortunately it's the dreaded "common mode failure" which ended up defeating built-in redundancy. The article is far from clear about where the "automatic response" was being generated but it seems that is an additional area of concern.

Quoting GRIVely (Reply 353):
I visited the Philips TV factory in Eindhoven a few years ago and their vice-president for production was going crazy because the software development for their new remote control logic was way over budget and behind time. He said that he couldn't believe 30% of the cost of their new flatscreen TVs was going to be software. It had something like 125,000 lines of code. Think how much more complicated the control, sensor and logic applications are in a modern spacecraft or airplane. Oh, and if some human being didn't think of some obscure, once in ten thousand flight hours, or more, condition that could happen and then got supervisor permission to spend the time and money to successfully write and test the equivalent of a rule for what to do in that exact case your software literally can't respond. Computers don't think their way through a situation. If someone didn't write the rules for that specific case there is no solution from the software.

And that VP's most likely solution will be to send that software to an off-shore location where he can hire three engineers for the price of one in the Netherlands because three must be better than one, and those newbies won't have any idea of any of the "obscure, one in ten thousand hours" conditions lurking in the code. In short, there is some value in maturity.

In this case, Airbus chose to go with new technology over old technology. This is explained in the Wikipedia article at: http://en.wikipedia.org/wiki/Airbus_A400M_Atlas .

Quote:

Originally the SNECMA M138 turboprop (based on the M88 core) was selected to power the A400M. Airbus Military issued a new request for proposal (RFP) in April 2002, after which Pratt & Whitney Canada with the PW180 and Europrop International answered; the latter was a new design. In May 2003, Airbus Military selected the Europrop TP400-D6, reportedly due to political interference over the PW180 engine.

We have no way of knowing whether the older, more mature options didn't have similar issues lurking within, but I think it's fair to say that even before this incident the engine software has been a major budget and schedule issue for A400M.

Quoting GRIVely (Reply 353):
Most of the time your fix ends up solving that particular problem and you go on to deal with other DRs waiting in the queue.

A typical response to this situation would be to order a major review of all outstanding issues in all the queues to see if any others could cause similar problems. That could be very costly for the A400M program. It seems obvious to me that one reason there was such haste to keep the test frames flying was to avoid a program stand-down. It should be interesting to see if such a stand-down is avoided or not.
Wake up to find out that you are the eyes of the world
The heart has its beaches, its homeland and thoughts of its own
Wake now, discover that you are the song that the morning brings
The heart has its seasons, its evenings and songs of its own
 
hivue
Posts: 2078
Joined: Tue Feb 26, 2013 2:26 am

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 2:58 pm

Quoting chuchoteur (Reply 355):
From what I understand, #4 ECU had an installation issue and was replaced pre-flight with an ECU that had a prior software load on it.

If only 2 engines had had installation issues...

If this is correct it means the putative software installation issue occurred prior to the physical installation of the ECUs on the engines. One would think (with 20/20 hindsight) that they would take an "ETOPS-type" approach and have one group prep 2 of the ECUs and a totally independent group prep the other 2. Or else apply the newer software to only 2 of the ECUs on the airplane.
"You're sitting. In a chair. In the SKY!!" ~ Louis C.K.
 
User avatar
7BOEING7
Posts: 3039
Joined: Fri Oct 12, 2012 5:28 pm

RE: Airbus A400 Crashes In Spain

Thu Jun 11, 2015 6:59 pm

In my experience in production flight test, any time we had new "engine" software that had never flown before, it was only installed on one engine. After the flight was completed successfully as far as engine checks were concerned it was then installed on the other engine(s) (if required).
 
User avatar
Revelation
Posts: 24584
Joined: Wed Feb 09, 2005 9:37 pm

RE: Airbus A400 Crashes In Spain

Mon Jun 15, 2015 12:50 pm

It's kind of odd to read how A400Ms are flying at PAS while the Spanish military has yet to approve flights of the new production airframes:

Quote:

Airbus test pilots Ignacio “Nacho” Lombo and Tony Flynn are putting the A400M through its full display paces this week, as the company continues to wait on a Spanish military decision on whether to allow it resume flight testing the type for its customers.

ref: http://www.flightglobal.com/news/art...to-resume-a400m-deliveries-413509/

While the ban is still in effect there are five UK and one FR airframes ready to resume flights. The delays mean that the payments upon delivery that Airbus expected to get for these frames are not happening, which is adding stress to the program. I would expect the ban to be lifted soon, but the article is not giving any dates.
Wake up to find out that you are the eyes of the world
The heart has its beaches, its homeland and thoughts of its own
Wake now, discover that you are the song that the morning brings
The heart has its seasons, its evenings and songs of its own
 
User avatar
KarelXWB
Posts: 26968
Joined: Sun Jul 15, 2012 6:13 pm

RE: Airbus A400 Crashes In Spain

Tue Jun 16, 2015 11:36 am

One of the RAF A400M's was airborne yesterday.
What we leave behind is not as important as how we've lived.
 
Okie
Posts: 4156
Joined: Wed Jul 16, 2003 11:30 am

RE: Airbus A400 Crashes In Spain

Tue Jun 16, 2015 2:36 pm

Quoting Revelation (Reply 358):
While the ban is still in effect there are five UK and one FR airframes ready to resume flights. The delays mean that the payments upon delivery that Airbus expected to get for these frames are not happening, which is adding stress to the program. I would expect the ban to be lifted soon, but the article is not giving any dates

According to Janes back in April, Airbus was going to ramp up production to 2.5 frames per month by the end of 2015.

I have no doubt Airbus has identified the problem.
The real question will be Airbus convincing the authorities that there are proper quality assurances and safeguards in place after their best efforts lead to this point.

Okie
 
redflyer
Posts: 3905
Joined: Thu Feb 24, 2005 3:30 am

RE: Airbus A400 Crashes In Spain

Tue Jun 16, 2015 5:29 pm

Quoting Okie (Reply 360):
The real question will be Airbus convincing the authorities that there are proper quality assurances and safeguards in place after their best efforts lead to this point.

After this relatively "dumb" mistake which wiped critical code from the software load, I would say Airbus needs to convince themselves that there are proper quality assurances and safeguards in place from one end of the production line to the other. They've been operating under a lot of pressure for years to resolve production and cost issues. Who knows where else shortcuts have been made for the sake of expediency? And this isn't a jab at Airbus; it's a jab at human nature.
A government big enough to take away a constitutionally guaranteed right is a government big enough to take away any guaranteed right. A government big enough to give you everything you need is a government big enough to take away everything you have.
 
JJJ
Posts: 3751
Joined: Wed May 31, 2006 5:12 pm

RE: Airbus A400 Crashes In Spain

Wed Jun 17, 2015 8:48 am

Quoting Revelation (Reply 358):
It's kind of odd to read how A400Ms are flying at PAS while the Spanish military has yet to approve flights of the new production airframes

They're flying for a few days already. Plenty of pics on twitter and the usual sites.

Official link from Spanish MoD

http://www.defensa.gob.es/gabinete/n...GC-150611-INTA-AIRBUS-REUNION.html

[Edited 2015-06-17 02:15:45]
 
User avatar
Revelation
Posts: 24584
Joined: Wed Feb 09, 2005 9:37 pm

RE: Airbus A400 Crashes In Spain

Wed Jun 17, 2015 12:25 pm

Quoting jjj (Reply 362):
They're flying for a few days already.

Also the UK released the planes for operation yesterday:

http://www.flightglobal.com/news/art...training-flight-resumption-413705/
Wake up to find out that you are the eyes of the world
The heart has its beaches, its homeland and thoughts of its own
Wake now, discover that you are the song that the morning brings
The heart has its seasons, its evenings and songs of its own
 
User avatar
KarelXWB
Posts: 26968
Joined: Sun Jul 15, 2012 6:13 pm

RE: Airbus A400 Crashes In Spain

Thu Jun 18, 2015 12:52 pm

Deliveries are restarting as well, France will receive its next A400M in the coming days:

http://www.flightglobal.com/news/art...next-a400m-39in-the-coming-413821/
What we leave behind is not as important as how we've lived.
 
User avatar
Grizzly410
Posts: 437
Joined: Sun May 10, 2015 8:38 pm

RE: Airbus A400 Crashes In Spain

Fri Jun 19, 2015 10:47 am

Two good news for the program in one week :

Fly permits could be granted in the coming days !!
http://www.reuters.com/article/2015/...rbus-flights-idUSKBN0OR2H920150611

And since this morning MSN19 can be seen on the tarmac after a few days in the paint shop, now with its customer registration, F-RBAG
Suggesting the customer will take his aircraft home soon !   
In order to be old and wise, one must first be young and dumb.
 
User avatar
Grizzly410
Posts: 437
Joined: Sun May 10, 2015 8:38 pm

RE: Airbus A400 Crashes In Spain

Fri Jun 19, 2015 1:19 pm

Can't edit previous post, or just don't find how   

Restrictions are now lifted !!!

"Airbus Defence and Space is about to recommence deliveries of Airbus A400M aircraft following the lifting of all remaining flight restrictions on new production aircraft by Spanish regulator DGAM yesterday."

http://airbusdefenceandspace.com/new...ce-set-to-resume-a400m-deliveries/
In order to be old and wise, one must first be young and dumb.
 
User avatar
Revelation
Posts: 24584
Joined: Wed Feb 09, 2005 9:37 pm

RE: Airbus A400 Crashes In Spain

Fri Jun 19, 2015 1:43 pm

Quoting Grizzly410 (Reply 365):
Two good news for the program in one week

Glad to see the program is progressing.
Wake up to find out that you are the eyes of the world
The heart has its beaches, its homeland and thoughts of its own
Wake now, discover that you are the song that the morning brings
The heart has its seasons, its evenings and songs of its own
 
User avatar
KarelXWB
Posts: 26968
Joined: Sun Jul 15, 2012 6:13 pm

RE: Airbus A400 Crashes In Spain

Mon Jun 22, 2015 10:27 am

MSN19 has been delivered to the French Air Force.

http://twitter.com/ja_almarza/status/612600739010113536

Well, that's faster than expected.
What we leave behind is not as important as how we've lived.
 
spacecookie
Posts: 213
Joined: Sun Jun 07, 2015 3:57 pm

RE: Airbus A400 Crashes In Spain

Thu Jun 25, 2015 8:51 pm

hello to all, this is my opinion about this crash, i am not a huge military plane fan at all.

i really think airbus made some mistakes here, and also europrop.
in europe are other manufactures who makes plane -turboprop- engines but they decided to go the europrop way, new manufacture to turboprop engine, and they had some trouble with the software from the first days,
the plane was manufactured-finished and on the ground because of the this, and they had a huge delay and all the money wasted.
  • 1
  • 4
  • 5
  • 6
  • 7
  • 8

Who is online

Users browsing this forum: No registered users and 20 guests

Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos