jgch
Posts: 18
Joined: Sun Nov 27, 2016 5:32 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 9:18 pm

moo wrote:
The ironic thing is that in this case the outsourced IT staff may be trying desperately to get onsite but they cant, for some reason, get a flight on the customer preferred airline...


Probably with Air India. The next question is: do they have a visa?
 
User avatar
11725Flyer
Posts: 1281
Joined: Mon May 30, 2016 4:51 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 9:24 pm

I just checked their site here in the U.S. and the flight status feature is now working. Hopefully, a positive sign.
 
redroo
Posts: 517
Joined: Tue Jun 07, 2016 10:28 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 9:43 pm

I used to work for a big organisation that refused to outsource its IT because it was seen as a core business function. We had two of every application on different servers in different buildings at least 30 miles apart. All the applications were designed to fail over onto the backup within a minute or so. This whole setup was regularly tested using multiple scenarios, every couple of months... e.g. could you flip onto backup, stay on it, then flip back to production; what would happen if the network in primary location died; what would happen if primary was blown up by a terrorist. It cost a lot of money for this setup, but we knew it would cost a ton more if we couldn't operate for even a minute. There is no excuse for a system outage like this.
 
Chemist
Posts: 665
Joined: Tue Oct 20, 2015 4:46 am

Re: BA experiencing computer problems.

Sat May 27, 2017 9:51 pm

I have worked in IT and done strategy around disaster recovery. I can tell you that many companies trying to save money do things like outsourcing. I've also seen year after year IT management requests budget for things that are important but not urgent, and year after year management who wants to live within their tight budgets to maintain earnings per share deny the requested funding. And then I've seen at least one case (not of the scale here though) where some VP realized they didn't have something and demanded it from IT and we scrambled. And that was something we had asked for year after year but never made the budget cut. It was only important when non-IT management decided THEY thought it was important.
 
jgch
Posts: 18
Joined: Sun Nov 27, 2016 5:32 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 9:56 pm

11725Flyer wrote:
I just checked their site here in the U.S. and the flight status feature is now working. Hopefully, a positive sign.


In Europe not. Still the emergency banner and NO news on twitter.
 
PITflights
Posts: 35
Joined: Sun Aug 14, 2016 8:33 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 10:09 pm

Looks like there are flights in the air on their way to LHR
 
Chelsearose
Posts: 12
Joined: Wed Feb 09, 2011 8:57 am

Re: BA experiencing computer problems.

Sat May 27, 2017 10:40 pm

This thread was interesting to read.

So, I've just seen some flights on BA .. mainly to the US and Southern Africa taking off. BA has mentioned ALL flights have been cancelled for the rest of the day, and indeed in their operational info these flights appear as cancelled. Why are those flights operating? Are they ferrying crew essential for tomorrow's flights?
 
User avatar
Blimpie
Posts: 303
Joined: Fri Mar 21, 2014 3:48 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 10:47 pm

smokeybandit wrote:
Do they run their website off Geocities or something?


You sir win the internet. That was the first time I actually laughed out loud in quite some time.
Now get the hell off of my lawn your dang kids!
 
LHRFlyer
Posts: 1027
Joined: Sun Apr 11, 2010 12:50 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 11:07 pm

Chelsearose wrote:
This thread was interesting to read.

So, I've just seen some flights on BA .. mainly to the US and Southern Africa taking off. BA has mentioned ALL flights have been cancelled for the rest of the day, and indeed in their operational info these flights appear as cancelled. Why are those flights operating? Are they ferrying crew essential for tomorrow's flights?


I think some long-haul flights have left LHR without passengers, partly to relieve congestion at LHR and to minimise ongoing disruption tomorrow.
 
concordeforever
Posts: 77
Joined: Thu Sep 22, 2016 6:51 pm

Re: BA experiencing computer problems.

Sat May 27, 2017 11:22 pm

Some flights were due to leave this evening as positioners to try to get some people back in the morning. All went out with no passengers but some took some cargo as this is now backlogged just as much. The next few days are going to be manic!
 
G-CIVP
Posts: 1561
Joined: Tue Mar 20, 2001 6:38 am

Re: BA experiencing computer problems.

Sat May 27, 2017 11:24 pm

BobPatterson wrote:
G-CIVP wrote:
"Apparently the data server got struck by lightning".

Like they really keep their servers in the open air!


Lightning hits cause power surges that can disable equipment attached to power lines of any type.

Did you have anything else to contribute to this thread or are nonsensical posts your specialty?

As it turns out, there is a great deal to be seen here.


Often server rooms have equipement that prevents power surges and regulates the current. Depends where the servers are housed, I would be surprised that a power surge has caused the problem.

Obviously, you have a knowledge gap, so it may be best to do some reading up on the subject matter.
 
User avatar
77west
Posts: 950
Joined: Sat Jun 13, 2009 11:52 am

Re: BA experiencing computer problems.

Sat May 27, 2017 11:35 pm

I often deal with servers used in small to medium business and we always make sure they have dual power supplies with double surge protection, IE, there is both passive and active surge protection in addition to line-interactive battery backup. The batteries cut in if supply voltage goes above or below 5% of rated. This disconnects the equipment entirely from the grid, and happens in milliseconds. In a datacenter this goes to a whole new level.

I am surprised a BA datacenter could be affected this badly, most datacenters even have independent power feeds from different substations in case one circuit goes down.

I wonder if there has been a failure of a backup circuit as well, but then, what about replication to an alternate datacenter? Surely large companies would be able to survive a single datacenter going offline.
77West - AW109S - BE90 - JS31 - B1900 - Q300 - ATR72 - DC9-30 - MD80 - B733 - A320 - B738 - A300-B4 - B773 - B77W
 
gunnerman
Posts: 918
Joined: Fri May 19, 2017 7:55 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 12:32 am

speedbored wrote:
Statement from Alex Cruz on BAs twitter feed suggests that it's a power failure.

Frankly, I'm flabbergasted that BA don't (a) have backup power supplies/generators, (b) don't seem to have any sort of backup/disaster-recovery server center and (c) don't seem to have any alternative "manual" plan for computer systems being unavailable. It should be standard practice for any sizeable business that relies heavily on computer systems.


The main data centre is Boudicea House near Heathrow Airport, with an unmanned backup data centre in nearby Cranebank. It remains to be seen how a power outage could have had such a serious sytems failure. Whatever the problem is, it's not reservations as you can make bookings, so the relevant Java platform, Oracle servers, links to Erding and the Amadeus system are all fine.
 
AngMoh
Posts: 982
Joined: Fri Nov 04, 2011 5:03 am

Re: BA experiencing computer problems.

Sun May 28, 2017 12:49 am

Just wondering one thing: supposedly the outage was global. But BA flights from SIN (BA15 to SYD, BA12 to LHR) and HKG (BA28 and BA32 to LHR) left on time even though the BA UK was still down. BA16 left SYD on time when the mess at BA was at its maximum but then became delayed by 6 hours in SIN even though the departing slot is close to that of BA12.
Were these flights on time because all info had been loaded beforehand (due to security requirements) while BA16 SIN-LHR was delayed as it needed to wait for final passenger list from the SYD-SIN leg and thus was caught up in the mess?
727 732 733 734 735 73G 738 739 739ER 742 743 744 752 753 762 772 77E 773 77W 788 A300 A310 A319 A320 A321 A332 A333 A343 A345 A346 A359 A35K A388 DC-9 DC-10 MD11 MD81 MD82 MD87 F70 ERJ145 E170 E175 E190 E195 ATR72 Q400 CRJ200 CRJ700 CRJ900 BAE146 RJ85
 
ashwinr
Posts: 1
Joined: Wed Aug 22, 2012 2:41 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:49 am

The knock on affect is unbelievable!
 
User avatar
Airbus747
Posts: 151
Joined: Thu Mar 13, 2014 12:18 am

Re: BA experiencing computer problems.

Sun May 28, 2017 2:23 am

This makes me wonder... what if all computers go down at one of those new "virtual ATCs" like the one at LCY ?
 
anshabhi
Posts: 2141
Joined: Thu Oct 20, 2016 10:40 am

Re: BA experiencing computer problems.

Sun May 28, 2017 4:24 am

BA flights to Delhi & Mumbai were not affected. There was some effect at Bangalore, Chennai & Hyderabad however.

Outsourcing should be complete, and not fragmented. I am quite confident, even if the software was written in US, many Indians would have been involved. Move high quality jobs here!
Last edited by anshabhi on Sun May 28, 2017 4:43 am, edited 1 time in total.
 
manny
Posts: 563
Joined: Sat Sep 09, 2006 8:59 am

Re: BA experiencing computer problems.

Sun May 28, 2017 5:02 am

redroo wrote:
We had two of every application on different servers in different buildings at least 30 miles apart. All the applications were designed to fail over onto the backup within a minute or so. This whole setup was regularly tested using multiple scenarios, every couple of months... e.g. could you flip onto backup, stay on it, then flip back to production; what would happen if the network in primary location died; what would happen if primary was blown up by a terrorist. It cost a lot of money for this setup, but we knew it would cost a ton more if we couldn't operate for even a minute. There is no excuse for a system outage like this.


Its very poor decision making and planning to have your backup facility only 30 miles away. They need to be further apart. Like hundreds if not thousands of miles away because in case of a major storm or some unforeseen incident both centers might not be available.
 
manny
Posts: 563
Joined: Sat Sep 09, 2006 8:59 am

Re: BA experiencing computer problems.

Sun May 28, 2017 5:07 am

If it indeed is a backup power systems not firing up as expected than its not an IT issue but a facilities management/disaster recovery issue. Most of the time companies retain control of their own facilities and disaster recovery.

The outsourcing usually is only for the software component and/or IT hardware maintenance.
 
smi0006
Posts: 2435
Joined: Wed Jan 16, 2008 7:45 am

Re: BA experiencing computer problems.

Sun May 28, 2017 5:24 am

I feel so sorry for the front line staff of BA- what do you tell angry customers when you have no information yourself?

Manual Checkin is all good and well - if immigration authorities give you approval. In my experience they are generally no longer willing to do this. Last week there was a SITA outage that impacted Australian airports. Certain airlines got approval, others didn't, and AU would give approval but the destination country wouldn't and in reverse. What can airlines do if approval isn't given?
 
LJ
Posts: 4945
Joined: Wed Nov 17, 1999 8:28 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 5:42 am

77west wrote:
I wonder if there has been a failure of a backup circuit as well, but then, what about replication to an alternate datacenter? Surely large companies would be able to survive a single datacenter going offline.


That's why you have to test your back-up plans regulary. The company I work for discovered that the generator didn't work and that the back-up faciltiies (we have two back-up faciltiies) didn't perform as planned a few years ago. After a large investment we now test all back-up systems and facilities and our power generator regulary. Why BA didn't have such a policy surprises me. One would expect from a company whose operations rely heavily on IT that they have a scenario (ensure you have a hard copy instead your computers/Ipad doesn't work) readily available. One thing for sure, BA's IT management will have some explaining to do to the IAG Board of Directors.

I personally think we take 24/7 power and internet too much for granted.
 
TC957
Posts: 3611
Joined: Wed May 23, 2012 1:12 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 7:32 am

Think it might be the other way round, like IAG board of directors explaining to IT management why they are so penny-wise, pound-foolish by not investing in proper updated back-up systems that I'm sure IT management have asked for.
 
Egerton
Posts: 864
Joined: Sun Jan 11, 2015 9:50 am

Re: BA experiencing computer problems.

Sun May 28, 2017 7:42 am

In early 2011 our firm had an IT outage.

We had from 1977 an IBM mid-range computer, a small mainframe later with mirrored discs. PC’s did not exist then. This had battery powered uninterruptable power supply (UPS), in initially a room full of Leclanche Cells, later a smaller box of tricks. There was also a stand-by 150kVa diesel powered generating set which took 2 minutes to automatically power up and chop in, to rest the UPS.

The UK National Grid delivered high voltage to us which we transformed to 440v/230v to supply the whole site with 500kva. In the early days, the grid dropped out about twice a year, improving when the local grid supply changed from being a single line to a loop. When the grid dropped out (unplanned) usually for a few only a few minutes, and exceptionally for half a day.

The wiring to the IBM box was a via a 60 amp supply from our incomer. Separately our various rings supplied to sockets for fused 13 plugs. We has a separate red ring via the battery UPS, for mission critical 13 amp plugged items, including comms to remote depots and some screens. These sockets were red, to avoid people plugging in items with heavy loads which would exceed the UPS capacity, electric fires for instance.

This all worked like a dream, until early 2011. An external contractor working with us had been issued with a brand new PC which he correctly had plugged into a white socket. This non-IBM PC failed in an exciting ‘flash bang’ manner. The chap was not hurt at all, but he was very surprised, not to say shocked, but not electrically. Those in the open plan office likewise.

The IBM box, which by then had evolved from an IBM System 38 to a powerful thing running c 150 screens over some 15 locations, came to an immediate halt, and was down for 6 hours, after which it was re-booted etc. Apparently, some sort of electrical shock wave (I am not an electrical engineer) had emanated from the flash bang of the new PC which was probably a Lithium Ion battery failure. This shock wave had apparently passed through the UPS which was unharmed, and knocked out one of the two independent power supplies within the IBM box.

We had not allow for, nor was aware of, such a failure mode. Technology can be a mysterious thing.
 
User avatar
cv990Coronado
Posts: 351
Joined: Thu Nov 08, 2007 4:38 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 8:38 am

TC957 wrote:
Think it might be the other way round, like IAG board of directors explaining to IT management why they are so penny-wise, pound-foolish by not investing in proper updated back-up systems that I'm sure IT management have asked for.


I think you are spot on there, but will it happen? To my knowledge, Turkey's don't vote for Christmas or Thanksgiving. I'm sure our little Irish friend and his colleagues will find a suitable scapegoat.
SSC-707B727 737-741234SP757/762/3/772/WA300/10/319/2/1-342/3/6-880-DAM-VC10 TRD 111 Ju52-DC8/9/10/11-YS11-748-VCV DH4B L
 
GDB
Posts: 13273
Joined: Wed May 23, 2001 6:25 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 9:56 am

TC957 wrote:
Think it might be the other way round, like IAG board of directors explaining to IT management why they are so penny-wise, pound-foolish by not investing in proper updated back-up systems that I'm sure IT management have asked for.


While I am not involved with this IT screw up, within Engineering there is a project costing several million. It's late, likely over budget, is creating problems before it's fully implemented (tests have not gone well so far).
At the start they had a team of long term, well known to us, IT contractors at LHR, alongside BA IT specialists. All were made redundant and outsourced, right in the middle of this project.

But hey, the person who did this got his/her/it's annual bonus for cutting costs and is no doubt either within BA just waiting for a chance to screw up again or has left. However they would also have done this on the orders of Alex 'Manuel' Cruz. A man who thinks you can run a full service, intercontential airline just he he did with Vueling - which did not turn out so well either.

And this is far from the first time this sort of thing has happened after an IT outsourcing.

Many staff will have been forced in to work on their time off to clear this mess up.
They had their IT cost savings, wonder how much this weekend has cost BA, in both money and pax goodwill - how many of them will book with BA again?
 
User avatar
cv990Coronado
Posts: 351
Joined: Thu Nov 08, 2007 4:38 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 10:16 am

@GDB Sounds like you are on the inside. This is what I imagined had happened, sadly I am not in the least bit surprised.
Someone mentioned a backup centre 30 miles away, that might just might, be suitable for a small company but IAG?
I would think they need one somewhere such as the Scotish Highlands with hydro, and wind powers sources. The might have even managed to get the EU or some rural development body to help with costs or tax breaks.
SSC-707B727 737-741234SP757/762/3/772/WA300/10/319/2/1-342/3/6-880-DAM-VC10 TRD 111 Ju52-DC8/9/10/11-YS11-748-VCV DH4B L
 
GDB
Posts: 13273
Joined: Wed May 23, 2001 6:25 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 10:45 am

cv990Coronado wrote:
@GDB Sounds like you are on the inside. This is what I imagined had happened, sadly I am not in the least bit surprised.
Someone mentioned a backup centre 30 miles away, that might just might, be suitable for a small company but IAG?
I would think they need one somewhere such as the Scotish Highlands with hydro, and wind powers sources. The might have even managed to get the EU or some rural development body to help with costs or tax breaks.


I honestly don't know the details of this weekend, I am not due back in until Wednesday, was last there Friday.
But there was a major IT outsourcing in the company beyond the project I mentioned.

It's terrible for pax and that is of course the most important thing but staff morale will be shredded too. I mean what's the point in doing the best you can in a customer facing role when a made in Waterside screw up renders those efforts useless?
 
jgch
Posts: 18
Joined: Sun Nov 27, 2016 5:32 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 10:48 am

smi0006 wrote:
I feel so sorry for the front line staff of BA- what do you tell angry customers when you have no information yourself?


These are the really poor guys in this game. But it probably highlights the real weakness of this company.
The IT melt down is one story. Even worse is the communication (or better the non-communication). In this particular situation BA, they missed the opportunity to impress their clients. Their clients felt left alone. There were no refreshments, no BA person visible. BA should have brought in all available ground staff to the airports to help.

All the measures could have been planned upfront and documented. Every member of the staff must know what to do in case of an emergency. Apparently, this was not the case at BA. IAG/BA management failed on multiple levels not only IT.
 
jgch
Posts: 18
Joined: Sun Nov 27, 2016 5:32 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 11:01 am

I just found a good article on Lord Marshall: https://www.independent.co.uk/news/obit ... 20614.html

Quote:
"The customer doesn't expect everything will go right all of the time – the big test is what you do when things go wrong".
 
G-CIVP
Posts: 1561
Joined: Tue Mar 20, 2001 6:38 am

Re: BA experiencing computer problems.

Sun May 28, 2017 11:24 am

GDB wrote:
They had their IT cost savings, wonder how much this weekend has cost BA, in both money and pax goodwill - how many of them will book with BA again?


I will confess as dedicated fan of BA, I going to start using other airlines. I don't like the general attitude and culture towards passengers, e.g. charge you a full fare but don't even give you a complimentary tea or coffee on board.

Anyway, on the brightside, look forward to the seat sale!
 
LJ
Posts: 4945
Joined: Wed Nov 17, 1999 8:28 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 11:52 am

TC957 wrote:
Think it might be the other way round, like IAG board of directors explaining to IT management why they are so penny-wise, pound-foolish by not investing in proper updated back-up systems that I'm sure IT management have asked for.


Since when does management listen to IT managers until something has gone wrong and takes the blames for srew ups (unless forced by shareholders or public opinion)? Wouldn't be surpised if Board is asking the IT manager and his reply would be something like "Well we did warn you but you still continued the cos cutting". Or probably that won't be said by an IT manager as he fears its job and doesn't know anything at all about IT.
 
User avatar
par13del
Posts: 9529
Joined: Sun Dec 18, 2005 9:14 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 12:07 pm

Bottom line is that they saved money by outsourcing, that is done and placed on the financial ledger.
Now the failure and the subsequent loss of funds is on the IT Staff and whatever contractors who gave assurances that they could handle it and obviously did not....
Its simple on the outside looking in, unfortunately, the actual CYA papers give a different story.

I see BA253 is enroute left 30mins late, hopefully all pax and bags made the flight.
 
User avatar
Aesma
Posts: 12138
Joined: Sat Nov 14, 2009 6:14 am

Re: BA experiencing computer problems.

Sun May 28, 2017 12:57 pm

oOfredOo wrote:
PatrickZ80 wrote:
musapapaya wrote:
I wonder are they able to rebook passengers onto other airlines? Because this procedure relies on the IT system too?


Problem is that they can't verify the bookings. Anyone can show up at their desk and claim they have a booking to a certain destination. BA can't check if that's true or not, so before you know it they'd be handing out free flights.

Better wait until the system is up again and then see how they're going to get everyone to their destinations.


So what? It's better than paying 600 euro compensation to passengers, and have a total meltdown of traffic. I've flown Transavia Oktober last year and they boarded everybody having a printout, voucher or email or a good story showing up at their desk after the systems failed. And it worked out just fine.


Wouldn't that be a security concern ?
New Technology is the name we give to stuff that doesn't work yet. Douglas Adams
 
WAC
Posts: 135
Joined: Sat Nov 29, 2008 10:31 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:26 pm

This kind of reminds me of a situation in global hospitality company I used to work for. They decided to outsource all IT including corporate, and a few weeks later it was hit with IT black out (apart from reservation system which is based on MDOS 80s system (it has only broken down for less than 4 hours in its 25+ history)).It emerged this was due to former disgruntled employees taking down the IT infrastructure...
At any rate I abandoned BA for long-haul a long time ago circa 2010 and only use them for Europe/ME where there is plenty of competition if things go wrong.
 
User avatar
Aesma
Posts: 12138
Joined: Sat Nov 14, 2009 6:14 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:30 pm

My large company had its main datacenter go down recently. Basically there was a power outage to the site, which is very rare, the battery backup in the datacenter worked fine, however it was night so the solar panels (a very large array) didn't produce electricity, and the 4 enormous 18 cylinders diesel generators had an unforeseen malfunction : their startup batteries had a too low voltage. UPS batteries got drained before electricians could get to the site and restore power, all the IT equipment shut down. Fortunately restarting everything went OK, aside from a few servers that didn't like it.

Clearly lessons were learned, but at the end of the day, IT is not that critical for the company, if it's down it just means office people are out of work, customers aren't materially affected, so I guess there is no desire for true high availability. There is a backup datacenter but I think the idea is that it can be started in a day or two, basically if the main one burns down or is flooded or something like that, it's not quick at all.

My team manages the phones and videoconferencing equipment and last year a phone server went down, it was the "worse" one as all the higher ups phones were affected, I spent half a day and half a night repairing it, with the vendor on the phone, soon after that a resiliency scheme we had proposed years before was accepted, so now if a server goes down, phones connect instantly to a backup one, conversations aren't even interrupted. Everything is still in the same datacenter, though...
New Technology is the name we give to stuff that doesn't work yet. Douglas Adams
 
asdf
Posts: 584
Joined: Tue Mar 18, 2014 12:03 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:35 pm

Aesma wrote:
oOfredOo wrote:
PatrickZ80 wrote:

Problem is that they can't verify the bookings. Anyone can show up at their desk and claim they have a booking to a certain destination. BA can't check if that's true or not, so before you know it they'd be handing out free flights.

Better wait until the system is up again and then see how they're going to get everyone to their destinations.


So what? It's better than paying 600 euro compensation to passengers, and have a total meltdown of traffic. I've flown Transavia Oktober last year and they boarded everybody having a printout, voucher or email or a good story showing up at their desk after the systems failed. And it worked out just fine.


Wouldn't that be a security concern ?


well, why?
worst case is that someone rides on a plane without proper payment
like paying for london - paris but flying london - LAX
could happen
but who really would take that chance?
without preperation?
and lost over there without a free backflight
without a visa there would only be very less destinations aviable
maybe some unemployed teens or twens
here is no harm done
not on money and not on security

if someone has a passport and if one has a kinda quick issued paper ticket one can legal and physically reach the gates
if one can reach the gates one can fly

if you have
a flightplan
a crew
a plane
passengers
and passengers on the other side of the trip who like to use that plane for the way back

its not rocket sience to figure out that it would be the economically best way to simply bring that bird in the air asap ....

but scenarios like this need to be evaluated before days like yesterday

there seem to be - besides a lot of other missing things - no "limit harm" team on duty
Last edited by asdf on Sun May 28, 2017 1:40 pm, edited 1 time in total.
 
VS239
Posts: 57
Joined: Fri Jul 28, 2006 4:29 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 1:38 pm

If nothing else, apart from a complete 'insert part of the body here' - up from a BA perspective,I just hope ALL airlines learn from this and prevent such things happening again. BA were caught short as they have been before. Let's hope they ALL learn and move on..but freaking LEARN. !!...

BA -->no way
Who...me??
 
Planeflyer
Posts: 1445
Joined: Fri Mar 14, 2014 3:49 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:44 pm

Tough few days for BA people and their customers. Too bad they have done a good job these past few years.

I'll admit from the start I have zero insight into the particulars but I imagine IT at an airline to be very tricky given the mix of legacy systems and all the new systems that have recently been added.

I can imagine that the IT system must be somewhat of a patch work nature. Can any of the BA IT folks speak to this?

Wonder if the IT failures at DL, LUV and United had a common thread?
 
Egerton
Posts: 864
Joined: Sun Jan 11, 2015 9:50 am

Re: BA experiencing computer problems.

Sun May 28, 2017 1:46 pm

I challenge the A.net assumption that out-sourcing is always wrong. There is little evidence for this assumption.

Sometimes, the mere threat of out-sourcing provides the only way achieve better performance.at lower cost. Occasionally, the threat has to be put into practice. This is usually the case where the arrogance of the existing crew exceeds their competence. Labour cartels feature strongly in these instances. Out-sourcing is almost always a high risk move. .When outsourcing is reluctantly introduced, it can prove to be a good move or a bad move, and this will take time to prove.

However, there is the probability that appointing senior management who have no experience or competence in the field of activity they will manage is unwise. This is the cause of many stupid decisions, over a wide field of activities, from baking to banking. Sending folk to Business School may give an overlay of new thinking but it cannot replace starting from the bottom and working your way up. Business Schools filled with generalist can have a negative effect. Good senior managers are born our of relevant experience, they cannot be manufactured in the flow-line production system of Business Schools.

If the BA experience of out-sourcing proves in this case and in due course to have been unwise, then the blame should fall upon those who appointed the wrong people to be in charge. Lions led by donkeys, as they say.
 
User avatar
par13del
Posts: 9529
Joined: Sun Dec 18, 2005 9:14 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 2:25 pm

Well, some thoughts...
1. Usually out-sourcing companies have their own management, so why exactly does the company doing the outsourcing need competent management, you only need an accountant to verify the charges and ensure cost are being saved.Most time, the local management undermines the activities of the outsourced company.

2. Threat to get lower cost, hhmmmm, if management does their job properly there would be SOP's and manuals to allow anyone from the street to come in read the manuals and do the job, If the laws and regulatory authorities require the staff to have some level of competence the outsourcing is not against your company but against the country who put such regulations in place. How do you lower those cost, move them to a jurisdiction that does not have such regulations has proven to be cheaper than getting involved in the politics of getting the regulations modified.

3. If the job is outsourced to a company that lacks competence, then those doing the outsourcing have issues themselves, and expecting them to have competent management to supervise the outsourced company is a bit much

In this case, we can be certain that after the dust settles and the investigations are complete, we will find that out-sourcing had very little if anything to do with the failure. Pax rely on BA IT Systems, if they do not have faith and confidence in the system they will take their business elsewhere, so......
Terminal 5 had major meltdowns when it went into operation, there was no where else for BA to go so Terminal 5 had to succeed, however it was done.
 
User avatar
SamYeager2016
Posts: 208
Joined: Wed Jun 15, 2016 6:22 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 2:57 pm

par13del wrote:
1. Usually out-sourcing companies have their own management, so why exactly does the company doing the outsourcing need competent management, you only need an accountant to verify the charges and ensure cost are being saved.Most time, the local management undermines the activities of the outsourced company.


Because you're just a customer, not THE customer. The company carrying out the outsourcing is always going to do things that are in their financial interest but which may not necessarily be in the best interests of the company that outsourced the work e.g. less regular testing of disaster recovery, cut down QA work to speed implementation. The company outsourcing needs people that understand the implications of proposed changes sometimes known as an "Intelligent Customer" function. Equally the work that is outsourced is unlikely to be static i.e. there will be enhancements to existing systems, new systems being developed that may be replacing all or part of existing systems and which will always be interfacing with existing systems.

Don't get me wrong, I'm not against outsourcing as such but it involves a lot of work initially as well as continuing work subsequently to make sure your company's interests are fully taken into account.
 
AngMoh
Posts: 982
Joined: Fri Nov 04, 2011 5:03 am

Re: BA experiencing computer problems.

Sun May 28, 2017 3:03 pm

Egerton wrote:
I challenge the A.net assumption that out-sourcing is always wrong. There is little evidence for this assumption.

Sometimes, the mere threat of out-sourcing provides the only way achieve better performance.at lower cost. Occasionally, the threat has to be put into practice. This is usually the case where the arrogance of the existing crew exceeds their competence. Labour cartels feature strongly in these instances. Out-sourcing is almost always a high risk move. .When outsourcing is reluctantly introduced, it can prove to be a good move or a bad move, and this will take time to prove.

However, there is the probability that appointing senior management who have no experience or competence in the field of activity they will manage is unwise. This is the cause of many stupid decisions, over a wide field of activities, from baking to banking. Sending folk to Business School may give an overlay of new thinking but it cannot replace starting from the bottom and working your way up. Business Schools filled with generalist can have a negative effect. Good senior managers are born our of relevant experience, they cannot be manufactured in the flow-line production system of Business Schools.

If the BA experience of out-sourcing proves in this case and in due course to have been unwise, then the blame should fall upon those who appointed the wrong people to be in charge. Lions led by donkeys, as they say.


I worked for a British company with an office and the data centre in the City of London. Most of the IT was outsourced to India. We had a water intrusion over the weekend and all the power supplies fried resulting in all servers being dead - there was no issue with the IT itself. All of Asia was down on Monday because the few staff responsible for the data centre were uncontactable and not even aware that the power was down until they turned up for work at 9 AM UK time and by that time it was the end of the day for us. The India team was trying to reach the site personnel in the UK but they did not respond because it was a weekend. A whole working day in Asia was lost because the responsible staff in the UK was literally sleeping.
727 732 733 734 735 73G 738 739 739ER 742 743 744 752 753 762 772 77E 773 77W 788 A300 A310 A319 A320 A321 A332 A333 A343 A345 A346 A359 A35K A388 DC-9 DC-10 MD11 MD81 MD82 MD87 F70 ERJ145 E170 E175 E190 E195 ATR72 Q400 CRJ200 CRJ700 CRJ900 BAE146 RJ85
 
kalvado
Posts: 2269
Joined: Wed Mar 01, 2006 4:29 am

Re: BA experiencing computer problems.

Sun May 28, 2017 3:33 pm

par13del wrote:
2. Threat to get lower cost, hhmmmm, if management does their job properly there would be SOP's and manuals to allow anyone from the street to come in read the manuals and do the job, If the laws and regulatory authorities require the staff to have some level of competence the outsourcing is not against your company but against the country who put such regulations in place. How do you lower those cost, move them to a jurisdiction that does not have such regulations has proven to be cheaper than getting involved in the politics of getting the regulations modified.

I agree, anyone with a valid driver license should be allowed to fly 777, period!
 
sharktail
Posts: 136
Joined: Sun Feb 12, 2012 6:41 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 3:58 pm

LJ wrote:
77west wrote:
I wonder if there has been a failure of a backup circuit as well, but then, what about replication to an alternate datacenter? Surely large companies would be able to survive a single datacenter going offline.


That's why you have to test your back-up plans regularly. The company I work for discovered that the generator didn't work and that the back-up faciltiies (we have two back-up faciltiies) didn't perform as planned a few years ago. After a large investment we now test all back-up systems and facilities and our power generator regularly. Why BA didn't have such a policy surprises me. One would expect from a company whose operations rely heavily on IT that they have a scenario (ensure you have a hard copy instead your computers/Ipad doesn't work) readily available. One thing for sure, BA's IT management will have some explaining to do to the IAG Board of Directors.

I personally think we take 24/7 power and internet too much for granted.


I am in IT as well, but happy I am not in an airline. Because unfortunately, you are not able to take any system offline in a DR exercise for an airline. Not even for 5 minutes.

Banks can generally do a DR test over the weekend so they can simulate loss of power in a data center to see what happens and how soon a system recovers. You can't shut down an airline for a day to do that. There are ways to do this in a test environment. But invariably, something unforeseen always goes wrong when you do the real test, as multiple people have commented on here. it's not impossible to solve, but a lot harder to pull off.
 
aahammayo
Posts: 5
Joined: Thu Feb 12, 2015 5:05 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 4:44 pm

 
xmyai
Posts: 7
Joined: Wed Nov 30, 2016 5:46 am

Re: BA experiencing computer problems.

Sun May 28, 2017 5:27 pm

par13del wrote:
Well, some thoughts...
2. Threat to get lower cost, hhmmmm, if management does their job properly there would be SOP's and manuals to allow anyone from the street to come in read the manuals and do the job, If the laws and regulatory authorities require the staff to have some level of competence the outsourcing is not against your company but against the country who put such regulations in place. How do you lower those cost, move them to a jurisdiction that does not have such regulations has proven to be cheaper than getting involved in the politics of getting the regulations modified.

I guess you have never worked on complex IT systems? SOPs only go so far. There will be occasions when there isn't a SOP that covers what you are trying to do or if there is one, it doesn't work as desired. It takes expertise based on knowledge gained over a long period of time, ideally involving commissioning the systems in the first place, to fix such issues.

par13del wrote:
3. If the job is outsourced to a company that lacks competence, then those doing the outsourcing have issues themselves, and expecting them to have competent management to supervise the outsourced company is a bit much

This is one of the issues with outsourcing. It's not always possible to know how good the outsourcing company are when the contract is signed.

As for outsourcing in general, reading another forum today someone claimed that all the big UK banks, bar one, are now moving their core IT systems back in house because they are not happy with the quality of service provided by the out sourcing companies.

Unfortunately I've seen first hand what happens when IT systems are outsourced to India. The company I worked for were a world leader in a particular area. Senior management thought it was easy so they outsourced the function to India. Numerous SOPs were put in place and the Indian developers even worked alongside staff in the UK for 2 years before taking over. However within 6 months of all the work going to India, we were no longer world leaders. A few years later we exited that market place completely.
 
GDB
Posts: 13273
Joined: Wed May 23, 2001 6:25 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 5:50 pm

VS239 wrote:
If nothing else, apart from a complete 'insert part of the body here' - up from a BA perspective,I just hope ALL airlines learn from this and prevent such things happening again. BA were caught short as they have been before. Let's hope they ALL learn and move on..but freaking LEARN. !!...

BA -->no way


As a BA employee for most of my adult life, I have to admit I really don't think they will.

It's bad enough operating from LHR, which running at near full capacity as normal, already makes us prone to events we cannot control (but do often get blamed for), but to then take big risks with pax goodwill...................you know what has just come into my head? That scene from near the end of the 1997 movie Fargo, where Marge Gunderson the heavily pregnant cop, has the surviving kidnapper/murderer in her car, after reeling off all the horrors in the story, 'all that, for a little bit of money?' she asks him, 'you do know there are more important things? Don't you?'

Edit; here it is, and it was a beautiful day at LHR when it happened too! (Though not in a snowy sense!)
https://www.youtube.com/watch?v=vmoYpJIUWhY

(Better to use that than me hurl a load of 'The Thick Of It' and/or 'Veep' style insults at the BA management).
 
jgch
Posts: 18
Joined: Sun Nov 27, 2016 5:32 pm

Re: BA experiencing computer problems.

Sun May 28, 2017 5:58 pm

@xmyai - good post! This brings it to the point.

Outsourcing as concept basically works. But it requires a very well managed hand over project and an active management afterwards. In the cases I have experienced, neither was the case.
I have seen a number of outsourcing/offshoring projects. None of them gave the expected ROI or quality gain thet was expected.
During the hand over phase, the local experts are made redundant. Hence their motivation is down to a minimum and they have all the possibilities to sabotage the project by telling the contractors staff only 90% of the truth. The missing 10% everyone forgets and no one realizes it was missing during the handover.
The managers on the other hand have often only a rough idea what the ground staff is really doing, so they have not the detailed know how to manage the project. Once everything is outsourced, these guys make sure to leave asap. Their successors have to clean up the left over crap.

When it comes to the day to day work, the outsourced companies loose their know how, hence becoming more and more dependent on their contractor. They then have a very rough idea on what is going on - hence they have lost control.
 
User avatar
Aesma
Posts: 12138
Joined: Sat Nov 14, 2009 6:14 am

Re: BA experiencing computer problems.

Sun May 28, 2017 6:16 pm

It's still a big mess, no ? From evening news I'm watching on French TV people are preparing for a second night of sleeping at the airport !
New Technology is the name we give to stuff that doesn't work yet. Douglas Adams
 
xmyai
Posts: 7
Joined: Wed Nov 30, 2016 5:46 am

Re: BA experiencing computer problems.

Sun May 28, 2017 6:24 pm

par13del wrote:
Well, some thoughts...
2. Threat to get lower cost, hhmmmm, if management does their job properly there would be SOP's and manuals to allow anyone from the street to come in read the manuals and do the job, If the laws and regulatory authorities require the staff to have some level of competence the outsourcing is not against your company but against the country who put such regulations in place. How do you lower those cost, move them to a jurisdiction that does not have such regulations has proven to be cheaper than getting involved in the politics of getting the regulations modified.


To add to what I have already written - read Jaded Geek's contribution over at https://forums.theregister.co.uk/forum/ ... ply_issue/

He covers the out sourcing issues in more detail than myself and has more relevant first hand experience too.

Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos