Moderators: richierich, ua900, PanAm_DC10, hOMSaR

 
Virtual737
Posts: 1471
Joined: Tue Jul 19, 2016 6:16 am

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 7:11 am

bennett123 wrote:
Surely BA has disaster recovery planning.


Yes. The recovery plan was a disaster.
 
jomur
Posts: 471
Joined: Sun Jun 19, 2016 6:36 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 8:23 am

How do we know the recovery plan didnt work? Maybe it just took a long time to do and it obviously wasn't a 5 minute fix. BA would have had to validate all the revalant data to make sure it was up to date.
Also there is nothing that says it would have een fixed earlier if BA hadn't outsourced its IT.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 8:28 am

SATexan wrote:
This is a local UK data center issue. They will learn from this and add checklists / regression testing whenever electrical and network upgrades are done in data centers. No need to drag India into this.


If you don't understand the issue, please don't post anything at all - because your post here indicates you have utterly NO idea about what the actual issue is.

Also, IMHO, India cannot be a site for hosting disaster recover servers.


Yes it can, its a fine place for hosting disaster recovery servers. Its geographically remote from a London site, it doesn't share the same power links, it doesnt share the same internet links, it doesn't share the same weather patterns...

Its a great place to put a DR site.

It is too difficult to manage a business continuity exercise when your DR site is that far away. It requires extraordinary coordination.


Its not difficult at all, you have an onsite crew responsible for physical maintenance, and everything else can be done by someone with a laptop anywhere in the world that there is an internet connection.

Further, most companies across the globe are unable to replicate their live environments in the DR sites.


Then those companies are incompetent, because there is NOTHING that should prevent a company from live streaming their data to a DR site. Nothing at all.

The live servers are managed by a different group of people that manage the server configurations, patches, security setup, databases, external/internal dependencies and keep the code base up-to-date. The software code and databases at the DR site are at least a 3-4 day earlier snapshot of live environment.


Not if you have a proper DR plan, a proper release plan, a proper run book, configuration management etc. Anything else is a simple excuse.

It is very difficult to have a "real time data" snapshot in DR.


No its not, its pretty easy to have "real time data" streamed to your DR site. In fact, in many ways you would have to go out of your way to have your DR site data lag your production data...

tommyarias wrote:
If anything went wrong with outsourcing, the problem is in the UK. You can outsource work but can not outsource responsibility.


Very true statement. Even after outsourcing to India the ultimate responsibility to ensure that the job gets "done" lies with the bosses in UK, USA or elsewhere. These guys get paid big bucks to merely "outsource"...


And the outsourcing company bid for and accepted the contract on the basis that it could do the job it was being asked to do.

So lets go back to your first sentence:

SATexan wrote:
This is a local UK data center issue. They will learn from this and add checklists / regression testing whenever electrical and network upgrades are done in data centers. No need to drag India into this.


Data centres typically have very very few operational staff - they have hundreds of cabinets of servers, but the operational staff there are there to babysit the physical hardware, replace dead drives, PSUs, plug in servers that have been shipped to them, remove old servers from cabinets and send them elsewhere etc etc.

Data centre staff are almost always orientated around ensuring the data centre runs. They typically have zero operational responsibility beyond that, including DR (beyond ensuring their data centre is up and running and meets requirements).

The responsibility for ensuring the DR systems are effective lie elsewhere in BA, not physically with the data centre staff. The people holding that responsibility can physically be anywhere in the world.

So the people saying "no need to drag India into this" are totally misunderstanding where the responsibility for the failure in the DR plan here lies - it lies with the outsourcing company that holds the contract where they took on this responsibility along with managing the rest of BAs IT setup. Yes, there may also be an element of a senior manager overseeing it in the UK, as a liaison between the contractors and BA, but at the end of the day there was a contract signed which said (paraphrasing) "we agree to look after this..." and that contract was breached over the weekend.

I've written DR plans, I've put them into effect for large companies, I achieved BS25999 accreditation for DR plans, and currently I am pursuing ISO 22301 accreditation for DR plans...
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 8:31 am

jomur wrote:
How do we know the recovery plan didnt work? Maybe it just took a long time to do and it obviously wasn't a 5 minute fix. BA would have had to validate all the revalant data to make sure it was up to date.


*Thats* how I know the recovery plan didn't work, those things you said right there...

If the plan worked, you wouldn't have noticed anything at all - and if you did, it would have been a few minutes of disruption, not days.

There should be no need to validate any data - thats an automated process done when the data is shipped moment by moment to the secondary and tertiary sites.

Also there is nothing that says it would have een fixed earlier if BA hadn't outsourced its IT.


No, there isn't, but they did outsource it so the blame lies with the current holders of the responsibility - which is the outsourcing company which signed a contract saying "we can do this".
 
jomur
Posts: 471
Joined: Sun Jun 19, 2016 6:36 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 8:44 am

So what happens when the point of failure is the point where the backups and original sytems integrate as it must, I asume, at some point? Its nat as if you are going to have 2 or more complete seperate systems all the way to the end user as its not practical on a cost or physical space point.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 9:03 am

jomur wrote:
So what happens when the point of failure is the point where the backups and original sytems integrate as it must, I asume, at some point? Its nat as if you are going to have 2 or more complete seperate systems all the way to the end user as its not practical on a cost or physical space point.


Thats precisely what you do - you have two physically separate infrastructures, at different geographical locations, using different internet connections and different power systems.

There are several ways to fail over as well, depending on how you are approaching the issue - the secondary site can be a purely DR orientated site, not being used unless necessary (typically called a cold or warm standby, depending on the exact approach used), or it can be used in a hot-hot approach, in that *all* your sites are handling traffic (say, 33% load for each site across 3 sites), you just build in space capacity at all locations so that each location can take 100% load should both other locations fail.

If you don't think thats practical, you have *no* idea about DR at this level - once you get to a certain site, a second mirror site for your IT ops is basically essential to your business. You have to weigh up the cost of "not doing business at all" with "having spare capacity lying around just in case"....
 
mjoelnir
Posts: 9833
Joined: Sun Feb 03, 2013 11:06 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 9:06 am

jomur wrote:
How do we know the recovery plan didnt work? Maybe it just took a long time to do and it obviously wasn't a 5 minute fix. BA would have had to validate all the revalant data to make sure it was up to date.
Also there is nothing that says it would have een fixed earlier if BA hadn't outsourced its IT.


The recovery plan, if there was one, did not work. Data is validated while the main systems are running, it is difficult to validate Date if the primary source is down and not available.

No it is not proven that outsourcing led to the bad recovery, but we can be pretty sure that insufficient investment in redundancy did.
 
Draken21fx
Posts: 260
Joined: Wed Jun 15, 2016 10:38 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 10:38 am

Listen to moo he knows what he is talking about ;)

Even for smaller companies a disaster recovery plan is a must. Companies do also test the disaster recovery systems during low traffic to ensure that the switch over goes according to plan. I cannot believe that such tests were not carried out by BA.

With a few exceptions (in which the company is obliged by law to maintain servers in specific locations) it does not really matter where the servers are located as long as the rules moo has mentioned are followed.

The issue was that BA, most probably, went with the cheapest contract they could find without thinking of potential implications and if the contracts signed could be honoured by the contracted (?) company.

Reminds me a lot of the disruption to Natwest a few years ago which cost the banking group millions https://www.theguardian.com/money/2012/ ... appens-now
Yes the issue was not a data centre meltdown but rather an update but the implications were very similar https://www.theregister.co.uk/2012/06/2 ... are_glitch

Again I am not trying to bash Indian IT but rather show what implications saving a few thousand €,£ or $ can have on big (and small) organisations. Subcontracting is popular out there nowadays but unfortunately when the s^&^ hits the fan you have inexperienced ops to deal with the issue which only aggravates the problem and that is not only a problem in IT.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 10:58 am

jomur wrote:
So what happens when the point of failure is the point where the backups and original sytems integrate as it must, I asume, at some point? Its nat as if you are going to have 2 or more complete seperate systems all the way to the end user as its not practical on a cost or physical space point.


If it is an equipment failure design should have redundancy within data center ie., redundant components and application running a local server cluster.

Depending on how mission critical your system you have a separate data center within same metro area(minimum 12 miles apart), different city, country or continent.

Most companies do have good handle with single component failure scenarios as those happen frequently, but they don't test remote fail-over as it is impossible to shutdown a mission critical production system for testing. They rely on table top exercises and hope for the best. Auditors are a OK with this approach.
 
oldannyboy
Posts: 2813
Joined: Wed Jun 15, 2016 8:28 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:06 am

Cheap outsourcing gone-bad all over again, as it's happened so many times in the past. Guess lessons are never learnt by those managers who love to re-invent the wheel. They love those at BA!
 
User avatar
Revelation
Posts: 27477
Joined: Wed Feb 09, 2005 9:37 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:06 am

WPvsMW wrote:
Power surges happen all the time. A data center by definition has at least two independent feeds from the grid AND a huge battery room and inverter AND a huge diesel genset. The deeper problem appears to be the UPS failed to switch to alternate mains supply (or there was no alternate mains supply) quickly enough, and that when alternate mains was also bad (or non-existent), the battery and/or inverter failed and/or the diesel genset failed. This is not incompetence in India, it is incompetence in power supply design, probably done long ago, and periodic testing overlooked. Delta's April meltdown redux.

I heard an interview with BA's CEO yesterday on the radio and he made it clear that the power equipment did its job but the networking hardware (i.e. "messaging system") did not come up after the power issue was resolved. Lord knows why that was. Maybe its hardware was damaged by the power spike. Maybe it reloaded a bad configuration. He didn't say. But I can see why this snowballed into a long outage. The power gets restored. Most of the servers come on line but the network is down so their memory fills with unsent messages and processes get stuck. This presents a state that no one has ever debugged before. Add to it that yes, many of the people who knew the systems the best are no longer with the company, and you have a recipe for a long drawn out outage.

cv990Coronado wrote:
But RBS Capital Markets analyst Damian Brewer has said BA’s apparent failure to have learnt lessons from rivals problems “suggests fundamental management and planning weakness”.

But Alex Cruz won't do the honourable thing, Willie Walsh who is not well known for silence seems to be very quiet. I really feel for the BA frontline staff who have no support from the top and are working in the dark having to face a very angry and frustrated public. A very long way indeed from Sir Colin Marshall's "The world favourite airline".

Different management teams take different strategies. I read one report where BP's handling of the oil rig disaster in the Gulf of Mexico has been used by some management teams as a lesson to NOT be forthcoming with information because it could later be used against you.

I'm not sure what BA staff could have been telling people during the crisis. None of their aircraft could move because they could not access the computers needed to print legally required documents. The people who knew the most about what was going on were also the ones deeply engaged in fixing it. The mechanisms for distributing this information were all largely not working. Does it make sense to have your employees go in front of the public just to say "We don't know what's going on either and we have no idea how long it will take to get customers to their desired destinations"?

So, yes, this to me indicates a massive lack of preparation, but I'm not sure if the communication aspect could or would have been done much differently if they had planned for this kind of outage.
 
Draken21fx
Posts: 260
Joined: Wed Jun 15, 2016 10:38 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:07 am

dtw2hyd wrote:
...

Most companies do have good handle with single component failure scenarios as those happen frequently, but they don't test remote fail-over as it is impossible to shutdown a mission critical production system for testing. They rely on table top exercises and hope for the best. Auditors are a OK with this approach.


From personal experience they do do it. Personally I have been awake at 3-4am a few times for the exercise in a few past companies.And yes you are right when it comes to auditors, they do not enforce the testing and presenting a top level DR plan is acceptable for them.

edit: Maybe the topic should be renamed to "Best practices for IT" with the combined knowledge of the forum I think we could have helped BA avoid the unpleasant situation of the past few days :D
Last edited by Draken21fx on Tue May 30, 2017 11:10 am, edited 1 time in total.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:20 am

dtw2hyd wrote:
but they don't test remote fail-over as it is impossible to shutdown a mission critical production system for testing. They rely on table top exercises and hope for the best. Auditors are a OK with this approach.


I've switched over a multi-million-dollar-a-day-revenue business to their secondary site and had no one notice before :) I've worked at a company in the UK which had a button marked "DO NOT PRESS UNDER ANY CIRCUMSTANCES" in their (unsupervised) client waiting area which triggered a failover to an alternative data centre for a system bringing in over a million dollars a day in revenue - it got pressed multiple times a month while I was there, with no negative effects at all. The company was so confident in its testing and setup that it simply did not matter when a failover happened, so it was used as a party trick :)

Seamless DR is a practical thing these days, and has been for more than a decade.

If you don't do real life testing, you don't have a plan worth counting on.
 
User avatar
Revelation
Posts: 27477
Joined: Wed Feb 09, 2005 9:37 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:22 am

moo wrote:
No, there isn't, but they did outsource it so the blame lies with the current holders of the responsibility - which is the outsourcing company which signed a contract saying "we can do this".

Currently being reported as Tata Consultancy Services by a few different media outlets ( here's one: http://www.theregister.co.uk/2016/02/25 ... y_service/ )

Revelation wrote:
WPvsMW wrote:
Power surges happen all the time. A data center by definition has at least two independent feeds from the grid AND a huge battery room and inverter AND a huge diesel genset. The deeper problem appears to be the UPS failed to switch to alternate mains supply (or there was no alternate mains supply) quickly enough, and that when alternate mains was also bad (or non-existent), the battery and/or inverter failed and/or the diesel genset failed. This is not incompetence in India, it is incompetence in power supply design, probably done long ago, and periodic testing overlooked. Delta's April meltdown redux.

I heard an interview with BA's CEO yesterday on the radio and he made it clear that the power equipment did its job but the networking hardware (i.e. "messaging system") did not come up after the power issue was resolved.

I found a quotable source:

Cruz gave his first interviews on the failure on Monday, telling Sky News: “On Saturday morning at around 9.30am there was indeed a power surge that had a catastrophic effect over some communications hardware which eventually affected all the messaging across our systems.”


Ref: https://www.theguardian.com/business/20 ... t-meltdown
 
User avatar
ojjunior
Posts: 1084
Joined: Thu Jul 13, 2006 12:31 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:24 am

Yet another brilliant decision to outsource services in India... the cheapest way. Quality to screw, passengers too.
Just do the math if the these days worth this outsourcing.
Well done BA! We welcome you to the club of huge corporations around the world who think that's a great idea.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:38 am

moo wrote:
.. Seamless DR is a practical thing these days, and has been for more than a decade. ...


It is feasible on technology front for decades, but business users will resist to give any downtime to completely test a data center fail-over, which needs lot more time than real fail-over. When there are dozens of tightly integrated applications in one data center owned by difference business groups, it gets even more complicated.

When it comes to downtime of mission critical production system for fail-over testing, business users are worse than a.net fanboys. I spend most of my time negotiating down time for testing.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 11:39 am

dtw2hyd wrote:
moo wrote:
.. Seamless DR is a practical thing these days, and has been for more than a decade. ...


It is feasible on technology front for decades, but business users will resist to give any downtime to completely test a data center fail-over, which needs lot more time than real fail-over. When there are dozens of tightly integrated applications in one data center owned by difference business groups, it gets even more complicated.

When it comes to downtime of mission critical production system for fail-over testing, business users are worse than a.net fanboys. I spend most of my time negotiating down time for testing.


If theres downtime, its being done wrong...

Thats basically the entire gist of most of my posts in this thread :)
 
User avatar
SomebodyInTLS
Posts: 1976
Joined: Wed Jun 15, 2016 12:31 pm

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 12:00 pm

AngMoh wrote:
If anything went wrong with outsourcing, the problem is in the UK. You can outsource work but can not outsource responsibility.


I would tend to agree with that from personal experience. I sometimes have to deal with issues in centrally-managed IT systems for two separate corporations and in both cases it's been an absolute nightmare since they've been outsourced - one in India, one to various European centres. I should add that this isn't even outsourcing to another company, as far as I know this is what you might call internal outsourcing to newly-created departments in other countries.

There are two problems:

  • you are always forced to go through a central helpdesk (which may be preceded by submitting through a mostly-useless issue-tracker website) - no-one knows anything and they usually annoy the hell out of you by forcing you to answer inane and utterly irrelevant questions about your IP address or whatever
  • the systems are hopeless at getting the right information sent to the right person, no matter how much you "escalate" things and try and steer it from your end - you're frustrated by a wall of bad organisation, bad UI, bad feedback

Honestly, I usually know more about the problem than the person who has to fix it and I start off by sending a detailed description of the issue and what to do about it and the first reaction is always on the level of "did you try turning it off and on again"... and then it will get passed on to a totally incorrect department, usually missing all that information I carefully compiled, and then it will get lost for days until I start ranting again.

I have sympathy for those at the outsourced end (even though most seem clueless, except for a couple of rare cases who went above and beyond and actually sorted everything out within minutes - those really impressed me) and I would put the blame at the manner in which things have actually been outsourced. When there's a local guy you can go visit and explain the problem to, it's nearly always immediately understood and promptly sorted out - when it's badly outsourced it just takes forever to get to the person responsible and then another forever to get the problem understood and fixed.
Last edited by SomebodyInTLS on Tue May 30, 2017 12:23 pm, edited 6 times in total.
 
User avatar
par13del
Posts: 11449
Joined: Sun Dec 18, 2005 9:14 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 12:05 pm

jomur wrote:
How do we know the recovery plan didnt work? Maybe it just took a long time to do and it obviously wasn't a 5 minute fix. BA would have had to validate all the revalant data to make sure it was up to date.
Also there is nothing that says it would have een fixed earlier if BA hadn't outsourced its IT.


BA has systems which caters to clients on a 24/7 basis, one would expect a recovery plan to take note of the 24/7 requirement.
Now if BA comes out and publicly states that its recovery plan is a 3 to 5 day process......
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 12:12 pm

moo wrote:
If theres downtime, its being done wrong...


Are you saying you promoted a system without testing?

I am not talking about downtime after a geographically dispersed data center pair went into production.

Not many applications are deployed directly on a metro or continental cluster. Most are upgraded from single data center local cluster to a geographically dispersed data center pair as business grows and systems become critical and auditors write-up multiple times for the lack of business continuity strategy.

Sure it will be a near continuous operation once it goes into production. Zero downtime if application is cluster aware and customer can afford the infrastructure.

With the advent of outsourcing, companies keep moving data centers from provider to provider, so the tests done at old data center pair may not work with new set.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 12:17 pm

dtw2hyd wrote:
moo wrote:
If theres downtime, its being done wrong...


Are you saying you promoted a system without testing?


Theres testing and theres what you describe...

Of course I test, but I don't get into the situation where someone has to OK it because a test might bring downtime, it simply doesn't get to that, I don't let it.
 
User avatar
Revelation
Posts: 27477
Joined: Wed Feb 09, 2005 9:37 pm

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 12:24 pm

SomebodyInTLS wrote:
Honestly, I usually know more about the problem than the person who has to fix it and I start off by sending a detailed description of the issue and what to do about it and the first reaction is always on the level of "did you try turning it off and on again"... and then it will get passed on to a totally incorrect department and lost for days until I start ranting again.

So I have sympathy for those at the outsourced end (even though most seem clueless, although I have been very impressed by a couple of rare cases who went above and beyond and actually sorted everything out within minutes) and I would put the blame at the manner in which things have actually been outsourced. When there's a local guy you can go visit and explain the problem to, it's nearly always immediately understood and promptly sorted out - when it's badly outsourced it just takes forever to get to the person responsible and then another forever to get the problem understood and fixed.

I just moved from a gig at a huge corporation to one at a privately owned firm with 600 employees worldwide. I love the difference. They have a ticket system to report problems. If I'm not getting the response I need in a timely fashion, my escalation process is to go upstairs and ask what the bleep is going on with my ticket. If it can't be solved locally and quickly, chances are it'll never be solved, so it's time for a different plan. No need to sit around waiting for unknown entities to do their job, or not do their job. It works today or it doesn't work at all and just deal with the consequences. If that means a project stops, there's only two layers between me and the CEO, and he sits in on engineering meetings twice a week so he'll find out damn soon. Things get worked out really quickly.
 
User avatar
readytotaxi
Posts: 8874
Joined: Mon Dec 11, 2006 2:09 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 12:27 pm

Don't think BA need a DR plan any more, sure as hell need a PR recovery plan :checkmark:
 
Amiga500
Posts: 2666
Joined: Tue Mar 03, 2015 8:22 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:08 pm

dtw2hyd wrote:
moo wrote:
If theres downtime, its being done wrong...


Are you saying you promoted a system without testing?


Why would you have to down the primary system to test the backup?

You simulate the failure(s).Receiving no network packages from the primary would be a pretty big failure - not sure how they could have a testing regime that wouldn't consider it.


The tricky bit is ensuring you have an available backup for the primary when your in the midst of testing the nominal backup. Just in case.
 
Amiga500
Posts: 2666
Joined: Tue Mar 03, 2015 8:22 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:08 pm

readytotaxi wrote:
Don't think BA need a DR plan any more, sure as hell need a PR recovery plan :checkmark:


They should ask UA :lol:
 
User avatar
golfradio
Posts: 947
Joined: Mon Jun 15, 2009 5:35 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:18 pm

dtw2hyd wrote:
When it comes to downtime of mission critical production system for fail-over testing, business users are worse than a.net fanboys. I spend most of my time negotiating down time for testing.


Huh! How can you ask for downtime of a mission critical application??

I am one of the architects of a SaaS application at the second biggest software company. We have SLAs of six-nines with penalties that will quickly escalate into millions for an outage of this kind. If I were to even suggest a downtime for testing DR, my chief architect will waste no time kicking my rear to the curb.

There are concepts like canary release testing, blue-green deployment testing etc which are proven. Downtime is non-negotiable.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:27 pm

Amiga500 wrote:
dtw2hyd wrote:
moo wrote:
If theres downtime, its being done wrong...


Are you saying you promoted a system without testing?


Why would you have to down the primary system to test the backup?

You simulate the failure(s).Receiving no network packages from the primary would be a pretty big failure - not sure how they could have a testing regime that wouldn't consider it.


The tricky bit is ensuring you have an available backup for the primary when your in the midst of testing the nominal backup. Just in case.


In theory it is possible, in real world, the only way to make sure data center fail-over works perfectly is thoroughly test when you implement a geographically dispersed data centers and test regularly because infrastructure is always changing.

No one does backups now-a-days, data gets replicated real time to storage at secondary data center. Infrastructure is the least of the issues.

Real life systems are much more complicated than ONE system gets the news. If there are 100 applications in primary data center, may be 10 are mission critical and customer is ready pay for the infrastructure and 99.999% up time. Rest they don't want to care because of the cost. Also no customer want to keep the infrastructure free at secondary data center, they want to use it dev and test environments.

Like I said companies do their best with table top exercises but there will always be nasty surprises. You cannot have two data centers with same IP range to truly test all applications in a fail-over scenario. If you use different IP range, you will find some developer 30 years back hard-coded IP into an app no one dared to touch, and it is very mission critical because that developer is now a direct report to CIO and no one can question him/her.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:38 pm

dtw2hyd wrote:
Amiga500 wrote:
dtw2hyd wrote:

Are you saying you promoted a system without testing?


Why would you have to down the primary system to test the backup?

You simulate the failure(s).Receiving no network packages from the primary would be a pretty big failure - not sure how they could have a testing regime that wouldn't consider it.


The tricky bit is ensuring you have an available backup for the primary when your in the midst of testing the nominal backup. Just in case.


In theory it is possible, in real world, the only way to make sure data center fail-over works perfectly is thoroughly test when you implement a geographically dispersed data centers and test regularly because infrastructure is always changing.


Yes, but testing doesn't have to be destructive, you just design the infrastructure that way.

No one does backups now-a-days, data gets replicated real time to storage at secondary data center. Infrastructure is the least of the issues.


Uh, you should still do back ups - there are many scenarios where replication off site isn't enough, for example in auditing and regulatory retention.


Also no customer want to keep the infrastructure free at secondary data center, they want to use it dev and test environments.


We seem to be sitting on very different levels in this industry, because I'm at the level where secondary data centres purely for DR is the norm.

Like I said companies do their best with table top exercises but there will always be nasty surprises. You cannot have two data centers with same IP range to truly test all applications in a fail-over scenario. If you use different IP range, you will find some developer 30 years back hard-coded IP into an app no one dared to touch, and it is very mission critical because that developer is now a direct report to CIO and no one can question him/her.


Plenty of ways to mitigate that without issue...
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:38 pm

golfradio wrote:
Huh! How can you ask for downtime of a mission critical application??

I am one of the architects of a SaaS application at the second biggest software company. We have SLAs of six-nines with penalties that will quickly escalate into millions for an outage of this kind. If I were to even suggest a downtime for testing DR, my chief architect will waste no time kicking my rear to the curb.

There are concepts like canary release testing, blue-green deployment testing etc which are proven. Downtime is non-negotiable.


Any system or application needs planned downtime to avoid unplanned downtime.

Your SLA is for your own software as a service. Your infrastructure and processes are built for that. A corporation needs more than one software to run You may not care about an in-house piece of the whole puzzle which is also critical.
 
lavalampluva
Posts: 1433
Joined: Tue Jun 24, 2014 7:33 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:40 pm

You get what you pay for. If you want to be cheap, expect cheap.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 1:57 pm

moo wrote:
Yes, but testing doesn't have to be destructive, you just design the infrastructure that way.

Just to be clear, are we having an academic or hypothetical discussion. I am sure BA and outsourced contractor did lot of non-destructive testing.

You can idle one engine and simulate IFSD or really shutdown and go thru relight process. Both are not the same.

Uh, you should still do back ups - there are many scenarios where replication off site isn't enough, for example in auditing and regulatory retention.

Implemented several tape-less backup recovery strategies meeting regulatory requirements. With current storage frame capacities and data de-dup and accelerator technologies there is no need for tape backups. Also major US storage vendor works real hard on golf courses with CIOs to make sure they don't buy tape libraries and buy few $$$millions worth of storage from them.

We seem to be sitting on very different levels in this industry, because I'm at the level where secondary data centres purely for DR is the norm..

Not in the world where corporations are run by Willie Walsh alike.

Plenty of ways to mitigate that without issue...

Sure you can, only if you knew the IP was hard coded into the app.
 
airzona11
Posts: 1922
Joined: Wed Dec 17, 2014 5:44 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 2:51 pm

dtw2hyd wrote:
moo wrote:
Yes, but testing doesn't have to be destructive, you just design the infrastructure that way.

Just to be clear, are we having an academic or hypothetical discussion. I am sure BA and outsourced contractor did lot of non-destructive testing.

You can idle one engine and simulate IFSD or really shutdown and go thru relight process. Both are not the same.

Uh, you should still do back ups - there are many scenarios where replication off site isn't enough, for example in auditing and regulatory retention.

Implemented several tape-less backup recovery strategies meeting regulatory requirements. With current storage frame capacities and data de-dup and accelerator technologies there is no need for tape backups. Also major US storage vendor works real hard on golf courses with CIOs to make sure they don't buy tape libraries and buy few $$$millions worth of storage from them.

We seem to be sitting on very different levels in this industry, because I'm at the level where secondary data centres purely for DR is the norm..

Not in the world where corporations are run by Willie Walsh alike.

Plenty of ways to mitigate that without issue...

Sure you can, only if you knew the IP was hard coded into the app.


Any publicly traded company, let alone an airline should have real-time failover data centers. If something so rudimentary in 2017 did not occur, lots of shareholders in addition to the public and regulators are going to be asking questions that are going to cost the jobs of internal and external partners.

IT Outsourcing has negative connotations, but when done right you get more expertise, skill and capabilities vs doing in house. As this unfolds it is looking very well like those in-house severance packages were given to lower the cost and the corner cut somewhere along the way cut quality.

BA has been cutting costs and getting bad press recently, this event is going to tarnish their brand. They had record profits last year. When the economy slows, fuels ticks up or something happens, they will be the first to add more fees and surcharges. The post-mortem is still developing but what a bad look for BA, "the favorite airline," that they are actually cutting corner in such critical areas.

I am really surprised that IAG doesn't have failover or shared systems within the group.
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 3:18 pm

dtw2hyd wrote:
moo wrote:
Yes, but testing doesn't have to be destructive, you just design the infrastructure that way.

Just to be clear, are we having an academic or hypothetical discussion. I am sure BA and outsourced contractor did lot of non-destructive testing.

You can idle one engine and simulate IFSD or really shutdown and go thru relight process. Both are not the same.


You seem to be missing my point - I'm saying that you don't need to take the risks that you are professing exist, if you take the right approach in the first place.

dtw2hyd wrote:
Uh, you should still do back ups - there are many scenarios where replication off site isn't enough, for example in auditing and regulatory retention.

Implemented several tape-less backup recovery strategies meeting regulatory requirements. With current storage frame capacities and data de-dup and accelerator technologies there is no need for tape backups. Also major US storage vendor works real hard on golf courses with CIOs to make sure they don't buy tape libraries and buy few $$$millions worth of storage from them.


Did I say anything about tapes? Nope, so stop fixating :)

I was challenging the very premise that "no one does backups now-a-days" :)

dtw2hyd wrote:
We seem to be sitting on very different levels in this industry, because I'm at the level where secondary data centres purely for DR is the norm..

Not in the world where corporations are run by Willie Walsh alike.


Except that BA has spare capacity in three data centres across the EU...

dtw2hyd wrote:
Plenty of ways to mitigate that without issue...

Sure you can, only if you knew the IP was hard coded into the app.


Nope, you simply don't let it be an issue - there are millions of networks out there that use the same internal IP range and yet can talk to one another through the magic of the internet. The technology for the routing exists and is well tested.

And its trivial to scan a codebase for IP addresses - that and passwords are two things that you should be automating scanning for anyway... neither have any place in a codebase and should raise red flags.
 
SATexan
Posts: 294
Joined: Wed Jul 22, 2009 6:49 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 3:57 pm

moo wrote:
SATexan wrote:
It is very difficult to have a "real time data" snapshot in DR.

No its not, its pretty easy to have "real time data" streamed to your DR site. In fact, in many ways you would have to go out of your way to have your DR site data lag your production data...


Oh really! and what exactly is your plan for streaming "live data" to a third party with proper encryption? What is the cost?

moo wrote:
I've written DR plans, I've put them into effect for large companies, I achieved BS25999 accreditation for DR plans, and currently I am pursuing ISO 22301 accreditation for DR plans...


All fine & dandy. What you have written is a standard text book case study of DR recovery. But you seem to be in a fairy tale world when it comes to the actual implementation. If BA had done everything per the textbook we wouldn't be discussing this failure today.

moo wrote:
The responsibility for ensuring the DR systems are effective lie elsewhere in BA, not physically with the data centre staff. The people holding that responsibility can physically be anywhere in the world.

So the people saying "no need to drag India into this" are totally misunderstanding where the responsibility for the failure in the DR plan here lies - it lies with the outsourcing company that holds the contract where they took on this responsibility along with managing the rest of BAs IT setup.


Whaat?? Come again..

The outsourcing company needs absolute information regarding the server setup and entire architectural framework to be handed to them. The servers have to be a 1:1 match with those existing in the Production environment. Same exact network, load balancers, switches, network capacity, memory, operating system, patches, hardware setup, software plugins, third party components, code, database..everything. So BA has to "give" this information out completely. How can you state so confidently that BA shared all of this information accurately? Whose responsibility is it to verify that the vendor has all the setup replicated to perfection?

Pal, the DR exercise is critical to BA's business. So in the Business continuity exercise and any DR preparation BA needs to be diligent and take it more seriously. Look, I've got no problem if you have a problem with offshoring to India or the country of India itself, but the buck stops with BA. There has to be a Recovery team based in UK tasked with coordinating and managing precisely this sort of failures. You CANNOT hand over that responsibility to a third party in India. Didn't your DR textbook tell you that??
 
User avatar
moo
Posts: 5117
Joined: Sun May 13, 2007 2:27 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 4:22 pm

SATexan wrote:
moo wrote:
SATexan wrote:
It is very difficult to have a "real time data" snapshot in DR.

No its not, its pretty easy to have "real time data" streamed to your DR site. In fact, in many ways you would have to go out of your way to have your DR site data lag your production data...


Oh really! and what exactly is your plan for streaming "live data" to a third party with proper encryption? What is the cost?


I don't get the contrived problem here?

If its your data, you own the encryption keys, so you can multicast it to the other data centres as required.

You can use distributed transactions to allow all the other data centres to be updated on demand.

Loads of ways.

And the cost is "staying in business".

SATexan wrote:
moo wrote:
I've written DR plans, I've put them into effect for large companies, I achieved BS25999 accreditation for DR plans, and currently I am pursuing ISO 22301 accreditation for DR plans...


All fine & dandy. What you have written is a standard text book case study of DR recovery. But you seem to be in a fairy tale world when it comes to the actual implementation. If BA had done everything per the textbook we wouldn't be discussing this failure today.


My implementations seem to work just fine. Nothing fairy tail here...

SATexan wrote:
moo wrote:
The responsibility for ensuring the DR systems are effective lie elsewhere in BA, not physically with the data centre staff. The people holding that responsibility can physically be anywhere in the world.

So the people saying "no need to drag India into this" are totally misunderstanding where the responsibility for the failure in the DR plan here lies - it lies with the outsourcing company that holds the contract where they took on this responsibility along with managing the rest of BAs IT setup.


Whaat?? Come again..

The outsourcing company needs absolute information regarding the server setup and entire architectural framework to be handed to them. The servers have to be a 1:1 match with those existing in the Production environment. Same exact network, load balancers, switches, network capacity, memory, operating system, patches, hardware setup, software plugins, third party components, code, database..everything. So BA has to "give" this information out completely. How can you state so confidently that BA shared all of this information accurately? Whose responsibility is it to verify that the vendor has all the setup replicated to perfection?


Its the outsource contractors JOB to ensure they can conform to the contract THAT THEY SIGNED. No contractor is going to take on a complex existing infrastructure without a period of discovery and remedial work, and only once that is done to the contractors satisfaction only then should they take on the responsibility.

If the contractor is taking on an infrastructure they haven't themselves audited and confirmed, then yes it is their fault top to bottom.

This isn't crayon drawing time before nap time, this is real world business. If the contractor is taking on infrastructure they cant support then they deserve to be driven out of business by being sued into oblivion for incidents like this.

See my other comments about institutional knowledge as well, they cover a lot of your rant.

SATexan wrote:
Pal, the DR exercise is critical to BA's business. So in the Business continuity exercise and any DR preparation BA needs to be diligent and take it more seriously. Look, I've got no problem if you have a problem with offshoring to India or the country of India itself, but the buck stops with BA. There has to be a Recovery team based in UK tasked with coordinating and managing precisely this sort of failures. You CANNOT hand over that responsibility to a third party in India. Didn't your DR textbook tell you that??


Yes, if the contractor failed, BA is to blame for using that contractor - but blame isnt an indivisible construct here, it can be spread pretty damn thinly and land on a lot of desks, and BA can feel perfectly happy laying quite a lot of that blame at the feet of the contractors, if it was the contractors at fault.

If the DR is done properly, the event management team can be based anywhere in the world - there are very very very very few instances where they have to be on site, because usually that means hardware and you can delegate the swapping out of failed hardware to a low level tech. You also seem to ignore the fact that if their DR was done correctly, there would be no need for panicked responses or rushed solutions - they would fail over onto an alternative environment and run on that for the duration, the original fault could be rectified and then you move back onto the primary environment (should you be doing warm-standby rather than hot).

I'm not sure exactly what your point here is, but at the end of the day, BA contracted this stuff out and the contractor failed in their role - ensuring BA can do business. BA can contract what they like out, and BA can hold who they like responsible, just as their shareholders are holding them responsible.
 
Flighty
Posts: 9963
Joined: Thu Apr 05, 2007 3:07 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 4:26 pm

BA had an incompetent IT executive who was too scared to test the recovery system because it was "mission critical" and frankly too complex/ill designed to cut over.

These things happen. Big airlines have a very complex IT system that must be live synchronized 24/7. For example, global payments systems have to synchronize with Rez which has to synchronize with alliances, GDS, operations, baggage, engine maintenance, customer loyalty emails. And many of those components are not "nice to have," they are mission critical essential. You never want your engine maintenance records to accidentally be a day old. You never want your payments system to go down. Or your GDS synchronization. Or your proprietary links to resellers and contracts. Anyway that is my soapbox speech. It is very complicated, in some cases life and death critical and only semi under the control of BA (hence it has to be robust to all external feeds going outside of norms). It's a hard job and big airlines routinely fail at it.
 
airbazar
Posts: 10859
Joined: Wed Sep 10, 2003 11:12 pm

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 4:32 pm

AngMoh wrote:
If anything went wrong with outsourcing, the problem is in the UK. You can outsource work but can not outsource responsibility.

100% agree.
I work in IT. When I started at my current company 15 years ago we had 0 workers in India. Today we have over 10,000 IT workers in India. We are a multi-billion (with a B) $$$ company. We have never had meltdowns or serious problems as a result of the offshoring. There are right ways to do it and wrong ways to do it but ultimately the responsibility is with those in charge and last I checked, BA was headquartered in the UK not in India. I don't know how BA is doing it but at my company the IT workers in India receive their guidance and "orders" from the U.S. So if they're doing something wrong it's because someone here in the U.S. told them to do it that way.
 
Amiga500
Posts: 2666
Joined: Tue Mar 03, 2015 8:22 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 4:37 pm

dtw2hyd wrote:
In theory it is possible, in real world, the only way to make sure data center fail-over works perfectly is thoroughly test when you implement a geographically dispersed data centers and test regularly because infrastructure is always changing.


I'm not even IT admin, I just help the guys out from time to time in between engineering - but even I'm aware of what you can do with virtual environments to test out infrastructure.



dtw2hyd wrote:
No one does backups now-a-days, data gets replicated real time to storage at secondary data center. Infrastructure is the least of the issues.


What kinda of cowboy are you? I take it you work for some commanche operation that has no need of meeting quality regulations?!?!

Duplication & Replication =/= Backup.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 4:49 pm

Amiga500 wrote:
dtw2hyd wrote:
In theory it is possible, in real world, the only way to make sure data center fail-over works perfectly is thoroughly test when you implement a geographically dispersed data centers and test regularly because infrastructure is always changing.


I'm not even IT admin, I just help the guys out from time to time in between engineering - but even I'm aware of what you can do with virtual environments to test out infrastructure.



dtw2hyd wrote:
No one does backups now-a-days, data gets replicated real time to storage at secondary data center. Infrastructure is the least of the issues.


What kinda of cowboy are you? I take it you work for some commanche operation that has no need of meeting quality regulations?!?!

Duplication & Replication =/= Backup.


LOL. Break a mirror/create a snapshot. Mount to backup host. De-dup to separate local storage. Sync the mirror back. Replicate to remote site. Retain per business/regulatory requirement. No cowboy ops here. This is how most modern infrastructures are designed. You have so called backups at both sites. This method takes less than 20% of your original used storage. No tapes at primary or secondary site or at a off-site storage location like iron mountain.

I am saying, you cannot mimic a test. If BA did a real test during planned downtime, we wouldn't be having this conversation.
 
dtw2hyd
Posts: 9100
Joined: Wed Jan 09, 2013 12:11 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:00 pm

moo wrote:
dtw2hyd wrote:
moo wrote:
Yes, but testing doesn't have to be destructive, you just design the infrastructure that way.

Just to be clear, are we having an academic or hypothetical discussion. I am sure BA and outsourced contractor did lot of non-destructive testing.

You can idle one engine and simulate IFSD or really shutdown and go thru relight process. Both are not the same.


You seem to be missing my point - I'm saying that you don't need to take the risks that you are professing exist, if you take the right approach in the first place.

dtw2hyd wrote:
Uh, you should still do back ups - there are many scenarios where replication off site isn't enough, for example in auditing and regulatory retention.

Implemented several tape-less backup recovery strategies meeting regulatory requirements. With current storage frame capacities and data de-dup and accelerator technologies there is no need for tape backups. Also major US storage vendor works real hard on golf courses with CIOs to make sure they don't buy tape libraries and buy few $$$millions worth of storage from them.


Did I say anything about tapes? Nope, so stop fixating :)

I was challenging the very premise that "no one does backups now-a-days" :)

dtw2hyd wrote:
We seem to be sitting on very different levels in this industry, because I'm at the level where secondary data centres purely for DR is the norm..

Not in the world where corporations are run by Willie Walsh alike.


Except that BA has spare capacity in three data centres across the EU...

dtw2hyd wrote:
Plenty of ways to mitigate that without issue...

Sure you can, only if you knew the IP was hard coded into the app.


Nope, you simply don't let it be an issue - there are millions of networks out there that use the same internal IP range and yet can talk to one another through the magic of the internet. The technology for the routing exists and is well tested.

And its trivial to scan a codebase for IP addresses - that and passwords are two things that you should be automating scanning for anyway... neither have any place in a codebase and should raise red flags.


Again either you are having hypothetical/academic discussion (or) you are very lucky. You haven't seen the application from hell.

Do you think BA and contractor didn't go through annual audit? They probably did pass with flying colors. "Only on paper". It is not just BA. Most corporations are in similar situation.

Unless you are privy to BA's IT infrastructure capacity, how can you claim there is spare capacity. With hardware virtualization and capacity-on-demand, there is no need waste that infrastructure. If there is a production fail-over, it gets priority.
 
User avatar
par13del
Posts: 11449
Joined: Sun Dec 18, 2005 9:14 pm

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 5:13 pm

airbazar wrote:
100% agree.
I work in IT. When I started at my current company 15 years ago we had 0 workers in India. Today we have over 10,000 IT workers in India. We are a multi-billion (with a B) $$$ company. We have never had meltdowns or serious problems as a result of the offshoring. There are right ways to do it and wrong ways to do it but ultimately the responsibility is with those in charge and last I checked, BA was headquartered in the UK not in India. I don't know how BA is doing it but at my company the IT workers in India receive their guidance and "orders" from the U.S. So if they're doing something wrong it's because someone here in the U.S. told them to do it that way.


Well I guess we are now getting into the details of what BA outsourced which can take many forms. Programmers for example usually expect you to tell them what to do (program), infrastructure outsourcing like server farms for example is different you don't need to tell them how to backup replicate or that you need access to your system.....
 
XXXX10
Posts: 707
Joined: Mon Jan 03, 2000 7:10 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:20 pm

I'm very impressed by the union!

Within one day of the faliure they have-traced the problem, investigated the cause and apportioned blame to the culprits.

Perhaps BA should have asked them to repair it!
 
indcwby
Posts: 336
Joined: Sat Dec 18, 2010 7:32 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:23 pm

Obviously sounds like if they had failover/backup/redundant servers at the same location of the primary server. Not the best way at all. Should always have a redundant data center in a different location, geographically, just in case of natural disasters. We did our drill to practice failover scenarios not too long ago because life happens.
 
WPvsMW
Posts: 2252
Joined: Thu Mar 23, 2017 7:30 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:32 pm

More from the link the Rev. posted:

"The explanation provoked further confusion, however, with IT specialists querying whether a power surge could cause such failure and electricity providers saying they had no evidence of such a surge. SSE said its network was operating as normal in the Harmondsworth area on Saturday morning.

A BA spokesperson said on Tuesday there was a “power supply issue” but that there still needed to be a full investigation. * * * Cruz ... said the outsourcing of jobs was not to blame for the “catastrophic” IT failure, ..."

Until an IT professional is allowed to speak, confusion will prevail. Marketing, finance, etc., types barely known SMS from email. Since the reports are "power supply" and "messaging", the inference I draw is two separate failures, fail-over "failed" in UPS messaging (therefore the served data center went down), and fail-over "failed" in switching BA's res, dispatch, and other comms to a live concurrent data center.
 
GDB
Posts: 15640
Joined: Wed May 23, 2001 6:25 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:43 pm

XXXX10 wrote:
I'm very impressed by the union!

Within one day of the faliure they have-traced the problem, investigated the cause and apportioned blame to the culprits.

Perhaps BA should have asked them to repair it!


Missing the point, their job is to represent BA staff, they warned the board last year that this loss of experienced IT professionals was a risk. Of course they would say that, however BA also got rid of local non BA contractors with a lot of history of working with BA. Tell me who has been proved right. To save £80 million BA could be looking at a hit of £500M. What a kick in the teeth to ALL staff.

I am not an IT person but if those who are could comment on this;

http://up2v.nl/2017/05/29/what-went-wro ... atacenter/
 
XXXX10
Posts: 707
Joined: Mon Jan 03, 2000 7:10 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 5:53 pm

My point was, that although what they have said may turn out to be accurate, it can surely be no more than a guess given the amount of time betwewn the event and the statement.

I wonder if we'll ever know the real facts.
 
fcogafa
Posts: 1434
Joined: Fri May 16, 2008 4:37 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 6:05 pm

The telegraph is reporting the 'full details', the system was powered up too quickly after the failure, which caused everything to go tilt. If it had been done slowly they could have handled it better
 
SATexan
Posts: 294
Joined: Wed Jul 22, 2009 6:49 pm

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 6:11 pm

Airline system backends in general are antiquated. I was at PHX airport last Tuesday and the AA customer service systems were down for a good portion of the day with no ETA. I was told that even their main reservations system was down. The lines at customer service were fairly long and there wasn't a damn thing that the customer service could do. They kept telling "check your mobile apps - may be that is working". So the fact that systems as critical as airlines don't have downtime or 100% reliable is codswallop.

moo wrote:
You also seem to ignore the fact that if their DR was done correctly, there would be no need for panicked responses or rushed solutions - they would fail over onto an alternative environment and run on that for the duration, the original fault could be rectified and then you move back onto the primary environment (should you be doing warm-standby rather than hot).


I am telling you the same thing. That their DR exercise was not done correctly. Where I disagree is in lumping all of that failures on the offshoring company. I was suggesting in general how many companies cut costs and do not establish a full fledged DR environment and also that BA has a lot of responsibility with regard to the current failure.

moo wrote:
I don't get the contrived problem here?


The problem here is that data encryption/decryption and a transfer of massive data in real time is time consuming. It is entirely possible that BA (or their vendor if you wanna blame them) did not test their DR systems adequately when they did their Business continuity exercise. Besides, those decisions to multicast to other data centers are to be made by BA. No contracts should allow third parties to instantiate recovery centers without oversight.
 
airbazar
Posts: 10859
Joined: Wed Sep 10, 2003 11:12 pm

Re: BA Union Blames Meltdown on Indian Workers

Tue May 30, 2017 6:30 pm

par13del wrote:
airbazar wrote:
100% agree.
I work in IT. When I started at my current company 15 years ago we had 0 workers in India. Today we have over 10,000 IT workers in India. We are a multi-billion (with a B) $$$ company. We have never had meltdowns or serious problems as a result of the offshoring. There are right ways to do it and wrong ways to do it but ultimately the responsibility is with those in charge and last I checked, BA was headquartered in the UK not in India. I don't know how BA is doing it but at my company the IT workers in India receive their guidance and "orders" from the U.S. So if they're doing something wrong it's because someone here in the U.S. told them to do it that way.


Well I guess we are now getting into the details of what BA outsourced which can take many forms. Programmers for example usually expect you to tell them what to do (program), infrastructure outsourcing like server farms for example is different you don't need to tell them how to backup replicate or that you need access to your system.....

Yes you do, if you don't want to have any meltdowns. Otherwise, good luck.
Every process no matter how insignificant it might seem, is thoroughly documented, reviewed, tested and reevaluated at a regular interval. It doesn't matter whether the "operator" is in the U.S., India, or China. Everyone has to follow the same processes. For something as critical as disaster recovery there are even external auditing companies that come in every so often to make sure we have all of our ducks in a row. That is what companies who are serious about their reputation, do. Mine is a financial company and you just don't mess around when it comes to dealing with other people's money.
 
User avatar
vegas005
Posts: 324
Joined: Mon Mar 21, 2005 12:25 am

Re: BA Union Blames Meltdown on IT Outsourcing to India

Tue May 30, 2017 6:47 pm

Stupid is as stupid does---saving a few pennies is a really big mistake. Believe me when I tell you airline ticketing is COMPLICATED and there are a few great systems available and a lot of iffy crap. BA made their bed and have to sleep in it.

Who is online

Users browsing this forum: 380Capt, DNDTUF, INFINITI329, KIRFlyer, PhilipBass, pspfan, PvdE, SaabFA71, SIVB, Spiderguy252, ZK-NBT and 143 guests

Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos