header-image

Current Affairs Discourse

Blog about Current Affairs

Current Affairs Discourse

Flight Control Systems and Systems Integration issues on Boeing 737 Max 8?

  April 2, 2019 April 2, 2019   By:   Autonomous, Root Cause Analysis, Safety and Security  , , , , No Comments

Update 7

The offset was 60 degrees between the two AOA sensors in the case of Ethiopian Air after an incident during take off? Bird strike?

Plus the following…

 

Cutting out the power to MCAS motors did not fix the steep dive? What does that mean?

The Integration issues:

Flight control systems issue: Major upgrade in the works. Plus AOA sensor input to MCAS is not the same on all aircrafts?

This is Epic systems design and implementation failure possibly?

Boeing 737 Max 8 MCAS – Is the proposed Fix good enough?

  March 29, 2019 March 29, 2019   By:   Autonomous, Root Cause Analysis, Safety and Security  , , , , , No Comments

Swiss cheese. You hear that a lot in systems design and engineering.

It is often not possible to fail safe a system with 100% certainty.

Especially true when there is an unforeseen and/or unintended consequence and a fix is proposed to a problem.

Root cause may not be known with certainty, however, a fix could be put in place as a short term remedy to alleviate the cascade catastrophic failure.

Swiss cheese.

If you stack up a few of them, then the likelihood of a through and through hole is quite low.

In fact, the entire aircraft systems design is based on this principle. With several redundancies and backup and backups for backups.

Engines being the most obvious example. There is computers, sensors etc that also have redundancies. Which is why the 737 Max and other Aircrafts cost $100Million.

So why then was this Autonmous MCAS System basing it’s entire activation on a single sensor input?

The proposed fix:

It appears that Ethiopian Air AOA sensor offset was also 20 degrees!!

What does that imply? MCAS won’t activate with 20 degree offset with the above fix.

However, it doesn’t fix the root of the issue. What is causing this offset?

Could there be a deeper issue here which could be impacting other systems as well?

Could this be more than the unintended consequences of the MCAS algorithm discussed here:

MCAS Algorithm & The Unintended Consequences – Boeing 737 Max

 

To be continued…thoughts in progress.

MCAS Algorithm & The Unintended Consequences – Boeing 737 Max

  March 18, 2019 March 18, 2019   By:   Autonomous, Regulatory Issues, Root Cause Analysis, Safety and Security  , , , , , , No Comments

An Exclusive by Seattle times reporter Dominic Gates has detailed some of the lapses in the design of the Boing 737Max8 MCAS control algorithm.

It raises several questions about flaws in the design, design process and safety reviews.

My hypothesis published soon after Lion Air JT610 crash is along the similar lines.

Lion Air Flight 610 Boeing 737 MAX 8

I will try to answer some of the questions raised in the Seattle Times article and also in my previous blogs about this issue since we now have more information.

 

  1.  Why was the MCAS algorithm put in place?

The algorithm was required to get FAA airworthiness certification for Boeing 737Max due to possibility of high speed stall. This situation was determined to be possible by the nature of the design of the aircraft with bigger engines and different mounting locations. Possibly via CFD analysis etc.

Possible. But how likely was it?

2. So what went wrong?

Sensors. Angle of Attack (AOA) sensors to be precise. It appears that either one or perhaps both of the AOA sensors were compromised. The MCAS is an algorithm. It needs input from the sensors to act. If the data is flawed, it will act in unintended operating envelopes. Unless of course, it is designed to account for that. Which it wasn’t perhaps?

3. Why was the sensor data flawed?

There could be several reasons for this. Let us see what is believed to have happened In the case of Air France flight 447 crash in 2009.

In that case (Airbus Air France Flight 447) it was determined that the Pitot Tubes (Speed sensors) were flash frozen due to super cooled condensation in the air. This caused the aircraft to stall. To bring it out of stall, the pilots needed to pitch the nose down. However, the stall alarm algorithm had a design flaw. It would come on when nose was pitched down and turn off when nose was pitched up since in the up condition with zero speed it assumed the aircraft was on the ground. So the pilots got confused.

In this case (Boing 737Max), the Lion Air FDR showed that there was an offset between the two AOA sensors. Lion Air Aircrafts do not have the optional AOA sensors mismatch light. However, the previous flights of the same aircraft had encountered similar flight control problems and it appears the maintenance crew determined that the sensor(s) were bad and replaced them.

The optional light just alerts the pilot to possible issue. Doesn’t impact the operation of the MCAS algorithm.

4. So if the sensors were replaced, then how come they went bad again?

Now it is possible that the new sensors were also bad. How likely is that? So possibly there is something else going on. A system issue (raised in my Lion Air Post above).

5. So how come just one bad sensor out of the two caused MCAS to trim down?

That is the big question. My initial thoughts on this was that MCAS was simply selecting the worst case/input. It appears that is not the case. It is only getting only one input? If this true, then why have two sensors?

My guess is that there is some pre selection of the signal via signal conditionining ASIC or ECU before feeding the input to MCAS algorithm ECU. The two ECU’s might not be the same – the signal selection and MCAS algorithm ECU. There could be several reasons for this.

Would be interesting to know the ECU vintages and commonality with prior and/or other 737 models. This could be the smoking gun in my opinion.

6. Why doesn’t MCAS look at other inputs and conditions and determine validity of the sensor signal before acting on it?

Good question. It should. Should have been flagged in safety reviews. Seattle Times article has good info on this.

7. What about how far it can trim and how much?

This is an  aeronautical engineering question. Outside my scope of expertise. Seattle Times Dominic Gates articles covers this well. I will add this though:

The trim limits and activation thresholds should have been engineering calibrations in the algorithm. It would have made it possible to “tune it” for each aircraft  variations.

 

The FAA CANIC (Continued Airworthiness Notification to International Community) published does seem to (?) address all three issues (5,6,7). See my Ethiopian Airlines post on this.

Ethiopian Airlines ET 302 Boeing 737 Max 8

8. So if the fixes are in the works, we should be good right?

No. It isn’t just about this MCAS feature anymore. The whole design and development of this aircraft and possible all aircrafts at Boeing is suspect now. It is imperative that it be investigated properly. In the works maybe?

Who is really incharge of the Boeing 737 Max investigation?

Update/Additions 1

Just came across this. Even Airbus Aircrafts have anti stall systems which get triggered falsely due to bad AOA sensors!!

Update 2.

Interesting twitter conversation with Peter Lemme (interviewed for the Seattle Times article above) back in November.

Note: Will update as needed.

Who is really incharge of the Boeing 737 Max investigation?

  March 17, 2019 March 17, 2019   By:   Root Cause Analysis, Safety and Security  , , , , , No Comments

Update: DOJ & US Department of transportation has launched an investigation into FAA’s approval of Boeing 737 Max airliner.

While the FAA, NTSB, BEA, Boeing and Ethiopian Airlines are investigating the ET302 crash and how it maybe related to Lion Air JT601 crash in Indonesia.

So far, we have had Ethiopian Airlines send the Black Boxes to BEA in France which seemed to imply that BEA might be leading the investigation.

However, it appears that is not the case. BEA simply downloaded the data and transferred it to Ethiopian airlines.

In the meantime, we have a leak from Air traffic control in Addis Ababa where a source provided information from the voice recorders to a Reuter’s reporter about the conversations with the pilots of flight ET302.

Boeing 737 Max investigation compromised by data breach?

So there appears to be some confusion as to who is really incharge of this investigation?

What about all the physical evidence from the crash site?

Without a proper chain of command, delegation and and work protocols, the likelihood of successful investigation is very low.

The Air France Flight 447 investigation was also bogged down due to these very reasons. Besides the fact that it took 2 years to find the black boxes.

Someone needs to step in and say, we are incharge and publish a document that shows the working relationships and work protocol between all these agencies invloved.

Boeing statement seems to imply it is NTSB. However, based on what has happened so far, that does not appear to be the case. Is that how it was handled in the case of Lion Air JT610 investigation? Via NTSB?

Perhaps the US Department of Justice and Department of Transportation investigation will help clear up the confusion?

 

Boeing 737 Max investigation compromised by data breach?

  March 17, 2019 March 17, 2019   By:   Root Cause Analysis, Safety and Security  , , , , , , No Comments

Updated.

Reuters aviation reporter Jamie Freed has published an exclusive based on a source involved in the Ethiopan Airlines ET302 investigation being assisted by BEA in France.

Hope this doesn’t mean the data is compromised. FDR data transfered (?) soon after I sent the above response.

Not sure why BEA decided to block me on Twitter. Got misinterpreted as malicious perhaps? It wasn’t. If it were me, I would say thank-you and investigate the leak. It could be anywhere in the Chain of possession but obviously after the data extraction?

Meant to say chain of possession not chain of command. Tried to point that out which is when I discovered I have been blocked.

Update 1

It has been pointed out that the source is air traffic control voice recordings and not from the Black boxes (CVR and FDR). So perhaps the data in the black boxes is not compromised. However, the airtraffic control data is still crucial part of the investigation and needs to be secured as well.

It still isn’t clear as to who is leading the overall investigation and where?

BEA appears to have simply extracted the data and sent it to Ethiopan Airlines. Very different from Air France flight 447 investigation where BEA led the entire investigation.

 

Ethiopian Airlines ET 302 Boeing 737 Max 8

  March 10, 2019 March 10, 2019   By:   Autonomous, Safety and Security  , , , , , , , , No Comments

It has happened again. Another catastrophic crash of a Boeing 737 Max 8 albeit with a different airline.

The MCAS feature is like those garage door sensors. Trying to mitigate a scenario that is possible but not likely (stall). However, it has lot of false positives. Detects obstructions all the time and kicks back the garage door for no reason. Unintended consequences. Assuming that what this is. Another false trigger of stall logic triggering MCAS steep dive.

The previous incident with Lion Air Flight 610 is detailed here:

Lion Air Flight 610 Boeing 737 MAX 8

New York Times is reporting that there were changes made to the control algorithm. Not sure what the reference is to. Original MCAS algorithm or some changes since the Lion Air Crash?

I sure hope the MCAS algorithm was turned off at the least.

Updated Statement by Boeing:

Reuters article on this:

Flight Radar 24 Tweet/blog info:

Stakes are high here for Boeing, the Airlines and passengers.

New York Times is reporting that all Chinese Airlines have stopped using the Boeing Max 8 effective immediately.

Confirmation from WSJ that China is grounding all Boeing 737 Max 8.

What is SouthWest Airlines in the US going to do? Can they turn off the MCAS algorithm effective immediately?

Something to think about though. Given service bulletin’s were sent to all Boeing 737 Max 8 pilots with details on how to override the MCAS.

https://twitter.com/processisinc/status/1104946625938669571?s=21

Indonesia follows China and grounds Boeing 737 Max 8 fleet. More to follow?

Just came across this interesting Seattle Times article. Concurs with my thesis.

FAA will not ground the Boeing 737 Max in the US. Mandates software fix to MCAS by end of April. However,

FAA statement and Continued Airworthiness notification to international community (CANIC). Design change is three pronged for MCAS algorithm.

  1. Activation enhancement
  2. Signal strengthening, and
  3. command limit.

The activation enhancement and command limit can and should be Engineering calibrations.

Signal strengthening requirement implies this is believed to be S/N deterioration issue.

It appears the signal enhancement/strengthening might be requirement to use inputs from both AOA sensors in tandem and modify the algorithm to accommodate all inputs to arrive at a best best possible value for AOA instead of just assuming the worst input as true. This is my educated guess. Since I do not have any inside information.

Updated statement from FAA below. Might revisit the Boeing 737 Max decision if Ethiopan Airlines flight ET302 data shows anything new.

Few questions to consider:

1) Did ET302 have the AOA sensor light? Did it ever come on?

2) JT610 did not have the light.

3) It appears SouthWest and American Airlines have the light

4) what are the airlines doing differently?

5) MCAS does not need the light. It knows there is a mismatch between the two AOA sensors. Needs to figure out the best possible course of action. Being addressed via the Software update in April (see above CANIC)

Grounded!

Ethiopian Airlines has chosen to send the black boxes to BEA in France. Interesting development given that Air France Flight 447 crash investigation by BEA led to the following  fixes in Airbus:

 

  1. Stall “Alarm” algorithm logic redesign and new software
  2. Pitot tubes redesign and new supplier
  3. Cockpit protocol changes
  4. Pilot training

The Boeing 787Max issue is now linked to its MCAS algorithm which is an anti stall system!!

To be continued…

Lion Air Flight 610 Boeing 737 MAX 8

  January 14, 2019 January 14, 2019   By:   Autonomous, Regulatory Issues, Safety and Security  , , , , , , , No Comments

I have been following this incident since October 2018. Reading and commenting on articles in New York Times and Wall Street Journal using Twitter thread to sort of catalog my findings.

The Flight Data Recorder was found back in November which enabled the investigators to piece together what may have happened to this aircraft.

It appears that a safety feature designed to prevent a stall condition seen in Air France flight 447 (an Airbus A330) crash created an unintended consequence of deep dive due to the perceived failure of the similar sensor as in the AF447 incident (pitot tubes) albeit a different failure mode! A functional safety failure.

Boeing may have finally realized that the algorithm design might be flawed and appears to be in the process of making design changes.

The cockpit voice recorder has been found now as per news reports (January 14, 2019). The WSJ article on this is reporting that there might be an issue with calibration update in the field.

My thoughts on this calibration issue (further elaboration of my Comment posted on WSJ article):

Field calibrations (or Service Trims as they are called sometime) can be changed in the field using service tools used by trained technicians. However, it does require a protocol, training, certification for techs etc. Usually all these are developed by engineers.

 

There was something wrong with one of the sensor inputs here. Replacing the sensor did not fix the issue. This happened several times. So most likely it wasn’t the sensor at fault. It appears to be a 20 degree offset.

Could be a mounting, wiring, signal conditioning instead of a sensor to sensor variation. Calibration fix while possible, a field technician would be not be capable of doing this. Unless of’course this was known issue, a protocol has been developed for it, it was in place, technicians were trained for it etc.

All this does not preclude the fact that this was just a poorly designed safety feature for the following reason:

1) Why would the MCAS algorithm decide to act on flawed sensor data? Especially when there is discrepancy between the two sensor inputs.

2) Why would the MCAS algorithm automatically put the aircraft into steep dive and then ignore pilot attempts to pull the nose up?

3) Why does it require the pilot to turn off the feature by manually disabling the control system?

 

Also, questions have been raised about pilot training and training manuals.

How do you write a training manual for a flawed design which may not be an intended feature?

You can’t. Which explains why there wasn’t a training protocol for this.

I hear Boeing is making design changes to the MCAS algorithm which once released would require a software update to the ECU in question.

Needless to say before making any such changes a comprehensive functional safety review of the entire system is warranted using methodologies like Failure Modes Effects and Diagnostic Analysis.

 

 

Time to Regulate the Autonomous Vehicles and Ride Share

  October 11, 2018 October 11, 2018   By:   Autonomous, Regulatory Issues, US  , , No Comments

It is time to regulate the autonomous vehicle and autonomous ride share services space.

Similar to the regulations we have in the healthcare space via FDA. Perhaps even more critical here since the repercussions are lot more severe.

In the medical field, the failure of a product or service only impact the patients receiving that product or service (NCDs).

Here, an autonomous vehicle has the potential to cause havoc way beyond just the person(s) receiving the service at the time of service delivery.

What business do non automotive manufacturers have selling an autonomous vehicle or a service?

They are basically modifying a vehicle with their own technology and are selling a service using a product that has been tampered with.

Who is regulating this?

Automotive manufacturers have decades of experience in bringing fail safe systems to market. The app makers don’t.

It is simply impossible to develop fault tolerant mission critical systems in overnight hackathons. It is a iterative process. Takes time to evolve.

So for starters, why not mandate that technology providers have to work with OEMs and only an OEM can sell an autonomous vehicle. A service provider can buy the certified vehicle from the OEM to sell a service.

This way the OEM is responsible;y for the system, and there is no finger pointing when there is an incident. This will make the entire autonomous ecosystem a lot safer.

The autonomous ecosystem includes smart infrastructure as well.  This needs to be developed in parallel.

Obviously the tech providers have to work with the public sector on developing the smart infrastructure.

The autonomous vehicles will be crucial part of the autonomous smart infrastructure.

It is imperative that its evolution be managed in controlled manner by working with the OEMs similar to what we do with infrastructure via the public sector.

The only way it will happen is via regulation. Need someone in US Congress (House or Senate) to sponsor the bill. We know it won’t be anyone from California for sure.

Why am I cancelling my Tesla Model 3 Reservation?

  August 31, 2018 August 31, 2018   By:   Culture  , , No Comments

 

Why am I cancelling my Tesla Model 3 reservation?

I ordered my Tesla Model 3 on December 27, 2017 after all the initial order rush was over. The email said Delivery estimate was mid 2018.

I didn’t believe it and was prepared to wait until 2020 to get my $35,000 Tesla Model 3 with the additional $7,000 tax credit which would bring my effective cost down to $31K with the sales tax factored in ($38K – $7k). Would be cash purchase so no financing cost etc. Don’t believe in borrowing money to buy a depreciating property – A topic for another conversation perhaps.

 

Then few months into 2018 I got another email saying the new estimate was late 2018. Really? Didn’t believe it. Still waiting for 2020.

Then came the great reviews of the product from early adopters and some consultants which added to the anticipation. Then the horror stories about bottlenecks, delay’s, quality problems, scrap rate etc. All understandable and to be expected since since we have a new product, new process, new supply chain, new robots, new software etc.

However, the car runs on Linux. Not acceptable to me as an embedded systems engineer in my early career. This is the reason I waited to order a Tesla since my expectation is that by 2020, Tesla will run on new Nvidia GPU that runs on QNX.

 

That and the fact that new products, especially cars, take years to mature to a point where the defect rate stabilizes to a manageable level. The 2020 Tesla’s shouldn’t have the durability issues that the 2013 Tesla’s are having now. Key word being shouldn’t.

 

However, it appears there is a huge culture issue at Tesla that permeates the entire organization driven my Mr. Elon Musk’s management style or lack thereof.

 

Details: (paywall) -> https://www.wsj.com/articles/elon-musk-faces-his-own-worst-enemy-1535727324

This will prevent Tesla from addressing the product issues and slowly improving the reliability (infant mortality) and long term durability issues.

 

Now that is a deal breaker for me. It doesn’t help that the $7,000 tax credit will expire soon as well, however, that was not a deal breaker.

 

Can this be fixed? Maybe. But not by 2020.

Also, by then there will be other choices in this space from competitors that are laser focused on making the cars instead of being distracted by flamethrowers, short shorts, submarines, rockets etc.

Now this (hobbies) would be ok if there was trust and empowerment to senior managers that manage supply chain issues, operations etc. There isn’t.

 

Making reliable and durable cars profitably is lot harder than it appears….

 

Whole Foods HealthCare?

  January 30, 2018 January 30, 2018   By:   Health Care, US  , , , , , No Comments

Well folks, someone had to take this on.

So why not a consortium of Amazon, Berkshire Hathaway and J.P. Morgan Chase to try to figure out and tackle the intractable Health Care problem in the US.

At the gist of it, the problem on supply side of the equation involves the incentive system.

The incentives of the service providers (hospitals, physicians, labs etc), the payers (private insurance, Medicare, Medicaid) and the pharmaceutical industry are not aligned with the real overall long term well-being of the patients.

On the demand side of the equation, patients themselves are not empowered, educated or incentivized to take charge of their own health.

Large corporations are usually self insured. Meaning that the claims are paid by the corporation. However, they do hire a service provider to manage the healthcare plan. Like Anthem, United Health, Aetna etc.

So in this case, this consortium of Amazon, Berkshire and JPMorgan, will instead create its own administrator as a non profit.

Second, They will also most likely be a provider as well by building their own hospitals and hiring staff. Could be a non profit as well.

Third, the patients (the employees of the three firms to begin with) will be incentivized to really care for their own health for the long term.

Last but not least, this consortium will roll out this system to other interested corporations or even patients directly. How about $2,000/year per person for continuous monitoring of ones  health and proactively addressing issues?

Whole Foods might be leveraged to be part of this solution. Perhaps ship stuff on demand to address issues identified by this self monitoring system.

 

Now we know the real reason for why Amazon acquired Whole Foods.

 

Health Care Maintenance As a Service offering once the service matures in its internal operations in collaboration with Berkshire and J.P. Morgan and Chase.

This is Very similar to the strategy Amazon used with AWS. System was first developed for its own retail operations and then sold as a service to others. Now half the internet is hosted on Amazon including Netflix!