Back to Squawk list
  • 41

FAA orders new 787 electrical fix to prevent power failure

Submitted
 
All Boeing 787 operators will be required to periodically deactivate the electrical system to avoid a problem with a newly-discovered software bug that could cause the aircraft to lose alternating current (AC) power, the US Federal Aviation Administration says in a new airworthiness directive. The agency adopted the final rule after Boeing reported the results of a laboratory test showing a total loss of power is possible if the generator control units run continuously for eight months, says the… (www.flightglobal.com) More...

Sort type: [Top] [Newest]


wayne007
Wayne Jeffrey 5
Why do I think of "The IT Crowd" when I read this?

"Have you tried turning it off and on again"!!!
preacher1
preacher1 7
Can we say "REBOOT".
vector4traffic
So it's possible that the electrical system was on continuously even during the various maintenance checks? Come on ...
jbqwik
jbqwik 1
Good point. I don't totally buy that, either. That being said, I have worked on systems that have ridiculously onerous "Keep Alive" power schemes -PITA to bypass.
vector4traffic
Is it even normal operating procedure to have the APU on while towing the plane into the hangar for the check?
xairbusdriver
Jim Smirh 1
[Is] the APU on while towing the plane?
That would sound like normal ops to me. I wouldn't want to depend on battery power to run the auxiliary hydraulic pumps to stop the rather large aircraft when the tow bar breaks! I'm sure there is a brake hydraulic accumulator, but why take chances! Maybe a reply from some 787 mechanics would be nice...
jbqwik
jbqwik 1
APU doesn't need to be running; the LiON pack has substantial capacity. You'd have to ask someone familiar with that model's power distribution / power management scheme. I've seen a basic flowchart and remember thinking "good luck!".. ..
btweston
btweston 1
You're right. I'm sure Boeing made it all up...
mhlansdell00
Mark Lansdell 1
If I read the article correctly, it said it MAY happen. It's my guess from that, that it has not happened yet but might under the right circumstances. The article doesn't talk about the necessary shut down time. It can't be determined from what's written whether the system has to be shut down momentarily or for a long period. From what others have written the "clock" has to power down in order to reset to zero or a positive value starting the cycle over.
sparkie624
sparkie624 1
You are right... with the term "It May" tells me that this happened in a test environment, however, I can't imagine why you would test a plane that has been powered fro that period of time...
randomguy
randomguy 2
Here's the link to the AD.

https://s3.amazonaws.com/public-inspection.federalregister.gov/2015-10066.pdf

It came up during Boeing lab testing.
sparkie624
sparkie624 1
Thanks for posting... The reset procedure will take less time than it will take to complete the paperwork... Interesting task, but it baffles me that they say they do not have to disconnect the battery.... The procedure must disrupt the power to the Hot Battery Buss.
joelwiley
joel wiley 1
Sparkie624, I don't know airplane innards, but do know something about computers. I read the AD and it sounds like they do a system reset for the software. Sort of like rebooting your laptop. I noticed the AD related that for the event to occur all four units had to be started at the same time. Is that a normal procedure?
sparkie624
sparkie624 1
On aircraft that I have worked on no... But apparently these GCU's have a reset button on them... Similar to our home computers... You would simply to to the units, push the button, and wait for the diagnostic to complete and the system comes back on line.. Easy task.
BaronG58
BaronG58 1
"to to the units" Never done that before..sounds like fun. 8--)
mhlansdell00
Mark Lansdell 1
I'm not so sure about that. Again, these articles leave a lot to speculation. If they wrote it all down there would be nothing for folks like us to talk over and get into arguments about. :-) I don't have the maintenance expertise that you have and I look forward to reading what you have to say. You and Preacher are rocks on this board and I learn something every time I read your postings. I think that's the way it was designed, so it's working.

The article did say exactly what you did but never asked the question. That's the difference between a "mechanic" and an "engineer", practical and ethereal that's not to say we can't have a little of both. I was glad to see you join this fray and we are both asking the same questions.
sparkie624
sparkie624 1
When they use the term "may" it tells me that it was not an actual occurrence, and that it was either a computer simulation or they set up a test bed on a bench and produced this finding. Just speculation... If it had happened on a live aircraft then they would have said that it did, not that it may... Also, think about your home computer.... Keep it powered for 248 days continuously working, Do not shut it down, and never restart it.... How well do you think it would work after 248 Days... My main computer that I use has been up for 4 days, fr my servers... For my servers they are 18, 18, 23, 18 Days respectively (I do not have your average home setup). The most that I run my servers before reboot is about 90 days...
mhlansdell00
Mark Lansdell 1
That's exactly the way I read the article. I think it actually described the test and scenario. I could go back and copy it but I don't feel the need to with you. May is always a "definite maybe" to me. There are times when I don't read as carefully as I could or should, but I can always depend on someone to catch my bads.
sparkie624
sparkie624 1
Yup... Like I have always said "it has a very high probability of a definite maybe!"
mhlansdell00
Mark Lansdell 2
Nice to meet you. :-)
sparkie624
sparkie624 1
Same here... I have made some great friends here..
mhlansdell00
Mark Lansdell 1
I feel like I just made another one
joelwiley
joel wiley 3
How many aircraft go 284 days without being shut down? When I first heard of this directive, it reminded me of something that happened with the Patriot missile batteries back in Gulf I. If I recall correctly, an internal floating point clock would gradually lose synchronization with real time over the span of a week or two. The developers never considered running the systems 24/7 for weeks. End points are always a challenge in programming.
sparkie624
sparkie624 1
Good point... LOL, the Electric Computer Planes needs to be shut down from time to time for reset, but do keep in mind that even when the aircraft is shut down, as long as the a/c has a battery connected onboard, the a/c is technically powered... I have seen times when we have to disconnect the a/c batteries for 20 minutes to allow the a/c to accept AC power.
Moviela
Ric Wernicke 6
A stack overflow in an aircraft power system? This is what happens when the sharp pencil boys (accountants) are allowed to influence good engineering practices. So they trimmed the local software engineering staff to the bone, then hire H1-B visa holders on the cheap.

These people work hard, but lack the chops to elegantly write code that works. You just can't expect that from graduates of the Institute for Computer Science, Cosmetology, and Small Engine repair.
jbqwik
jbqwik 3
I'm with you. Would be laughable except Boeing (and, you know, that *other* agency wink wink, nod nod ) knowingly allowed this to fly. Just more fuel for those pointing to the management hubris that has been the recent hallmark of this company.
bishops90
Brian Bishop 1
The H1-B people who work for us are more highly educated than most US Person candidates we can find, and due to ITAR can't legally work in a number of those positions anyway. Our people don't write software, but we deal with a lot of them in design and FEA and CFD work.
tcmarks
Tim Marks 2
Please, someone provide an example of any aircraft that has ever been run continuously for 284 days? Anyone? That is the AC generators mounted on the engines cited in the AD that have to run continuously and has occurred only under lab conditions at Boeing. This is another example of sensationalism on the part of the FAA and the news media. And BTW, losing the AC generators on the 787 will not cause 'loss of control' when there is hard wired DC bus power from the battery system and the Ram Air Turbine (RAT) that will automatically deploy as the last resort to provide power to primary systems. So no, the 787 will not fall out of the sky with this rediculous scenario defined by the FAA.
linbb
linbb 2
But but but that's beside the point you are missing the sensational
headlines that make news print!!!!!!!!!!!!!! LOL. You are so right
thanks for the post.
mhlansdell00
Mark Lansdell 1
The article fails to say how long the system has to be shut down. That alone would tell you how simple or complex the procedure is.
preacher1
preacher1 3
It don't say how long it has to be down for. I just says that a problem MAY develop if units are kept up for 8 months continual. Even most severs need to be rebooted after 6 months.
mhlansdell00
Mark Lansdell 1
All true. I guess my point was too subtle. The article was horribly incomplete and could have easily included the sequence timing. It's one thing that it would have to be shut down over 4 or 8 hours it's quite something else if it's a reboot and only momentary.
akayemm
Experts engaged in crop dusting know a lot about killing bugs and insects. They may help in 'killing' this software "bug" ... ha ... ha
btweston
btweston 1
I guess you'd call that a good catch.
larrykoch
Larry Koch 1
The bigger question is, why a counter is necessary. There must be a good reason why the designers felt the generators should not be operated more than 248 days continuously. I can't imagine a programmer inserting a kill-counter that was not included in specifications.
randomguy
randomguy 2
Likely a clock. And the 248 days is roughly a 32-bit integer worth of milliseconds.
dbaoracle
They used the word "overflow". This sounds like a software bug where some value was, for example, defined as an integer with a max value of 32767. When the i t hits 32767 it "overflows" and recycles the number. This can cause all sorts of software issues. Also there have been some pretty famous x number of days kinds of bugs with OS software.
randomguy
randomguy 1
It is likely a 32-bit signed integer counter, probably running a clock of some sort.

100 ms/second * 60 seconds/minute * 60 minutes/hour * 24 hours/day * 248 day = 2142720000 ms.

2^31-1 = 2147483647
MrAflac9916
Mr Aflac 1
This is why all planes should have mechanical backup. If the computers fail, there needs to be a backup system.
linbb
linbb 1
I agree am on a homebuilt web site and those building new ones have all of the
great stuff and latest but not one steam gauge that could save ones life if the battery
fell out of it, I know its extreme but get the drift. Also when looking at those
screens seems would take more time than a scan of needles from steam gauges. Don't
have any time in the new ones or have flown in many years either.
sparkie624
sparkie624 1
That is the reason that I do not like airbus and it's computer system... It is a bad enough to have fly by wire, but having a computer between the stick and the flight control is too much.
johnnyaviator
johnnyaviator 1
It's One thing to read something on a paper that wants to sell and it's a totally different thing knowing and understanding the potential danger if any! It is not just a coincidence that after millions of flight hours by all operators no one has experienced it yet. Every 5 to 6 legs all the airlines for one reason or another they cycle power. No need to worry people ... relax!
MH370
MH370 0
(Duplicate Squawk Submitted)

Updated: FAA finds Boeing Dreamliner could lose all power, issues maintenance mandate

The Federal Aviation Administration on Friday issued a directive mandating "a repetitive maintenance task" for that model of airliner due to issues with its power supply. Specifically, the FAA explained testing revealed that 787s could lose all AC electrical power after being continuously powered for 248 days, a problem that, if left unchecked, would leave an aircrew unable to control the plane.

http://www.cnn.com/2015/05/02/us/boeing-787-dreamliner-faa-directive/
MH370
MH370 -1
(Duplicate Squawk Submitted)

FAA warns Boeing 787 bug could shut off aircraft power

AFP. In a new problem for the Boeing 787, the US aviation regulator has ordered repairs to correct a software bug that could cause the aircraft to suddenly lose all power.

The Federal Aviation Administration issued a directive dated Friday warning that after a 787’s generators have run continuously for 284 days, they could abruptly shut down, leading to a loss of aircraft control.

The FAA said Boeing itself identified the problem: an internal software counter in the generator control units (GCU) will overflow after 248 days of straight use.

http://www.aviationnews.eu/59870/faa-warns-boeing-787-bug-could-shut-off-aircraft-power/
vdi74
James T -1
(Duplicate Squawk Submitted)

FAA: Boeing 787s need to be powered off every 120 days

Boeing is working on a software fix for 787 generator control units (GCUs), but in the meantime FAA has ordered 787 operators to power off Dreamliners at least once every 120 days to avoid the potential loss of electrical power in flight.

During laboratory testing, Boeing discovered a problem with 787 GCUs: After 248 days of continuous power, all four main GCUs will go into failsafe mode simultaneously, which would result in the loss off all electrical power. Boeing advised FAA of the problem and the agency has issued an airworthiness directive (AD) with immediate effect that requires “a repetitive maintenance task for electrical power deactivation.”

http://atwonline.com/components/faa-boeing-787s-need-be-powered-every-120-days
larrykoch
Larry Koch -1
The bigger question is, why a counter is necessary. There must be a good reason why the designers felt the generators should be shut down after 248 days of continuous use. I don't see this as an IT problem as much as a design issue. I can't imagine a programmer inserting a kill-counter unless is was in the design specs.
TorstenHoff
Torsten Hoff 1
The generators weren't designed to be shut down after 248 days of continuous use -- they malfunction because a 32-bit counter that represents 100 millisecond intervals will overflow after 248.5 days. Whoever wrote the software probably thought "Hey, nobody will ever run this thing for that long, so I don't need to guard against an overflow condition."

As far as why a counter is necessary, measuring time is useful for many things, it helps answer questions such as "has the device entered the correct state within the expected time after receiving a command". You would also want to measure time for keeping track of maintenance intervals.
mhlansdell00
Mark Lansdell 1
Let's say all that is so. Why "create" a failure in the first place. Does a counter have to overflow? There must be a reason that someone at Poeing thought it was a good idea to shut down the system entirly. These articles leave a lot to imagination and conjecture.
TorstenHoff
Torsten Hoff 1
The counter has to overflow eventually because it can only represent a fixed number of values. Once it reaches the largest number it can, incrementing it again makes it wrap around and start over with the lowest number it can represent.

There are two types of integers used in computing, signed and unsigned ones. Unsigned ones start with zero,signed ones can be negative. The fact that the integer in question overflows after 248.5 days indicates that this is a signed 32-bit integer. The code that uses the integer is probably not prepared to handle negative values, causing the malfunction.

This programmer screw-up will be talked about for years in software and embedded systems development.
mhlansdell00
Mark Lansdell 2
I almost understood all that, thanks.
joelwiley
joel wiley 0
Buffer overflow has been an known issue since computers had vacuum tubes. Those who do not learn the lessons of history....
randomguy
randomguy 2
This isn't a buffer overflow, but an integer overflow.
joelwiley
joel wiley 0
True, and integer overflow has been around as long.
joelwiley
joel wiley 1
Failure to address the overflow issue sounds like an inadequacy in laying out the design specifications.
mhlansdell00
Mark Lansdell 1
I'm not so sure it was a failure nor an inadequacy. It's hard to know what is in soemon eles's head. Maybe the software engineer never figures the system would be running that long in any single session. After all, the problem has not manifested itself only the possibility exists according to the article, the way I read it. It seems to me the solution is a scheduled service of shutting down the system during one of the inspections or other required servicees.
joelwiley
joel wiley 1
While it is theoretically possible to think out every possible detail in systems design, it is unlikely that it will be accomplished, and the probability drops with system complexity. Designing for 7/24/356.25 operations is 'complex'. I certainly would not think of running an airplane in that category. At the end of the day, it gets shut down would seem more likely.

I noticed in the AD "If the four main GCUs (associated with the engine mounted generators) were powered up at the same time after 248 days of continuous power, all four GCUs will go into failsafe mode at the same time, resulting in a loss of all AC electrical power regardless of flight phase...." Do they usually get powered up at the same time? I don't have any idea. If not, the likelihood of an unlikely event gets less likely. No reason to disregard the dictates of prudence and reboot it every few months.
JerrySteinberg
(Duplicate Squawk Submitted)

FAA Warns of 787 Dreamliner Glitch That Could Cause ‘Loss of Control’

The high-tech Boeing Dreamliner can’t seem to escape the limelight. On Thursday, the Federal Aviation Administration issued a warning about the Dreamliner’s computer system, saying a software glitch could cause a total loss of electrical power that would endanger an aircraft in flight and “result in loss of control of the airplane.”

...

http://www.frequentbusinesstraveler.com/2015/05/faa-warns-of-787-dreamliner-glitch-that-could-cause-loss-of-control/
sparkie624
sparkie624 -1
(Duplicate Squawk Submitted)

FAA finds Boeing Dreamliner could lose all power, issues maintenance mandate

The headaches for Boeing over its 787 Dreamliner continue.

The Federal Aviation Administration on Friday issued a directive mandating “a repetitive maintenance task” for that model of airliners due to issues with its power supply. Specifically, the FAA explained testing revealed that 787s could lose all AC electrical power after being continuously powered for 248 days, a problem that — if left unchecked — would leave an aircrew unable to control the plane.

The order took effect immediately, with the federal agency finding that there’s no good reason to delay the decision.

“The FAA has found that the risk to the flying public justifies waiving notice and comment,” the agency said.

The maintenance mandate was characterized as temporary, until software is developed to resolve the problem.

http://wreg.com/2015/05/02/faa-finds-boeing-dreamliner-could-lose-all-power-issues-maintenance-mandate/

Login

Don't have an account? Register now (free) for customized features, flight alerts, and more!
This website uses cookies. By using and further navigating this website, you accept this.
Dismiss
Did you know that FlightAware flight tracking is supported by advertising?
You can help us keep FlightAware free by allowing ads from FlightAware.com. We work hard to keep our advertising relevant and unobtrusive to create a great experience. It's quick and easy to whitelist ads on FlightAware or please consider our premium accounts.
Dismiss