The F-35 program has been a litany of glitches and problems, many as a result of the program’s pork distribution approach.
It’s hard to know where to start but the current pain-points appear to be: a logistical software system that exports a tremendous amount of information about the state of an aircraft to the builders back in the US, and an engine maintenance program in which only a few locations can perform engine overhaul – one being in the US and the other in Turkey. It never seems to have occurred to anyone that mutually distrusting European powers wouldn’t be thrilled about having to ship their very expensive stealth aircraft to Turkey for engine repairs, or that the US shouldn’t be able to tell the location and deployment status of every aircraft. Apparently amazon.com-style tracking is not appreciated when we’re talking about super secret aircraft. “Duh!” comes to mind, but it’s impossible to assume that these decisions were made by dummies, because dummies don’t build next-generation stealth attack jets. It’s a bit mind-boggling, to me.
The current state of advanced military gear is that a significant component of it is software. Software is great because it provides mutability and capability upgrade – but the cost of developing and maintaining such software is extreme. Look at some of the things Tesla can do with their cars: they can add a “beast mode” in which the car’s engine limiters are set with different parameters, as is the steering, and suddenly you get brief supercar performance in return for some shortened component lifecycles. With a 5th-generation military aircraft, you get the same thing but – like with the Tesla – there’s a system log that can turn up at awkward times, “why were you taking your engine past redline 3 times over Syria, German aircraft?” But improvements are what they are, i.e.: [bi] the user interface and underlying data for a weapons system can be upgraded in place without requiring any changes to hardware. What happens is that you no longer have an airplane, you have a weapons system that includes an airplane somewhere at the end of a long logistical and management software system.
The Pentagon is upgrading mission systems avionics as part of a tech refresh effort for the F-35 Joint Strike Fighter that improves memory, weapons delivery, storage, processing speed, display video and aircraft parametric data, industry developers said.
Faster processors improve F-35 delivery of weapons enabled by the latest 3F software drop, such as the AIM-9X air-to-air missile. Also, improved radar warning receiver technology will more quickly identify enemy aircraft and integrate with the aircraft’s mission data files, or threat library.
As a “security guy” I immediately focused on the words “threat library” – oh, so if your plane is not patched with the latest update about “who the enemy is” your battlefield sensor fusion system might experience contradictions? This is a real issue: is a Turkish airplane a ‘friend’ or a ‘foe’ over Syria this week? It might be good, too: you can program the threat library to recognize commercial 727s and flag them as “civilian plane, do not shoot.” At least, until people start false-flagging aircraft (which is what got Korean Airlines flight 007 shot down). I’m pretty cynical about software (i.e.: realistic) and its bugginess – one possibility is that, reading between the lines, the software was buggy and is getting successively less so. There are security questions, but also just questions of basic system reliability – humans are not particularly good at writing perfect software, and a fly-by-wire aircraft needs software that is not quite perfect, but damn close.
[source lmco]
“click LOADOUT, then MISSILES then DELETE” See? [af mac]
However, Robert Behler, the Pentagon’s independent weapons tester, characterizes the current schedule for C2D2 as “high risk” and said the program office is struggling to stay on schedule, he said in an annual report published Jan. 30 by the Operational Test and Evaluation Office.
“The current Continuous Capability Development and Delivery (C2D2) process has not been able to keep pace with adding new increments of capability as planned,” the office’s director wrote. “Software changes, intended to introduce new capabilities or fix deficiencies, often introduced stability problems and adversely affected other functionality.”
“often introduced stability problems and adversely affected other functionality” is code for “this is one buggy piece of shit.”
Coding faster won’t make it less buggy; that’s baked-in to the existing software architecture, by now. When adding new features breaks other things, that means the code was not adequately modularized and the underlying architecture does not adequately define and scope primitive operations.
The sound of the nails being hammered into the coffin, though, are when the countries that were supposed to be buying it, bail out of the program and buy existing stuff that works: [bloomberg]
Germany will order 45 fighter aircraft from Boeing Co. to replace the Luftwaffe’s aging Tornado jets, Der Spiegel magazine reported on Sunday.
Defense Minister Annegret Kramp-Karrenbauer emailed her U.S. counterpart Mark Esper on Thursday to inform him of the decision, the magazine said, without identifying the source of its information. Germany will order 30 F/A-18 Super Hornets and 15 EA-18G Growlers, the report added.
Germany is buying stuff that is time-tested and works; they’re smart enough to realize that they don’t need the cutting edge because a) Russia is not going to invade them after all and b) if all you’re doing is bombing and strafing insurgents, then you don’t need stealth. Stealth is only necessary if you plan on going on the offense against an enemy equipped with 4th-generation gear. In a sense, by buying the F/A-18s, Germany is telling Russia that they are not planning to attack them any time soon.
“Not a bad model” – the guys at AT&T’s “UNIX room” who were a collection of great programmers, indeed, used to do what they called “developing a toy” first, then once they had figured out how the toy behaved, they’d come back and write a fully-developed version. That was how UNIX scripting became a model for remarkably poor software development: UNIX bypassed having to consider cryptic, incompatible, poorly-designed configuration files as “software” – which it is. Scripting is also software. I’ll go out on a limb and say that even a database (because the structure of the tables is relevant) is software.
“cut corners perceived as slow” – I was in a meeting the other day with someone from Google who said that the Google development model is to have design documents, and if those weren’t detailed enough you had a link to the code. I nearly choked on my coffee: “read the source code” is not design, or documentation. It’s like saying “this is my fall fashion line-up” and handing someone a bolt of fabric.
They shoulda used node.js for the F-35.
Isn’t it entirely possible they dived into a Big Architecture before they really understood the problem? The Big Picture was “the next great American airplane!” …. and it’ll be so great everybody will buy it! (for the crypto equivalent, see Ross Anderson’s “Keys Under Doormats” talk (https://www.youtube.com/watch?v=LWwaVe1RF0c): “Clipper was designed by people who didn’t really emotionally or intellectually acknowledge the existence of other countries”).
The entire mountain of death-march stupidity that is ALIS can possibly be blamed on using UML as a design tool for a distributed system. Thinking within the limits of UML guarantees you trip up on every single one of the 8 fallacies of distributed computing. 400-800MB aircraft data files, from shore to ship (or from US to remote theatre). Hoping crypto keys don’t expire during a test. Wrong variants for planes with different hardware. But it works in the lab!
“Apparently amazon.com-style tracking is not appreciated when we’re talking about super secret aircraft. “Duh!” comes to mind, but it’s impossible to assume that these decisions were made by dummies, because dummies don’t build next-generation stealth attack jets. It’s a bit mind-boggling, to me.”
Well, we always hear how everything needs to be run like a business to make it better. Maybe some businessfolk got their claws into this and decided to market an opt-out model to create an ‘additional revenue stream’: For a monthly fee you can disable tracking*. That’s definitely not backdoored and there are certainly no ToS-clauses stating that the service provider may, at any time, temporarily disable the feature.
*Called ‘enable HYPERSTEALTH (TM)’ after the merketeering department has finished with it. Opens a webshop page in-cockpit (in-helmet?) if you click it without a subscription or while not logged in.
““click LOADOUT, then MISSILES then DELETE” See? ”
But don’t accidentally hit the FIRE ALL button placed right next to it. The software’s supposed to stop you from firing stuff while parked on the ground but that doesn’t always work. We’ll fix it in some future release.
Incidentally, regarding the Turkish maintenance facilities, I am genuinely looking forward to the first reports of confused or angry pilots who can’t operate their plane after service, all because the techs set it to Turkish language and forgot to switch it back.
“No, tower, I can’t find the language setting because it’s not called language right now. Telling me the word doesn’t help either because I can’t read half the letters, over”
“When adding new features breaks other things, that means the code was not adequately modularized and the underlying architecture does not adequately define and scope primitive operations.”
I do not and promise not to ever do anything safety-critical or important. Stiil, I can’t help the guilty look I’m wearing right now.
—
On the whole, where critical software is concerned, I believe people should pay proper attention to the the space industry. No, I’m not saying they do excellent software work (I’m sure they generally do). I’m pointing to all the very expensive, often trivial mistakes that can kill you. The space sector has made great sacrifices in order to provide these example to us. The least we can do is pay attention the next time a space probe decides it is 50 miles underground and moving at 4c, when it is actually hovering just above its landing site and supposed to drop off the trillion-dollar rover.
Or you could point to the 737 max, but that seems to be more of a trifecta of shoddy programming, cheapskating and general short-sightedness.
There have been lots of wry comments lately about secession from the USA – either the WestPact or the NEPact or whatever pulling out to make a better country, or “allowing” a RedPact to leave the USA and thereby (hopefully) improving the residual USA.
During one such online conversation recently, someone suggested the latter, “allowing” republicans to leave and take a few states with them. I suggested that as a gesture of generosity and goodwill, we give them the entire F-35 project as well.
===========================
Separately: When the F-35 was proposed, a significant part of the con job was that we had already invented a super-amazing next gen plane in the F-22, but those weren’t affordable. The F-35 was supposed to save money by being smaller and less capable but more numerous with a streamlined, cross-service supply chain.
I haven’t looked at the most recent numbers, but they did significantly exceed $100m/copy when I looked at them not terribly long ago. Ironically, that was right about the number I remember being bandied about as the “too expensive” price tag for the F-22. IIRC, the price they quoted for the F-22 was 120m/copy.
Turns out the Air Force could have saved all their R&D dollars and just kept buying F-22s and been billions ahead on the budget. I suspect that the F-22s wouldn’t have worked for the Navy (space and possibly weight considerations aboard carriers, though I’m happy to be corrected by someone who actually knows something about areas where I’m merely speculating), and they certainly couldn’t have worked for the marines (no VTOL), but the Navy & Marines couldn’t possibly have been worse off and the Air Force would have been decades ahead of where we are now.
True the stealth skin for the F-22 was (again, IIRC) easily damaged by common weather conditions, rendering it non-stealth, but putting the same skin on a new F-35 aircraft body wouldn’t have helped anything. Since R&D on a new stealth skin is less expensive than R&D on a new stealth skin AND an entire new plane to put it on, dumping the F-22 seems more than a bit stupid in retrospect.
I laughed so hard I nearly choked on my whisky.
I’ve been practicing agile development for a decade or so, and I absolutely believe that it is the best approach for most software projects… But the control software for a fighter jet is absolutely not “most software projects”.
“It’s like saying “this is my fall fashion line-up” and handing someone a bolt of fabric.”
Well, AIUI, that is actually the way it is done: The ultimate arbiter of what has been agreed between designers and producers is sample garments. The production has to agree with these.
See e g https://dl.acm.org/doi/pdf/10.1145/240080.240257 (Old paper, but lots of interesting info on how clothes are made.)
komarov @ #2: Or you could point to the 737 max, but that seems to be more of a trifecta of shoddy programming, cheapskating and general short-sightedness.
Don’t forget regulatory capture, which is what allowed them to get away with the three things you named (for a while).
Marcus: node.js
Use npm to keep it up to date (left-pad costs extra).
I think agile, like every other programming process, depends on having a system architect who does a good job of decomposing the problem, factoring out common subcomponents, and working well with team leads. If you have that, you’ll likely succeed regardless of organizational model (see: google) – I believe it was Brian Kernighan who said, “a lot of modern software engineering is ‘how to program, for those who cannot'” mic drop.
When I see agile proposed as an alternative to “slower, heavier processes” I assume they mean they want to cut the corners that were put in place because they can’t get things done on schedule – which means they need new system architect and team leads not a new process.
Nothing really about the software but The new avionics are intended to enhance the F-35’s sensor fusion so that information from disparate sensor systems can be combined on a single screen for pilots to lower the cognitive burden and quicken the decision-making process. screams incompetence in the human factors/ergonomics area too.
Did the designers get anything right on the F-35?
@ 3 Crypt Dyke
Apparently one minor problem with the F-22 is that it requires ~ 6 days maintenance for each ~1 flight.
I remember hearing that Saudi F-18s required 24 hours maintenance for an hour of flight time so it looks like the F-22 is worse without the G””awful Saudi environment for planes.
@jrkrideau:
Is that including the maintenance for the stealth surface? Because I had heard (and this isn’t my area of expertise of even a primary area of hobbyist interest, so I could WAAAY easily be wrong) that the problems with the stealth coating were either a or the primary delay in turn-around time.
If that’s true, then my original comment stands – they had to come up with a better stealth coating (and did) that required less maintenance. Slapping this on an existing airframe is still cheaper than developing the new coating AND a new airframe.
If that’s not true and it was engines or something else that required the excessive maintenance time, then I don’t know what the solution would be, but unless it was a fault with the actual airframe structure, it still would have been something that could at least be fixed in upcoming copies of the craft that had not yet been built. And R&D would have been far less for that than for creating a new aircraft.
I am of course happy to hear from people who know more from me on the topic.
And also of course, I actually would prefer that they dispensed with both programs. Stealth, as Marcus has said, is a first-strike weapon. The USA is already far too ready to strike first. The world would be a more peaceful place if the USA became much less militarily aggressive. Part of it is simply our credibility (damage to which was entirely self-inflicted, and eagerly so). If a neutral party is needed to keep the peace somewhere, no one trusts the USA to actually be neutral. So pulling back and paring down our arsenal to be consistent with something like France + UK + Germany would save a ton of money and relieve threat-stress in a lot of areas of the world.
Some would say that emboldens violent authoritarians and war mongers, but we already encourage, embolden, and enrich the violent authoritarians and war mongers of the world, so long as they give us what we want. Cutting our propping-up-thugs budget would do far more to cut that shit out than our current military overspending. And it would save money yet again.
I wonder what we could do with all that cash?
Whenever anyone discusses failures in software development processes, I’m reminded of the old adage that Marxism is the perfect form of government, but nobody has yet implemented Marx’s vision. It’s always been corrupted by the humans who sought to get it working.
I worked on a project for 17 years where my marching orders were “do whatever the COTR tells you to do,” and the COTR happened to sit in the cubicle next to mine and would sometimes redirect me multiple times in a week as he sat at his desk dreaming up spiffy new ideas for our project. The COTR was a reasonable-enough guy that I could push back on some of his wackier notions, but so long as I kept him and his bosses happy, my bosses at my company were happy. Dude even had me spend time creating special, annoying, pop-up messages for a user that annoyed us, and didn’t say “boo” when I’d leave work at my regular time after he’d chewed up multiple hours of my day ’cause he wanted to discuss Star Trek: TNG or Space: 1999 instead of whatever he was supposed to be doing, or letting me do whatever I was supposed to be doing. We got a lot done, but it was coding anarchy. Fun times.
Got switched to a project that mandated the use of Agile in the contract, so I started learning about Agile, but in its theoretically perfect form. The form in which the developers on the team are insulated from the customer via the proxy “Product Owner” who’s supposed to be someone working for your company who would negotiate all sprint goals and changes in direction with the customer and then report back to the “Project Manager” (another person working for the same company as the devs) and other members of the team regarding how the customer wanted to proceed.
Yeah, no. This particular contract mandated customer staff as both POs and PMs for every team, so the Agile model was seriously broken from the start. In general, the PMs didn’t know Agile, and so would set deadlines for milestones months away but in the middle of sprints and other absurdities, as well as change directions on the team mid-sprint and then get upset when the original sprint goals weren’t met. The POs were generally been absent from all discussions (the first PO for my team I met only twice, and it was clear the second time that he didn’t remember ever meeting me), and few of the POs or PMs were technical, so we’d wind up with ridiculous demands. “Push back” was only an option if you were highly skilled in diplomacy, because these people were more driven by ego than mission, and saying things like, “we shouldn’t do that because {explanation}” were considered “push back.” It’s been a disaster, both for the customer and for the company I work for.
I’ve had to explain to multiple people that this isn’t how Agile is supposed to function, so they shouldn’t blame the problems on Agile itself.
Marcus, I’m entirely with you and Brian Kernighan on this one.
I see processes as a way to cheat at being an expert. They do help if you’re almost there, as a team (some of the team grok developing software, some not just yet). They don’t do much if you’re already all excellent, nor if you’re all mediocre. I think this is why processes seem to work beautifully for their proponents and their disciples, but then don’t port nearly as well once they start selling them more widely.
@Dave W#11:
That sounds more like “consulting for a client from hell” than software development. When the customer gets their hands on a development process and starts asking for bits and pieces and specific features, the system architecture goes all to hell and the whole schedule along with it. I can imagine designing a development process that could withstand such a customer but it’d look like rapid prototyping using shell scripts on UNIX – an entirely throw-away system. I did a consulting gig like that back in the early 90s and wound up metaprogramming my solution, then I implemented the system as I understood it (in the programming language I designed) and documented it, then told the client, “here! you can add your own features all you want!”
I’ve been in that situation, too. You’re supposed to be coding something but instead you’re randomly chasing some poorly-communicated idea on powerpoint. Nothing ever results from that, though I used it as a chance to level up my techniques and to play with metaprogramming and learn FORTH.
I have infos about the stealth surfaces; I’ll try to do a post about them soon(ish) including some things about the maintenance load they represent.
Fun spoiler factoid: F-22 coatings are different from F-35 coatings. Why? I have no idea because it’s classified!
The US government decided the F22 was not to be exported to protect its top secret doohickies. The Japanese really wanted them, but even they were told no. If they had let the Japanese buy them it would have brought the unit cost down and led to more USAF purchases. Part of the goal of the F35 was to make a stealth aircraft that could actually be exported.
Finland looks to be dodging the F-35 as well. What if they gave a super-stealth jet and nobody came?
[national interest]
@ 16 Marcus
I just noticed that the Saab is bidding on the Canada contract against Lockheed Martin and Boeing.
I still think we should have Sukhoi in the competition. Both the Su-35 and the Su-57 look like good deals.
It says something that up here in Canada we’ve decided it might be a better idea to just replace our fighters with either a newer version of the same fighters or the also-30-years-old Saab Gripen.
ColeYote@#18:
It says something that up here in Canada we’ve decided it might be a better idea to just replace our fighters with either a newer version of the same fighters or the also-30-years-old Saab Gripen.
It says a lot of things. One of those being that Canada apparently has realized that you don’t need ultra-expensive high maintenance 5th-generation stealth fighters to… um… what is it that the Canadian Air Force does, anyway?
This is a particular problem for the Marines and other groups that desperately need to replace aging Harriers. My brother is a Marine combat engineer, and his stories about the great lengths they go to in order to keep those things in the sky are hilarious. There’s actually a special group in NATO dedicated to hoarding components for repair and maintenance, and provide funding to do things like fix airfields so there’s less wear and tear on the Harriers. There is pretty much nothing to replace them in many operations, and all their eggs are in the F-35 basket. Derp.
DrVanNostrand@#20:
There is pretty much nothing to replace them in many operations, and all their eggs are in the F-35 basket. Derp.
The F-35 has destroyed the UK’s air power, in a unique way that no other weapons system could.