How Silicon Valley is helping the Pentagon in the AI arms race

For eight years, a docile fleet of bright orange, unmanned sailboats bobbed around the Bering Sea near Alaska, counting pollock and feeding data to the US government’s oceanic exploration agency.

Amassing an unrivalled database of ocean maps which could then be analysed by machine learning programs, the autonomous vessels made by Saildrone, a start-up founded in 2013 by young British engineer Richard Jenkins, made significant contributions to scientific research on climate change.

But as geopolitical tensions between the US and China increased, Saildrone landed a much bigger fish, one with a fresh sense of urgency and a significant budget: the US Department of Defense.

By 2021, the San Francisco-based company was a key contractor helping the US Navy to develop an armada of artificial intelligence systems to conduct surveillance in international waters, including the Arctic Ocean surrounding Russia and the South China Sea.

Silicon Valley venture capitalists rushed to back it, investing $100mn in the small company in 2021 compared to $90mn in total prior to that. Shortly after, it started developing its “Saildrone Surveyor” for the US Navy — a 65-foot autonomous vessel around the length of two whale sharks designed for deep ocean intelligence, such as surveillance and reconnaissance.

“We were 10 years ahead,” says Jenkins, the start-up’s chief executive, arguing that his company is a “good example of where commercial tech is so much more advanced and more nimble” than the systems being developed by the US military.

It is a blueprint that could prove crucial for the US as it races to evolve its defence strategy from a reliance on heavy hardware such as tanks, ships and aircraft to more nimble investments in disruptive systems, such as new tech like artificial intelligence that has the power to transform modern warfare.

Three factors are driving that shift: China’s rapid development of advanced weapons systems that negate US defences, the war in Ukraine which has highlighted the advantages of integrating commercial tech into a nation’s military and the startling advances in AI.

For all these reasons, the US government has become a far more motivated customer, while a downturn in deals and valuations in Silicon Valley has made the public defence sector seem like a stable and reliable option for start-ups.

The whiff of opportunity has spurred a gold rush among investors in California, who are piling billions of dollars into defence and weapons technology start-ups. Venture capital in the sector has doubled from around $16bn in 2019 to $33bn in 2022, PitchBook data shows.

But getting the defence department to reallocate some of its mammoth $886bn budget from its five incumbent prime contractors, which include Lockheed Martin and Boeing, to the thousands of entrepreneurs producing cutting-edge systems remains an obstacle. Tech entrepreneurs and investors have accused military leaders of engaging in “innovation theatre” — paying lip service to the benefits of disruptive technology while holding back lucrative contracts.

“For the first time ever, the US military is dependent on commercial tech to win a war, but they’re not organised to deal with commercial tech,” says Steve Blank, a tech veteran and founding member of the Gordian Knot Center at Stanford, which was set up to train innovators in national security.

“China operates like Silicon Valley,” he adds, in reference to the tech sector’s speed of innovation and agility. “On a good day, the DoD operates like Detroit”, the midwest city that never recovered from the decline of auto-making. “It’s not a fair fight.”

Cherissa Tamayori, director of acquisition at the Defence Innovation Unit (DIU), an arm of the DoD that was set up in 2015 to push commercial technology and help companies navigate the bureaucracy of military procurement, agrees that Silicon Valley is “significantly important” to national security.

“We need to make sure our military is equipped with the best tech out there, wherever it comes from,” she says. “Our adversaries overseas are using commercial technologies, and that is increasing the urgency and need for us to figure this out.”

A slow start

Until recently, the Pentagon’s efforts to tap the Silicon Valley innovation machine for national defence had been underwhelming.

A small group of start-ups has reaped some rewards. Six of them — ShieldAI, Hawkeye 360, Anduril, Rebellion Defense, Palantir Technologies and Epirus — have been valued at over $1bn. Only a handful of aerospace or space companies that provide defence capabilities have attracted colossal investment, such as Elon Musk’s SpaceX, which is nearing a valuation of $150bn.

Anduril has won a blockbuster public contract, worth nearly $1bn, from the US Special Operations Command for technology that can detect drones and shoot them out of the sky. However, large government contracts of the scale required to manufacture complex systems remain a rarity. Instead, venture capital firms such as Andreessen Horowitz, Lux Capital and 8VC have provided much of the early backing, while a slow and frustrating public procurement process means many of the early winners have had to rely on billionaire founders to survive their formative years. These include data analytics group Palantir, set up by tech entrepreneur Peter Thiel, and Anduril, founded by Palmer Luckey after he sold his virtual reality start-up Oculus to Facebook.

“I have faced this problem every day for eight years,” says Brandon Tseng, founder of ShieldAI, which launched in 2015 and is now valued at $2.3bn. “Yes, we have scaled, but we have to continue to scale, and the amount we have been able to capture has been minuscule in comparison to the prime [contractors].” For Saildrone founder Jenkins, the system worked — eventually. “Could we have got there sooner with better contracting? Absolutely.”

The source of frustration is the rigid planning, programming, budget and execution buying framework, known as PPBE, used to allocate resources across the military. It was established in the 1960s to end conflicts of interest, but the layers of bureaucracy make it notoriously slow and difficult to navigate. Competition waned as the defence market consolidated. Both Palantir and SpaceX resorted to suing the Pentagon for the right to compete for business.

Designed to acquire physical assets like aircraft parts and tanks, critics say the framework is unsuitable for the kind of software that is set to revolutionise future warfare. It takes around two years to land a serious contract, a timescale that has pushed many inventors into the so-called Valley of Death in which desirable prototypes are lost because the companies behind them wither and die waiting to win work.

Speed is a factor for another important reason. “By the time you get through the slog, it is two years later and the technology is obsolete,” says Thomas Tull, a billionaire investor and chair of the multibillion-dollar US Innovative Technology Fund. “It is not set up for the velocity of innovation.”

Initiatives from the DoD, such as the DIU and Afwerx, which was set up in 2017 to help young companies sell to the US Air Force, have had limited budgets. In-Q-Tel, the venture arm of the CIA, which was founded as early as 1999, has made hundreds of investments in start-ups such as Keyhole, which later became Google Earth, and Palantir.

But its funds remain relatively small. It invests between $500,000 and $3mn, and companies the DIU has backed have been awarded around $5bn in contracts from US defence agencies, a small proportion of the trillions of dollars spent on defence procurement since its launch.

China’s hypersonic missile tests, followed by Russia’s invasion of Ukraine in 2022, focused minds at the Pentagon. The need for reform was becoming essential.

The catalysts for change

Ukraine’s deployment of dual-use technology — capabilities that have both commercial and defence applications — such as satellite imagery and autonomous drones is among the biggest catalysts for the US to bridge the chasm between Washington and California.

When Musk’s SpaceX opened up the Starlink internet service that is resistant to Russian interference, it was the first time a commercial company provided the backbone for a country’s military capability during wartime.

“What’s happened in Ukraine has been a game-changer. More commercial technology is being used than during any other conflict,” says Mike Brown, a venture capitalist at Shield Capital and the former director of the DIU. “That has got the wheels turning for the US military, which is saying, ‘We need to adopt far more of this’.”

There is also an increasing pressure to tap into the brilliant minds of Silicon Valley and its deep-pocketed investors if the US wants to catch up with China’s advanced technology. Beijing’s testing of anti-ship ballistic missiles and long-range hypersonic missiles that can probably evade US defence systems underscored how critical innovation is as a deterrence.

Blank, of Stanford, puts it like this: “The question [for the Pentagon] is what else should we be putting in the air or in the water to make a problem for China so complicated that they have to think about peace, not conflict.”

And, then of course, came the biggest game-changer of all: the rapid development of AI, already believed to be the most significant invention to the future of war since the US developed the atomic bomb in the 1940s. The existing military “kill chain” — shorthand for the process by which warfighters identify, track and kill targets — would be rendered obsolete by AI: kills could potentially be carried out at hypersonic speeds without human involvement.

Mass firing of long-range anti-ship missiles could wipe out hardware such as aircraft carriers — on which the US spends tens of billions a year — on day one of a conflict with China. The battlefield infrastructure of the 20th and 21st centuries would stand little chance against swarms of autonomous drones, unmanned attack submersibles and synthetic aperture radar that can observe practically any movement on the planet.

“The advancement [of AI] over the past six months has changed everything,” says Tull, of the ITF, pointing out that the technology is still in relative infancy. “This is ‘Wright Brothers-take-flight’ kind of stuff.”

Several defence tech start-ups that harness the power of AI have already changed the way the US military gathers and deploys intelligence. Almost half of Palantir’s $1.9bn revenues last year came from US government contracts, including to provide AI software that uses surveillance technology and data analytics to track and follow suspected terrorists, for example. In April, Palantir demonstrated how its AI could be used to rapidly analyse a battlefield situation, generate potential courses of action and submit an operational plan to “neutralise” the threat.

San Francisco-based PrimerAI, which scrapes thousands of sources of “open intelligence” and uses natural language processing to analyse it, delivered intelligence to the US shortly before Russia invaded Ukraine. It was a powerful part of the US intelligence on Vladimir Putin’s intentions.

AI is also changing military hardware. In December 2022, ShieldAI piloted the first unmanned F-16 fighter jet at an airfield north of Los Angeles. It was a breakthrough for the US Air Force, which has contracted ShieldAI to develop sophisticated unmanned aerial vehicles.

American start-ups like BlackSky, Capella and PlanetLabs — remote sensing companies that fuse AI and satellite technology to provide real-time detailed overhead images — have allowed Ukraine to pinpoint the precise location and status of advancing Russian convoys. Now they have prototype deals with the DIU.

The US “had its eye off the ball from 2001 to 2016, focusing on the war on terror. During that time, Russia and China built capabilities that countered what we have been using,” says ShieldAI boss Tseng. “We needed a plan to modernise [in order to deal with] these threats.”

Peacetime is over

So far, that plan has not been borne out by the numbers.

The 100 largest venture-funded defence start-ups have collectively raised $42bn from investors in their lifetimes. By contrast, the total revenue they have made from government contracts is between $2bn and $5bn, according to the Silicon Valley Defense Group — a non-profit organisation aimed at increasing collaboration between the region and politicians. Of those 100 companies, two-thirds are start-ups that have developed space or machine-learning technologies.

In a report published in July, the SVDG accused the DoD of dishing out “door prizes but no sustained commitments” to include start-ups producing cutting-edge systems in major defence acquisition programmes. This echoed the sentiment of a letter from founders and investors of Silicon Valley start-ups to defence secretary Lloyd Austin weeks earlier, criticising the government’s “antiquated” process for buying military technology. The letter, whose signatories included the heads of Anduril, Palantir and Lux Capital, also warned the US was rapidly losing ground on the “technological battlefield”.

It would be wrong to claim the US is doing nothing. Last year’s Defense Authorisation Act established a Congressional commission to examine ways to modernise military acquisition. In April, the DoD reorganised the DIU, elevating its new director, former Apple executive Doug Beck, to sit directly under Austin. Beck, who served in the US Navy for 26 years before joining Apple, where he reported directly to chief executive Tim Cook, is regarded as key to accelerating plans to bring the military and Silicon Valley closer together. Congress allotted $111mn to fund the DIU’s mission in 2023, about $45mn more than DoD had requested.

That the DIU introduced 100 new vendors to the Pentagon under his tenure shows a shift is taking place, says its former director Mike Brown. “But it’s not yet happening at scale,” he adds. “If you analyse total spending, for now it looks like we’re buying the same large hardware we always have. But over the next couple of years, that will start to change.”

With ever increasing incentives, the stage is set for the Pentagon and Silicon Valley to forge far greater ties, bringing technology used by the warfighter in line with the tech used in commerce and industry, and accepting a much bigger role for the private sector in doing so. The challenge for the Pentagon will be increasing digital fluency at a time of rapid technological innovation, and incentivising private capital to help build an armed force capable of keeping up with advancements by China.

The setbacks do not just lie with the US military. Some Silicon Valley companies, such as Google, have turned away from some defence work under pressure from staff over ethics issues such as providing software for autonomous drones capable of identifying targets.

Concerns about being seen to support controversial US conflicts overseas — particularly in traditionally anti-war California — have also put off some large venture capital funds, though there are signs these anxieties are fading as the scale of the potential revenues on offer becomes apparent.

Some start-ups have also missed out on public contracts because they have not done what it takes to pass the US military’s rigorous tests and security clearing protocols, such as adequately securing their own systems and training AI models on military data. “There’s a lot of tourists in this space,” cautions one veteran defence and AI investor. “So many more things can go wrong in a conflict scenario for this tech.”

For some in the industry, such as Blank, the stakes are too high for complacency from the US military, tech start-ups or society at large. “We’re still operating in the US as if we’re in peacetime,” he warns. “China now sets the pace for us and we’re no longer a peer or a near-peer.”