Spread the love

Neon can spin up databases in milliseconds, making it ideal for building, and running, agentic AI.

A photograph of a Databricks website on a phone being held in front of a larger, blurry version of the Databricks logo.
Credit: rarrarorro / Shutterstock

Agentic AI requires a whole new type of architecture; traditional workflows create serious gridlock, dragging down speed and performance.

Databricks is signaling its intent to get ahead in this next generation of app building, announcing it will purchase open-source serverless Postgres company Neon. The startup’s platform can spin up new database instances in less than a second, making it purpose-built to support agentic workflows.

This move “allows Databricks to strengthen its AI infrastructure capabilities, specifically in areas like AI-driven database provisioning and the development of AI agents, a notable gap they are seeking to fill as their competitor Snowflake currently lacks these features,” said Scott Bickley, advisory fellow at Info-Tech Research Group.

AI building AI

After the $1 billion acquisition, Neon’s serverless Postgres architecture will be integrated with the Databricks Data Intelligence Platform, allowing developers to quickly build and deploy AI agents without having to concurrently scale compute and storage. This approach can prevent performance bottlenecks while simplifying infrastructure and reducing cost, Databricks says.

Many of the tasks AI agents perform require launching a database for information retrieval, but that can take several minutes, slowing responses, the company noted. Neon’s ability to create near-instant database instances allows AI agents to react quickly. Additionally, databases can be flooded with requests from multiple agents, also hampering speed; Neon can create separate copies of database content for each individual agent.

According to recent internal telemetry, 80% of databases on the Neon platform are created automatically by AI agents rather than humans. Neon says it can spin up a fully-isolated Postgres instance in 500 milliseconds or less, and supports instant branching and forking of database schemas as well as data, so production isn’t halted.

“Traditional database systems can’t keep up with the scale and variability of agent-driven architectures, where thousands of temporary databases are spun up and shut down rapidly,” explained Robert Kramer, VP and principal analyst at Moor Insights and Strategy.

Neon’s serverless Postgres model, when integrated with the Databricks platform, will provide instant provisioning, separation of compute and storage, and API-first management, he noted. “Organizations can reduce infrastructure costs, speed up deployment cycles, and improve experimentation without disrupting production.”

According to Databricks, Neon’s platform is 100% Postgres-compatible and works out of the box with several popular extensions It also provides a cost structure that scales with usage.

“We’re giving developers a serverless Postgres that can keep up with agentic speed, pay-as-you-go economics, and the openness of the Postgres community,” Ali Ghodsi, co-founder and CEO at Databricks, said in a statement.

Setting out to disrupt the database industry

Neon’s clients include Replit, Retool, Boston Consulting Group, Vercel, Cloudera, and Cloudflare, according to its website.

In an announcement, its founding team said they set out to “disrupt the database industry” and create a new architecture that separates storage and compute and introduces a “branchable, versioned storage system.” The idea wasn’t to build a wrapper around Postgres or offer managed hosting, “it was a fundamental rethink of how Postgres should work in the modern era,” they said.

Neon was launched publicly in 2022, and soon became one of the fastest-growing developer databases on the market. Once the transaction closes, many of its team members are expected to join Databricks.

The latest in a string of strategic moves

Databricks has been making strategic purchases to position itself as a top platform for building, testing, and deploying AI. In 2023, it purchased open source large language model (LLM) training platform MosaicML for $1.3 billion, and last year it scooped up data storage company Tabular for more than $1 billion.

“Databricks has been aggressive in its acquisition of companies that have accelerated their core technology platform,” said Bickley.

He added that linking generative AI model-building capabilities from Mosaic with Apache Iceberg and Delta Lake formats has been a “powerful enhancement,” and lent itself to the recently-announced SAP-Databricks partnership, which will merge contextual ERP data with external data sources.

What IT buyers should keep in mind

Integrating Neon’s model into legacy systems and rethinking database governance for agent-driven architectures will take time and careful planning, Kramer emphasized. He noted that the AI data infrastructure market is crowded, so Databricks may face a challenge in differentiating itself.

“They must ensure that Neon scales reliably, integrates with enterprise environments, and has the proven track record of success,” he said. He added that the true test will be whether customers can effectively utilize these new capabilities at a large scale without introducing additional complexity.

Bickley also urged IT buyers to be cautious, particularly when it comes to pricing. While a consumption-based subscription model can offer cost efficiency, if not properly governed or contractually structured it can “bleed enterprise budgets with runaway, unmanaged costs,” he noted.

“In its current form, Neon offers robust capabilities to control costs via its scale-to-zero feature,” he said.

Adopters should also focus on product absorption timelines, as well as preservation of Neon’s open source culture and community and Apache 2.0 licensing. It’s reasonable to expect some proprietary, fee-based products (such as managed Neon instances), he noted, and Databricks will likely integrate features with those from Mosaic and Tabular.

“All things considered, this acquisition enhances Databrick’s comprehensive capabilities as it builds a leading data management suite, providing buyers with the option to rationalize vendors in the data management space,” said Bickley. “Bringing best-in-class serverless database capabilities into the fold and extending their use via AI agents sets Databricks apart for now.”

More on Databricks:

If you found this article helpful, please support our YouTube channel Life Stories For You

Facebook Comments Box
Tags: , ,
Categories: