Why code-testing startup Nova AI uses open source LLMs more than OpenAI

It is a universal truth of human nature that the developers who build the code should not be the ones to test it. First of all, most of them pretty much detest that task. Second, like any good auditing protocol, those who do the work should not be the ones who verify it.

Not surprisingly, then, code testing in all its forms — usability, language- or task-specific tests, end-to-end testing — has been a focus of a growing cadre of generative AI startups. Every week, covers another one like Antithesis (raised $47 million), CodiumAI (raised $11 million), and QA Wolf (raised $20 million). And new ones are emerging all the time, like the new Y Combinator graduate Momentic.

Another is year-old startup Nova AI, an Unusual Academy accelerator grad that’s raised a $1 million pre-seed round. It is attempting to beat its competitors with its end-to-end testing tools by breaking many of the Silicon Valley rules of how startups should operate.

Whereas the standard Y Combinator approach is to start small, Nova AI is aiming at mid-size to large enterprises with complex code-bases and a burning need now. Smith declined to name any customers using or testing its product except to describe them as mostly late-stage (Series C or beyond) venture-backed startups in e-commerce, fintech or consumer products, and “heavy user experiences. Downtime for these features is costly.”

Nova AI’s tech sifts through its customers’ code to build tests automatically using GenAI. It is particularly geared toward continuous integration and continuous delivery/deployment (CI/CD) environments where engineers are constantly shipping bits and pieces into their production code.

The idea for Nova AI came from the experiences Smith and his co-founder Jeffrey Shih had when they were engineers working for big tech companies. Smith is a former Googler who worked on cloud-related teams that helped customers use a lot of automation technology. Shih previously worked at Meta (also at Unity and Microsoft before that) with a rare AI specialty involving synthetic data. They’ve since added a third co-founder, AI data scientist Henry Li.

While OpenAI promises that the data of those on a paid business plan is not being used to train its models, enterprises still do not trust OpenAI, Smith tells us. “When we’re talking to large enterprises, they’re like, ‘We don’t want our data going into OpenAI,” Smith said.

The engineering teams of large companies are not the only ones that feel this way. OpenAI is fending off several lawsuits from those who don’t want it to use their work for model training, or believe their work wound up, unauthorized and unpaid for, in its outputs.

“In this case, instead of using OpenAI’s embedding models, we deploy our open source embedding models so that when we need to run through every file, we aren’t just sending it to OpenAI,” Smith explained.

While not sending customer data to OpenAI appeases nervous enterprises, open source AI models are also cheaper and more than sufficient for doing targeted specific tasks, Smith has found. In this case, they work well for writing tests.

“The open LLM industry is proving that they can beat GPT 4 and these big domain providers when you go narrow,” he said. “We don’t have to provide some massive model that can tell you what your grandma wants for her birthday. Right? We need to write a test. And that’s it. So our models are fine-tuned specifically for that.”

Open-source models are also progressing quickly. For instance, Meta recently introduced a new version of Llama that’s earning accolades in technology circles and that may convince more AI startups to look at OpenAI alternatives.