To scale agentic AI, Notion tore down its tech stack and began recent

Many organizations could be hesitant to overtake their tech stack and begin from scratch.

Not Notion.

For the three.0 model of its productiveness software program (launched in September), the corporate didn’t hesitate to rebuild from the bottom up; they acknowledged that it was obligatory, in actual fact, to help agentic AI at enterprise scale.

Whereas conventional AI-powered workflows contain express, step-by-step directions primarily based on few-shot studying, AI brokers powered by superior reasoning fashions are considerate about device definition, can establish and comprehend what instruments they’ve at their disposal and plan subsequent steps.

“Quite than making an attempt to retrofit into what we had been constructing, we wished to play to the strengths of reasoning fashions,” Sarah Sachs, Notion’s head of AI modeling, informed VentureBeat. “We've rebuilt a brand new structure as a result of workflows are completely different from brokers.”

Re-orchestrating so fashions can work autonomously

Notion has been adopted by 94% of Forbes AI 50 firms, has 100 million complete customers and counts amongst its prospects OpenAI, Cursor, Figma, Ramp and Vercel.

In a quickly evolving AI panorama, the corporate recognized the necessity to transfer past less complicated, task-based workflows to goal-oriented reasoning techniques that permit brokers to autonomously choose, orchestrate, and execute instruments throughout related environments.

In a short time, reasoning fashions have turn into “much better” at studying to make use of instruments and observe chain-of-thought (CoT) directions, Sachs famous. This permits them to be “much more unbiased” and make a number of choices inside one agentic workflow. “We rebuilt our AI system to play to that," she stated.

From an engineering perspective, this meant changing inflexible prompt-based flows with a unified orchestration mannequin, Sachs defined. This core mannequin is supported by modular sub-agents that search Notion and the online, question and add to databases and edit content material.

Every agent makes use of instruments contextually; as an illustration, they’ll determine whether or not to go looking Notion itself, or one other platform like Slack. The mannequin will carry out successive searches till the related info is discovered. It might probably then, as an illustration, convert notes into proposals, create follow-up messages, monitor duties, and spot and make updates in information bases.

In Notion 2.0, the staff targeted on having AI carry out particular duties, which required them to “suppose exhaustively” about how one can immediate the mannequin, Sachs famous. Nevertheless, with model 3.0, customers can assign duties to brokers, and brokers can really take motion and carry out a number of duties concurrently.

“We reorchestrated it to be self-selecting on the instruments, relatively than few-shotting, which is explicitly prompting how one can undergo all these completely different eventualities,” Sachs defined. The intention is to make sure every part interfaces with AI and that “something you are able to do, your Notion agent can do.”

Bifurcating to isolate hallucinations

Notion’s philosophy of “higher, sooner, cheaper,” drives a steady iteration cycle that balances latency and accuracy by fine-tuned vector embeddings and elastic search optimization. Sachs’ staff employs a rigorous analysis framework that mixes deterministic exams, vernacular optimization, human-annotated knowledge and LLMs-as-a-judge, with model-based scoring figuring out discrepancies and inaccuracies.

“By bifurcating the analysis, we're in a position to establish the place the issues come from, and that helps us isolate pointless hallucinations,” Sachs defined. Additional, making the structure itself less complicated means it’s simpler to make adjustments as fashions and methods evolve.

“We optimize latency and parallel considering as a lot as potential,” which results in “method higher accuracy,” Sachs famous. Fashions are grounded in knowledge from the online and the Notion related workspace.

In the end, Sachs reported, the funding in rebuilding its structure has already offered Notion returns by way of functionality and sooner fee of change.

She added, “We’re totally open to rebuilding it once more, when the following breakthrough occurs, if now we have to.”

Understanding contextual latency

When constructing and fine-tuning fashions, it’s vital to grasp that latency is subjective: AI should present essentially the most related info, not essentially essentially the most, at the price of pace.

“You'd be stunned on the alternative ways prospects are prepared to attend for issues and never look forward to issues,” Sachs stated. It makes for an fascinating experiment: How gradual are you able to go earlier than individuals abandon the mannequin?

With pure navigational search, as an illustration, customers might not be as affected person; they need solutions near-immediately. “If you happen to ask, ‘What's two plus two,’ you don't need to wait to your agent to be looking out in every single place in Slack and JIRA,” Sachs identified.

However the longer the time it's given, the extra exhaustive a reasoning agent might be. As an illustration, Notion can carry out 20 minutes of autonomous work throughout a whole bunch of internet sites, information and different supplies. In these cases, customers are extra prepared to attend, Sachs defined; they permit the mannequin to execute within the background whereas they attend to different duties.

“It's a product query,” stated Sachs. “How will we set person expectations from the UI? How will we verify person expectations on latency?”

Notion is its largest person

Notion understands the significance of utilizing its personal product — in actual fact, its staff are amongst its largest energy customers.

Sachs defined that groups have energetic sandboxes that generate coaching and analysis knowledge, in addition to a “actually energetic” thumbs-up-thumbs-down person suggestions loop. Customers aren’t shy about saying what they suppose ought to be improved or options they’d wish to see.

Sachs emphasised that when a person thumbs down an interplay, they’re explicitly giving permission to a human annotator to investigate that interplay in a method that de-anonymizes them as a lot as potential.

“We’re utilizing our personal device as an organization all day, on daily basis, and so we get actually quick suggestions loops,” stated Sachs. “We’re actually dogfooding our personal product.”

That stated, it’s their very own product they’re constructing, Sachs famous, so that they perceive that they might have goggles on on the subject of high quality and performance. To stability this out, Notion has trusted "very AI-savvy" design companions who’re granted early entry to new capabilities and supply vital suggestions.

Sachs emphasised that that is simply as vital as inside prototyping.

“We're all about experimenting within the open, I believe you get a lot richer suggestions,” stated Sachs. “As a result of on the finish of the day, if we simply have a look at how Notion makes use of Notion, we're probably not giving one of the best expertise to our prospects.”

Simply as importantly, steady inside testing permits groups to judge progressions and ensure fashions aren't regressing (when accuracy and efficiency degrades over time). "The whole lot you're doing stays trustworthy," Sachs defined. "You understand that your latency is inside bounds."

Many firms make the error of focusing too intensely on retroactively-focused evans; this makes it troublesome for them to grasp how or the place they're bettering, Sachs identified. Notion considers evals as a "litmus check" of improvement and forward-looking development and evals of observability and regression proofing.

“I believe an enormous mistake a whole lot of firms make is conflating the 2,” stated Sachs. “We use them for each functions; we take into consideration them actually in another way.”

Takeaways from Notion's journey

For enterprises, Notion can function a blueprint for how one can responsibly and dynamically operationalize agentic AI in a related, permissioned enterprise workspace.

Sach’s takeaways for different tech leaders:

• Don’t be afraid to rebuild when foundational capabilities change; Notion totally re-engineered its structure to align with reasoning-based fashions.

• Deal with latency as contextual: Optimize per use case, relatively than universally.

• Floor all outputs in reliable, curated enterprise knowledge to make sure accuracy and belief.

  She suggested: “Be prepared to make the onerous choices. Be prepared to sit down on the high of the frontier, so to talk, on what you're creating to construct one of the best product you may to your prospects.”