Your AI assistant forgets you every morning. This benchmark proves it.

For all the breathless demos of AI agents booking flights or writing code, one uncomfortable fact rarely gets airtime: most of these agents start each conversation as a stranger. They have no memory of what you asked yesterday, no model of your preferences, no sense of the habits that define your decision-making. A new benchmark, VitaBench 2.0, aims to quantify exactly how bad that problem is. The numbers are sobering.

Developed by researchers from the Beijing Institute of Technology, Tencent AI Lab, and other institutions, VitaBench 2.0 runs on a simple but punishing premise. Tasks are presented not as isolated requests but as temporally ordered sequences for individual users. Preferences are deliberately embedded in fragmented, heterogeneous interactions. The agent must continuously extract, utilize, and update those preferences across the timeline. It is the benchmark equivalent of a new hire who must learn a boss's quirks by observing everyday behavior, but without the new hire's natural ability to infer and remember.

What's at stake

The benchmark's focus on personalization and proactiveness targets a blind spot in current evaluation. Existing agent benchmarks, from ToolBench to SWE-bench, test reasoning, tool use, and code generation. They rarely test whether an agent can build a model of the user it serves. That omission matters more as agents move from one-shot Q&A to long-term collaboration in health, finance, and daily scheduling.

VitaBench 2.0 introduces two core evaluation dimensions. First is personalization: can the agent infer that a user who always picks window seats for flights would also prefer a corner table at a restaurant? Second is proactiveness: can the agent recognize when information is missing, say dietary restrictions, and ask for it before making a decision, rather than assuming the default?

The results gap

The paper benchmarks a range of proprietary and open-source models, including GPT-4o, Claude 3.5 Sonnet, DeepSeek-V3, and others. The exact scores are best read in the full paper, but the headline finding is unambiguous: even the best models achieve success rates well below what would be required for practical deployment. The gap is widest on tasks that require the agent to revise its user model when preferences change mid-interaction, a scenario that humans handle intuitively but LLMs systematically mishandle.

One illustrative failure mode: agents that correctly memorized a user preference early in the interaction would rigidly apply it later, even when the user's subsequent actions contradicted it. The agents lacked the meta-cognitive ability to detect when the user model needed updating. Another recurring failure: agents that failed to ask clarifying questions when the user's request was underspecified, despite having been explicitly trained with instruction-following data. The benchmark's proactiveness dimension caught models that could paraphrase a user's request but could not recognize the gaps in it.

Architectural lessons

The paper also provides an extensible memory interface that enables controlled comparison across different memory architectures. This is a practical contribution: rather than simply declaring that current models are bad at personalization, the benchmark allows researchers to isolate whether the bottleneck is the model's reasoning capability, its memory retrieval mechanism, or the raw attention capacity for long contexts.

Early experiments with sliding-window memory, summarization-based memory, and retrieval-augmented memory suggest that no single architecture solves the problem. Retrieval-augmented approaches help with factual recall but struggle with inferential tasks, such as knowing that a preference for quiet environments in restaurants implies something about preferred café types. Summarization approaches compress too aggressively, losing nuance. The best-performing architectures combine multiple memory mechanisms but still fall short of reliable personalization.

Why this matters beyond the lab

VitaBench 2.0 arrives at a moment when several companies, including Google, OpenAI, and Anthropic, are shipping or preparing agents that operate over longer time horizons. Google's Project Mariner, for instance, aims to help users complete web-based tasks by remembering context across sessions. The benchmark's results suggest that these products will face a harder version of the personalization problem than their marketing acknowledges.

There is also a privacy angle that the paper touches lightly. Persistent user models, if implemented, create new vectors for data leakage. An agent that builds a detailed profile of a user's habits and preferences is a more useful assistant, but also a more sensitive one. The benchmark does not evaluate privacy trade-offs, but the architecture choices it enables could influence how much user data needs to be stored and for how long.

The takeaway

VitaBench 2.0 does not claim to be a complete solution. The authors note that their tasks are simulated and that real-world user interactions are messier than any benchmark can capture. But it draws a clear line between two kinds of AI capability: the ability to answer a question correctly, and the ability to understand a person well enough to anticipate what they will need next. The gap between those two is where the next generation of agent research will be fought.

For now, the benchmark's verdict is that LLM agents are excellent strangers. Teaching them to become friends is going to take a lot more work.