Many teams struggle to find and apply trustworthy evaluation. This is especially the case when results are noisy, benchmarks are hard to maintain, and risks like data leakage or contamination are easy to overlook.

This research builds scalable, realistic evaluation systems through benchmark mining and generation, careful dataset curation, and techniques that reduce leakage and overfitting. Our aim is to make it much easier for both internal teams and customers to create and maintain high-quality benchmarks, speeding up evaluation workflows and improving the reliability of AI-assisted coding tools.

Ongoing projects:

• BenchRoom: We have a pipeline for automatically collecting SWE-Bench-like benchmarks on scale.
• Anonymization for SWE benchmarks: We apply metamorphic testing to reduce data leakage in benchmarks.

When agents fail, their reasoning is often opaque, execution is spread across many steps and systems, and errors may only surface late in the process. All this makes debugging and agent improvement both slow and unreliable.

This research builds tools to capture, visualize, and analyze agent behavior, including detailed execution traces, anomaly detection, and methods for selecting high-quality traces for training. Our goal is to give both IDE users and internal agent developers better observability, so they can iterate on AI agents more quickly and with greater confidence.

Ongoing projects:

• Trace selection for LLM post-training: We build a tool to identify the agent traces that improve the quality of post-trained agentic models.
• Anomaly detection for AI agents: We analyze agent execution traces to detect anomalies in agent execution.

Today’s agents are fragile: small prompt tweaks, tool updates, or shifts in context can unexpectedly break behavior. On top of that, manual prompt engineering does not scale.

This research investigates automated ways to improve robustness, such as self-optimization, adaptive prompting, and dynamic system configuration. Our goal is to simplify prompt engineering, automate choices like agent topology and tool descriptions, and streamline debugging of unexpected agent behavior, ultimately giving both product teams and internal agent developers more reliable, maintainable AI systems.

Ongoing project:

• Auto-optimizing agents: We use evolutionary algorithms to optimize prompts and agents.

Code generation only delivers value when outputs meet real specifications, yet current agents often produce solutions that look plausible but are wrong. In addition, verification can be a major bottleneck.

This research develops ways to enforce correctness through testing, formal checks, and other validation strategies, and explores how to weave these checks directly into the generation loop. The aim is to strengthen automated test generation in IDEs and make benchmarks like SWE-Bench more robust by expanding and tightening their test suites.

Key Projects:

• Test generation agent: We implement an agent and compare it with general-purpose SWE agents.
• AI-based test repair: We fix the outdated tests that led to build fails on CI/CD.