JetBrains logo

Education Research

Education Research

Designing and developing innovative education-oriented features for JetBrains products and beyond

We also share our knowledge across the company and the scientific community by organizing conferences and hackathons, as well as participating in education-focused events.

Our research interests

Generative AI in programming education

Intelligent tutoring systems

Systems & design thinking

Multimodal
learning

Recent publications and research activities

SE Simulation

The core idea of this project is to simulate the full software engineering process for a learner as if they were a real employee working on real-world tasks. The primary goal is not only to improve programming proficiency but also to strengthen system design skills.

Learners will develop features within an existing codebase that already contains reference implementations (used internally for context and not visible for participants). Their work can then be evaluated from multiple perspectives, including architectural decisions, code quality, complexity, modularization, and overall engineering approach.

To make the experience realistic, we aim to integrate with the same tools used in modern development environments, such as GitHub, Slack, YouTrack, and others, so that the entire workflow mirrors that of an actual company. Ultimately, this system is designed to help learners grow from an intern or junior level toward a mid-level engineer by practicing real, end-to-end system design and implementation tasks.

The project has two tracks:

  • A technical track focused on implementing the system, automatically evaluating system designs, and developing benchmarks to measure quality.
  • A research track aimed at understanding which skills need to be taught, which interaction scenarios are effective, and in what form they should be presented.

Education-focused LLM evaluation

Existing programming datasets are often Python-centric and include tasks that are already familiar to LLM models, which limits their utility for unbiased evaluation. Additionally, the lack of standardized evaluation pipelines results in inconsistent and incomparable results across studies. To address these gaps, this project focuses on building a new dataset of programming problems and creating a standardized evaluation pipeline to ensure reliable and reproducible assessments.

The goal is to develop a structured dataset sourced from JetBrains Academy tasks. The dataset will start with Kotlin tasks and later expand to additional programming languages. Alongside this, a user-friendly evaluation system will be built to deliver essential test metrics while allowing for potential expansion into advanced metrics like code quality and complexity.

By introducing a fresh, multi-language dataset and a robust evaluation pipeline, this project aims to overcome the limitations of existing benchmarks. It will provide consistent, reproducible, and richer model assessments, with the potential to set a new standard for evaluating programming models while supporting internal and external research efforts.

AI-generated metaphors

This tool extracts coding concepts from task descriptions and generates visual and text-based metaphors to explain these concepts to students.

The prototype for this solution is built on the open-source manim library, and it implements a pipeline that automatically extracts computer science concepts, generates metaphors as text, and then creates and refines videos illustrating the concepts.

The generation process involves four main distinct steps, which include: (1) the analogy's decomposition into concrete visual elements, (2) the creation of a high-level screenplay to define the narrative flow, (3) the generation of a manim script, and (4) its execution. Additionally, the generated code undergoes automated analysis and refinement cycles before the process is completed.

This video was created by the solution to explain the concept of Booleans. It was generated completely automatically without any manual editing.

Publication

SciPulse

SciPulse is envisioned as a hybrid-intelligence assistant designed to reduce the routine overhead of working with academic literature. Its core purpose is simple: to help researchers, especially junior ones, spend less time on tedious mechanical tasks and focus more on meaningful discovery. Combining traditional search methods with LLM-based reasoning, SciPulse offers more accurate retrieval, filtering, and evaluation of academic papers than any single approach could provide.

Additionally, SciPulse features a learning mode tailored for junior researchers. This interactive component teaches young researchers to evaluate papers, extract data, and critically assess quality, an often-overlooked aspect of academic training.

In short, SciPulse aims to be a practical and reliable companion for researchers at any stage of their career, helping them boost productivity and enhance learning through a unified, intelligent workflow.

KOALA

A tool for tracking student behavior during in-IDE learning

KOALA collects fine-grained logs of controlled experiments in the IDE. It allows you to track all step-by-step changes in code files, as well as all interactions with the IDE. You can customize it easily using YAML configuration files, and you can even send the collected data to a remote server, providing a convenient setting for remote experiments.

We have collected several datasets on task-solving processes, interactions with an AI-powered next-step hint system, and AI debugging. If you need this data for your research, feel free to contact us. We could also help you to set up your own study with using KOALA as the data collection tool.

Examples of data collected using KOALA include:

  1. Activities performed in the IDE, such as running, debugging, etc.
  2. Current student code.
  3. The opening, closing, and refocusing of files.
  4. Open tool windows.
  5. Survey data.

In the image, you can see how different data is written to different CSV files.

Teaching materials

The educational materials developed by the Education Research Lab combine our teaching and research experience to make programming classes more engaging for students. The materials are fully open source and can be easily used in any classroom.

Data Visualization in Python

This in-IDE course teaches the principles of effective data visualization. It covers how to use Python libraries, as well as how to choose the appropriate plot type in different scenarios.

Data Visualization in Python

Kotlin Onboarding

Split into three parts, this project-based in-IDE course covers CS1 concepts. Students work on interactive console and web projects, such as a Hangman game and an Alias board game.

Introduction , OOP, Collections

Introduction to IDE Code Refactoring

This in-IDE course helps students understand the concept of refactoring and teaches them how to use refactorings effectively in IntelliJ IDEA. The course is available for both Kotlin and Java.

Kotlin version, Java version

Kotlin Open Materials

This set of open materials is designed to help you develop your own Kotlin course. The resources include presentations with notes, quizzes, and homework assignments. If you're planning to create a Kotlin course, feel free to use these materials!

Kotlin Open Materials

About us

Comprising talented researchers and software engineers, the JetBrains Education Research team specializes in conducting high-quality research and developing cutting-edge technological prototypes. Beyond our core team, we collaborate with external partners worldwide, including in the Netherlands, Switzerland, the US, and Canada.

  1. Anastasiia Birillo
    Education Research Team Lead

    "My team is a group of incredibly talented and passionate people, and I’m proud to work alongside them!"

1 / 10(Current Item: 1)

Ongoing collaborations

We actively collaborate in various formats with universities around the world.

Utrecht University
Netherlands

We collaborate with Hieke Keuning on the SE Simulation project, focusing on designing and conducting studies together and co-authoring joint publications. Our partnership also includes organizing joint events and seminars to exchange ideas for research projects. Additionally, Hieke Keuning is the daily supervisor for Anastasiia Birillo's PhD.

TU Delft
Netherlands

We lead the Intelligent Teaching Assistant in Programming Education track from the industry perspective within the AI4SE collaboration. Our primary focus is the AI Metaphors project, which forms a key part of Yuri Noviello's PhD. In this initiative, we oversee the technical aspects and actively contribute to shaping the direction of the research.

University of Toronto
Canada

We collaborate with Michael Liut on the AI Metaphors project, focusing on a joint evaluation of the using this type of representation in a CS classroom, followed by a joint publication. Looking ahead, we plan to expand our collaboration by working on joint projects like SE Simulation, or Education-focused LLM evaluation in a for of thesis projects and exchanging research ideas.

ETH Zürich
Switzerland

We recently collaborated with April Wang on the AI-Debugging project, focusing on designing and conducting a joint evaluation of the proposed approach, followed by a joint publication. As part of this partnership, we also delivered a guest lecture at the university. Looking ahead, we plan to expand our collaboration by working on joint thesis projects and exchanging research ideas, particularly in connection with the Cognifire project and creating engaging in-IDE courses together.

Collaborate with us

We are happy to take part in seminars or guest lectures. You can come and meet our team, or we can present our work to your group or at your event.

We are also open to collaborating on existing projects or setting up new ones in our areas of interest. Possible formats include validation design, user studies, or other types of research involving human participants.

Write to us at edu-research-team@jetbrains.com to learn more.