JetBrains logo

Education Research

Leveraging KOALA for Programming Data Collection

This half-day tutorial is designed for EDM-2025 attendees interested in collecting programming-problem-solving data from JetBrains IDEs for research purposes.

Tutorial structure

Tutorial overview

This tutorial is divided into four parts, guiding participants through the process of setting up a data collection process for in-IDE courses and a custom repository, and analyzing the data collected:

Overview of KOALA and the in-IDE learning format

Setting up KOALA, a tool for collecting data during problem-solving

Collecting data for a predefined in-IDE course and a custom repository

Analyzing the collected data

Tutorial structure

Prerequisites

Please ensure you have the following installed:

A JetBrains IDE

You can use either IntelliJ IDEA (Community or Ultimate) for Kotlin and Java courses or PyCharm for Python courses.

The JetBrains Academy plugin

This plugin for in-IDE learning can be installed directly from your IDE.

The KOALA GitHub repository

This repository should be cloned on your laptop so it will work during the tutorial session.

Instructors

The practical part of this tutorial will be led by three instructors, with Education Research team members Rustam Sadykov and Aleksey Rostovskiy providing classroom support.

Daniil Karol
Researcher with the JetBrains Education Research team
  • 5+ years of programming experience
  • 1.5 years of research experience
  • 1 year of teaching experience
Katharina Dzialets
Researcher with the JetBrains Education Research team
  • 4 years of product management and product research experience
  • 5+ EdTech products and projects built
Anastasiia Birillo
Head of JetBrains Education Research
  • 8+ years of programming experience
  • 5 years of research experience
  • 5 years of teaching experience
Tutorial structureDetailed description

Tutorial structure

Each part of this tutorial is designed to be independent. If you miss a part, you can download the materials and continue at your own pace.

Part 1. KOALA tool and in-IDE learning format overview (~30 min)

Part 2. Setting Up KOALA (~45 min)

Part 3. Data Collection (~45 min)

Part 4. Data Analysis (~30 min)

Q&A (~30 min)

We will have a 10-minute break after each part.

Detailed descriptionUseful links

KOALA tool and in-IDE learning

Part 1

The goal of this part is to provide participants with an overview of the in-IDE learning format and the KOALA tool to give the audience more context.

It consists of the following parts:

  1. General introduction: Start with a general introduction to this tutorial.
  2. In-IDE Learning: Describe possible ways to learn programming inside JetBrains IDEs, including the JetBrains Academy plugin and standalone coding projects.
  3. KOALA tool: Describe the tool's purpose and possible ways to use it for data collection processes.

By the end of this part, participants will be familiar with the framework for the upcoming practical part.

Setting Up KOALA

Part 2

This section focuses on setting up the KOALA tool and demonstrating its data collection capabilities.

It consists of the following parts:

  1. Introducing the KOALA server: A basic overview of the server for data collection and storage.
  2. KOALA plugin configuration: Setting up the configuration files according to your needs (Task Content, Info, Agreements, Settings and Inspections, Server Error Message and Final Page, Surveys, and Scenarios).
  3. KOALA plugin student view: A look at the plugin from the students' perspective.

All configuration files are language-independent except for the Task Content configuration files, which list the files that will be tracked.

The KOALA starting page

The KOALA task page for an in-IDE course

Data Collection

Part 3

The goal of this part is to complete a predefined in-IDE programming course and complete programming tasks from a standalone project in student mode in order to collect task-solving data.

It consists of the following steps:

  1. Collecting data for an in-IDE course: Use KOALA to collect data during the task-solving process inside an in-IDE course and observe the process from the students' perspective.
  2. Collecting data for a custom repository: Use KOALA to collect data during the task-solving process inside a standalone repository with dynamic configuration of the tasks and the environment.

Completing tasks for an in-IDE course in student mode

Completing tasks for a custom repository

Data Analysis

Part 4

The goal of this part is to download the collected data and perform basic analysis.

It consists of the following tasks:

  1. Downloading the collected data: Download the collected data from a remote server.
  2. Performing basic analysis: Perform basic analysis using an interactive dashboard in a web browser.
  3. Converting to ProgSnap2: Convert the data into the standard ProgSnap2 format to make it compatible with many other available research tools.

Visualization dashboard main page

Example of basic statistics calculated using a sample of collected data

Example of the Top Actions window showing statistics with a sample of collected data

Useful links

Download these products and materials relevant to the tutorial:

Free JetBrains IDE licenses

JetBrains offers free educational licenses for all IDEs for teachers and students to use in the classroom.

JetBrains Academy
plugin

The JetBrains Academy plugin enables educators to design and manage courses directly in JetBrains IDEs.

KOALA
plugin

KOALA is a plugin for collecting data during the task-solving process and gathering information about code snippets and IDE interactions.

Tutorial
materials

This repository contains all the materials from this tutorial for future use.

Collaborate with us

The Education Research team is happy to take part in seminars or guest lectures. You can come and meet our team, or we can present our work to your group or at your event.

We are also open to collaborating on existing projects or setting up new ones in our areas of interest. Possible formats include validation design, user studies, or other types of research involving human participants.

Write to us at edu-research-team@jetbrains.com to learn more.

Contact usBack to tutorial structure