Measuring How Mastory Works: A Pilot Study

Aug 22, 2021

What do you do if you have a wonderful idea about how to make learning mathematics really exciting? You develop a product that is available for any student to benefit from it. But how do you convince math teachers, school principals, and district leaders to give it a try? You provide evidence for its effectiveness.

We will provide this evidence by having hundreds of students use the Mastory App over a whole school year. During that period, we will track students’ activities and ask them about their experiences several times. At the beginning and end of the school year, an external research partner will conduct research with the students and their teachers about their experiences. This is the rough outline of our pilot study.

Dimensions of improvement

Current norms, beliefs, and practices of mathematics education tend to perpetuate existing hierarchies that privilege white middle and upper-class students, native English speakers, and boys while marginalizing Black and Latino students, students from families facing economic disadvantage, English learners, and girls. 1

By focusing our efforts on redistributing classroom power to students and elevating their assets and deep interests, we want to overcome the “math people myth” – the misled but still widespread belief that mathematical proficiency is naturally limited to a select group of students. At Mastory, we believe that mathematics must not be considered a privilege. We make math accessible to every student, independent of their origin, gender, cultural heritage, or family’s socioeconomic status.2

In our pilot study, we evaluate the effects Mastory has on Black and Latino students and on students experiencing poverty from large urban areas of the United States who learn Algebra 1 in 9th grade. We focus on three aspects of students’ relationships to mathematics:

1. Positive experiences in mathematics learning environments;

2. Students’ positive mindset about math;

3. Mathematical growth and proficiency.

We also aim to change teachers’ attitudes in several respects. We hope to increase positive beliefs and mindsets about Black and Latino students’ learning math. We believe that the adventures offered to Mastory students and teachers can bring about these positive changes. But how can we be certain that these positive changes will really occur?

Catching the wave of change

We plan to measure the outcomes mentioned above by combining a correlational and a comparative study approach. In the correlational study, we will distribute a baseline survey at the beginning of the school year and then we will ask the same questions in a follow-up survey at the end of the school year. This way we can keep track of and quantify changes in students’ attitudes over time. Teachers will be interviewed as well about their attitudes and opinions at the beginning and end of the school year.

In the comparative study, we will compare reports from students who experience Mastory throughout the year with those from a control group, in which students learn mathematics without Mastory. In this way, we will be able to compare the experiences and changes in attitudes of the two groups and develop a better understanding of how Mastory affects the way students learn mathematics.

To ensure independent research, these surveys and interviews will be conducted by an external research partner, the American Institutes for Research (AIR).

Measuring all components

Mastory is composed of four different components:

1. Students’ App: The core piece of our solution. Before and after their regular classes, students will use this application on their cell phones to meet the characters who drive the science fiction story. After class, students can apply their new knowledge to solve challenges in the Students’ App, which in turn will reveal the next levels of the story.

2. Teacher’s Dashboard: On this platform, teachers can follow their students’ activities and connect the contextualized Mastory challenges to the topics covered by their mathematical instruction.

3. IoT Rover: In order to make Algebra I even more tangible, this programmable real-world robot car will be used in the classroom in connection with the story.

4. Webinar: Teachers will be made familiar with the Mastory concept and storyline through an onboarding webinar before the pilot study begins.

The development and testing of all these components have already begun with priority students and their teachers in Chicago. We believe it is crucial for our priority students (Black and Latinx students and students who experience poverty) to really enjoy Mastory and to stay engaged with the story and its characters. Thus, we have dedicated ample time and resources to the co-creation, testing, and improvement of the Students’ App and Dashboard before the pilot study year.

Research methods

In order to measure the changes that we are aiming for, we need to know when and how the four components of Mastory were applied (utilization), how they were used (usability), and how useful students and teachers found them (usefulness).

We defined twenty learning questions divided into those three categories. These questions will be answered throughout the pilot study year by analyzing the collected data at regular intervals.

To ensure that we can capture the authentic voices of priority students, we will apply a variety of data collection methods, including focus groups, interviews, in-app surveys which include open-ended questions, and digital usage data logs. Digital data collection will happen continuously, while surveys, interviews, and focus group meetings will be conducted before the midterm and near the end of the school year.

We will analyze the collected data quarterly and at the end of the pilot study year with the assistance of our research partner, the AIR. As a relatively small organization, we can quickly respond to the findings that emerge from our data analysis. We can adjust the course just in time and determine with great accuracy which changes in the project lead to what consequences.

After the final analysis, we will publish the results not only as a scientific study but also as a public booklet to inform students, parents, teachers, school directors, and districts about the study results.

Conducting such an extensive study involving several researchers, approximately 500 students, 15 teachers, and other school personnel over a year is quite costly.

To finance our pilot study, we received funding as part of the Balance the Equation Grand Challenge of the Bill & Melinda Gates Foundation.

Follow us!

To learn more about the concept and implementation of Mastory, please stay tuned for our next blog entries and read the ones we have already posted. Be a part of our exciting journey towards a better math education for everyone!

————————