Questions are scarce, answers are not

1 Questions are scarce, answers are not

For decades, educational inequality in Brazil was described as a problem of access: access to books, to laboratories, to well-trained teachers. That diagnosis aged quickly.

Content explained patiently, at whatever reading level the student asks for, at any hour, is no longer scarce. What remains scarce is something else: knowing what to ask, understanding the answer, and doubting it. That is where inequality reorganises itself.

The Center for Curriculum Redesign frames school in four dimensions — knowledge, skills, character and meta-learning — and argues that an age of AI shifts the weight towards the last three (Fadel et al., 2024). Our contribution is not the framework; it is what happens when a small organisation tries to operate it with untrained volunteers in public schools, and then tries to measure it.

The hypothesis has an immediate practical consequence. If what is missing is the question, the lesson does not need a subject expert to transmit content; it needs someone who triggers the question and then makes the student present and revise the answer. A volunteer trained in a fixed protocol can do that. We set out to test it, and this paper reports what happened — including what went wrong in the measurement.

Three questions organise the report:

• RQ1. Can volunteers without a teaching background, trained in a fixed protocol, run AI-mediated lessons at scale?
• RQ2. Is there a relationship between exposure to the programme and performance on a content test?
• RQ3. What happens to the validity of that measurement when the person being assessed has AI available?

2 February 2023

To answer those three questions we start at the beginning: how an idea from two entrepreneurs became, in three years, a network of more than a thousand students.

ChatGPT reached Brazil in January 2023. The first Professor Voluntário lesson took place in February, already designed entirely around the tool. Both founders are technology entrepreneurs with degrees in business administration who taught themselves to program by searching Google — a path that, in their own words, is now "one prompt away".

The first cohort had twenty students, closely followed. Out of it came games built by the students themselves, e-books written and sold online, music produced from independent research, and admissions to competitive selection processes. We record this for what it is: anecdotal evidence from a small, intensively supported group, with no instrument and no counterfactual.

From 2024 the network grew from fifteen to more than a thousand students, across public schools and social institutions, and individual follow-up became impossible. That loss is what led us to introduce a standardised test — and with it, the problems this paper discusses.

3 We do not teach; the student learns

Before turning to what the test measured, it helps to say what exactly is being tested: the method behind the lesson.

The method fits in one sentence and is deliberately unoriginal: the volunteer does not explain the subject first. They provoke, send students to research, and require a presentation.

The five-step protocol

• Provocation. A closed question or short activity that creates curiosity about the topic.
• Research with AI. The student writes prompts, tries different formats, and rephrases when the answer is unclear.
• Study. Reading and interpreting what came back.
• Presentation. The student tells the class what they understood. This is the step that separates the method from copy-and-paste.
• Teacher explanation. Closes, corrects and connects.

The rule for the volunteer is never to leave the five steps, whatever the subject.

Triggering, not teaching

Step two is where the volunteer's real job becomes visible, and it is narrower than "teaching": it is triggering. The student already has the tool and is comfortable with it. What they lack is a reason to open it, a first question worth asking, and a format that turns the answer into something they have to process rather than copy. So the volunteer carries a small, fixed repertoire of prompt templates — six of them, in plain Portuguese, written for a teenager to type verbatim and then modify.

The repertoire is not a menu of equivalent options. Each template has a job, and the job determines when in the lesson it may be used. Three roles, mapped onto the five steps:

The distinction between the three roles is the part volunteers get wrong most often, and it is what training insists on. A provocation prompt used after the study adds nothing; a consolidation prompt used instead of research produces a student who can recite a meme and explain nothing. The comic strip is the one template that legitimately works at both ends — as the opening provocation, or afterwards as a summary — and even then never in place of the explain prompt.

Two design decisions matter more than the list itself. First, research is never optional: the playful formats provoke or consolidate, but comprehension happens when the student reads a real explanation and asks again about the part that did not land. Second, every template ends in something the student must do in front of other people — compare, present, defend the joke, answer the next question. The prompt starts the process; the classroom is what keeps it from ending in a copied paragraph.

Framed this way, the role of the volunteer is not to know the subject. It is to trigger: to choose the provocation, hand over the right template at the right moment, and then demand that the answer be turned into speech.

The participation scale

The central pedagogical instrument is not the test; it is the scale the volunteer uses to rate each student's participation:

The goal of each lesson is to move up one level, never to jump from zero to four. Level 4 is the operational definition of our thesis: the student who doubts the machine and asks the next question. One limitation is worth stating up front: until now this scale has lived in the volunteer's informal judgement and was never recorded as data — a gap section 8 sets out to close.

The future belongs to those who can ask artificial intelligence the right questions, read and interpret the answers, and have the critical sense to challenge them and ask new questions that deepen the knowledge.

This is said aloud at the start of every lesson. It is not an institutional slogan: it is an instruction to the student about what is expected of them in the next hour.

4 How you manufacture a teacher

This method is only useful if it survives without the founders in the room. Hence the question of this section: how do you produce, in series, someone able to run it?

A method that depends on its founders does not travel. Our answer was to turn practice into material: every lesson documented interaction by interaction, with a reference code, interaction type, objective, suggested script, estimated time and measurement method. There are 257 documented interactions across the two financial-literacy modules. The volunteer receives the plan and is free to teach their own way, provided the five steps survive.

The commitment asked of a volunteer is two hours a week, twelve lessons, plus training in the method. Each one is assigned a "concierge teacher", available on WhatsApp for questions and classroom emergencies.

Recruitment closes the loop, and deserves attention because it is what sustains scale without a budget: after each lesson the volunteer fills in a form and uploads photos; the organisation publishes a post tagging whoever taught; the volunteer shares it with their own network; new candidates arrive. The person delivering the service is also the acquisition channel, and what they get in return is public recognition. The marginal cost of an additional classroom is, essentially, coordination.

Of 130 who answered that question, 90 (69%) came through social media and 35 (27%) through a referral from a friend or acquaintance; one came from an event, and four picked "other". Ninety-six per cent of our volunteer pipeline, in other words, arrives through channels the volunteers themselves feed — no advertising spend, no paid recruitment.

The funnel, as recorded in our own system: 124 people applied, 61 were approved (49%), and 52 of those reached the volunteer WhatsApp group (85% of the approved). Thirty-two volunteers have taught in more than one cohort, a 37% return rate among the 86 with at least one recorded cohort. Six schools are registered, and 86 post-lesson forms cover 85 distinct lessons. We report these numbers with an admission attached: allocation to a specific classroom, and the dates of each funnel step, are not stored in a way that lets us compute conversion times — a gap in our own instrumentation that the next cohort fixes.

5 The filter that manufactured a result

What follows is what we nearly failed to see — which is why it comes before the numbers.

Students answer the test with a phone in hand. Very high scores kept appearing in pre-tests from classes that had never had a lesson, concentrated on particular days and in particular classrooms, sometimes with identical answer patterns among classmates. Faced with that, the team adopted a cleaning rule that looked sensible: discard pre-test responses above 15 points, as "impossible".

Nobody proposed the symmetric rule. A high score in the post-test did not look impossible — it looked like success.

It is worth being explicit about the magnitude: roughly three quarters of the "result" we could have reported came from the ruler, not from the lessons. And there is an aggravating factor we only found when rebuilding the analysis: because the test grew from 28 to 31 questions in mid-2025, the same 15-point cut meant different things in different periods.

We dropped the rule. We now report the within-student gain and the dose-response relationship, both published here with n in front, and we keep the sensitivity analysis as a permanent part of the report, so any reader can see what each cleaning choice does.

6 How we measured, and what we found

On the volunteer side, our measurement still has gaps. On the student side there has been a dedicated instrument since 2024.

The instrument is a financial-literacy content test administered by online form, answered before and after the lessons. Students who report never having attended a module lesson count as pre-test; those who report having attended count as post-test and also report how many lessons they attended. The design is therefore a pre-post comparison of non-equivalent groups — not the same people, not randomly assigned, merely comparable — without randomisation and without a control group — the weakness section 8 tries to fix going forward.

Two technical notes, for honesty. First, the form changed size midway: 277 responses were collected on a 28-question version, between February 2024 and May 2025, and 366 on a 31-question version from June 2025 onwards. Every figure here recomputes the score over the 28 common questions, so that the two periods are comparable. For quick reading: the test has 28 questions, so means of 14.6 and 16.2 are 5.2 and 5.8 on a 0-10 scale, and the 3.1-point within-student gain is 1.1 points on that scale. Second, the answer key was not given to us: it was inferred from the responses, and it reproduces the exact recorded score in 271 of the 277 original-version rows.

The whole sample first

Comparing everyone who took the test before any lesson with everyone who took it after, scores go from 14.6 (n = 416, SD 5.8) to 16.2 (n = 286, SD 6.2). The difference is 1.6 points on a 28-point scale. The statistical test says it is not chance (p = 0.0006), and the 95% confidence interval puts the real effect between 0.7 and 2.5 points. As an effect size — the difference divided by the natural variation between students — that is d = 0.27, a small effect.

The obvious objection is composition: the two groups are not the same students, and some schools are over-represented on one side — Pão dos Pobres and Nações Unidas lean pre, Bahia leans post. So we re-estimated the difference within each school and within each year of administration, and weighted the results back together. Neither control removes the effect: the school-weighted difference is 1.7 points and the year-weighted difference is 1.5, against 1.6 raw. Whatever this is, it is not an artefact of which schools answered when.

The highlighted row is the one we would defend hardest. It compares students who all attended the programme, differing only in how much of it they got, which removes the question of who chose to be measured when. It is also the row that carries the least statistical baggage and lands at roughly half a standard deviation.

For scale: the meta-analysis of PreK-12 tutoring by Nickow, Oreopoulos and Quan (2020) finds a pooled effect of 0.37 SD across 96 randomised trials, with weaker effects for non-professional tutors than for teachers and paraprofessionals. Our 0.47 sits in that neighbourhood, and it is delivered by volunteers with no teaching background at a marginal cost close to zero — but it comes from a comparison without randomisation, so the right reading is "same order of magnitude", not "better than tutoring".

Dose matters

Among students who had already attended lessons, performance tracks the number of lessons attended. This is the most robust result we have, because it does not depend on identifying the same student twice and because the comparison happens entirely within the exposed group.

The first step is instructive: students with one to three lessons are no better off than students with none. The effect appears after the third lesson and consolidates after the sixth. That is consistent with a method that depends on students changing behaviour rather than receiving information — a change that takes some weeks. Treating the four bands as an ascending scale, each step adds 0.87 points on average, with a margin of error of 0.18. The counterpoint: on its own, the number of lessons explains only 3% of the variation in scores — the rest comes from things we did not measure.

The same student, as a check

Identifying the same student at both ends proved harder than expected. Names and e-mail addresses are free-text fields, full of abbreviations and typos, so we applied a cascade of criteria: identical e-mail; full name plus school; approximate name by edit distance within the same class; e-mail with a mistyped domain. The cascade yielded 81 pairs. The ceiling is low for a reason that is not technical: in most classrooms the pre-test and the post-test were answered by different students, in sessions months or years apart.

Across those 81 students the mean gain was 3.1 points out of 28, with a median of 3 and half the cases between 0 and +7. Sixty-nine per cent improved, 6% stayed level and 25% got worse. Gains are larger among those who started lower — 6.6 points in the bottom quartile against 1.3 in the top — but the negative correlation between initial score and gain (ρ = −0.37) is also what regression to the mean and a ceiling effect would produce, so we do not read it as evidence that the programme favours weaker students.

One caveat belongs here rather than at the end: the students we can see at both ends are the ones who were still there and filled the form properly. We measured the unpaired students to check — their gain is 1.0 point against 3.1 for the paired ones, and the two groups start level (14.60 vs 14.58) but end apart. The paired figure is therefore the effect among those who stayed, not the average effect of the programme. The same caveat applies to the dose table: students who reach the sixth lesson are not a random sample of those who started.

Self-reported confidence

The form also asks students to rate themselves from 1 to 5 on two dimensions. The first barely moves: comfort studying with AI tools arrives high, 4.1 before and 4.1 after, with half the students picking the maximum before their first lesson. The second does move: comfort talking to someone about the economy rises from 2.5 to 2.8, and among paired students the mean change is +0.24. The reading is straightforward: they do not need us to stop fearing the machine; they need us to have something to say.

What they write about the tool

The same form has carried two open-ended questions since March 2024 — what do you understand ChatGPT to be? and how do you think it can help your education? — which give us 1,272 free-text answers. Comparing the vocabulary before and after, with inflections and synonyms grouped into families and a z-test on the share of answers containing each family, two movements stand out and the rest of the vocabulary sits still.

Students describe the object differently: fewer name an entity ("an artificial intelligence") and more name an instrument ("a tool", "a resource") — a shift that is sharper in the question asking what ChatGPT is, where the tool family rises from 24.3% to 33.6% of answers (p = 0.011). And they stop declaring ignorance — the twenty-five catalogued ways of writing "I don't know", from "n sei" to "never used it", fall by a factor of two and a half among those who wrote anything at all. What does not move is the language of critique: mentions of error, checking, sources or dependence sit at about 4% before and after, with no gradient by number of lessons.

The two clouds

The same corpus, seen word by word: on the left, students who had not yet had a lesson; on the right, those who had. Size is the share of answers containing the word. The clouds illustrate; the table above decides — a cloud shows who wrote more, not who changed.

We read this as a boundary rather than a victory. Twelve lessons of using AI change how students name the tool and remove the "I don't know", but they do not, on their own, produce students who talk about the tool's limits. That is the gap section 7 tries to address. Two cautions apply: before and after are mostly different students, and answers average twelve words either way — brevity that a richer instrument would avoid.

7 The test is dead; long live the question

The underlying problem is not our rule. It is that, in a content test answered with AI available, the signal of learning and the signal of consultation are identical on the answer sheet. No statistical criterion separates them defensibly, and every attempt to separate them embeds the expectations of whoever wrote the criterion. The problem is general: in Brazil, 70% of internet-using secondary students already use generative AI for schoolwork research, and only 32% have received any school guidance on how to use it (ICT in Education 2024).

Our proposed way out is not to ban the tool, but to measure what it cannot deliver on its own. Four axes, offered as an agenda to be tested rather than a validated instrument:

• Question quality. Given a concrete situation — deciding whether to pay in instalments, say — the student writes the prompts they would use. Scored on context, specificity, and the presence of a follow-up question that goes deeper.
• Critique of an answer. The student receives an AI answer with a planted error, typically numerical, and is asked what is wrong, how they would check it, and what they would ask next.
• Oral explanation without reading. One minute of unscripted explanation, scored on a simple rubric. It is what the participation scale already does in class, and it is cheating-resistant by construction.
• Transfer. Applying the concept to a new case presented in class, with no prompt text available to feed a machine.

The last two already exist in our practice and only need to be recorded as data. The first two are what is missing. Frameworks for deciding how much AI an assessment should allow already exist — the AI Assessment Scale of Perkins, Furze, Roe and MacVaugh (2024) is the clearest — but a framework that sets the permitted level is not the same as an instrument that scores the quality of a student's question or their critique of an answer. We found no validated instrument for either, which is at once our gap and our proposed contribution.

There is reason to take pedagogical design seriously, rather than access to the tool alone: in a field experiment with about a thousand secondary students, access to a generic assistant improved performance during practice but left students worse off on an exam taken without AI, while a version with pedagogical safeguards removed the harm (Bastani et al., PNAS, 2025). Our protocol bets on exactly that point: the oral presentation is the safeguard that stops research from ending in transcription.

8 Limitations, next steps, and an invitation

None of this fixes the weaknesses of the design we already ran. They deserve to be named before we propose what comes next.

What this study cannot claim

• There is no control group: nothing here isolates the programme's effect from maturation, other classes, or selection.
• Exposure is self-reported, in bands.
• The answer key is inferred, and the instrument changed length midway through the series.
• Schools, classes and years are pooled; small cells have unstable means.

The protocol for the next cohorts

• Pre-test and post-test administered in the same classroom, with a system-generated student identifier instead of a typed name.
• Cleaning rules fixed before looking at outcomes and applied symmetrically, with the sensitivity analysis published alongside.
• The 0–4 participation scale recorded at the start and end of each module, with anchor examples per level and double coding on a sample to measure agreement between volunteers.
• An AI-allowed assessment block covering the first two axes above.

The invitation

We keep fifteen to twenty public-school classrooms running simultaneously, taught by volunteers trained in the same protocol. That makes possible something this study lacked: assigning classrooms to variations of the method and observing the difference. We offer that network to researchers interested in designing and testing assessment for the AI era — including tests against our own hypothesis. The data and analysis scripts behind this report are available in de-identified form.