At the University of Johannesburg, the CALTSTEAM (Centre for Advanced Learning Technologies in STEAM) Research Center is pioneering a unique approach: using VR, AR, and AI not just as teaching tools, but as bridges between African cultural knowledge and modern science education.
A lot of scientific knowledge has a grounding in cultural knowledge. Unfortunately, due to colonization, much of that knowledge has been appropriated. We decided to acknowledge the cultural knowledge and use it to support the understanding of STEM concepts.
The Three CALTSTEAM Projects
KAVAS
Culturally Anchored VR/AR Simulations
LABIR
VR Microteaching with Learning Analytics
AI Tutor
Socratic Inquiry-Based Learning
Using African cultural knowledge—traditional beer brewing, drumming, moon phases—to teach abstract science concepts like chemical reactions and sound waves.
KAVAS (Culturally Anchored Virtual and Augmented Reality Simulations) acknowledges that students' cultural background profoundly influences how they develop science concepts. The project creates VR experiences that connect indigenous knowledge with scientific understanding.
Four Cultural-Science Connections:
- Inkomboti (Traditional Beer): The complex chemical reactions in brewing become tangible—students can pull molecules apart, break chemical bonds, and form new ones in VR
- Cultural Drumming: Sound wave transmission at the molecular level, using African music traditions to teach Grade 10 physics
- Phases of the Moon: Interviews with the Khomani San (indigenous people) about lunar beliefs, transformed into VR simulations teaching astronomy
- Bioethanol Distillation: Village-level energy production teaching the distillation process
Umesh Ramnarain: "Within virtual reality, the learners are able to engage with the molecules and they are able to pull them apart, break the chemical bonds and then put the atoms together to form new chemical bonds. So we decided to provide this sort of experience to the kids on the making of the traditional beer in Inkomboti... We actually visited a province which is in the Northern Cape and there we have got the indigenous community which is the Khomani San... We conducted interviews with them and tried to understand their concept of the phases of the moon because this is very important because this actually informs the way in which they live their lives."
Student teachers practice in a VR classroom with 5-6 avatar learners. Every gaze, movement, and interaction is tracked and fed back for reflection.
LABIR (Learning Analytics in Virtual Reality) puts student teachers into a fully simulated VR classroom where they practice microteaching before facing real students. The system captures rich data that was previously impossible to measure.
Data Captured:
- Lesson Timing: How time is allocated across introduction, middle, and conclusion phases
- Teacher Gaze: Where is the teacher looking? One student? The whiteboard? The entire class?
- Hotspot Tracking: Movement patterns as participants navigate the VR classroom
- Dialogic Interaction: Conversations between learners and with 3D models
- Spectator Recording: Full lesson recording for reflective playback
All of this feeds into post-lesson reflective sessions. The teacher educator scores on a rubric, and the student teacher receives feedback that was previously "intangible."
Umesh Ramnarain: "An important aspect of teaching is how do you focus yourself when you are teaching? For example, if you are doing a PowerPoint presentation, do I only focus on my slides or do I make the interaction, the facial contact with learners? So we looked at that. It's more from a pedagogical perspective. It's about creating awareness for the teachers. Sometimes what we also see is that when a teacher is teaching, a teacher tends to maybe only focus on maybe one or two kids within the classroom."
Unlike most AI tutors that give immediate answers, this one is deliberately Socratic—it scaffolds, prompts, and leads you toward understanding. Three tries, then you get the answer.
The AI Tutor is built on customized ChatGPT (GPT-3.0) and specifically designed to not give direct answers. Instead, it embodies the spirit of inquiry-based learning itself.
How It Works:
- Scaffolded Dialogue: Ask "What is photosynthesis?" and it responds with "What is your understanding of synthesis? What do you mean by photo?"
- Thought-Provoking Questions: Leads students toward synthesizing understanding rather than receiving information
- Three-Strike Rule: If you don't get anywhere after three attempts, it provides the answer
- Dashboard Tracking: Tracks question types over time—basic vs. advanced, factual vs. conceptual
Key Finding:
Over a year of use, the questions evolved. Initially: "What is guided inquiry?" Later: "How do you scaffold a learner who's stuck without taking control of the learning situation?" The level of questioning increased with engagement.
Umesh Ramnarain: "So it is a Socratic tutor, so it's a dialogue that you have with the tutor. We kind of coded it that when you ask a question, it's not going to give you the answer. It's kind of going to lead you towards the answer... If you ask what is photosynthesis, it kind of would scaffold you and say, 'What is your understanding of synthesis? What do you mean by photo? Do you know words with photo?' So it kind of leads you towards synthesizing something using light."
Forum Q&A: Deep Dive into the Research
Janice Gobert asks about leveraging data for AI assessment. Umesh explains the shift from quantitative to qualitative measures—and why interviews reveal what tests cannot.
Janice Gobert probed the practical question: with all this VR data being collected, what can actually be extracted and used for AI-based assessment?
Key Findings:
- Qualitative Over Quantitative: One-on-one interviews and focus groups reveal more than standardized tests
- Misconception Correction: Student teachers arrive with their own misconceptions—the technologies help address them
- Conceptual Improvement: After 2 years, clear improvement in conceptual understanding of phenomena
- Dual Growth: Students improve both pedagogically AND conceptually through technology interaction
Janice Gobert: "Can you tell me more about the different sources of data? What is it outputting? What are the data outputs so that you could leverage AI for assessment?"
Umesh Ramnarain: "We have not relied entirely on quantitative measures. It's been mainly interviews... We have noticed that as they engage more and more with these advanced learning technologies, they are able to address the misconceptions. These student teachers, they come from historically disadvantaged schools. So as much as we talk about inquiry-based learning, they have not experienced it firsthand. Through the interaction with technologies, they improve pedagogically and they also improve conceptually."
Knut Neumann raises the dashboard problem: 200+ students generating hundreds of questions. The solution? Aggregation, AI support, and inductive teaching.
A key question from forum participants: with hundreds of students asking questions to the AI tutor, how do teacher educators make sense of all that data?
The Aggregation Approach:
- Class-Level View: Data is aggregated across 200+ student teachers to identify global patterns and deficiencies
- Inductive Teaching: Real student questions become entry points for teaching theory—connecting practice to theory
- Individual Tracking: Students stuck at Level 1/2 questions are flagged for one-on-one lecturer intervention
- Selective Extraction: Lecturers pick "interesting scenarios" to illustrate different scaffolding strategies
Knut Neumann: "If I have 25 students in my class, there'll be hundreds of questions. How do you feed back that information? It takes teachers a lot of time to really process all this information."
Umesh Ramnarain: "So it's kind of aggregated with the entire class... We are able to now zoom in on aspects. This information is then available to our teacher educators or the lecturers. And then they are able to use it and address it in their lectures. So that's aggregated data. We are also able to generate data for individual students so we can track their development."
Yasemin clarifies the VR setup: students are real people using avatars, feedback comes after the lesson via learning analytics, and the system uses a rubric—no real-time AI intervention.
Yasemin Copur-Gencturk sought to understand the technical architecture: are the VR learners simulated or real? When and how does feedback occur?
Technical Clarification:
- Real Students, VR Avatars: 5-6 fellow student teachers use headsets and choose avatars—they're not AI-generated characters
- No Real-time AI: The VR classroom has a learning analytics dashboard, not AI intervention during teaching
- Post-Lesson Feedback: Micro-lessons are 15 minutes max; all feedback is shared afterward in reflective sessions
- Rubric-Based Scoring: Teacher educators score using a built-in rubric, now able to measure previously "intangible" aspects
Yasemin: "So for this virtual reality environment, students are not virtual. Students are real students?"
Umesh Ramnarain: "They use the headset and then they choose the avatar... The virtual reality classroom does not have AI. It's got a learning analytics dashboard. All of that would be shared with them—the report which would be generated would be later. But now, there is so much that can be shared, because you are able to track all of these, and previously these would be intangibles."
Yasemin explores the AI tutor's mechanics. Key insight: three tries to guide you, then it gives the answer. The goal is scaffolding, not information delivery.
The AI tutor is explicitly designed NOT to give direct answers—a deliberate pedagogical choice that embodies inquiry-based learning principles.
Walkthrough Example:
- Student asks: "What is photosynthesis?"
- Tutor responds: "What is your understanding of synthesis? What do you mean by photo? Do you know words with photo?"
- Student tries again: (guided toward "synthesizing something using light")
- Three strikes rule: After three unsuccessful attempts, the tutor provides the answer
Yasemin: "So if you walk me through one of the examples... So I think one of the questions was, let's say, what is photosynthesis? How does it work?"
Umesh Ramnarain: "If you ask what is photosynthesis, it kind of would scaffold you... 'What is your understanding of synthesis? What do you mean by photo? Do you know words with photo?' So it kind of leads you towards synthesizing something using light."
Yasemin: "Three tries. If you don't get anywhere, it's gonna give you the answer?"
Umesh Ramnarain: "Okay. All right."
Janice Gobert connects to Mickey Chi's ICAP framework: question types reflect learning orientation. The data becomes an entry point for teaching theory inductively.
Janice Gobert raised a deeper epistemological question: how are teachers being oriented toward causal mechanisms rather than surface-level summarization?
The ICAP Connection:
- Mickey Chi's Framework: Question types reveal cognitive orientation—summarization vs. causal understanding
- Orienting Tasks Matter: When you tell students "you're going to draw after this" or "you're going to explain after this," their information processing changes
- Inductive Approach: Real student questions become teaching examples—extracting "interesting scenarios" to illustrate scaffolding strategies
- Theory-Practice Bridge: The technology enables connection between the practice (questions asked) and the theory (types of scaffolds)
Janice Gobert: "Mickey Chi—do you know her? She has this ICAP framework and she looks at the types of questions... Some of them stay very focused on the summarization level. What is the extent to which you are kind of prompting them to seek understanding of the causal mechanisms? Because if you have that kind of epistemological framing as a goal, they will approach this differently."
Umesh Ramnarain: "So it's kind of become now more of an inductive approach, where through the manner in which the students interact, these are great examples that you pull into the classroom when you're teaching the theory. The type of questions need to be cognitively probing questions, prompting questions, clarifying questions. We notice that the types of questions the students are asking are great examples to illustrate different types of scaffolds."
Through interaction with these technologies, student teachers improve both pedagogically and conceptually. Many come from historically disadvantaged schools and have never experienced inquiry-based learning firsthand. Now they can.