Language Learning App (LLA)

Virtual Reality UX Prototype for ESL Students

The Problem

Project to build an advanced UX prototype simulating Virtual reality.

The client, Elaine Nichols, ran a special private school for immigrant kids, often from low-income backgrounds. Since learning English is required for jobs, the client was interested in creating a self-learning smartphone app that taught ESL (English as a Second Language). In addition, the client wanted to add gamelike qualities to the app’s teaching, and exploit Immersive media – virtual and augmented reality – in the user interface.

In fall 2018, I was asked to design and develop an application prototype. The goal of the prototype was testing the general concept for ESL learning, along with creating a good simulation of virtual reality in a UX prototype.

What Was Needed

Product Research and User Storytelling

Initial Research was conducted into existing language apps, especially those targeted at children and tweens, rather than adults.

A series of visits with the school's students allowed construction of typical users for the app. Based on these discussions, I developed a list of user types for the application. Users were defined by demographics, likes, and dislikes. Since the app had strong gamification, user attributes like "type of battle preferred" during a test, challenge or student duel were defined along with regular UX attributes.

Sample User Persona for LLA application

Student Learning Models

In the first stage of research, I did a study of various learning styles. The goal was determining what kinds of things engage, and actually teach, for casual access. We envisioned ESL students, often on slow bus trips, going to the app to pass the time. The app had to enable casual learning. After analyzing the literature on casual learning, along with problems with current language apps, we came to the following product specifications:

apps showing stage1 , stage 2 stage 3 learning models

Competitive analysis of language app learning models.The leftmost screen shows Stage 1 style learning (memorization). The middle screen shows less common Stage 2 learning (context). The rightmost screen shows Stage 3 learning challenges, which are currently found in only a few language apps at present (communication)

Interaction Design

Research and User Models resulted in our choosing a "virtual world" model for the app, as opposed to the "flashcard" model common in other apps. Instead of memorization, learning a language let students manipulate an avatar in the virtual world. Similar to Pokemon, the avatar could be entered in challenges with other learners, and its success in battle depended on the student's understanding of the language.

diagram demonstrating interaction within virtual world
Methods of Interaction

Since the app's virtual world had to function on both high-end and low-end devices, we implemented a "Responsive Virtual Reality" using Progressive Enhancement, a model developed by Arturo Paracuellos of (his example below):

example of responsive WebXR app

(Source: Click image to link to original article)

We modified his Progressive Enhancement model as follows; VR, "magic window" smartphone VR, or an SMS conversation between a user and the app system or app avatars.

progressive enhancement of VR app, desktop for teachers, VR and magic windows for students, sms for parents

After defining user stories, defining interaction began. Since we were building a prototype, interaction focused on onboarding.

Low-Resolution Conversational Design

I used Conversational Design to define interaction in low resolution. Writing interaction as a screenplay between users and game avatars had ensured that many virtual world interactions could be accomplished by "texting" an avatar or student in the virtual world.

interaction dialog between user and avatar

I also created a conversational diagram for a challenge between two students

interation dialogue during student duel
Medium Resolution

Medium resolution design consisted of simple wireflows, mostly defining the user task completed on each screen during onboarding.

medium resolution onboarding using low-resolution wireflow

High-Resolution Interaction

High resolution was a standard Interaction Design Flowchart, incorporating conditional branches within onboarding for all users. Each step in the interaction was matched by number to the conversational design model created earlier.

high-resolution interaction diagram, initial onboarding high-resolution interaction, parental onboarding

Developing an Interactive Prototype Simulating Virtual/Augmented Reality

The client felt that paper prototypes would not work for the target student and parental audiences. So, we needed to create an app prototype that simulated Virtual Reality, as well as SMS communication and voice control. To create the app, we used ProtoPie ( Unlike most other UX prototyping tools, ProtoPie supports smartphone sensors including tilt/movement, camera, SMS, compass, and voice.

Simple Avatars and scene UI elements were created for the grey-box wireframes.

sample avatar and scene UI for grey-box wireframe

A series of onboarding screens were created. ProtoPie allowed us to make the screens actually work. For example, users actually had to type into the registration and login fields, and ProtoPie validated their input. Controls were touch-driven and animated. We were able to make "grey-box" wireframe screens of the app, and make elements onscreen interactive. The app could also simulate VR in "magic window" format. Student testers could move their smartphones around to explore the virtual world, interact with avatars, and manipulate objects.

three sample screens showing onboarding in the LLA app. Select language, select user type, select avatar

Interactive prototype screens for onboarding. Left: select language, Center: select user type, Right: select Avatar.

three sample screens from the prototype showing basic level 1 questions.

Interactive prototype screens for onboarding. The prototype challenges students with basic (Level 1) questions in order to help them learn the interface. Questions are functional and scored by the prototype.

prototype screens showing aspects of the virtual world simulation, viewable in 3d via 'magic window' movement of the users's smartphone. Left: check voice commands in vitual world, Center: Choose a virtual world, Right: see other students and challenges in the virtual world

Interactive prototype screens related to the virtual world. The left screen is a functional microphone that allows students to test voice control of virtual world elements. The center screen allows them to choose the virtual world landscape. The right screen is a leaderboard showing challenges and other students.

sample screens demonstrating possible interfaces for advanced questions and tests (Level 2, 3, 4)

Interactive prototype screens for advanced questions. To the left, a draggable element allows students to choose endings for worlds. In the center, students must create a sentence telling their avatar how to plant a crop. On the right, phrase choice to answer questions.

a sample screen showing SMS interaction in the LLA prototype

Interactive prototype screen for SMS communication. Users can address their avatar within the virtual world, or by SMS. Parents can check up on student activity via SMS, and teachers can send messages to students.

User Testing and validation

Live tests of the avatar were conducted with the primary "tween" student audience, ESL students aged 9-14. Usability tests were conducted for the onboarding strategy, and students were asked to give their subjective evaluation of the app's value to them.

scenes from tests of the interactive prototype with ESL students


Exit Polls

Students also completed surveys which helped to quantify their experience with the app prototype. Questions were designed to see whether testers were able to recognize specific user interface (UI) elements.

quantitative results from user testing with the interactive prototype
quantitative results from user testing with the interactive prototype

Based on the results, I constructed a "pitch deck" for the client to show to investors along with the interactive app prototype.

LLA Pitch Deck

(Click above to access the pitch deck)


The resulting app prototype is unique in re-creating a virtual world/game VR environment without hardcore coding. The app's use of smartphone sensors allowed testers to explore the virtual world created, and interact with it. In contrast to most UX prototypes, the logins and questions actually worked! This was essential, since the testers (9-15 years) were reluctant to comment on static wireframes or comps.

Other Immersive Media Projects: VR Real Estate App