Immersive Language Learning 5/2017 -5/2019
Employer: Cognitive and Immersive System Lab (CISL)
Location: Troy, NY
Position: UX researcher
-
Project goal​
Help students acquire Chinese (Mandarin) by using our immersive system:
​
- Students can practice their reading, speaking, and listening in Mandarin by completing
tasks in our system.
​
- Meanwhile, students can learn new vocabulary and phrases during interactions.
-
Primary research questions and challenges
- Do all the new technologies we implemented (AR, 360 Panoramic Display, Gesture
Recognition, Speech Recognition, Watson from IBM) really help users learn Mandarin
better?
​
- How do we bridge the gap between our users and those new technologies?
​
- What are some potential usability problems this system might have?
​
- Can this system fit into an everyday classroom setting?
This student is trying to order a cup of water in our virtual restaurant. But can she complete that?
-
Methods & Findings
When:
Used at the beginning stage of the project
​
To answer:
- How students usually study in the
classroom?
- How teachers involve technology during
teaching?
Field Study
Findings:
- Students need more help with character
recognition and pronunciation.
- Teachers use a lot of structured dialogue during
teaching. Technology-wise, teachers only use
PPT.
​
Deliverables:
- Develop new teaching functions:
provide pronunciation facilitate and character
recognition facilitate if students asked for help.
- A few system interactions and dialogues are
also modified accordingly.
When:
With the development of other prototypes
​
To answer:
- How does other software or systems
help students learning a language?
- What usability problems do they have?
Can we prevent that during development?
- What functions in other systems users
found useful?
​
Analyzed components:
- Pronunciation practice
- Characters writing practice
- Useful features
- Technologies involved
- Strength and Weakness
- Usability issue
Competitive
Analysis & Testing
Findings:
- A more advanced pitch contour platform from
Wai Chinese
- Usability issues in other software:
-
Confusing evaluation matrix - Various Softwares
-
Unable to cancel a written strike - Hello Chinese
-
No tone display in the pronunciation help
function - Busuu
-
Instruction and translation has no visual
separation - Ed Wonderland
-
No space between character tone - Memerise
-
... ... (Over 40 different software, 12 other problems identified )
​
Deliverables:
- Reference list of possible usability issues to
prevent. This list prevents us from making
worthless mistakes, thus saving us a lot of time.
- A new design for the pronunciation help
function. (See in design section below)
When:
From beginning to end, whenever we have prototypes. GoGo AGILE!
​
To answer:
- What are the potential usability problems?
- How do users solve usability problems
while using?
- Can users complete tasks using our
system?
Usability Testing
Findings:
- Usability issues
-
No visual cues for the help functions, users are still confusing after the demo
-
Even users can recognize the displayed tone in picture, they cannot pronounce it right without teacher using hand gestures to demonstrate the tone
-
Users barely use the speech recognition output as a reference since the output is in Mandarin
-
Users cannot pronounce the wakeup word of the system, caused extreme frustration
-
The system has a low tolerance for wrong pronunciation
-
... ...
- Unnatural dialogue flow
- User behavior patterns
-
When the system cannot recognize vocal input, users with beginner knowledge tend to repeat the same phrases. Users with more knowledge in Mandarin tend to use shorter phrases or rephrase the sentence.
-
Only one vocal help function is used more frequently: "what is this?"
- Corner cases
​
Deliverables & Solutions:
- Provide automated skip function and help
function. Provide help function suggestion
in a pop-up window when pronunciation
issues happen.
- New wake up word that is easy to pronounce
- A new design for the pronunciation help
function. (See in design section below)
​
Other methods we used including Focus group, Benchmark testing, and Persona. However, I was not leading those projects. With all respect to my awesome teammate.
Does the student order a drink successfully? Yes, she does.
After that, she wants to order a dish. But due to the lack of pronunciation instruction and skip function, the system cannot process her pronunciation and is stuck. Our avatar gives her 10 more cups of water. Oh No! But luckily we catch this during usability testing.
-
Design
The original pronunciation help function:
Some usability problems:
- The numbers on the left-hand side appear confusing to most beginner
level users.
- The users cannot pronounce characters correctly even with the amplified
tone trend.
The new prototype I proposed:
- Students can repeat after the avatar. The system will generate a trend
graph to help correct the pronunciation.
- Easier to use. More interactive. It provides more feedback.
- More vivid trend implication to help correct pronunciation.
- We get more positive feedback on this prototype.
- This is the most frequently used function among all other help functions.
-
Learnings
​
It is challenging and exciting to work in a cross-functional team. Since everyone has a different perspective and approach to the project, we hear different voices all the time. As a UX researcher, it is important for me to corporate those various requirements into my daily research. I gradually learned to prioritize different needs and collaborate with people from different backgrounds. Together, we created a great working environment that is warm, supportive and full of critical thinking.
​
​
​