Hi, I'm
I'm a researcher, maker and creative thinker studying digital economies of 'trash' through decolonial feminist economics, STS, and art.
ipsA home
A Home Automation system for Mental health and Human Communication.
Intelligent Home Systems for self and mutual care.
Methods
Competitive Analysis, Heuristic Evaluation, Affinity Diagramming, Task Analysis, Literature Review, Observation, Personas, Scenarios, Storyboards, Wireframing, low and medium fidelity Prototyping, Critiques, Heuristic Evaluation, User Testing
Tools
Miro, Google Docs, Powerpoint, whiteboards, Paper, Photoshop, Canva, Draw.io, Adobe XD, Axure RP, Python, Camera, Home server
Team
Sara Milkes Espinosa (UX Research and Design), Kiran Bhattacharyya (development)
My contributions
Conducted all preparatory and evaluation research and lead the design process, Wireframing, Low and Medium Fi prototyping, coordinated
Duration
6 months
Project
IpsaHome is an intelligent assistance system for the home that emphasizes communication and mental wellbeing. It is meant to be used by households and families where collocation is limited because of differences in schedules. Household members can log and collectively share their moods to other members in the system, facilitating mutual communication and care. Sensors in the system monitor features such as times when lights are on and off, time watching television, food logs and exercise to provide feedback on household habits. The system is scalable and encourages technical transparency, providing all users with access, customization and understanding of the technical artifacts in the household environment connected to the IpsaHome system.
Challenge
An ongoing design and user experience challenge of the project is to match the music-making systems to Computer Science concepts in collective, fun and easily comprehensible ways for our target users. However, without a synthesis of the results of previous iterations, it was difficult to discern the useful system features from what had not worked. Additionally, from the research perspective, the new design had to be adapted to the technical and physical constraints of a multitouch capacitive table. Finally, with grant funding nearing an end, the focus of this version was to facilitate the evaluation process.
COVID created a radical challenge to our process that redefined the scope of the project.
Proposed systems
Pre-COVID: Tangible Version
I lead the research, design and evaluation of the tangible version of Xylocode, a system that enables a physically engaging creative experience with music and code at museums and public installations for families and groups of children. Users collectively manipulate tangible objects to modify the system's rules, learning basic computer science concepts like conditionals, booleans, variables and arrays; and music concepts like rhythm and composition.
Post-COVID: Web Version
I lead the redesign and user testing of Xylocode as a web version, keeping the constructivist design of the tangible version and making it appropriate for a web experience. The system is redefined to allow single-user interaction through a browser with a laptop or a tablet. Tutorials emphasize the educational content and a logging backend collects system data for further analysis. This version is currently part of the Amazon Future Engineer program.
Xylocode Web has a stronger narrative design: users help voiceless birds sing with a deconstructed xylophone. The emitters shoot balls that make sounds as they impact the geometric constructions built by users. The Code boxes to the left hold the conditional logic controlling the system.
Process
NOTE ON COVID The Tangible version and the Web version of Xylocode followed a similar research /development/ evaluation pipeline, sharing much of the insights from the literature review phase, iterative and prototyping process. The transition towards the Web version required complementing these phases with additional research and steps as noted below. Whereas an in depth evaluation and deployment of the Tangible version is not possible at the time due to social distancing constraints, the Web version has been deployed and is undergoing Beta testing.
RESEARCH
-
Literature Review
-
Expert Interviews
-
Requirements Analysis
Tangible version
Web version
ITERATE
-
Material exploration & Paper Prototyping
-
Design Critique
-
Agile Prototyping
-
Weekly Review
PROTOTYPE
-
System Consolidation
-
Hi-fi Prototyping: Micro-interactions
-
Research Questions: Pilot
-
Alpha Testing
EVALUATE
-
Research Questions: elaboration
-
Beta Testing
-
Instrument Refinement
IMPACT
-
Large scale deployment
-
Online assessments
-
Preliminary Results
Literature Review
I conducted a focused literature review with two other members of the team about the previous publications of the TuneTable project to analyze the designs and results of their deployment at museums. We synthesized the results of testing from unpublished documentation and evaluation videos. I complemented this analysis with a review of literature about related projects such as Osmo, as well as literature in computer science education and HCI in museums. I synthesized a list of best practices and design guidelines from this step.
The transition to the Web version required reviewing additional literature about Scratch, and Ed Tech guidelines about computational thinking with online programming platforms.
Expert Interviews
I carried out informal, semi-structured interviews with three experts in the field of co-creative installations involved in previous iterations of the TuneTable. On expert was a former PhD student at the lab, a second one was a Music Tech researcher, and the last one was a professor at Northwestern University working in tangible computer science education. I asked questions concerning the systems design, narrative design and the target users. This step helped me cross-validate findings from the literature review regarding the functionality of previous designs and the relationship to published literature. In addition, the expert interviews helped me gain insight into the logic behind previous designers decisions and to take away useful heuristics for future design.
Sketch from an expert interview with a former researcher involved in the TuneTable project.
Requirements Analysis
I met extensively with the directors of the TuneTable Project and the evaluation team to discuss my synthesis of the literature review and expert interviews, and to gather the requirements this project needed to fulfill. The project directors wanted to simplify the design so rules would be clearly understandable for a 5 minute engagement. This has been referred to as 'immediate apprehensibility' by other researchers. Together we identified five main pain points from previous designs. The evaluation team required that the design followed the criteria of the APEX coding system, an evaluation instrument used by the lab with the following categories: Physical Interaction, Intellectual Engagement, and Emotional engagement for Interest formation.
For the web version, the interaction and technical requirements changed significantly: instead of a 5 minute interaction requiring immediate apprehensibility for a group of users, the system was required to follow curriculum guidelines for an interaction possible in formal education. Another expert dealing with CS education with music was added to the team to develop tutorials.
Summary of the research Phase
I performed a synthesis of these initial research steps to create the design specifications to start the prototyping process. Here i present a brief summary of the result. You can contact me to find about the detailed report.
Literature Review
+
Expert
Interviews
+
Requirements Analysis
-
Previous publications' results
-
Gap analysis
-
Ed Tech Guidelines
-
Museum HCI insights
-
Previous systems' design
-
Internal understanding
-
Heuristics for design
-
Pain Points
-
Evaluation requirements
-
PI requirements
Design Specifications
-
Measurable CS concepts
-
5 minute interaction (tangible version) / 15 minute interaction (web)
-
Extensive Alpha testing
-
Immediate apprehensibility
-
Abundant visual feedback
-
Adaptable learning goals
Material Exploration & Paper Prototyping
We conducted an iterative process with several cycles of paper prototyping > Design Critique > Agile prototyping > Expert review. The iterative process was flexible to accommodate emerging needs and to allow the creative process space for exploration. Informed by the research phase, I guided the design and development with 1) 3D material exploration to understand the tangible interaction 2) lots of whiteboarding and paper prototyping to communicate with the project PIs, the evaluation team and the programmers.
We started with paper prototyping various system designs based on the results of the research phase. The main ideas for the interaction were inspired by a digital artifact called 'ball droppings' by JT Nimoy. To iterate through the system design in order to figure out the interactivity, the co-creative aspects, the music mapping and the tangibles, we followed the paper prototyping by design critiques with members of the lab. Once a feature had been refined on paper, we followed an agile development cycle then show the prototype to the whole team in a weekly meeting.
Early Sketches of the User Interface for discarded design and physical prototyping of the conductive tangibles.
Paper prototypes to determine spatial distribution of four simultaneous users before the Ideum Table arrived in the lab.
Initial sketches with PI Brian Magerko, PhD of what would become the final interaction with lines and ball emitters affecting conditionals that control logic of sounds and visual elements.
Design Critique
Design critiques were carried out often with two or three regular members of the lab that were familiar with the project. These meetings took place in person before COVID and online after transitioning to the web version. Following the requirement for constant testing of interaction features for robustness, we iterated through paper prototypes for main features, Wireframes for micro-interactions and options of visual design. This was critical for the tangible version considering the only developer at the time (Chloe) was hired part time. Later, the hired programmer (Michael ) reformulated the system with added complexity to the sound system adding MIDI.
Spread of paper prototypes with details of the user interaction features to be implemented in the agile prototyping pipeline.
Agile Prototyping
We carried out a continuous cycle of prototyping where I was in close communication with the developer (meeting twice a week) and even jumped into development as needed. We used a Kanban board, Slack (later Microsoft Teams because of Georgia Tech requirements), and in person or online meetings for our updates. Our code was hosted on the Expressive Machinery Lab's Github. We developed in Unity (C++ scrpting) and later Unity WebGL builds hosted in a Georgia Tech server.
Weekly Review
Reviews took place weekly or bi-weekly during the joint lab meeting. I presented our WIP to the Principal Investigators and the evaluation team and collected their feedback on the software prototypes. Te purpose of the meetings was to make sure that the features passing to the development phase were agreed upon by our multi stakeholder lab, including the Principal Investigators who came from different disciplines (Cognitive Science and Music Technology), the evaluation team and other members advising on education guidelines and musical design.
Views of the lab during the weekly review, white boarding and prototype review process.
System Consolidation & Micro-interactions
As the different interaction features were chosen through iteration, I started managing the project with a Traceability Text Matrix to coordinate requirements, features requested, quick tests results, bugs, UX needs and development needs. Since the Web version had overlapping features with the Tangible version, the system consolidation for the web version happened faster and allowed us to go into higher fidelity prototyping earlier. To compare different interaction paradigms for the Code Boxes where the conditional logic was presented to users, we carried out a decision matrix.
Screenshots of the four UI prototype versions and decision-matrix to decide on system and user interface. Live prototype here.
Alpha Testing
Tangible Version
We conducted six sessions of Alpha Testing with members of the lab, Georgia Tech Colleagues and research collaborators. Among these we carried out 3 black box and 3 white box sessions. We identified bugs, scenarios and usability issues previously not considered.
The development of the Tangible Version was put on halt at this point due to COVID. The lab decided to carry out a translation of the main elements of the system to an online version.
Web Version
I carried out four user testing sessions with children in our target range to pilot both the system and establish benchmarks of usability, as well as to refine the evaluation instruments. This step validated the system usability, provided us with feedback from children in our age range and helped us identify features that would make the system more desirable.
Evaluation
With the results from Alpha testing and our Usability sessions, we refined the research questions as we prepared a beta testing in partnership with the Amazon Future Engineer program. An important addition to the system was a set of tutorials prepared by EdTech collaborators in our team that introduced children to specific terminology of computer science.
Research Questions: Refinement
Although our research through design methodology was initially concerned with how to design for co-creative, music experiences to increase interest in computer science, our research questions for this last phase of the project focused on the evaluation.
RQ 1:
What are the creative affordances of Xylocode? How do children use it and what kinds of artifacts are built?
RQ 2:
Does Xylocode foster Interest Formation in CS? Do children enjoy using the system? Do children retain the computer science concepts?
Beta Testing and instrument refinement
To answer RQ 2, we created a set of comprehensive instruments, including Pre/Post Surveys that itemized demographic and individual features, evaluated interest formation with likert scales and verified content retention through illustrated questions.
To study real time use and differences across individual use, we created a logging system that recorded individual moves on the platform, such as when children dragged elements or entered conditional combinations on the code boxes.
RQ 1 required a different evaluation approach of a more qualitative nature. We are currently carrying out observation sessions and interviews with individual children.
The Beta Testing was carried out thanks to a partnership with Amazon Future Engineer in a class of 18 students of an elementary school during Computer Science week.
Impact
Xylocode has been deployed in several elementary and middle schools across the U.S. in partnership with Amazon Future Engineer and has been used by more than 150 children.
Our preliminary data suggests that Xylocode has a positive effect on Interest formation, with some indication that this effect is particularly strong in girls (RQ 2).
Our findings have allowed us to understand some of the novel creative affordances of Xylocode, providing an answer to RQ 1.
Creative affordances of Xylocode as per our observation sessions and artifact analysis
Narrative
Children create stories with the bird characters and build their environments.
Symbolic / Semiotic
Children play with word associations and explore signifiers in their artifacts.
Graphic
Children express creatively by creating abstract constructions and musical landscapes
Logical / Spatial
Children create mazes and entrap balls in geometric structures.