Capturing Creative Cognition and Emotions using Facial Expressions
Background
This research captured the in-the-moment dynamics of emotions during the creative process using facial expression analysis.
Project Type​
PhD Project (Solo)
Experiment Design
Scientific Research
​
Role
User Research
Diary Study
Headgear Design
Participant Recruitment
Conducted the experiment
Data Analysis
Wrote a 40-page research paper
​
Year
02 Years
(2019-2021)
The Problem
Creative person and product components are already explored in depth
Multiple Frameworks are proposed
Creative process needs to be explored in-depth
​
Several stage-based models have been proposed in Lit.
How Might We Statement (phase 1)
How might we gain better insight into the creative process of the individuals so that we can take steps to improve it?
Initial Research Questions
RQ1: What are the steps involved in the Creative Process?
RQ2: What are the factors that influence the Creative Process?
Literature Review
Desk Research
Before working on anything, I studied various white paper articles in-depth to get a clear picture of the creative process of the individuals
As an example, the literature mentions an anecdote about a French scientist, August Kekule, discovering the benzene ring while he was dozing off. An idea of the ring came to him suddenly after seeing a snake writhing in loops.
I kept several other anecdotes at the back of my mind and moved on to do other modes of research to expand my horizons.
1:1 Interviews
I aimed to understand the real-life experience of the creative process of the individuals to -
-
Uncover their motivations,
-
Thoughts, and cognitive thinking processes
Primary Research
1:1 Interviews
Online and Field interviews were conducted with 10 creative experts
Some of the important questions asked were:
What’s your creative process; how do you get your stuff done?​
​
Individuals work differently. Tell us about when you realized you had a unique creative process.
​
When are you most creative?
​
How has your background influenced your creativity?
​
Have you ever struggled with your creative process?
​
What are the factors that influences your creative process?
Diary Study
5 creative experts, for a period of 1 week.
In this way, deeper insights were gained through a bi-angulation approach.
Prompts were asked on WhatsApp thrice a day:
When did you feel like doing your creative work today?
​
What factors influenced your creative process?
​
Which factors were the most valuable to your creative process. Why?
​
Did you struggle with your creative process?
​
How were you feeling when you were completely immersed during your creative process?
Analyzing the Data (Phase 1)
-
A journey map was constructed to decipher the themes
-
Diverse set of experience data points was gathered
Themes + Insights (Phase 1)
Creative process is dynamic, and one moves from one stage to another. This is not linear.
Theme 1: The Dynamic Process
There are several stages involved during the creative process.
Each stage/phase has their own characteristics and attributes that helps to achieve the desired outcome.
Its an emotional process that results in creative outcomes.
Theme 2: The Emotional Process
Different moods helps to achieve different outcomes.
Positive and negative emotions both have different impact on the creative process.
I wanted to find
the relationship between the emotions and the creative stages using a more objective and behavioral measure.
RQ3
What could be a novel way of capturing and analyzing in-the-moment emotions of the individuals during the creative problem-solving process?
RQ4
Do different stages of the creative process exhibit different emotional profiles? (+ve, -ve or neutral)
RQ5
Does the amount of cognitive effort spent during the creative problem-solving process (CPS) differ from that involved in a simple problem solving (SPS) process?
How Might We Statement (phase 2)
How might we capture the role of emotions during the creative stages in a more objective way in order to provide a framework on which several creative support systems could be built?
Deconstructing the construct of Creative stages
![Creative Process [Recovered]-01.jpg](https://static.wixstatic.com/media/212a88_813e9c2085c84ab4a46330d57e34edbb~mv2.jpg/v1/fill/w_1018,h_720,al_c,q_85,usm_0.66_1.00_0.01,enc_avif,quality_auto/Creative%20Process%20%5BRecovered%5D-01.jpg)
7 stages were finalized based on similarities of the construct.
Construct of Emotions
Measures of emotion are diverse, and different tools can be used to do so -
I chose Facial expression analysis of measure/metric because it -
-
Conveys emotions without saying a word,
-
Can be observed by others,
-
Is a robust and validated objective measure over others, and
-
Is used in many industries (e.g., animation, acting, film, intelligent agencies)
Self-Report Scales
Heart Rate &
Skin conductance
Body Postures
Facial Expressions
Pupil Dilation
EEG
fMRI
Facial Expressions as measures of emotions
Facial expression relies on facial action units (AUs).
​
Each AU represents a specific facial muscle movement.
​
Different combinations of these AUs represent different emotions.
Hypothesis
H1: Individuals will vary in spending their cognitive efforts at different stages of the creative process.
H2: Each stage of the creative process will exhibit different emotional profiles.
Methodology (Behavioral Experiment)
Once the metrics were chosen for measurement, the experiment was designed and conducted to capture how emotion and creativity interact together.
Participant Recruitment
Headgear Design + Tasks
Custom headgear
to capture the frontal facial profiles
Three divergent thinking tasks were chosen
(based on the task validity and reliability for both the CPS and SPS groups)
Experimental Setup
Experiment was conducted in a 2-way mirror room with 3 dome cameras installed
Total duration to complete the tasks was 30 minutes/participant
Retrospective Think Aloud Protocol
The participants described their creative process post-activity, using their video recordings and recalls. The participants were asked to think aloud about their thought processes while watching the video. The audio device recorded their utterances.
Synthesis and Analysis
After the Experiment............
Protocol Analysis (segmentation and coding) based on verbal Utterances
Markov chains to study the state transition between stages
Markov Chain models are used to predict, estimate, or guess the result of an event based on the preceding event.
The event here refers to the stage of the Creative Process.
The image here is for representational purposes only and does not contain the actual diagram of the analysis
OpenFace 2.0 ML algorithm to extract the facial AUs automatically
I utilized SVM poly degree 4 to classify the stages.
We got different accuracy scores for various stages, based on the emotion categories.
The higher the accuracy score the better the classification
Support vector machine to classify the stages based on different emotions
For the huge dataset generated by OpenFace, I used the support vector machine to classify 7 stages of the creative process using 7 different emotions.
The image here is for representational purposes only and does not contain the actual diagram of the analysis
For the huge dataset generated by OpenFace, we used the support vector machine to classify 6 creative ‘stages’ using 7 different ‘emotion’ categories.
Findings and Insight
After many data points and analysis later…….
Stage Transition Diagram (Based on Markov's analysis)
CPS Group
Individuals spent most of their cognitive efforts in transitioning from
Preparation to Preparation (57%)
Illumination to Ideation (65%)
Production to Ideation (57%)
Block to Preparation (38%)
Ideation to Ideation (46%)
Ideation to Production (35%)
SPS Group
Individuals spent most of their cognitive efforts in transitioning from
Production to Preparation (82%)
Illumination to Preparation (100%)
Preparation to Production (50%)
Refinement to Preparation (50%)
Evaluation to Evaluation (37%)
Block to Preparation (38%)
Accuracy Scores by the SVM in classifying the stages based on different emotion categories
(for both the CPS and SPS Groups)
(0% represents no accuracy and 100% denotes maximum accuracy)
Summary of the Findings
This is the first study to capture in-the-moment emotional experiences of the individuals during the CPS and SPS process using facial expressions.
Recommendations and Takeaways
Novel methods could be devised to regulate emotions at different stages of the creative process.
​The results may serve as a framework for developing creative support systems for various professionals based on the automatic facial expression data.
Translating the Findings
Taking the simplest hypothetical example, where the app can capture the real-time emotions to provide different cues/hints to the user for overcoming the stage of block
Learnings and what will I do next?
