In the fall of 2019, I was working on a class project in Dr. Robert Jacob's class called "Object-Oriented Programming for Graphical User Interfaces" to build a game-based learning system that computes user's eye gaze location to select an answer. As I was working on designing and implementing this interface, I got more interested in what happens between humans and digital systems. I started to look up literature that discusses interaction design and human-computer interaction (HCI), and I discovered a hybrid area between HCI and art. As a person who loves synthesizing ideas from various disciplines, I was naturally fascinated by the interdisciplinary approach of those works and the depth and breadth of discussions these works could create.
I wanted to create a thing that does not seek to promote efficiency but help users to carve out some time of their daily routine, slow down, and connect with their inner self--similar to how a Catholic confesses to the priest. This would require a strictly solitary interaction. (No one would enjoy someone else taking a peek at their journal without permission.)
Many tangible and interactive artworks are designed for a single-session interaction. I wanted to create an artifact that retains the vestige of user's past narratives so that they can engage in a continuous dialogue that could take place over the span of days, months, or even years.
I brought this project idea to Dr. Robert Jacob at the Tufts HCI lab. At this point, I had a clearer idea of what I wanted to work on. As many artists in the past have imagined, I was interested by the idea of a magical creation that mirrors their owner's inner state (mind, soul, emotions, etc.) and changes over time with the owner. The Picture of Dorian Gray written by the Irish writer Oscar Wilde illustrated this idea very well. In this work, Dorian Grey’s portrait, which originally depicted his youth and beauty, slowly transforms over time into a grotesque depiction of the character, indicating the gradual corruption of Dorian’s soul.
Though measuring one's moral depravity could be interesting, I decided to create something different yet somewhat similar. Mark Rothko's color fields paintings seemed particularly relevant to me. Throughout his artistic career, Rothko sought to capture emotionality while transcending any specific reference to culture or history, and his use of large abstract rectangular color fields was quintessential in his works.
While doing a literature review on affective computing, aesthetic interaction, slow technology, and ambient media, I collaborated with the graduate students at the Tufts HCI lab to build the artifact. I mainly worked on the visualization algorithm in HTML, CSS, JavaScript, and Processing that was used to map the values from the emotional and language tone analyses to specific values that determine the colors and form of the color fields. Our decision on which color to use for each emotion was based on Sutton and Altarriba’s research (2016) on the color associations to emotion and emotion-laden words.
We started by first completing the visualization algorithm and tested it by feeding some texts from literature into the system. This picture on the left is generated based on an excerpt from the last chapter of Pride and Prejudice written by Jane Austen (which definitely had its moments of happiness and some bitterness).
Then we began to integrate the visualization algorithm with the Watson Tone Analyzer API and speech-to-text generator that allowed us to classify user's emotional and language tones in their narratives and deploy that data to generate color fields.
On the left, the blue, yellow, and dark grey color fields represent sadness, joy, and fear, respectively. On the right, the red and black color fields represent anger and fear, respectively. The color fields on the right are less blurred because the text input was classified as more confident.
As I learned about reading Rothko's color fields from phenomenological perspective, I got fascinated by how phenomenology attempts to explain the interaction between subject and object and how the true perception of an object inevitably requires the presence and active involvement of the subject. This led me to take Phenomenology and Existentialism in Spring 2021, which ended with a final paper on my phenomenological analysis on Alberto Giacometti's sculptures and sketches. This was a great opportunity for me to gain depth in my ideas that helped me better articulate the philosophy of this work in the paper after getting rejected by another conference once. I focused on discussing the artistic novelty of the work and its potential as a tool for performative and improvisational art-making.
In contrast to surveillance-style affective computing (bio-sensing, computer vision, etc.), Affective Color Fields gives users more authority on what they are willing to feed into the system and how they would frame and reinterpret their narratives. They could even deceive the artifact by giving a fake account of their life. Through this playful interaction that makes the user an improvisator and the human-machine interaction a play, Affective Color Fields aims to trigger their imagination and provide richer aesthetic experiences. Over time, the color fields slowly evolve with the user, and this synchronizing experience is expected to promote a personal bond.
Quite obviously, Affective Color Fields does not aim to seamlessly blend into an ordinary user's daily routine and effaces itself for the sake of their efficiency. Instead, it prompts users to engage in a behavior that can be seen as "unproductive" or "meaningless" and deliberately carve out some time that will be spent with this artifact. I liked how our work promotes a ritualistic experience to users and challenges their preexisting world and original values. Through the process of unearthing a new meaning from the color fields and casting the new meaning to their memories and future instances in life, their newly conceived thoughts and beliefs get weaved into everyday life, creating ripples that affect them in the long term.
I worked on revising our paper for months after receiving a rejection, and in the summer of 2021, I was delighted to hear that our paper was accepted by the ACM Multimedia 2021 for Interactive Artworks track. It was an incredible experience as an undergraduate student to plan an art research project from start to finish, collaborate with other researchers, and witness our work getting published by a conference. I realized how exhilarating it can be to create knowledge and share it with the research community.
VIEW PAPER