The Milieux Institute and The Gina Cody School are collaborating to develop cutting-edge interdisciplinary research on artificial intelligence.

There are no simple solutions to the complex issues and dilemmas that define our times. As we grapple with rapid technological developments that bring multifaceted challenges, traditional approaches and siloed disciplines can only provide a limited understanding of the complex phenomena that surround us. Producing impactful research today demands making disciplines more porous to other ways of knowing and doing. Cultivating such spaces for creative exploration and critical experimentation across disciplines within the academic sphere is at the heart of Milieux’s mission. 

It is in this spirit that the Milieux Institute has collaborated with The Gina Cody School of Engineering to develop three multidisciplinary projects over the course of 2022-2023. This partnership provides resources and funding to sponsor interdisciplinary research projects at the intersection of arts and technology. Led by a group of Concordia University’s faculty, the winning proposals bring together students from diverse disciplines and backgrounds – from STEM fields to Social Sciences and Fine Arts – to develop cutting-edge research on artificial intelligence (AI). 

AI technologies such as large language models like ChatGPT and text-to-image systems have been rapidly disseminated and adopted, and have shown surprising capacity to delve into domains traditionally reserved for human judgment, including language, arts, and engineering. “We don’t understand what they could be used for and what their social, cultural, political, and educational ramifications are,” explains Concordia’s professor Rilla Khaled, principal investigator of one of the awarded projects. The three projects detailed below are situated at multiple angles of interdisciplinary experimentation, providing a platform for graduate students to explore the confluence of AI and arts as they learn from each other in a playful environment. 

ChatGPT as Game Interface 

Concordia professor of Design and Computation Arts Jonathan Lessard is joining forces with Computer Science and Software Engineering professor Leila Kosseim to develop the research project “ChatGPT as Game Interface”. This project is a unique blend of Lessard’s extensive experience in game design coupled with Kosseim’s expertise in natural-language processing and artificial intelligence and proposes to investigate whether large pre-trained language models such as ChatGPT can give voice to fictional characters in games and other interactive media. 

“Imagine being able to ask any question to any character in a game, or engaging with them on social media platforms like Discord, and receiving contextually relevant and immersive responses”, describe Lessard and Kosseim in their proposal. By transforming ChatGPT’s vast knowledge and capabilities into a personalized, situated source of discourse for specific characters, their project seeks to transform the way players interact with virtual worlds and their inhabitants. 

For Professor Lessard, this kind of multidisciplinary research in artificial intelligence allows for an exploratory and creative approach to technology, as opposed to a problem-solving approach: “I think the advantage of multidisciplinary research – having a designer or an artist in the mix – is to look at the technologies’ potential as opening a design or creative space that can be explored without having predetermined use cases to solve, to be an exploratory stance or see the creative potential for it.”

As a game designer, Lessard is interested in what fun we can have with these technologies. “I want to take this approach willingly and not as a by-product of the design of the project”. The project will leverage the emergent narrative experimentation platform, Chroniqueur – developed by the LabLabLab experimental games research group led by Lessard – as a testing ground for automated conversations with its multitude of simulated fictional characters. 

Lessard aims to collaborate with Kosseim and her students to not only investigate the integration of ChatGPT and similar technologies into the game, but also to evaluate the feasibility of training custom models on their own hardware, thus addressing concerns regarding ownership and control.

This project not only has significant artistic implications, but also commercial potential in the video game industry and beyond. 

Research through Design into AI Edge Cases

The rapid introduction and adoption of AI tools in everyday life has led to a number of controversial use cases. For instance, people are using ChatGPT for therapeutic purposes, and a chatbot made headlines for professing love to users and other unsettling messages following Microsoft integration of AI into their search-engine Bing.

Concordia professors Rilla Khaled (Design and Computation Arts), Sudhir Mudur (Computer Science and Software Engineering) and Ursula Eicker (Next Generation Cities Institute) formed an interdisciplinary research team to investigate these kinds of AI edge cases. 

Their project broadly asks: “are our social and cultural assumptions evolving in step with the rapid pace of AI innovation that we are currently experiencing, and if not, how might we classify what we are ready to experience now, what we are not yet ready for but might be soon ready to experience, and what we currently deem inappropriate or off-limits?”

Deploying Research through Design (RtD), the team will explore the social, cultural, and ethical implications of AI-powered services and platforms such as ChatGPT and DALL-E. By rapidly prototyping experimental functionality on top of these services, they strive to engage the public in critical reflection and conversation about the potential impact of these technologies on society. 

Simultaneously, a Speculative and Critical Design (SCD) approach that “focuses on people as opposed to users”, allows the team to not only imagine potential uses of AI, but to create experiences of what such uses might entail. 

The project will have four different teams, and they hope to develop eight small projects through an iterative process, involving two sprints of one month each: “This design research approach shows that the more iterations you have, the more likely you’re going to find something that works. This involves doing lots of experiments and parallel prototyping.” By regularly regrouping and reassessing progress, students will identify emerging themes, opportunities, and challenges. 

Ultimately, their project seeks to better understand the nuances of AI’s integration into our lives and to guide responsible development and usage.

The Aesthetics of Music Generation Using Artificial Intelligence

What does AI-generated music look like? We know that AI can create music hits, such as the controversy around the AI generated Drake song that went viral. But beyond recreating, how can generative AI enable the emergence of new patterns of creativity and aesthetic? 

Dr. Eldad Tsabary, professor in the Department of Music and co-director of the Performing Arts (LePARC) Research Cluster, is collaborating with Dr. Sabine Bergler (professor in the Department of Computer Science and Software Engineering) and Dr. Yong Zeng (professor in Information Systems Engineering) to study the intersection of AI and music generation. 

“We believe that this collaboration between the Gina Cody School and the Milieux Institute will best serve a double goal: that of solving an engineering problem and finding new paradigms for artistic creation.” 

The project aims to develop AI-based tools at a scale that is accessible to musicians and composers, fostering new creative possibilities and promoting the integration of AI in the artistic process. Their objective is to produce a musical work using these innovative AI techniques, which can be performed live by instrumentalists for an audience.

The research team believes that AI should not replace human competencies, but rather provide fresh perspectives on existing techniques and challenge conventional paradigms in the arts. “We are interested in discovering new patterns of creativity, new forms and new aesthetics that will arise from these new techniques. The goal of any AI approach is not to get the expected output, but rather to be surprised by the output of our creations.”

The three projects will produce cutting-edge research on artificial intelligence, bringing students from diverse backgrounds together to experiment and learn from each other. 

In discussing the potential challenges of interdisciplinary collaboration, Professor Khaled emphasized the experimental nature of their project as a positive ground: “It really helps that we have this kind of experimental frame, and we understand that what we’re doing is we’re all learning off of each other.”

This collaboration between the Milieux Institute and The Gina Cody School of Engineering represents an exciting opportunity to advance multidisciplinary research, explore the potential of emerging technologies, and the confluence of AI with the arts. As these projects unfold, they will undoubtedly inspire further exploration and collaboration, paving the way for future breakthroughs at the intersection of technology, arts, and society. Stay tuned for news about how these projects keep developing.

The cover image has been AI generated with DALL·E 2.
By Ariana Seferiades (MA Social and Cultural Anthropology)