Evolving in-game mood-expressive music with MetaCompose

Marco Scirea, Julian Togelius, Peter Eklund, and Sebastian Risi

Demo




Have you ever played a game for a long time?

Even games with really high music production values risk to induce boredom

Motivation

In games (and other interactive media) we need dynamic music

The techniques currently used are not very adaptive and are expensive

There’s a lot of research on procedural music generation but not a lot of focus on the expression of emotional significance

Important questions

  • Dynamic vs Adaptive music
  • Affect: Emotions vs Moods
  • Why is it hard to have Adaptive/Affective systems?

Dynamic vs Adaptive music

Dynamic = music is able to change
Adaptive = music adapts to the player's experience, or the state of the game

Affect

The experience of feeling or emotion.

  • Emotion: spontaneous mental state (no conscious effort), often accompanied by physiological changes
  • Mood

Emotions vs Moods

Emotion generally has a defined focus, moods tends to be more unfocused and diffused (Martin, 2003)

"Involves tone and intensity and a structured set of beliefs about general expectations of a future experience of pleasure or pain, or of positive or negative affect in the future" (Batson, 1992)

Can last much longer (Beedie, 2005)

J. A. Russell, “A circumplex model of affect”, Journal of Personality and Social Psychology

Perceived and elicited affect

Difference between recognizing what something is trying to express and feeling it

We mostly focus on perceived affect

Related systems: GenJam

J. Biles, “Genjam: A genetic algorithm for generating jazz solos”, in Proceedings of the International Computer Music Conference

Related systems: Deep Bach

G. Hadjeres and F. Pachet, “Deepbach: A steerable model for bach chorales generation”,

*Based on pre-composed pieces of music

D. Brown, “Mezzo: An adaptive, real-time composition program for game soundtracks”, in Proceedings of the AIIDE 2012 Workshop on Musical Metacreation
S. R. Livingstone and A. R. Brown, “Dynamic response: Realtime adaptation for music emotion”, in Proceedings of the 2nd Australasian Conference on Interactive Entertainment

*Based on pre-composed pieces of music

D. Brown, “Mezzo: An adaptive, real-time composition program for game soundtracks”, in Proceedings of the AIIDE 2012 Workshop on Musical Metacreation
S. R. Livingstone and A. R. Brown, “Dynamic response: Realtime adaptation for music emotion”, in Proceedings of the 2nd Australasian Conference on Interactive Entertainment

Why is it hard to have Adaptive/Affective systems?

Online games suffer the most from repetitive music

Why is it hard to have Adaptive/Affective systems?

  • Multiplayer Battletech: 200 music segments
  • Anarchy Online: 750 sequences

This only to achieve Dynamic/Affective music!

Tradeoff between control on the Affective dimension and the Adaptive

Traditional composition techniques are not well suited!

Generative methods: Spore

Brian Eno, Kent Jolly, and Aaron McLeran. Maxis (2008)

Risk of losing communicative functions

MetaCompose

Small music compositions connected to particular scene or level

Possibility of variations for added interest

Real-Time change in the music triggered by game events

Demo Time

Positive/Calm

Negative/Calm

Positive/Excited

Negative/Excited

In short

  • Transitions in expression are generally correctly perceived
  • Arousal is perceived correctly
  • Valence is still a problem, especially when looking distictions between positive/neutral valence
  • Arousal changes have an effect on valence perception

For more details see: "Can You Feel It? Evaluation of Affective Expression in Music Generated by MetaCompose" in Proceedings of GECCO 2017

Research Question


Can we use MetaCompose to change the perception of a game of Checkers?


  • Can we observe any difference in player experience (emotionally) when presented with affective-dynamic music compared to static music?
  • Can we observe any difference when the music is supporting the game’s internal narrative/state?

Why Checkers?

Experiment Design

3 experimental setups: consistent, static, and random expression

Participants were tasked with playing 2 games randomly chosen from these groups, both self-report answers and physiological data were collected

29 participants (19m/9f/1o), average age 28.9 years

What do you mean consistent?

Valence = how good is the current state for the player

Arousal = how much is at stake for the next move

How does it look?

Self report - Questions

All questions are in the form of 5-point Likert scales

  • Which game did you find more engaging?
  • In which game was the music best?
  • In which game did the music better match how exciting the game was?
  • In which game did the music better match how well you were playing?

Self report

The consistent setup seems to be better perceived, and is consistently rated as having higher music quality

Less significantly, it also appears to better match the excitement of the game and to be more engaging

Self report

The static and random groups are more similarly perceived, with static being slightly better perceived

Physiological data

Data from heart rate and electromyographic activity is too unclear to make any strong assertions

Limitations

  • Not enough participants
  • No real assurance that our valence/arousal metrics are correct

What now?

There is more than music in games

Games are made of many parts, and music is only one of those

More games, with different characteristics!

What about making people *feel*?

The future?

Thank you :)


Webpage: marcoscirea.com Twitter: @MarcoScirea

The Mood Features

  • Volume - Arousal
  • Timbre - Valence
  • Rhythm
    • strength - Arousal
    • regularity - Valence
    • tempo - Arousal
  • Dissonances - Valence

Based on: D. Liu, L. Lu, and H.-J. Zhang, “Automatic mood detection from acoustic music data”, in Proceedings of the International Symposium on Music Information Retrieval

High-Level Architecture