Exploring pose estimation in instrumental composition: the Body Fragmented project is published on Frontiers in Computer Science, part of the Research Topic: Embodied Perspectives on Sound and Music AI. It is published as open access.
Abstract
Introduction: In contemporary and experimental music composition, the integration of physical movement via sensors and digital technology offers new pathways for music composition, performance and interdisciplinary practice. This research explores how pose estimation can be used to generate movement-based notation and support embodied musical expression.
Methods: Through the Body Fragmented composition project, this study introduces a methodology that centres the composer and performer’s bodies in instrumental composition using pose estimation technology. The approach supports non-linear collaborative processes between composer and performer, facilitating a movement-led instrumental composition practice. A guiding question “what does that movement express and how could that sound” provided a conceptual anchor and encouraged an openness to the interpretation of the pose estimation outputs, allowing it to suggest sonic material and musical expression.
Results: Findings reveal that pose estimation can effectively capture expressive movement for compositional development, support performer interpretation through visual scores, and enhance collaborative dialogue. The study also identifies limitations in pose estimation’s ability to convey nuanced musical gestures, and detailed musical information, prompting the integration of supplementary notation materials.
Discussion: The reflective methodology enabled the collaboration to explore new forms of movement-led methodologies in instrumental composition.
The article discusses the composition, Body Fragmented, commissioned by Larissa O'Grady, funded by the Arts Council of Ireland and supported by the Contemporary Music Centre through their Contemporary Artists Network. Open access publication costs were covered by University of Liverpool.