Integrating Sound in Living Architecture Systems (2018)
Poul Holleman
category:
Journal
date:
August 2018
authors:
Poul Holleman, Salvador Breed, Paul Oomen
published by:
Spatial Sound Institute, Living Architecture Systems Group
Journal
date:
August 2018
authors:
Poul Holleman, Salvador Breed, Paul Oomen
published by:
Spatial Sound Institute, Living Architecture Systems Group
The paper elaborates upon the results of collaboration between the Spatial Sound Institute and Living Architecture Systems Group throughout 2017-2018. Rather than adding layers, it is argued that spatial sound interweaves meaningful fabric to sculptural form and living architecture. It is able to embed an architectural design within a sonic field (exterior), or spatial sound can be embedded itself inside sculptural objects (interior). As part of this work, 4DSOUND has evolved in applying irregular speaker setups for enhancing sound projection and creating applied instrumental possibilities to compose with spatial sound in the sculptural realm. Further, the paper discusses developments regarding the integration of the 4DSOUND’s spatial sound engine for the control of other media like light, kinetics, and sensor interfaces. Finally, the paper briefly discusses interesting future research and objectives.
Projects
ASTROCYTE, DXedit, Toronto, Canada - fall 2017
AMATRIA, Indiana University, Bloomington, Indiana, USA - spring 2018
NOOSPHERE | AEGIS, Royal Ontario Museum, Toronto, Canada - summer 2018
Introduction
Sound is an essential medium in understanding space, expressing emotions, and abstracting organic and artificial phenomena like growth and electronic textures. When a physical sculpture is present within the soundscape, the spatial and physical relations between sculpture and sound requires a clearer definition. The in-depth collaboration with Beesley et al. has challenged 4DSOUND to extend the control of sound beyond an empty space that hosts sound objects to a world where the virtual is seamlessly integrated with the actual.
The 4DSOUND paradigm considers sound by its sculptural and architectural qualities. Sounds are objects that can be sculpted with dimensions, construed sub particles, and applied properties through sound design. These objects are subsequently placed in a (virtually) infinite space where they can move following particular trajectories or according to a certain behaviour. The setup allows sound to become intimate as close as almost touching the body, or to move like a flock of birds on a distant horizon. Inside this virtual environment, sounds are embedded with acoustic reflections to model the flexibility of sonic experience from inside a small room to a vast space. The flexibility affords the virtual environment to take on surreal forms, creating otherworldly textures and gestures that evoke imagination beyond recognizable.
4DSOUND is an instrument, a set of tools that enables composing and performing intuitively with spatial sound in great sculptural detail. In the context of Living Architecture Systems, the challenge has been multiple. Not only we need to create a virtual sound space - virtual sound sources that live in a 3D space which manifest through a range of speakers - that can be explored, but to integrate this sound world inside a sculpture giving it a voice and allowing its compartments to be meaningful actors within a designed physical environment. Three R&D topics have been central in achieving these goals:
Exterior and Interior
To embed a distributed network of speakers within the sculpture required a redefinition of the 4DSOUND panning algorithms. The new algorithms expand sound spatialization within a singular omnidirectional grid, hence a distinction between exterior and interior speakers was necessary.
A field of exterior speakers can be considered similar to 4DSOUND’s regular layout. It covers the omnidirectional field of the actual space, and virtually extends this field beyond the actual space to infinity by adding spatial sound synthesis. 4DSOUND’s equally distributed grid system provides a social listening area not concentrated into just one sweet spot (P. Oomen, P. Holleman, L. de Klerk: A New Approach to Spatial Sound Reproduction and Synthesis; Living Architecture Systems Symposium: White Papers, 2016). The listeners are empowered to move around to self-compose experiential perspective and orientation.
The interior speakers are the voice of the sculpture. The configuration of the sculpture’s components and their spatial relations impact how the 4DSOUND interior speakers are set up and run as internal constellations. In projects that have been realized to date, we have successfully generated sounds that behave like a dense crackling neural network, or the omnipresence of smaller objects within the sculpture - like electronic birds in a tree.
For the most part, sounds in either one or both of the fields share the same extensive set of controllable parameters to form movement, physics and perception of sounds in space. In terms of control, this added sonic versatility to cover both the internal and external fields of different sculpture’s components requires only a minor expansion in parameters. A simple matrix interface allows routing sound sources to the desired speaker constellations. Sounds that live within the one sculptural composition, can easily be transferred to, or spread over other components. Indirect connections are also possible, like external reflections as responses to sounds that originate from inside can potentially create echoes that seem to come from spaces that are beyond the one that the listeners are in.
For the installation Astrocyte (2017) we introduced the ‘Sphere Engine’, a prototype application that complemented the regular 4DSOUND Engine. The Sphere Engine was designed to drive ten speakers inside the Large Sphere that created the central constellation of the installation. The speakers were spread almost equally across the central semi sphere, which was slightly tilted and raised above the heads of the listeners. The result was a defined focal listening point within the expanded space of the sculpture. This centralization together with the gentle directional quality of the speakers and granular sound design evoked a sensation of proximity, and a feeling of intimacy with this large, radiating object.
Image: the Large Sphere of ASTROCYTE, the inner semi sphere contains 12 speakers Photo courtesy by Philip Beesley
Image: 3D visualisation of the Sphere Engine spatialisation developed for ASTROCYTE
We found that the results of technical sound tests - such as exposure to continuous noise and a variety of waveform pulses to different spatial projection onto the sphere - offered a broad palette of characters and abstract associations. For example, synchronised pulses can accentuate the shape of the sphere as a whole, where diffracted sounds dissect its parts and seem reveal the granularity of inner processes within the sphere. As a result of high speaker density the perception of small, moving sound objects became extremely clear. Altogether, the sphere created a ‘sonic painting’ with a broad palette onto a tightly focused frame.
Image: user interface of the Sphere Engine developed for ASTROCYTE
Irregular Speaker Configurations
4d.pan, the proprietary panning algorithm developed by 4DSOUND, was initially limited to symmetrical rectangular speaker grids. To provide asymmetrical sculptures with varying densities of interior speakers, we expanded the 4d.pan algorithm to support irregular setups of varying sizes and shapes. The principle remains to position exterior field speakers as equally spaced as possible to assure balanced sound projection and refined continuous spatial definition in all directions. For the interior, all configurations are allowed to support any artistic or acoustic tasks and to create more irregular grids. A combination of an equally spaced exterior field with an irregularly spaced interior determined by the sculpture can deliver a convincing and powerful soundscape with explicit control over the sonic presence in the space.
Any given constellation of speakers can be divided into subsets to allow extremely detailed control in spatial sound design. The setup differentiates speaker groups (areas) for sound sources to exist within. The internal structure of the sculpture divides interior speakers in ways that mimic its components. This results in an almost organic design where the sonic and material interpretation of the sculpture dictates the speaker configuration, instead of the other way around. We are excited that backend complexities can realize an elegant frontend for intuitive system handling and sound design.
The permanent environment Amatria (2018) at Indiana University is a follow up on the Astrocyte installation (2017). The environment consists of ten speakers distributed throughout the larger part of the sculpture. We identified the Large Sphere (speakers A1-A3), the Small Sphere (speaker B1), and the Canopy (speakers SF1-SF6). By defining three sub constellations we were able to develop the spatial sound design around two actor objects (the Spheres), and a more diffracted area (the Canopy).
Image: irregular speaker configuration of AMATRIA
As the protagonist, the Large Sphere is equipped with three speakers, while the Small Sphere has only one. The differentiation made possible by using a triad configuration creates a spatial depth and dimensional perception of the sounds that are projected onto the field. The single speaker in the Small Sphere is timid compared to its neighbours, but the sharp contrast in sonic magnitude and its distinctive character defines a focal point within the shared space.
While designing the spatial sound for the installation, we experimented with unique role assignment for each character in the sculptures. For example ‘breathing’ spheres would have subtle reflections on the canopy with patterns resembling textures of dense fog, light raindrops or breaking glass. The flexibility of the system in filling the entire space or carefully occupying specific areas proved to be essential in constructing the integration of sculpture and sound.
Image: AMATRIA installed at Indiana University, Bloomington USA.
Shared Worlds: Merging the Virtual and the Actual
Another important R&D goal has been the integration of sound spatialization with other kinds of actuators in the sculpture such as granular light, kinetic mechanisms and sensor interfaces. To create meaningful integrated interaction we developed the prototype Distance Based Interaction Engine (DBIE) in collaboration with Philip Beesley Architecture Inc. members Rob Gorbet and Adam Francey. The DBIE is a simplified interpretation of the spatialization algorithms that already exist within 4DSOUND. In the 4DSOUND Engine, spatial properties of virtual sound objects are mapped to speaker amplitudes and audio signal processing (spatial synthesis). Meanwhile, the DBIE maps basic spatial properties of an ‘Excitor Object’ like position, size, and force to lights, kinetic objects, or any type of actuator.
The basic principle of Distance Based Interaction is the intensity of an actuator (e.g. the brightness of a light, or the speed of a motor) which is determined by its distance to a virtual object. Every actuator holds position properties that match their actual relative locations within the sculpture. The virtual Excitor Objects also live inside this 3D map where their presence can excite the actuators by moving closer and further away. An excitor can be an infinitely small point, or of any spherical size. An increase in scale means it covers more virtual space and effectively comes closer to more actuators resulting in more actuators being excited. The final parameter is Force, a multiplier of Distance Intensity acting as the upper limit of excitation. With scale and force combined, one can create a large but weak excitor, or a small and strong one.
Sensor systems are integrated in the same Cartesian space as the sound objects. Their positions in an infinite continuum of space are defined by XYZ-positions (P. Oomen, P. Holleman, L. de Klerk: A New Approach to Spatial Sound Reproduction and Synthesis; Living Architecture Systems Symposium: White Papers, 2016). Infrared modules register the presence of visitors and their positions inside the sculpture, as well as their hand gestures. Virtual sound objects and excitors can react consistently and immediately in attractive or repulsive reflexes triggered by infrared sensors. There is a subtle balance between direct basic responsiveness (e.g. triggering short samples on the location of the sensor) and more complex movements that evolve over time (e.g. ripple effects in the surrounding actuators, or the progression of the sound composition on a higher level). The correlation of sound events in space and actuation triggered by movement and behaviour enhances the immersive experience between the visitors and the sculpture.
In this setup, virtual sound objects, virtual excitors, speakers, actuators, and sensors all share the same environment, and can interact accordingly. This produces a versatile and intuitive interface to design coherent movements and behaviour. Background behaviour can be small excitors, moving fast and slow, emerging and vanishing like snowflakes or a soft breeze through the woods. Bigger excitors could be phantom creatures, or abstract magnetic forces beating through space. The technical essence is that single indexed actuators are no longer considered in artistic design, instead, the interface is elevated to a higher level of control where conceptual translation becomes more immediate and inspirational.
Image: Excitor Objects and their distance-based relationships to actuators present in a sculptural environment
The first implementation of the DBIE was realised for Noosphere and Aegis (2018) at the Royal Ontario Museum in Toronto, Canada. Background behaviour is generated by small, semi-random excitations throughout the sculpture. In the larger sphere Noosphere, an excitor is active with transformational size, force, and trajectory. They mark the first modest but profound step in unifying algorithms for behaviour in sound, light, and movement, and in interaction with human input through sensor interfaces.
Image: human input as excitation for kinetic sound, light and movement within NOOSPHERE
Future Research
In future research we will refine panning algorithms, explore the characteristics of different kinds and combinations of speakers, and continue stretching the flexibility of sound spatialization using varied speaker configurations. The divide between exterior and interior sound may be further contrasted to enhance depth of field. At the same time, the differentiation will also invite explorations in the coherence between the perception of the diffusion field and focused objects, emerging from the same technological principles in hardware and software.
A shared origin with all technological disciplines in the same spatial interaction paradigm will be further developed in an even more integrated software reality. A right balance between utilising processing power of a personal computer, and performing algorithms on the distributed network of microprocessors that are not necessarily aware of structural properties, can enable each sculptural character act more like individual agents within the network.
Other objectives are designing an offline prototyping environment to develop interactive algorithms and behaviours that can develop independently from a hardware testbed. Since all virtual properties will share the same physical environment, they should be included in a shared virtual environment. An extensive 3D visualization of all activity will complement the possibilities to exploit spaces as an instrument and provide insightful feedback. The tool will allow detailed preparatory work, stimulate creative design of the space as a whole, and increase efficiency during the final phases of installing the sculpture on site.
ASTROCYTE, DXedit, Toronto, Canada - fall 2017
AMATRIA, Indiana University, Bloomington, Indiana, USA - spring 2018
NOOSPHERE | AEGIS, Royal Ontario Museum, Toronto, Canada - summer 2018
Introduction
Sound is an essential medium in understanding space, expressing emotions, and abstracting organic and artificial phenomena like growth and electronic textures. When a physical sculpture is present within the soundscape, the spatial and physical relations between sculpture and sound requires a clearer definition. The in-depth collaboration with Beesley et al. has challenged 4DSOUND to extend the control of sound beyond an empty space that hosts sound objects to a world where the virtual is seamlessly integrated with the actual.
The 4DSOUND paradigm considers sound by its sculptural and architectural qualities. Sounds are objects that can be sculpted with dimensions, construed sub particles, and applied properties through sound design. These objects are subsequently placed in a (virtually) infinite space where they can move following particular trajectories or according to a certain behaviour. The setup allows sound to become intimate as close as almost touching the body, or to move like a flock of birds on a distant horizon. Inside this virtual environment, sounds are embedded with acoustic reflections to model the flexibility of sonic experience from inside a small room to a vast space. The flexibility affords the virtual environment to take on surreal forms, creating otherworldly textures and gestures that evoke imagination beyond recognizable.
4DSOUND is an instrument, a set of tools that enables composing and performing intuitively with spatial sound in great sculptural detail. In the context of Living Architecture Systems, the challenge has been multiple. Not only we need to create a virtual sound space - virtual sound sources that live in a 3D space which manifest through a range of speakers - that can be explored, but to integrate this sound world inside a sculpture giving it a voice and allowing its compartments to be meaningful actors within a designed physical environment. Three R&D topics have been central in achieving these goals:
- Categorizing speaker constellations in type and function to gain control over precise sound projection
- Introducing irregular speaker configurations to cater to irregular sculptural configuration
- Making virtual sound objects, actuators, and sensor interfaces part of the same virtual space and enabling integrated interaction.
Exterior and Interior
To embed a distributed network of speakers within the sculpture required a redefinition of the 4DSOUND panning algorithms. The new algorithms expand sound spatialization within a singular omnidirectional grid, hence a distinction between exterior and interior speakers was necessary.
A field of exterior speakers can be considered similar to 4DSOUND’s regular layout. It covers the omnidirectional field of the actual space, and virtually extends this field beyond the actual space to infinity by adding spatial sound synthesis. 4DSOUND’s equally distributed grid system provides a social listening area not concentrated into just one sweet spot (P. Oomen, P. Holleman, L. de Klerk: A New Approach to Spatial Sound Reproduction and Synthesis; Living Architecture Systems Symposium: White Papers, 2016). The listeners are empowered to move around to self-compose experiential perspective and orientation.
The interior speakers are the voice of the sculpture. The configuration of the sculpture’s components and their spatial relations impact how the 4DSOUND interior speakers are set up and run as internal constellations. In projects that have been realized to date, we have successfully generated sounds that behave like a dense crackling neural network, or the omnipresence of smaller objects within the sculpture - like electronic birds in a tree.
For the most part, sounds in either one or both of the fields share the same extensive set of controllable parameters to form movement, physics and perception of sounds in space. In terms of control, this added sonic versatility to cover both the internal and external fields of different sculpture’s components requires only a minor expansion in parameters. A simple matrix interface allows routing sound sources to the desired speaker constellations. Sounds that live within the one sculptural composition, can easily be transferred to, or spread over other components. Indirect connections are also possible, like external reflections as responses to sounds that originate from inside can potentially create echoes that seem to come from spaces that are beyond the one that the listeners are in.
For the installation Astrocyte (2017) we introduced the ‘Sphere Engine’, a prototype application that complemented the regular 4DSOUND Engine. The Sphere Engine was designed to drive ten speakers inside the Large Sphere that created the central constellation of the installation. The speakers were spread almost equally across the central semi sphere, which was slightly tilted and raised above the heads of the listeners. The result was a defined focal listening point within the expanded space of the sculpture. This centralization together with the gentle directional quality of the speakers and granular sound design evoked a sensation of proximity, and a feeling of intimacy with this large, radiating object.
Image: the Large Sphere of ASTROCYTE, the inner semi sphere contains 12 speakers Photo courtesy by Philip Beesley
Image: 3D visualisation of the Sphere Engine spatialisation developed for ASTROCYTE
We found that the results of technical sound tests - such as exposure to continuous noise and a variety of waveform pulses to different spatial projection onto the sphere - offered a broad palette of characters and abstract associations. For example, synchronised pulses can accentuate the shape of the sphere as a whole, where diffracted sounds dissect its parts and seem reveal the granularity of inner processes within the sphere. As a result of high speaker density the perception of small, moving sound objects became extremely clear. Altogether, the sphere created a ‘sonic painting’ with a broad palette onto a tightly focused frame.
Image: user interface of the Sphere Engine developed for ASTROCYTE
Irregular Speaker Configurations
4d.pan, the proprietary panning algorithm developed by 4DSOUND, was initially limited to symmetrical rectangular speaker grids. To provide asymmetrical sculptures with varying densities of interior speakers, we expanded the 4d.pan algorithm to support irregular setups of varying sizes and shapes. The principle remains to position exterior field speakers as equally spaced as possible to assure balanced sound projection and refined continuous spatial definition in all directions. For the interior, all configurations are allowed to support any artistic or acoustic tasks and to create more irregular grids. A combination of an equally spaced exterior field with an irregularly spaced interior determined by the sculpture can deliver a convincing and powerful soundscape with explicit control over the sonic presence in the space.
Any given constellation of speakers can be divided into subsets to allow extremely detailed control in spatial sound design. The setup differentiates speaker groups (areas) for sound sources to exist within. The internal structure of the sculpture divides interior speakers in ways that mimic its components. This results in an almost organic design where the sonic and material interpretation of the sculpture dictates the speaker configuration, instead of the other way around. We are excited that backend complexities can realize an elegant frontend for intuitive system handling and sound design.
The permanent environment Amatria (2018) at Indiana University is a follow up on the Astrocyte installation (2017). The environment consists of ten speakers distributed throughout the larger part of the sculpture. We identified the Large Sphere (speakers A1-A3), the Small Sphere (speaker B1), and the Canopy (speakers SF1-SF6). By defining three sub constellations we were able to develop the spatial sound design around two actor objects (the Spheres), and a more diffracted area (the Canopy).
Image: irregular speaker configuration of AMATRIA
As the protagonist, the Large Sphere is equipped with three speakers, while the Small Sphere has only one. The differentiation made possible by using a triad configuration creates a spatial depth and dimensional perception of the sounds that are projected onto the field. The single speaker in the Small Sphere is timid compared to its neighbours, but the sharp contrast in sonic magnitude and its distinctive character defines a focal point within the shared space.
While designing the spatial sound for the installation, we experimented with unique role assignment for each character in the sculptures. For example ‘breathing’ spheres would have subtle reflections on the canopy with patterns resembling textures of dense fog, light raindrops or breaking glass. The flexibility of the system in filling the entire space or carefully occupying specific areas proved to be essential in constructing the integration of sculpture and sound.
Image: AMATRIA installed at Indiana University, Bloomington USA.
Shared Worlds: Merging the Virtual and the Actual
Another important R&D goal has been the integration of sound spatialization with other kinds of actuators in the sculpture such as granular light, kinetic mechanisms and sensor interfaces. To create meaningful integrated interaction we developed the prototype Distance Based Interaction Engine (DBIE) in collaboration with Philip Beesley Architecture Inc. members Rob Gorbet and Adam Francey. The DBIE is a simplified interpretation of the spatialization algorithms that already exist within 4DSOUND. In the 4DSOUND Engine, spatial properties of virtual sound objects are mapped to speaker amplitudes and audio signal processing (spatial synthesis). Meanwhile, the DBIE maps basic spatial properties of an ‘Excitor Object’ like position, size, and force to lights, kinetic objects, or any type of actuator.
The basic principle of Distance Based Interaction is the intensity of an actuator (e.g. the brightness of a light, or the speed of a motor) which is determined by its distance to a virtual object. Every actuator holds position properties that match their actual relative locations within the sculpture. The virtual Excitor Objects also live inside this 3D map where their presence can excite the actuators by moving closer and further away. An excitor can be an infinitely small point, or of any spherical size. An increase in scale means it covers more virtual space and effectively comes closer to more actuators resulting in more actuators being excited. The final parameter is Force, a multiplier of Distance Intensity acting as the upper limit of excitation. With scale and force combined, one can create a large but weak excitor, or a small and strong one.
Sensor systems are integrated in the same Cartesian space as the sound objects. Their positions in an infinite continuum of space are defined by XYZ-positions (P. Oomen, P. Holleman, L. de Klerk: A New Approach to Spatial Sound Reproduction and Synthesis; Living Architecture Systems Symposium: White Papers, 2016). Infrared modules register the presence of visitors and their positions inside the sculpture, as well as their hand gestures. Virtual sound objects and excitors can react consistently and immediately in attractive or repulsive reflexes triggered by infrared sensors. There is a subtle balance between direct basic responsiveness (e.g. triggering short samples on the location of the sensor) and more complex movements that evolve over time (e.g. ripple effects in the surrounding actuators, or the progression of the sound composition on a higher level). The correlation of sound events in space and actuation triggered by movement and behaviour enhances the immersive experience between the visitors and the sculpture.
In this setup, virtual sound objects, virtual excitors, speakers, actuators, and sensors all share the same environment, and can interact accordingly. This produces a versatile and intuitive interface to design coherent movements and behaviour. Background behaviour can be small excitors, moving fast and slow, emerging and vanishing like snowflakes or a soft breeze through the woods. Bigger excitors could be phantom creatures, or abstract magnetic forces beating through space. The technical essence is that single indexed actuators are no longer considered in artistic design, instead, the interface is elevated to a higher level of control where conceptual translation becomes more immediate and inspirational.
Image: Excitor Objects and their distance-based relationships to actuators present in a sculptural environment
The first implementation of the DBIE was realised for Noosphere and Aegis (2018) at the Royal Ontario Museum in Toronto, Canada. Background behaviour is generated by small, semi-random excitations throughout the sculpture. In the larger sphere Noosphere, an excitor is active with transformational size, force, and trajectory. They mark the first modest but profound step in unifying algorithms for behaviour in sound, light, and movement, and in interaction with human input through sensor interfaces.
Image: human input as excitation for kinetic sound, light and movement within NOOSPHERE
Future Research
In future research we will refine panning algorithms, explore the characteristics of different kinds and combinations of speakers, and continue stretching the flexibility of sound spatialization using varied speaker configurations. The divide between exterior and interior sound may be further contrasted to enhance depth of field. At the same time, the differentiation will also invite explorations in the coherence between the perception of the diffusion field and focused objects, emerging from the same technological principles in hardware and software.
A shared origin with all technological disciplines in the same spatial interaction paradigm will be further developed in an even more integrated software reality. A right balance between utilising processing power of a personal computer, and performing algorithms on the distributed network of microprocessors that are not necessarily aware of structural properties, can enable each sculptural character act more like individual agents within the network.
Other objectives are designing an offline prototyping environment to develop interactive algorithms and behaviours that can develop independently from a hardware testbed. Since all virtual properties will share the same physical environment, they should be included in a shared virtual environment. An extensive 3D visualization of all activity will complement the possibilities to exploit spaces as an instrument and provide insightful feedback. The tool will allow detailed preparatory work, stimulate creative design of the space as a whole, and increase efficiency during the final phases of installing the sculpture on site.
Related:
‘4DSOUND: A New Approach to Spatial Sound Reproduction & Synthesis’ (2016) [PUBLICATIONS]
‘Sound Holograms’ (2013) [PUBLICATIONS]