✘ How to: accessible tech for emerging artists
And: discover Eveline Ypma’s AI-driven soundscapes - explore Jan Modaal's avatar-enhanced performances - dive into AR innovations for live music venues
Innovations like synthesizers and platforms like Napster have transformed the music industry, introducing new genres and consumption methods. Today, AI and immersive experiences (IX) play a major role, with artists like Gorillaz and Grimes using advanced tech to boost creativity and autonomy. However, only a small group of artists and venues, mostly in the US and UK, have the resources to work independently with this tech. Most musicians, like those in the Netherlands, are limited to restrictive applications that control intellectual property.
Prominent researcher Timnit Gebru writes about this: “We do not agree that our role is to adjust to the priorities of a few privileged individuals and what they decide to build and proliferate. We should be building machines that work for us, instead of “adapting” society to be machine readable and writable.” Open Culture Tech champions this, investing in autonomous tech to empower artists. Thunderboom Records, a non-profit lab, develops and shares responsible AI and mixed reality tools to promote inclusivity and equal opportunities in the music community.
Thunderboom Records is a non-profit innovation lab that supports the music industry by developing and sharing responsible technology, such as AI and mixed reality tools. Thunderboom Records aims to strengthen creativity, resilience, and equal opportunities in the music community. Through collaborative projects, interactive workshops, and open-source tools, Thunderboom promotes tech for good, focusing on inclusivity, transparency, and complementarity. In this newsletter we share three collaborative use cases
Through Open Culture Tech, we collaborate with 12 artists to create an open-source toolkit for AI, AR, and avatars, ensuring these tools are democratic, inclusive, and transparent. We share all tech and use cases in our newsletter on www.openculturetech.com/newsletter
Jan Modaal
Jan Modaal is a punk singer from Amsterdam. Jan ran into the issue that it is hard to interact with the audience when he is playing because he needs to focus on singing and playing guitar. For this reason, Jan and Open Culture Tech are exploring how Avatar technology can enhance his body movement and help him connecting with the audience
Open Culture Tech is working with Reblium to build an accessible, open-source and web-based Avatar tool based on Unreal Engine technology. For the collaboration with Jan Modaal, a test-ready prototype was developed that we used in a performance from Jan called “Jandroid Modaal”.
The starting point of our Avatar tool is a 3D scan of Jan’s head, made with Polycam. This 3D scan was processed into a character that Jan could edit himself in the tool. Jan eventually made 5 versions of himself that he placed in front of 5 different backdrops. These backdrops were AI generated with Runway. Both the avatars and backdrops were placed in Reblium and projected on a big screen that was placed in front of the screen. For this we used a very low budget regular projector and fabrics bought at the market as a projection screen, which worked well.
We tracked Jan’s body movements and facial expressions using two camera’s that were connected to the Reblium system. Jan himself was performing behind the screen, breaking through this barrier halfway during the show to literally break with the virtual representation of himself.
Based on the findings and feedback from this performance, further prototypes are being built that we will test again in the fall. The goal is to publish our tool for free on our website by the end of this year. With the latest version of our Avatar creator tool, it is possible to upload yourself as a 3D character (with Polycam) and animate this avatar in real-time using a webcam that copies your movement.
Eveline Ypma
Eveline Ypma is a soundscape artist from Amsterdam. She has created a production called Laugarvatn, consisting of three parts of 5 minutes each. The performance is named after a place where Eveline Ypma made several field recordings during her residency on Iceland. These field recordings form the basis for three soundscapes in which she combines these field recordings with live vocals and bass guitar. Together with the Open Culture Tech team, a fourth part of 10 minutes has been created in which the Icelandic field recordings have been replaced by AI-generated sound samples, in the style of her original Icelandic field recordings.
To generate original samples, Eveline played with various text-to-music tools (ChatGPTs for music). During her residency on Iceland, Eveline never saw the Northern Lights, so she decided to use AI to generate unique sound samples based on the prompt “Northern Lights Soundscape”. In this way, Eveline was able to create new music inspired by her journey and add a musical piece to her already existing work Laugarvatn.
The result of the collaboration between Eveline Ypma and Open Culture Tech is not only a showcase in which we have used generative AI to generate unique samples for live performance, but also the first version of a new open-source AI tool that allows anyone to sample their own samples. can create based on prompts. If you are curious about the process of creating this tool, and want to know more about how this application came about, read the detailed article here. And stay tuned, the Open Culture Tech AI-driven sample tool will be published soon.
Next Stage AR
We also collaborate with music venues to discover new possibilities for them in servicing artists and their audiences and finding new ways to monetize their services in a declining market. In our Next Stage AR project we investigate the potential of AR technology for venues.
AR applications have rapidly become part of the live music sector. The technology has already proven that it has the potential to enrich a live experience in an unforgettable way and it is expected that this will become a permanent part of the tasks of live music organizers in the near future. It is currently unclear how the various AR technologies can be used in an accessible, sustainable and profitable way by venues and festivals to enrich the immersion and social interaction of live shows. Together with concert venues, design studio Superposition and the VNPF we investigate in the Next Stage AR project what the potential is of mobile augmented reality for smaller concert venues and their audiences.
During a series of live concerts we test various applications of AR such as wayfinding, stagebuilding and marketing purposes. Alongside we develop our own open-source AR builder that can be used freely by venues all over the world.
We just had our first live test concert at Paradiso in Amsterdam. The below video displays one of the AR experiences of the night, created by VJ Bikkel with our open-source AR tool.
We also give workshops, lectures and keynotes. Curious to hear more about our projects? Drop us a line on max@thunderboomrecords.com or joost@thunderboomrecords.com.
LINKS
👩🎤 Open Culture Tech website
✘ This website provides an overview of all use cases within the project. All tech will also be released here before the end of the year
🤳 Why AR is a technology to take seriously in the music industry (Alberto Martinez Jr)
‘‘AR has the potential to revolutionize the interaction and connection between the artist and audience. It breaks down traditional boundaries, offering fans enhanced access to their favorite musicians and performances, and providing artists innovative ways to express their creativity and engage with their followers.’’
🖥️ AI and the industry: more use cases (David Henkin)
‘‘Artists like Travis Scott have pioneered virtual performances, leveraging AI to create hyper-realistic avatars and interactive environments that transcend physical limitations, reaching global audiences in unprecedented ways.’’
✨ A changing industry due to new tech opportunities (Guido Segers)
‘‘Great artists are already playing around with the possibilities. Drake, for example, has a bizarre shop, where you can look around his house 'The Embassy' and click on products. This way you buy the same fridge magnets as the rapper - from which he probably makes some money.’’
MUSIC
Our use cases are so fresh from the press that the accompanying music from Jan Modaal and Eveline Ypma has not been released yet. That is why we share some earlier work from both artists.