✘ Listening to A.I. Music
And: Tech's broken promises; The DJ is still human; Localisation is vital for emerging global stars; Moving Castles; Large lore models
Hi all, this is the second installment of our partnership with the MUTEK Festival and Forum. This article is written by Yotam Mann and Sarah Rothberg. Their focus is squarely on bringing in the listener into the debates about AI, music, and creativity. It doesn’t sound radical, but it’s a rare occurrence in the discourse so far. It’s not surprising that in their conversations Sarah and Yotam got to this point. Yotam has long dedicated his career to discovering how new technologies can progress music. Sarah is an interactive media artist, whose work takes form as playful, poetic, usually-a-bit-weird experiences that invite you to reconsider your relationship to the world around you.
Love, Maarten
“...they were trying to tell me that A.I. could make a voice that sounds just like me, but it’s not me, because I’m amazing. I’m like, is this A.I. thing going to be amazing too? Because I am naturally, organically amazing. I’m one of a kind. So actually, I would love to see that thing try to duplicate this motherfucker.”
Can A.I. make “amazing music”?
When you think of A.I. music, what comes to mind? Is it an artificial popstar? Is it infinite clones and variations of an artist or style you love? Is it a genre-bending new sound? Is it muzak?
Is it good? Will it ever be?
As an A.I. music software creator and an artist working with A.I., this is something the two of us have been talking about a lot lately. Unfortunately, in trying to figure out what A.I. music is, and what parts are good (let alone amazing), the conversation gets messy fast. In part, because there are so many ideas - true and false - about what A.I. music is or can be, but also, because it hinges on the unanswerable question: what is good music?
But, in the mess we found some interesting ways to organize the conversation. In particular, we felt it was helpful to define different aspects of what A.I. music is and might be, as well as how we listen to it and appreciate it…. or don’t!
What is A.I. music?
There are a lot of different things people might mean when they say "A.I. music." We created this little graph as one way to distinguish them. On one axis is how the music was produced: a spectrum from human intuition to a fully-automated production. On the other axis is the listener's mental model – their understanding about who the artist is, probably a product of storytelling or marketing. In other words: to what degree does the listener feel that the music was "made by an A.I."?
Placing music (both existing and hypothetical) on different points on this graph was fun and generated a lot of ideas and weird examples. On the top right are most songs that have ever been made: think of someone with an instrument in front of an audience. Towards the bottom right, you might imagine a producer using only samples from MusicGen, a recent text-to-music model, to make a song.
On the left side, we tried to place the Fake Drake song, Heart on My Sleeve, and got stuck. The story of Fake Drake was: this is music made by an A.I. (putting it in the bottom left). But the reality is that some producer(s) created it using a machine-learned voice model (so top left). Fake Drake blew up in part because the lore of the song – “A.I. makes Drake’s music” – is an exciting story!
The fact is, there's no such thing as "fully automated." There are always humans involved, and to say otherwise is to totally erase human labor. Humans are always involved in A.I. music, from producing or labeling data, researching new algorithms, to editing, curating, or even marketing. But despite the fact that it’s impossible, people keep picturing and promising a “fully-automated A.I. artist” because it makes a good story. People love stories!
Listening to listeners
Let's imagine the bottom left quadrant was somehow possible, that we had a “true” A.I. artist. Would we want to listen to it and the music it produces? That brings us back to: “is it amazing”?
Unsurprisingly, we don't have the answer to that question. But – the question itself reminds us to consider the listener, as opposed to just "how it's made."
We tried to enumerate the different ways we personally listen to music, and found some situations might be better suited than others for this imaginary fully-automated A.I. artist. One that kept coming up was "functional music": music that is only listened to for utility. It helps you accomplish some goal like study, sleep, or working out. “Optimized towards some utility” is what these algorithms are built to do, so that type of A.I. music seems like a more plausible fit.
But, is useful music what we mean by “good music”? We decided, for the sake of the discussion, that “functional music” is a separate category (unless you want to imagine a world where all music is purely "functional"... which seems sad).
Imagine a song you love – one that's really important to you. Imagine why you listened to that song, what it means to you. It's probably more than just the sounds in the song – but the associations you have with it: maybe you love the artist, where and when you first heard it, what the sounds remind you of. Maybe it's a song that helps you process a feeling, or that shapes your identity.
Every time we tried to imagine the (actually impossible) fully-automated song connecting to us emotionally, we either couldn't imagine or didn’t want to imagine it. Why? Art is not just the output. A recent paper on the subject, AI Art and its Impact on Artists, put it nice and simply: “Art is a form of communication, it communicates.”
The closest we could come to imagining loving something from that bottom left quadrant was to imagine ourselves as DJs or curators, listening to something generated with the goal of sharing it with someone else - communicating through it. Maybe in order for any of this music to be “good” in that elusive, transcendent, non-functional way, someone has to listen to it, and then say: this is good, and I want to share it with someone else.
And in that case, the A.I. just ends up back as a tool for an artist to create with. On the other side of the graph!
A.I. is a tool to make music for listeners
Once listeners become more attuned to what people do with A.I. (and less captivated by the technology itself), the effect these tools will have on music might be similar to other technological breakthroughs. Technology has in countless ways changed the way artists create, what we listen to, and even how we listen.
The first synths were built to imitate other instruments, but ended up making their own sounds altogether. Autotune was originally designed to correct off-pitch singing, but turned into its own style with its own virtuosity.
What changes might this technology make to music?
Machine learning (which is a more accurate description, but doesn’t have the same hype-y ring to it) will also change the way music is made, first through practical improvements – like stem splitting and style-transfer – but eventually will produce its own sound through musicians pushing it to its limits and breaking it.
And, maybe there is some version of an A.I. popstar, playing auto-generated songs. But, there might be (or hopefully will be) many, many creators attributed – and valued – for its production.
Back to whether any of it is good? Listeners still decide!
LINKS (via Maarten)
💔 Tech's broken promises: Streaming is now just as expensive and confusing as cable. Ubers cost as much as taxis. And the cloud is no longer cheap (Alistair Barr)
“Take video streaming. In search of better profitability, Netflix, Disney, and other providers have been raising prices. The various bundles are now as annoyingly confusing as cable, and they cost basically the same. Somehow, we're also paying to watch ads. How did that happen?
✘ One of my favourite pastimes is to compare the music industry to other tech industries, especially video streaming with films and series. Not that long ago, most music people thought Amazon was crazy to have a big array of different price points for the music service. Then, it became brilliant as the one-size-fits-all model of Spotify et al started creaking. As with Netflix & co, we now see that model going full into full ad-mode again. Lots to learn from those cycles when it comes to music and potentially opening up new revenue models - albeit just for the incumbents.
🗿 AI is remixing music - but the DJ is still a human (David Beiner)
“Collaboration between humans and technology is critical and has always been evolving. Our relationship with technology throughout the creation process will continue to be essential.”
✘ Squint, and you can see a future where a whole new creativity mode is unlocked through AI-powered tools. Interestingly, and especially in light of the piece above by Yotam and Sarah, David doesn’t consider the listener in his article. Will they also welcome a new era of creativity and adapt their ears?
🛝 Why localisation is vital for global stars from emerging markets (Srishti Das & Michelle Yuen)
“It’s easy to point fingers and say a successful artist must give back to the community, but must that be the main objective for successful artists? I disagree. The fate of a growing market should never be in the hands of one artist or a small group of people. It must lie in the hands of the larger local industry itself. Instead, an influx of investment and initiatives is more likely to drive the artists at the top to give back and participate in growing their home markets.”
✘ There’s an interesting parallel here to the Canadian model where this ‘giving back’ model is incorporated into law. So maybe it’s not a main objective for successful artists, but it could well be that it’s up to local governments to look into policy solutions.
⛫ Modular and Portable Multiplayer Miniverses (GVN & ARB)
“In result, there is an emerging need to supplement new distributed funding and decentralised governance mechanisms with new routes between private communities and public participation. We need new participatory media formats, as well as changes to existing formats, to better represent and facilitate collective agency and value distribution so that communities can avoid exactly the effects they are trying to escape from; Web 2.0 influencer dynamics and their power laws, where all the value and control flows to a few central nodes. We think of these private platforms as arid wastelands that still need to be ventured into for loot, the rescue of new members and the building of our new vehicles, until we can replace them with decentralised alternatives.”
✘ I actually want to share the article below with you today, but you’ll need to get your head wrapped around these moving castles first. I recently wrote about how we retreat into smaller, closed-off communities away from the Web2 platform economy. However, the experiments we run there, need to find ways to reach back out to our commons. Moving castles are one way to think about that. Next up, lore.
🪁 Large lore models (dmstfctn, Eva Jäger, Alisdair Milne)
“Having established the potential of Autonomous Worlds to become autonomous knowledge commons, given the right infrastructure, we look to define lore and “lore generation” in concrete terms by using a procedural spoken language game, I Went To the Shop, as an example. From there, we will consider how to hardcode this process of lore generation into a viable tool (LLoreM) that combines the value of player lore generation with the unique affordances of onchain games, considering the implications of such a tool in comparison to legacy Web2 lore-recording.”
✘ Take the time to read the whole article, but basically what they’re suggesting is a form of decentralized lore generation and a mode to capture it and make shareable. Any community, or culture, requires their stories to grow, sustain, and build. It’s hard to keep these stories going and then harder to record and archive them. Doing this consciously by embedding that into the tech stack is a great solution.
MUSIC (via Maarten, but with Yotam and Sarah)
A slightly different style of music recommendation today. Because I personally love a lot of what Yotam and Sarah do, I also wanted to take this opportunity to showcase their work a bit more. There’s a lot of cool stuff, but jazz.computer is one of my favourites. It brings to together our endless scrolling with music. Try it and have some fun.
 | A guest post by
|