synwrks

The Imagination Machine Today

February 12, 2022

Feelings and thoughts live within ourselves, sometimes for a day, sometimes for a decade, until they either cease to exist, or a kind of osmosis slowly filters them through the constraints of physical reality into their tangible, worldly manifestation. Just as anything fragile and cherished, you wish for them to be protected from the ephemeral nature of your memory and being.

This article is the manifestation of one such thought/feeling that finally found its way out. But in doing so, it had to face a number of limitations that irremediably shape it and distort it. A "thought" feels like an unordered blob of connected concepts, which slightly change every time it is queried, and often blends with personal memories and experiences. A blog article on the other hand is a precise set of words, connected concepts that have to be ordered into a culturally pre-defined structure, stripped of superfluous diversions and photographed into a solid, immutable object. It's a brutal coercion.

This same translation process is incurred by any artifact, be that functional gadgets or pretty paintings. This article will focus on the way it affects artistic creations. More specifically, I will dive into how digital technology unlocks a paradigm shift in the process of depicting our imagination. The intent is to express a renewed appreciation for something that we otherwise tend to overlook as obvious and granted.

The inspiration for portraying a landscape often comes from examining the tiny water flows and detailed crevices on a mountain that has always, and obviously, been there for everyone to see. The portrait does not reveal the existence of the mountain, but rather depicts a way to see it by choice of a specific lighting and the underlining of specific details. Let this be one portrait of nowadays' creative process. And if it speaks to you, let it be a source of inspiration.

Superpowers

The Imagination Machine 2121 describes technology allowing two people to experience each other's imagination. While that device only lives in science-fiction, it constitutes a reasonable future milestone for a present-day trend. This trend is the improvement of our creative tools and processes thanks to technology.

The focus of Synesthetic Works on digital art and electronic music derives from a fascination towards the immense freedom and precision of expression granted by these forms of art. It allows the art piece to represent the original thought, as it lived in the artist's imagination, with incredibly high fidelity.

While these new possibilities allowed the birth of novel artistic genres, I believe that is not because of new tools imposing different constraints, but rather because fewer constraints brought more possibilities into the open. At the same time if one had at their disposal only a laptop, and wished to express a fascination for classical music, they'd be able with this one instrument to compose and record an orchestral piece coming fairly close to what their imagination of it is. Of course it won't perfectly emulate the recording of a real orchestra playing the same piece, but it will get close enough for the idea to be considered accurately represented. The fact that the resources to achieve this went from a large team with tons of equipment and decades of training each, to an individual with one instrument and perhaps a few months of training gives a better perspective on the kind of transformation these tools brought about.

These are some examples of how skill, resources and time investment to represent one's imagination have been shifted towards an effort of introspection. There is no intent here in declaring one method lesser or better than the other, but rather to highlight an implication of the accessibility shared by the methods in the right column.

Original Thought Process One Century Ago Process Today (through software)
An apple resting on a table Place the pigments on a surface in such a way that it resembles the shape of the apple, the table, the diffusion and reflection of light, its bouncing a red tint over the table, its getting caught in the tiny shadows between the wooden table's nervatures. Win the fight against your hand's inability to stay still and the propensity to make frequent, unforgiving mistakes. The result will be a stylized apple. Decompose the image into its makeup of ideas: sculpt virtual clay to resemble the desired shape, describe the light source, then use a graph to combine concepts of randomness, reflectivity, color, bumpiness into the apple's skin material. Let the machine infer a photo-realistic image based on an understanding of physics.
A walking animated character Time and talent limitations will usually constrain this representation to a cartoonish silhouette or stick figure. Draw that a number of times (likely at least 50) in a way that resembles the intended movements. The final product can incorporate the illusion of movement through the use of additional tools/process. Describe the concept of a skeleton, describe its movement (in the simplest case, by enacting it in front of a camera, letting an algorithm understand its relationship to the skeleton). Describe the looks of your character (similarly to the apple above). Combine these concepts, let the machine throw in the physics, derive and draw each single frame. The result will likely have much higher fidelity per amount of time invested.
A melody played by an harp You will need to be trained into playing that specific instrument, and obtain one. Then you'll have to obtain the recording equipment and attempt a number of "takes". Select the best one. Describe the melody in terms of pitches and their "intensity". Describe the instrument's characteristics, either by describing its physical attributes to be simulated, or by using a library of audio samples. Simulate the "room" by fine-tuning a reverb. Combine and iterate on all these concepts until you are satisfied with the representation.

Many of the processes in the left column, if executed by professionals within the specific field, can often be faster than the ones described in the right column, which themselves still require some amount of training. The commonality between the examples above however lies in the fact that the examples on the left require high-cost, highly specialized equipment and training, while the ones on the right all rely on one common piece of equipment, and very generalizable knowledge that can rapidly be acquired. This better caters to the wild variety of our daydreams: as long as we can break them down into simpler concepts, there is likely a very short path to their manifestation.

The enabling and multiplying effect digital tools have on one's expressive abilities, through a usually negligible investment in resources, makes them akin to superpowers: unlocking a new dimension of self expression only takes a few minutes or weeks of training. Each a declination of shaping electricity into tangible manifestations of your imagination. Each seemingly "limitless" (in possibilities) and "sufficient" (towards the achievement of any vision).

These are all however distinct specializations of the same magic: the ability to describe thinking itself.

Thinking Replicas

What sets digital art aside from any predecessor, is that it provides tools for the automation of thinking. Thinking replicas are not a new thing: if for example you bought a synthesizer in the 80s, you would be relying on a similar form of automation. Somebody had found a way to capture a way to think about something - the parameters by which you could describe sounds with specific tones, timbres and dynamics - and captured it into a box with knobs: oscillators, filters, effects and more. The synthesizer would repeat that thinking process indefinitely for you to represent thoughts with: a melody, a noise, a robot voice... This can be said of other automation scenarios, even with more practical ends.

Computers immensely improved that process by making it easy and accessible to describe the way we think through coding. As we have seen in the examples about producing images or melodies, our thinking process can be broken down into a series of abstractions, which can be described through code for a machine to execute. It is no longer necessary to understand electrical or mechanical engineering, source materials, open a factory and hire employees to build the machine that "gets it". All of that has been reduced to writing some words on a laptop.

And that is why code can be seen as a new form of poetry, and perhaps a very intimate form of it. So intimate that it requires us to pierce through the surface of our conscious thoughts, and question their very nature. It requires us to learn to explain how a poem is born, rather than pinning down one possible outcome. What you produce is a replica of a portion of yourself, parts of you that you recognize in the art you make.

While these flowcharts are not representative of every artistic process, they can help represent the paradigm shift from the representation of thought:

To the automation of it:

Replace "Thinking Replica" above with "Music Sequencer", or "3D Modeling Software" and you might quickly find yourself in the map. The artist could also coincide with the software developer, which is perhaps the opportunity this article tries to advertise. Input could be generated by a creative, but also by the audience engaging with an interactive installation. Once thought is automated it can have a life of its own: meeting new people, taking part to their projects or entertaining them.

It is important to note that the automation of thought itself leverages and produces a continuum of abstractions. In this context, an "abstraction" can be seen as a facade of sorts: a choice is made on how to "represent" a collection of possibilities through fewer, simpler ones. When you describe what the skin of an apple looks like, you'd rather not have to write the mathematical formula that given the position and characteristics of the light, and its projection on the surface of the apple, outputs numbers for the red, green and blue channels of a pixel. You'd probably just prefer to explain how the material of the apple's skin varies between a number of colors, with certain random imperfections in its reflectivity and bumpiness. The latter can be considered as an abstraction to the former. You'd probably describe this material by connecting colorful blocks together, dragging levers and picking shades of red from a rainbow wheel: these themselves are abstraction, maybe built using a scripting language like Python. Python itself offers a simple syntax and a few handy utilities in the effort to abstract away more complicated aspects of computer programming that are instead surfaced in lower-level alternatives. This simplification pyramid can grow indefinitely in either direction.

If you are a digital artist, you are no longer using an instrument or tool. You are using a meta-tool, which allows you to expand your creative agency in the direction of building the tools themselves. Building tools, in this context, means describing your thinking itself, rather than one of its outcomes. Automating your thinking helps you better and more rapidly approximate your thoughts, feelings and imagination.

Synesthesia

Teaching computers to write poems is perhaps an unintuitive example. There are just too many elements involved, and in fact all examples of this I encountered so far feel closer to random text generators than actual replicas of the author's minds. Perhaps a form of meta-art exploring and exposing the process itself, rather than focusing on self expression.

The word "synesthesia" describes the phenomenon in which unrelated sensorial stimuli trigger each other. The beauty of this mechanism is that it is low-level enough that it can indeed be easily described to a machine for automation, while at the same time manifest different qualities for each individual experiencing it. Some synesthetes associate different colors to different letters, some associate words to flavors. Perhaps the most popular forms of it, and what inspired Synesthetic Works, are those correlating sounds and images.

As it happens to be true for the author, the experience of Synesthesia is a multi-faceted one, involving multiple elements of one's imagination and perception. If its strict definition is followed, there is already plenty of beauty to be expressed: abstract patterns or shapes describing the qualities of each sound, colors describing pitches, enough movement and detail to fill any screen. Zooming out from that experience, one can easily think of the wider evocative power of music: an entire soundscape can invoke places, a rhythm can invoke speed, melody can tell a story. The fascination for the ideas described in this article found its perfect outlet in the automation of this synergy.

Representing the synesthetic experience of a music track, or even a hour-long DJ mix, is rarely something a pre-digital individual could undertake. A walking character is one thing, but 20 instruments playing a continuously evolving soundtrack for 6 minutes could deter any synesthete from producing a cartoon to describe their experience (especially if this synesthete also needs to work for a living). And yet, synesthesia being different person by person, makes it an extremely intimate reality which will rarely stay authentic through the dynamics of a large team (more on this later). If the characteristics of music are described in a way that can be understood by a machine however, the only missing piece would be the description of how the two relate. This would effectively put the burden of animation on the computer, which would be "automating" your very specific perceptual glitch.

Fortunately, this is within many people's reach today, especially in the field of electronic music: most of the information is neatly organized for a machine to parse already, as a machine will ultimately generate the sound itself. Describing the relationship between notes, sound and visuals can be done in many ways, some involving code, some involving visual languages, or a combination of things. The beauty of a meta-tool is that, if nothing out there can help you describe your specific flavor of synesthesia within your specific time constraints, you might be able to prototype something that does.

This musical detour is only meant to be an example: what is it that only lives within your mind, that you wish you could share with the world, and just needed a bit of automation to become feasible?

What do Machines Have to Do with Intimacy

I mentioned how digital art can push us to inspect the source of our imagination, and describe our thinking mechanisms rather than just some of its possible results. I also mentioned how this, as any other automation process, makes it possible for an individual to take on the work entire teams used to tackle. Digital art is sometimes seen as a somewhat cold endeavor: so much of the tactile, intuitive, "human" characteristics of art-making are lost to the plastiky feel of keyboard and mouse.

I would like to argue the opposite to be true: I think never before in history, have we been like now in the position to truly manifest our inner universes to the world. The fidelity in which our dreams can be approximated, the ease and immersiveness with which they can be shared, has never been higher. While I don't mean to argue against the indisputable potential of large teams working together, when it comes to the search for authentic, personal, unfiltered expression, the more agency an individual has in the creative process the better.

In fact, when we imagine the painter confiding his point of view to the canvas, what we are witnessing is the struggle of an artist pushing that insight through the infinite constraints of physical reality. The wind keeps moving the tripod, the colors are dripping out of place, a mistake will need to be disguised by an extra tree, the shade doesn't look right, the gradient is too abrupt... Even if everything goes well, and a new masterpiece of realism is given to light, it is confined to the bidimensional reality of "still" life. What if the dream slowly evolved through the seasons? What if it was inhabited by the chirping of otherwise invisible birds? The artist hasn't even contemplated the possibility of representing all of that. Because how could they?

They would need to build a team, hire animators to draw and sound designers to mix. They would need to hire artistic directors to make choices on their behalf, and throw in one or two completely unrelated advertising elements to be able to pay everyone at the end of the month. Grow the team a little more, and storylines are written by lawyers trying to make the most of the IP, colors are chosen by marketers trying to keep the brand identity intact. The final product will likely still be extremely appealing and successful, but the original dreamer will now be running from meeting to meeting while managing people: the resulting enterprise no longer represents any one individual's imagination anymore, and the precious state of "flow" found in artistic pursuits has now been lost to a job title.

Of course I'm not arguing that one form of art is lesser than the other. I would never want to give up watching Star Wars movies, or playing The Legend of Zelda games. I'm rather trying to highlight the inherent warmth that there is in how digital technologies empower the individual. I think the extension of artistic skills to include means of thought-automation truly opens a door on the "limitlessness" of digital art creation: scarcity of time and resources can be overcome, or medium constraints can be expanded. The work required by an artistic vision can be compressed to occupy the time available. A hobby of drawing static, flat images can be expanded to produce scenery that is immersive, 3D, animated, interactive or all of the above.

And if that "plasticky" feeling about it all persists, don't let existing products limit your way of expression: build your own instrument if you need to. You can sculpt shapes in virtual reality, or sketch on a touch screen, blend it all in a 3D software, write some code to automatically generate environmental sounds based on a caption, build a program that assigns visuals to your music collection, repurpose the buttons of a DJ console. Cover your room in wires, screens, experimental MIDI guitars, stereoscopic cameras and speakers as you put together your very own "imagination machine".

- C