Publications — July 2016

A brief introduction to video mapping

Published in the journal ADE Teatro No. 161. 2016

We have always tried to touch the stars with our fingers. Our thirst for wonder, the need to believe beyond our limitations, to flee our reality or dress it up, has been and will be present in everything we do or imagine. We play with reality and we deform it, stretch it, change it, or simply present it as we feel it or believe we feel it. We confront it.

In this way, and in many others, we mirror ourselves as spectators and as makers; we involve ourselves in our daily life in society, whether from inside the stage or from outside it.

We want, we long, to be deceived, to believe that what we are seeing is real, if only for an instant, and in that way, thanks to a lie we all take on, we create a truth and go beyond reality. That is the game; it has always been so.

Why open with an introduction more proper to the performing arts, or to art in general, in order to talk about a set of techniques we call video mapping? Because in some way we go on doing what we have always done: whatever moment we happen to live in, we makers have pursued the same things and caressed whatever technology was available at the time. Because we are the heirs of our past, and video mapping is hardly an exception. Although it was not born within the performing arts, and its natural habitat is computing, street art, electronic music, VJs and architecture, we can see ourselves reflected in it and take up and fold into our aesthetic, visual and narrative discourse the opportunities this technology offers us.

What we do with it will depend on us, who are, as makers, ultimately responsible for what we make and show.

Video mapping is relatively new — so new that there is barely any research on it and reliable information is hard to find. What is more, nobody agrees. It is born, properly born, at the start of the twenty-first century, in venues where VJs begin to play with video samplers.

The first serious attempts to dignify video mapping come in 2005, thanks to the Mapping International Festival in Geneva, Switzerland. This festival, specialising in the world of the VJ, is one of the reasons why video mapping began to be looked at as an art.

Yes, it is true that mapping as we know it today is born around then. But what happened before? How did we get here? Where are we going?

When our ancestors painted in a cave thousands of years ago, they did so making use of the shape and texture of the stone, folding into it what they wanted to set down, producing a work of art intimately bound to the place where it was made. The rock is no longer a rock: it is now a bison, and it is hard to believe the animal was not there before, because its muscles, its head and its whole physiognomy fit perfectly with the original shape of the stone. In some way it had always been there, waiting to be painted and to stop being rock, taking on in that way, thanks to the artist, a new identity.

From the days of the Roman empire, but especially during the Renaissance, the Baroque and the Rococo, trompe-l’œil paintings use perspective to create the illusion that the painting comes out of the place where it was painted, generating a three-dimensional space that exists only in the eyes of the beholder and extends beyond the limits of the architectural element it was painted on. Baroque quadratura is a clear example: it makes the beholder feel they are looking at a dome far larger than it really is, all through a pictorial technique based on geometric distortion.

I cannot fail to mention anamorphosis, a technique that uses mathematical procedures to generate an optical illusion visible only from one particular point. If we look from the right place we can see the image, which otherwise appears distorted before our eyes. A clear example of anamorphism is found in the popular toys that became so fashionable during the Romantic period. These magic mirrors consisted of a drawing that could only be seen properly when reflected in a mirror, usually cylindrical or conical in shape. Leonardo da Vinci, Hans Holbein and William Scrots are just a few of the first artists to have played with this technique through history, which reaches its height in the sixteenth and seventeenth centuries.

Nor can I forget the phantasmagoria illusions, born at the end of the eighteenth century, which consisted of sophisticated projections that, using at least one magic lantern, painted glass plates, play with sound and image, dissolves and moving pictures, could overwhelm the spectator in a sensory experience to the point of making them believe that what they were seeing was real.

I mention these examples not only as a matter of history, although it is that too. I include them because each one has at least one essential characteristic that has shaped what we now call video mapping.

But what is video mapping? Basically it consists of projecting images onto a given surface or space, most often three-dimensional, making use of its structural features to create new elements that interact with the object being projected onto, which takes on a new identity born of the mixture of the projection and the space in question. The projected images are usually accompanied by a soundtrack, and there may be various special effects — smoke, lasers and a long list besides — all in perfect synchrony.

Since it can be done in many ways and in any number of spaces, there are many technical processes that will or will not be used depending on what we are doing.

We can map onto great buildings or tiny shacks, onto animate or inanimate objects, models, sheets of paper, forests, mountains, an entire village, and also onto our body, our face or the passer-by crossing the street. Indoors or out, inside a theatre on the stage or in the stalls. The limit is set by our imagination and by the money we have to make the production.

The first step would be to capture data on the geometry of the “object” we are going to project onto. We can do it by taking photographs, by 3D scanning, by using the construction drawings if they exist, and so on. It all depends on the size, the characteristics of the surface and how complex the project is.

Once we have the data, we use it to create a map — hence the name — or template, so that it serves as a reference (for us or for the various computer algorithms) throughout the process of creating the images and fitting them to the space.

Next we decide the framing, the position and the various views of the virtual cameras (each of these cameras will later become a real projector), where the spectators will be, the distance of the projector; we study the blind angles and how to solve them, the power and the optics of each projector, the lighting conditions of the place, the ideal image resolution both for editing and for projection (there is no point using 4K if the projector only takes HD).

We also decide whether to use a single projection or several at once, whether we use a 3D graphics system or opt for 2D vectorisation, and as many other things as the kind of project and the narrative we choose demand of us.

When the time comes to edit and animate the video, there are any number of programmes to work with, some more specialised than others or with more 2D or 3D functions. Programmes for photo retouching, for special effects and post-production, for three-dimensional creation, for graphic design, and whatever else we need.

During the performance we will also need specific applications: sequencers, programmes to interconnect sensors if we want interactive mapping, audio–video synchronisation systems, and so on and so on.

And of course an application that lets us adjust the projection so that it fits precisely where we want it. This is what we call “warping”, and it consists of digitally manipulating the image to correct the distortion that inevitably occurs when projecting onto a three-dimensional surface, or when the projector is not perfectly centred. Imagine we lay a grid over the image and can control the position of each of the points formed where the vertical and horizontal lines meet. Imagine that when we move a point, the image deforms in order to sit exactly where we have put that point. This gives us control over the geometry of the projection and lets us fit it precisely to its surroundings. It is one of the commonest ways of doing it.

If we use several projectors, we will have to join the different projections cleanly, and we achieve this with a technique called “blending”, which consists of mixing both images, adjusting the projections and correcting the joins so that the “seams” do not show.

Now let me try to draw a clear distinction between what is mapping and what is not.

Mapping and projection are often confused. Not every projection is mapping. It is not a matter of placing an image somewhere so that it fits more or less well. It is a matter of using the geometry of the place and the projection together, so that both become part of a new object born of the fusion of the two. Mapping is a site-specific work and, as such, can only be done in the place it was created for. It would be impossible to do the same mapping in a different space, both by definition and for technical reasons. By space or place, understand the object that receives the projection.

There are as many kinds of mapping as we can imagine, and as many again still to be discovered. Here are a few:

Architectural mapping: done, as the name says, on architectural elements, mainly façades, and usually making use of emblematic places — bridges, houses, great buildings, castles

Land art mapping: done in nature or in parks, the landscape being the object projected onto.

Micro mapping: small in scale, especially compared with other kinds of mapping. Made on small installations, models, three-dimensional elements and even crumpled paper. It can cover the whole object or be confined to one part.

Indoor mapping.

Outdoor mapping.

Immersive mapping: mapping that covers 360 degrees in every direction, placing the spectator — or not — at the centre of the projection.

Real-time mapping:

Interactive mapping: through sensors, digital image processors that read information in real time, cameras, microphones, big data and so on, the projection reacts there and then according to the data reaching it. Another form of interactive mapping would be the one in which the VJ decides live what to put up, according to a set of variables determined beforehand, or even undetermined.

Mapping with object tracking: with the right tools we can make the projection follow a moving object.

Mapping with body tracking: a specialisation of the above, focused on the body, either covering it entirely or confining itself to a part, such as the face or a hand…

Generative mapping: the projection comes into being at the very moment it is projected. There are no pre-recorded videos. It can be done in many ways, using particle systems, vector systems and many others.

Any and every possible combination of those listed above, or still to be listed.

The possibilities mapping offers us are tremendous; it is currently used in any number of fields and for very varied purposes.

Advertisers have embraced the technique, sometimes to promote films, election campaigns or the latest model from a well-known car maker. The impact on the viewer is very great, and its sheer spectacle gets the event talked about for weeks. Just imagine a twelve-storey building and what can be done with it.

The field of the performing arts would deserve a chapter of its own, since the length of this article does not let me dwell on it as I would like. I will only tell you that it is currently being used not just as a substitute for scenic elements — which is in fact the least interesting part — but as a tool that lets us investigate new narrative forms; a tool that, together with the performers and dancers, plunges us into a complex world where everything really is possible and leads us to create new languages. This, I believe, is where we have to place the emphasis. We can recreate an entire set, it is true; we can even generate the floor, the walls, the ceiling, the objects too. We can create a space in which everything that appears is projected and controllable — but that on its own is no narrative advance. As always, it will depend on what we do with it, on the use we make of it.

Technology today lets us do more than map the stage space. We can also project onto the performers’ bodies while tracking their movements in real time. And not only that: we can create generative projections produced according to the movements of the performer or the audience. We free the performer from having to repeat exactly the same movement over and over to match the projection, and we offer them the chance to build, together — performer and projection — a new reality that changes at every moment, that never repeats itself, that acts and reacts. I personally dream of this every night, and now and then during the day as well.

What does the future hold? Because everything I have said so far belongs to the past… What is the road ahead? In what way will the discoveries and inventions to come shape our artistic thinking? They will change everything, and change it greatly; of that I am sure. In fact they are already changing us. We no longer perceive our reality the way we did in the twentieth century.

Quite possibly the most important thing is missing from everything I say: the thing that has not yet been imagined.

Mapping has brought an advance as important as the liberation of the screen: we no longer need a two-dimensional support to project onto, we can now adapt to the medium, and this offers us endless possibilities. As technology advances, the quality of projections improves exponentially. Resolution, contrast, the amount of light and the colour ranges now let us project onto coloured and even black elements, modifying the source pixel by pixel so that it can be seen in the best conditions, whatever the structure and formal characteristics of the object projected onto.

Imagine fusing these projection techniques with the generation of holograms; imagine those holograms reacting to touch… It is already being done. Imagine the possibilities of mapping combined with optical effects that alter our perception. Imagine, too, that we can turn the stage upside down in a fraction of a second and rotate it at will, deform it, change the planes, change the spectator’s point of view. Imagine mapping as a character as well, or as a tool that adds information to what is happening. Imagine a stage where the lanterns have given up their place to video projectors. Imagine how many new ways of telling, of building a discourse, of encoding information… Imagine, play… We have new toys. Let us not stop playing.