In conversation
How Machines Learn to See Our World The Humans Behind Machine Learning
Bruno Moreschi is a Brazilian visual artist, filmmaker, and currently an early-career fellow at the Collegium Helveticum. In his fellowship project, with the support of Felix Stalder from ZHdK as an associate fellow, Bruno investigates human practices linked to artificial intelligence (AI), which we are not paying due attention to. The question of how we are teaching machines to “see” the world guides his project. During this conversation, Bruno talks about how his fascination for the topic has evolved, how the project benefits from his time at the Collegium and a workshop he has recently organized.
The workshop is titled “Working the Image.” What can participants expect from it?
Gabriel Pereira from the University of Amsterdam helped me conceive this workshop. He is an assistant professor interested in AI and digital culture. Together, we will guide people through a couple of exercises that help us better understand and critically discuss the association between what we see, the metadata in pictures, how tagging processes work, and how the collaboration of humans and machines could expand the informational power of images. In a second part of the workshop, we will have two speakers, who will share their experiences as crowdworkers.
“AI is often described as automatic and autonomous—but it crucially depends on work done by humans.”
What is a crowdworker?
AI is often described as automatic and autonomous—but it crucially depends on work done by humans. A considerable part behind the idea of “automation” is only made possible by people describing and tagging visual material. All these small tasks are performed by crowdworkers, also called turkers (see here why). Via digital platforms such as, for example, Amazon Mechanical Turk, these people will complete tasks for a very low wage. The tagged images become part of large datasets, which are used to train what we call “computer vision.” Things like facial recognition, social media content moderation, or the automated car all rely on these workers.
How did you become aware of this? And how did you get in touch with the turkers?
I became interested in the topic during an artistic residency at the Dutch Van Abbemuseum. Together with Gabriel Pereira, I analyzed how big-tech tools read the museum’s conceptual artworks. Many misinterpretations occurred and we wanted to understand certain dominant ideologies within AI. At some point, we wondered who the humans training these tools were. When we realized that they stay completely anonymous, with virtually zero chance to contact each other, we came up with the idea to establish a digital platform called Exch w/ Turkers. In this online space, not only can turkers communicate with each other, it is also an interface to connect with the public. We published an open call on Amazon Mechanical Turk and hired five crowdworkers to work for 30 minutes a day, answering questions asked by the participating public. We have collected the best parts from all the conversations and collated them in a printed publication, which will be launched during the workshop in January. The project began in the middle of the pandemic and social distancing. I was therefore genuinely interested in the experiences and know-how of these workers, at that time when when all of us were suddenly in work-situations that was already their reality. One could even say that, as more and more labor moves online and involves us feeding information into machines, they are prototypes for the future of work.
Who are the turkers? Who are some of people behind this kind of work?
The majority of the workers are located in the United States, as most of the services needed are in English. But there are an increasing number of workers from the Global South—for example, Indians, Brazilians, Venezuelans, Argentines, or Kenyans. And everyone has their own story. Anand from India, for example, needs to get up at the crack of dawn, since those services that pay best take place during US business hours. Brianne from the US works on a pink computer, as she has loved this color since childhood. A Brazilian woman, Sonia, uses part of her earnings to order Baby Alive dolls for her daughter and Lego for her son. If you had worked there, you would also have your own story. These crowdworkers are no different from me and you. This may seem obvious, but this aspect gets camouflaged when we just repeat terms like “artificial intelligence” or “machine learning.” We’re not talking about intelligent machines, we’re talking about machines with intelligent humans.
What insights did you gain from the exchange with the crowdworkers?
Perhaps the main insight is realizing the importance of paying attention to the first stages of machine learning. And that there are people involved in this and lives affected. I have been working with engineers and programmers in the Group on Artificial Intelligence and Art at the University of São Paulo for some years. Based on my research, some of them also established an exchange with crowdworkers and thus learning how many of these services are poorly designed and paid. I am convinced that we will only have more “accuracy”—a term that engineers love—if we improve the situations of humans in this machine learning process.
How are you building up on these insight in your current fellowship project?
As a final result of my fellowship I will produce an experimental feature film called Acapulco. It will consist of images that train machines, but which we humans almost never see. The name of the film comes from a story a crowdworker told me. For one week, his task was to identify which beach appeared in a picture. There were always three alternatives and—probably because of a bug in the system, every time he chose Acapulco—he got it right and earned more money. Neither of us had ever been to Acapulco before, but he told me that he really liked it there. What interests me most about art is its possibilities for prototyping, testing, and confabulation. The core of my project at the Collegium is to speculate, test, and practice new ways of using images to train computer vision. And in this workshops like the one I’ve organized, are essential to try new ideas—without the obligation to get it right.
Why did you choose to continue your research at the Collegium?
As someone who deals with the arts, but also with technology, data, and human relations, I benefit a lot from the interdisciplinary environment. Working in the same building with people from different backgrounds means that exchanges can happen at any time and in both less and more formal ways. This can immensely enrich any research. At the Collegium, people have the time and focus to carry out something specific. In my case it’s a film. But it could also be a book, an exhibition, or any other type of event. At the same time, it is also an opportunity to not only deliver results, but also to have the privilege of formulating the next steps in your academic or artistic journey. The Collegium provides and encourages intellectual independence, which is very important. Everyone here trusts in your abilities and will do everything they can to support you. This is rare to find—not unlike finding a rare star. On that note, I also find it incredibly poetic and inspiring to work in a place where people once observed the sky and the stars.
Discover more
