Read/Write/Rewrite
An interactive installation that visualizes how a machine can learn to ‘read’ and ‘write’

READ/WRITE/REWRITE was an interactive installation exhibited at Typojanchi 2017 in Seoul, South Korea, that visualizes how a machine can learn to ‘read and write’ by using machine learning applied to natural language in the form of written text. 

Ongoing research in machine learning was transformed into a tangible interactive installation able to reconfigure itself through different contexts and contents.

​With ​this ​project ​we ​explored ​how ​principles ​of ​machine ​learning ​could ​be ​applied ​from ​the ​perspective ​of ​the ​cultural field, ​the ​perspective ​of ​artists ​and ​designers. ReadWriteRewrite is an interactive installation that visualises how a machine can learn to ‘read and write’ by using machine learning applied to natural language in the form of written text. We reworked our ongoing research in machine learning into a tangible interactive installation able to reconfigure itself through different context and content.  

Visualization of a machine learning process of categorizing 3 million english words

Computational ​models ​for ​natural ​language ​primarily ​regard language ​as ​a ​sequence ​of ​symbols, ​as ​such ​the ​meaning ​of ​words ​can ​only ​be ​described ​as ​a product ​of ​word ​context, ​that ​is, ​to ​which ​other ​words ​a ​given ​word ​appears. ​This ​is ​in ​contrast ​to Chomsky’s ​linguistic ​theory, ​which ​holds ​that ​the ​principles ​underlying ​the ​structure ​of ​language are ​biologically ​determined ​in ​the ​human ​mind ​and ​hence ​genetically ​transmitted. The ​key ​difference ​between ​the ​perception ​of ​language ​by ​machine ​and ​by ​man ​lies ​in ​how language ​is ​embodied; ​For ​machines, ​language ​is ​a ​process ​of ​computational ​operations ​on symbols ​stored ​in ​computer ​memory, ​for ​man, ​language ​is ​an ​experience ​of ​the ​body. ​Here ​is where ​the ​origin ​of Read/Write/Rewrite ​lies: ​

If ​both ​machine ​and ​man ​read ​and ​write ​the ​same word, ​do ​they ​mean ​the ​same ​thing?

The ​work ​visualises  ​how ​machine ​learning ​algorithms ​organize ​roughly 3 ​million ​english words ​on ​the ​basis ​of ​their ​meaning ​and ​semantics. ​The ​end ​result ​of ​this ​process ​is ​a landscape ​of ​words ​in ​which ​words ​with ​a ​similar ​context ​are ​placed ​in ​near ​proximity. ​The ​word contexts ​are ​learned ​from ​a ​large ​body ​of ​digital ​texts ​of ​that ​publicly ​available ​such ​as ​news ​and Wikipedia ​articles. 

What is the meaning of word? We start by thinking that natural language is a product from the human and its environment, for man every word can be associated to sensory experiences. A word is a complex association of actions, memories and so on.

Since most software runs on computers, often without any sensors, natural language is reduced to written words only. A word is just a series of symbols. Thus the meaning of a word can only be explained through other words.

Turning words into space
A word embedding is a numerical representation for a set of words in which each word is a point in a high dimensional space. Such an embedding can be constructed using a neural network and a large body of texts. The neural network is trained to guess the blinded center word in a short text fragment by repeatedly feeding it fragments from example texts. When the neural network fails to guess the blinded word correctly it updates the representation of all the words in the fragment such that it is more likely to successfully guess the word. By applying the guess- and-update steps a great many times all the word-points in the space organise into an embedding that contains semantic qualities.

A listing of words that the neural network produced, that are similar to the word ‘typography’. We see a list of words related to typography, but also words that are in fields related to typography

Once a word embedding is found we can query the model to find words that are close to a given word. 

Examples of learned relations between words

A quality of word embedding is the ability to somehow learn word analogies. For example we can find a direction by taking the line between the words Italy and Rome. Instead from Italy we now leave from France and we end up at the word Paris. So the language model has implicitly learned a land to capital relationship, just from reading a lot of texts. The language model is surprisingly good at some relations, like directions, gender-based word forms and the stereotypical food for a given country. We also see that it is somewhat limited in certain other things.

We use the Word2vec model, a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space. Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.

Since Word2Vec constructs such high-dimensional spaces, and we can’t really look beyond 3 dimensions we have to find a way to fit all that information into 2 dimensions. In order to “flatten” this high-dimensional space into two dimensions we used the t-SNE algorithm. 

Documentation of the installation

The installation
The installation takes the shape of a four sided cube. The various visual perspectives make it possible to explore word similarities and interrelations that are encoded in the language model in different manners. The examples in this section are simplified illustrations of core visualisation principles applied to each side of the cube.

The​ ​installation​ ​aimed​ ​to​ ​make​ ​abstract​ ​concepts​ ​as​ ​machine​ ​learning​ ​and​ ​neural​ ​networks accessible​ ​and​ ​understandable​ ​for​ ​a​ ​wider​ ​audience,​ ​meanwhile​ ​addressing​ ​themes​ ​as​ ​design, future​ ​of​ ​typography,​ ​context​ ​vs​ ​meaning,​ ​etc..​ ​Machine​ ​learning​ ​principles​ ​will​ ​be​ ​applied​ ​more and​ ​more,​ ​and​ ​it’s​ ​important​ ​to​ ​give​ ​insight​ ​into​ ​how​ ​these​ ​principles​ ​work,​ ​to​ ​give​ ​a​ ​counter​ ​voice to​ ​‘empty’​ ​terms​ ​as​ ​Artificial​ ​Intelligence.​ ​By​ ​addressing​ ​this,​ ​the project aims​ ​to​ ​load​ ​these​ ​themes​ ​with cultural​ ​significance.​ ​

Three sides of the cube structure and their interaction

Interaction
The​ ​interaction​ ​of​ ​the​ ​work​ ​is​ ​based​ ​on​ ​visitor​ ​proximity.​ ​When​ ​nobody​ ​is​ ​near​ ​the​ ​work​ ​it​ ​will​ ​show and​ ​”work​” ​on​ ​the​ ​organisation​ ​of​ ​the​ ​word​ ​data.​ ​In​ ​this​ ​mode,​ ​words​ ​are​ ​visualised​ ​as​ ​anonymous data​ ​points.​ ​As​ ​a​ ​visitor​ ​approaches​ ​the​ ​cube​ ​the​ ​work​ ​will​ ​switch​ ​to​ ​a​ ​language​ ​appropriated​ ​to man.​ ​By​ ​moving​ ​closer​ ​to​ ​the​ ​cube​ ​the​ ​visitor​ ​zooms​ ​in​ ​on​ ​the​ ​language​ ​model.​ ​Through​ ​three visual​ ​perspectives​ ​it​ ​is​ ​possible​ ​to​ ​explore​ ​word​ ​similarities​ ​and​ ​interrelations​ ​that​ ​are​ ​encoded​ ​in the​ ​language​ ​model.

A more technical article we wrote about the machine learning part can be found on Medium. 

About Typojanchi
Typojanchi​ ​2017​ –​ ​the​ ​5th​ ​International​ ​Typography​ ​Biennale​ was​ ​a​ ​combination​ ​of​ ​exhibitions,​ ​talks and​ ​publications​ ​with​ ​contributions​ ​from​ ​local​ ​and​ ​international​ ​artists,​ ​designers​ ​and typographers.​ ​ ​The​ ​theme​ was​ ​​Body and​ ​Typography​.​ ​In​ ​comparison​ ​to​ ​most​ ​art​ ​biennials​ ​and​ ​art​ ​fairs​ ​Typojanchi​ ​is​ ​less​ ​self-regarding: it​ ​tries​ ​to​ ​interpret​ ​the​ ​current​ ​era​ ​and​ ​its​ ​socio-cultural​ ​environments.​ ​All​ ​different​ ​components create​ ​a​ ​vast​ ​range​ ​of​ ​intersections​ ​between​ ​visual​ ​languages​ ​and​ ​perspectives​ ​-​ ​literature,​ ​music, movie,​ ​city,​ ​politics​ ​and​ ​economy.

Date
2017
Location
Seoul, South Korea
Design and Concept
Typojanchi Director
Ahn Byunghak
Curators
An Hyoijn,Kim Namoo
Media/Technical support
Multi Tech
Construction, Build up
Gom Design
Translator
Jeong Daye
Photography
Kim Jin Sol
Video
Jeong Moon Ki
Software
Oracle
Visual design system and cover generator for 150.000 TU Delft publications
Read more
OPENRNDR
Open source framework for creative coding
Read more
Air Pollution in Europe
Interactive installation at the Clean Air Forum
Read more
Growth maps of Amsterdam
150 meters long wall graphics for a new underwater bike parking
Read more
Panartis conference
Identity for an online conference on authentication in art
Read more
RoBoGo
Fully customizable 3D printed media facade elements
Read more
Load more

Studio

RNDR is a design studio for interactive media that develops ‘tools’ that are only finished by how they are used.

To achieve this, we develop processes, create structures, design visualisations, code programs, and create interactions. The end result can manifest itself across different media, ranging from interactive installations, data visualisations, generative identities, prints and everything in between – often real-time. We are triggered by how information and technology transforms networks, cultures, societies, relationships, behaviours, and interactions between people. Our work explores and engages with hybrid space as it embraces both digital and physical realms.

RNDR was founded in 2017 in The Hague, (NL). Its main members have years of experience as partners, computer scientists, designers, art directors and developers at LUST and LUSTlab.

One of our core projects, and basis for most of our projects, is OPENRNDR, an open source framework for creative coding –written in Kotlin for the JVM– with over 13 years of development. OPENRNDR simplifies writing real-time audio-visual interactive software. OPENRNDR is fundamental for the capacity of RNDR as a studio, as it allows us to realize complex interactive works. OPENRNDR was awarded the Dutch Design Award 2019

Fields of work
Interactive design (ui/ux), data visualisation, information systems, software tools, interactive installations, media architecture, immersive experiences, interface design, visual identities, generative video, creative coding, exhibition design, graphic design systems, hybrid spaces and platforms, machine learning and artificial intelligence (ai), code and design workshops.

People
Jeroen Barendse (NL), partner. Design and art direction. Former partner of LUST and LUSTlab. Awarded BNO Piet Zwart Oeuvre Prize 2017
Edwin Jakobs (NL), partner. Computer scientist, creative coder and visual artist. Creator of OPENRNDR
Boyd Rotgans (NL), partner. Creative coder & interaction designer
Viola Bernacchi (IT). Design & data visualisation
Els van Dijk (NL), office manager
Marco Dell'Abate (IT), Royal Academy for the Arts, The Hague, Creative coder, 2023
Nike Kuschick (DE), Munich University of Applied Sciences, Interaction Designer, intern, 2024
Previously at RNDR
Thomas McElmeel (US), College for Creative Studies, Detroit, intern, 2023
Marco Dell'Abate (IT), Royal Academy for the Arts, The Hague, intern, 2021
Rein van der Woerd (NL), Data Art Technology Arnhem, Artez, intern, 2021
Derrek Chow (CA), University of Waterloo, Waterloo, intern, 2021
Jon Packles (USA), Parsons School of Design | The New School, intern and freelance, 2021
Jasper Kamphuis (NL), Willem de Kooning Academy, Rotterdam, intern, 2020
Ferdinand Sorg (DE), Hochschule für Gestaltung Schwäbisch Gmünd, intern , 2020
Jaekook Han (KR), Graduate Interactive Telecommunications Program, NYC, intern
Gábor Kerekes (HU), developer
Amir Houieh (IR), developer
Łukasz Gula (PL), Royal Academy of Art, The Hague, intern
Noemi Biro (RO), Willem de Kooning Academy, Rotterdam, intern
Selection of exhibitions
Oracle unfolded @Highlight Delft Festival, 2023
Perpetual Beta; Encounters in open space, The Grey Space, The Hague, 2021
Open Highway, Raum, Utrecht, 2018
Typojanchi Typography Biennale Seoul, main exhibitor (as LUST), 2017
Cartographies of Rest, interactive installation at Mile End Art Pavilion in London (as LUST), 2016
Hyperlocator, What’s next, Future Tomorrow, 38CC Delft (as LUST), 2016
Type/Dynamics exhibition, Stedelijk Museum Amsterdam (as LUST), 2014
Selection of lectures
Creative Coding With Kotlin and OPENRNDR, online presentation for Jetbrains, 2023
Cloud Salon presentation at Parsons Design & Technology, New York
Experimental Data Visualization for Master of Arts in Interaction Design, SUPSI, Mendrisio, 2022
Lecture Information Design masters. Design Academy Eindhoven, 2021
Talk for Graphic Days Turin, Eyes on the Netherlands (IT), 2021
Lecture for TAAALKS conference, Munich, 2020
Presentation at GitHub Satellite, 2020
Lecture at Britisch Library for Central Saint Martins' Form, Reform, Perform: futures of writing, 2019
MiXit conference, Lyon, 2019
Google Span conference, presentation and demo, Helsinki, 2018
JFuture conference, Minsk, 2018
Creative Coding Utrecht, Launch of OPENRNDR, 2018
Selection of workshops
Workshop OPENRNDR for Graphic Days, Turin (IT), 2021
OPENRDNR workshop for Digital Society School, Amsterdam, 2020
Weeklong workshop OPENRNDR @Politecnico di Milano, 2020
OPENRNDR x Machine Learning workshop @ KABK during Tech Week, 2020
Two week OPENRNDR workshop at Tumo, Armenia, 2019
Workshop OPENRNDR at Artez Interaction Design, Arnhem, 2019
Processing Community Day 2019, CCU and Sensorlab, Utrecht, 2019
Royal Academy of Art (KABK), Techweek, The Hague, 2019
Utrecht School for the Arts (HKU), OPENRNDR workshop, 2018
La Scuola Open Source, 1-week workshop, Bari, Italy, 2018
Two-week workshop at TUMO foundation, Yerevan, Armenia, 2019
OPENRNDR x Machine learning workshop @SUPSI, Mendrisio, CH, 2022
Teaching
Information Design, Master program Geo Design, Design Academy Eindhoven, 2023
Creative coding, Master Non-linear Narrative, KABK, The Hague, 2023
Interaction Design, Design Art Technology Arnhem, Artez, 2008
Generative Design, Bachelor Man and Communication, Design Academy Eindhoven, 2019

Contact

RNDR

Paviljoensgracht 20
2512 BP, The Hague
+31 (0)70.3635776

info@rndr.studio


INTERNSHIPS

We are mostly looking for interns that have a background in interactive design. Some experience in coding is preferred. 

interns@rndr.studio

OPENRNDR
Open source framework for creative coding that simplifies writing real-time interactive software

info@openrndr.org
openrndr.org