Language as tactile user interface

The ease with which we’re able to summon and dismiss texts on glowing rectangles makes us forget that language isn’t weightless. The ways in which we call out and respond to each other are deeply anchored within physical things. Heavy things. We make meaning by spilling oceans of ink, crushing mountains of herbs and minerals into pigments, and by sliding slabs of quivering muscle against each other.

And even when we summon an idea from the internet that idea is bound to this plane by physical objects. Language exists within at least three dimensions.

So if communication can shape mass (indeed, if language is mass), what will new forms of communication mean for the things we build, and the way we build? Can we incorporate content into spaces in ways that go beyond merely turning walls into display screens? How does this communication influence our relationships with our tools, and with ourselves?

As companies such as Google, Nest and Jawbone experiment with networked devices, there is an opportunity to take advantage of the materiality of these objects as communication interfaces. Colour, texture, weight and temperature are just some of the ways in which our devices can speak to us.

And what about how we speak to them? Bret Victor’s A Brief Rant on the Future of Interaction Design is a jeremiad against the idea that the primary user interface of The Future will be variously-sized pieces of glass. Victor rails against this vision — he calls it “pictures under glass” — because it robs us of haptic feedback:

Go ahead and pick up a book. Open it up to some page. Notice how you know where you are in the book by the distribution of weight in each hand, and the thickness of the page stacks between your fingers. Turn a page, and notice how you would know if you grabbed two pages together, by how they would slip apart when you rub them against each other. Go ahead and pick up a glass of water. Take a sip. Notice how you know how much water is left, by how the weight shifts in response to you tipping it. Almost every object in the world offers this sort of feedback. It’s so taken for granted that we’re usually not even aware of it. Take a moment to pick up the objects around you. Use them as you normally would, and sense their tactile response — their texture, pliability, temperature; their distribution of weight; their edges, curves, and ridges; how they respond in your hand as you use them.

Victor thinks “pictures under glass” are a transitional technology. Possibly, the coming wave of networked devices is what we’ll transition to — tools that are once again an instinctive extension of ourselves, and that recognize mass as speech.

This was originally posted on the Mass + Text blog, which is a scrapbook of ideas about the relationship between language, physical objects, and the communities they anchor.

(If you enjoy my writing and want to support my personal research projects, the best way is to buy me a book!)