---
title: Thinking machines
slug: thinking-machines
description: "The computers will start thinking, and people will stop."
cover: "/2025-04-24--computers-will-start-thinking.jpg"
---

There's an exchange early in the classic '80s movie [TRON](https://www.themoviedb.org/movie/97-tron). Some scientists are talking shop:

![Still from TRON, left to right Linda, Alan, and Gibbs, in conversation.](/2025-04-24--computers-will-start-thinking.jpg)

> **ALAN:** I tell ya, ever since he got that Master Control Program, system's got more bugs than a bait store.
>
> **GIBBS:** Ehh, you gotta expect some static. After all, computers are just machines, they can't think...
>
> **ALAN:** Programs will be thinking _soon_.
>
> **GIBBS:** (wryly) Hahaha, won't that be grand -- *computers and the programs will start thinking, and the people will stop!*

Gibbs has a point. The modern vision of a utopian future is one where work is relieved, and people are free to pursue leisure, or exercise their creativity with art, writing, and poetry. Thinking computers are here now, in the form of "large language models" (LLMs) like ChatGPT. Setting aside the irony that creative works are the first and most visible applications of LLM technology -- is that imagined future actually a good one?

## Mom is always right

When I was a kid, I remember a day going to yard sales with my mom in the family minivan. It was early summer, a hot day. The windows were down, and I complained that if the vehicle has good air conditioning, we should use it. What was the point in getting all hot? "To get used to the warm weather," came the answer. What an injustice! We were sweating back there! Later in life, I took a short trip to Arizona in August. Everyone scurried from building to building. Where the sun was doubled, reflected off of glass skyscrapers, the temperature jump was alarming. It was actually unsafe to spend long stretches outside unprepared. But when I returned to Massachusetts, for the rest of the summer 85 or 90 degrees Fahrenheit felt like nothing.

All that to say, the work that LLM technology offers to relieve isn't just about achieving a result. The effort maintains and builds our abilities. Work pushes us to connect to each other for help, or to persevere in doing something difficult. Outsourcing that work eventually means losing the ability to do it yourself.

## Attention must be paid

Simply put, an LLM is a document completion engine. You give it text, and it extends it. The result doesn't have to be true, it just has to be convincing. No amount of pre-training or guard rails will make it truthful. It does often say true things, but that's not the point, it's more of a happy accident.

Because they are built from essentially the whole public internet, LLMs also have a strong connection to The Algorithm. Algorithms that run social media feeds and online advertising are designed to attract human attention, a precious thing. Social media algorithms and LLMs are oriented towards capturing that attention. The foundational LLM paper is even called, "Attention is all you need". A prescient title. LLM intelligence is not like ours. It can't know what it's like to be a human.

If this was a person, someone who wanted your attention and had this kind of indifference towards truth, they would be considered a con-man or bullshitter. !![On Bullshit](https://archive.org/details/on-bullshit-by-harry-frankfurt), _Harry Frankfurt_!! Untrustworthy.

## Don't create the torment nexus

LLMs clearly manifest a type of intelligence. Sure, it's "just" some linear algebra and a ton of data. But it does exhibit a type of intelligence. One without empathy. !![AI chatbot pushed teen to kill himself, lawsuit alleges](https://apnews.com/article/chatbot-ai-lawsuit-suicide-teen-artificial-intelligence-9d48adc572100822fdbc3c90d1456bd0), _AP News_
[Belgian man dies by suicide following exchanges with chatbot](https://www.brusselstimes.com/430098/belgian-man-commits-suicide-following-exchanges-with-chatgpt), _Brussels Times_
[AI-Fueled Spiritual Delusions Are Destroying Human Relationships](https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/), _Rolling Stone_!! Not being human, it *can't* have empathy---and intelligence without empathy can be dangerous.

Science fiction is littered with cautionary tales about inhuman intelligence. For that matter, so is myth: genies give people whatever they want, but because people have self-destructive desires (like the desire to avoid work), it goes wrong. In TRON, Infocom has the MCP (Master Control Program), an overgrown chess program that is given access to whatever information it can consume, until its intelligence and capabilities are seemingly endless. The company leadership comes to rely on the program so completely that it becomes their entire interface for understanding and operating the business. There is also the irony that Infocom's success was built on the misuse of intellectual property, much as LLM companies have done !![AI, Copyright, and the Law: The Ongoing Battle Over Intellectual Property Rights ](https://sites.usc.edu/iptls/2025/02/04/ai-copyright-and-the-law-the-ongoing-battle-over-intellectual-property-rights/), _IP & Technology Law Society_
[Generative AI Has an Intellectual Property Problem](https://hbr.org/2023/04/generative-ai-has-an-intellectual-property-problem), _Harvard Business Review_!!.

I don't think I am wise enough to safely use a genie in a bottle. And I don't want to outsource my creative efforts to an addictive, bullshitting alien intellect, even if it might save time and effort in the short term.

![End of line](/2025-04-24--end-of-line.jpg)
