More from the point of inflection
The April 76 issue of Computers and Automation includes a POWERFUL article by Joseph Weizenbaum, the author of ELIZA. He got everything right, and saw the dismal consequences.

Weizenbaum had originally wanted to demonstrate the limits of computers. He wanted to show people that computers were just machines. He was truly shocked to find that ELIZA had the opposite effect....

DOCTOR, as ELIZA playing psychiatrist came to be known, soon became famous around MIT mainly because it was an easy program to demonstrate. Most other programs could not vividly demonstrate the information-processing power of a computer to visitors who did not already have some specialized knowledge, say, of some branch of mathematics. DOCTOR, on the other hand, could be appreciated on some level by anyone. Its power as a demonstration vehicle was further enhanced by the fact that the visitor could actually participate in its operation.

Participation and the use of language were certainly key differences. In the '60s ordinary students punched a pile of cards and submitted them to the Computing Center, which then returned a printout several days later. No connection with the process. Nearly all instruction in computers and programming started with number theory, the most abstract and least useful part of math. Prime numbers are mental masturbation for math freaks.

The shocks I experienced as DOCTOR became widely known and "played" were due principally to three distinct events.

1. A number of practicing psychiatrists seriously believed the DOCTOR program could grow into a nearly completely automatic form of psychotherapy. I had thought it essential, as a prerequisite to the very possibility that one person might help another learn to cope with his emotional problems, that the helper himself participate in the other's experience of those problems and, in large part by way of his own empathic recognition of them, himself come to understand them. What must a psychiatrist who makes such a suggestion think he is doing while treating a patient, that he can view the simplest mechanical parody of a single interviewing technique as having captured anything of the essence of a human encounter?

This point isn't surprising. Freudians knew they weren't doing anything real. Eysenck had disproved the human value of the process, but the practice continued because it was lucrative.

2. I was startled to see how quickly and how deeply people conversing with DOCTOR became emotionally involved with the computer and how unequivocally they anthropomorphized it. Once my secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room.

We think computers are private. In 1964 this was more or less true, though ELIZA obviously had a logfile generator. Otherwise the transactions couldn't have been written up in articles. The secretary had been editing those logs, so she knew it wasn't private. The same illusion continues. The vast majority of iPhone users worry about "Russian hackers", and go along with all the "security" needed to block "Russian hackers". The same iPhone users don't notice that Siri is listening all the time. If Siri WASN'T listening all the time, she wouldn't be able to jump in and answer your questions when you say her name.

3. Another widespread, and to me surprising, reaction to the ELIZA program was the spread of a belief that it demonstrated a general solution to the problem of computer understanding of natural language. In my paper, I had tried to say that no general solution to that problem was possible, i.e., that language is understood only in contextual frameworks, that even these can be shared by people to only a limited extent, and that consequently even people are not embodiments of any such general solution. But these conclusions were often ignored.

Deeply correct, and AI still proves it all the time. NSA and Google have been working hard on this problem for 40 years, and it's still not solved.

Here's the strongest and most prophetic part.

...the question of whether or not human thought is entirely computable. That question has, in one form or another, engaged thinkers in all ages. Man has always striven for principles that could organize and give sense and meaning to his existence. But before modern science fathered the technologies that reified and concretized its otherwise abstract systems, the systems of thought that defined man's place in the universe were fundamentally juridicial. They served to define man's obligations to his fellow men and to nature. The Judaic tradition, for example, rests on the idea of a contractual relationship between God and man. This relationship must and does leave room for autonomy for both God and man, for a contract is an agreement willingly entered into by parties who are free not to agree. Man's autonomy and his corresponding responsibility is a central issue of all religious systems.

See the loss of two-way obligations.

The spiritual cosmologies engendered by modern science, on the other hand, are infected with the germ of LOGICAL NECESSITY. They no longer content themselves with explanations of appearances, but claim to say how things actually are and must necessarily be. In short, they convert truth to provability.

Surely, much of what we today regard as good and useful, as well as much of what we would call knowledge and wisdom, we owe to science.

But science may also be seen as an addictive drug.

Not only has our unbounded feeding on science caused us to become dependent on it, but, as happens with many other drugs taken in increasing dosages, science has been gradually converted into a slow-acting poison.

Beginning perhaps with Francis Bacon's misreading of the genuine promise of science, man has been seduced into wishing and working for the establishment of an age of rationality, but with his vision of rationality tragically twisted so as to equate it with logicality. Thus have we very nearly come to the point where almost every genuine human dilemma is seen as a mere paradox, as a merely apparent contradiction that could be untangled by judicious applications of cold logic derived from a higher standpoint.

Even murderous wars have come to be perceived as mere problems to be solved by hordes of professional problemsolvers.

As Hannah Arendt said about recent makers and executors of policy in the Pentagon:

"They were not just intelligent, but prided themselves on being 'rational'. They were eager to find formulas, preferably expressed in a pseudo-mathematical language, that would unify the most disparate phenomena with which reality presented them; that is, they were eager to discover laws by which to explain and predict political and historical facts as though they were as necessary, and thus as reliable, as the physicists once believed natural phenomena to be. An utterly irrational confidence in the calculability of reality became the leitmotif of the decision making."

And now we're back in the territory of Conelrad and CDC. Predictive models control everything, and the psychopaths decide the variables for the predictive models. Science has passed beyond slow-acting poison. It's now a full-fledged HOLOCAUST, obliterating life and soul and universe FAST.

The demons are simply hiding behind the computer. Most people STILL don't understand the mechanical nature of computing. Most STILL believe that computers think rationally. The latest New Superstitionist STILL says computers will achieve true thought with more power and more speed. (Translation: More grants.)

Only programmers know how powerful the programmer is.

In '76 these trends were just starting to show up, and only a few prophets saw them. At that point we could have tamed the trends, could have applied the brakes with some strong negative feedback mechanisms keyed on empathy. But we didn't. Nobody listened.


= = = = =

Irrelevant personal sidenote: I've never humanized a computer, perhaps because as a programmer I know it's just a machine. I do humanize my house to some extent, and fully humanize the air conditioner. This odd choice started from a charming Japanese ad for Hitachi ACs. The word Kura stuck in my head, and I started calling the AC Kura. When Kura fights through an especially hard day, I pat it and praise it. When I pull Kura out of the window in October, I lay it down on a soft towel and cover it with a soft towel, and give it a sendoff to its annual long vacation. Six months of rest and dreams, with no exposure to sun and wind and rain and snow.

Labels: endless hell, TMI

  ¶ 6:18 AM