Tuesday, October 26, 2021
  More from the point of inflection

The April 76 issue of Computers and Automation includes a POWERFUL article by Joseph Weizenbaum, the author of ELIZA. He got everything right, and saw the dismal consequences.

Weizenbaum had originally wanted to demonstrate the limits of computers. He wanted to show people that computers were just machines. He was truly shocked to find that ELIZA had the opposite effect....
DOCTOR, as ELIZA playing psychiatrist came to be known, soon became famous around MIT mainly because it was an easy program to demonstrate. Most other programs could not vividly demonstrate the information-processing power of a computer to visitors who did not already have some specialized knowledge, say, of some branch of mathematics. DOCTOR, on the other hand, could be appreciated on some level by anyone. Its power as a demonstration vehicle was further enhanced by the fact that the visitor could actually participate in its operation.
Participation and the use of language were certainly key differences. In the '60s ordinary students punched a pile of cards and submitted them to the Computing Center, which then returned a printout several days later. No connection with the process. Nearly all instruction in computers and programming started with number theory, the most abstract and least useful part of math. Prime numbers are mental masturbation for math freaks.
The shocks I experienced as DOCTOR became widely known and "played" were due principally to three distinct events.

1. A number of practicing psychiatrists seriously believed the DOCTOR program could grow into a nearly completely automatic form of psychotherapy. I had thought it essential, as a prerequisite to the very possibility that one person might help another learn to cope with his emotional problems, that the helper himself participate in the other's experience of those problems and, in large part by way of his own empathic recognition of them, himself come to understand them. What must a psychiatrist who makes such a suggestion think he is doing while treating a patient, that he can view the simplest mechanical parody of a single interviewing technique as having captured anything of the essence of a human encounter?
This point isn't surprising. Freudians knew they weren't doing anything real. Eysenck had disproved the human value of the process, but the practice continued because it was lucrative.
2. I was startled to see how quickly and how deeply people conversing with DOCTOR became emotionally involved with the computer and how unequivocally they anthropomorphized it. Once my secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room.
We think computers are private. In 1964 this was more or less true, though ELIZA obviously had a logfile generator. Otherwise the transactions couldn't have been written up in articles. The secretary had been editing those logs, so she knew it wasn't private. The same illusion continues. The vast majority of iPhone users worry about "Russian hackers", and go along with all the "security" needed to block "Russian hackers". The same iPhone users don't notice that Siri is listening all the time. If Siri WASN'T listening all the time, she wouldn't be able to jump in and answer your questions when you say her name.
3. Another widespread, and to me surprising, reaction to the ELIZA program was the spread of a belief that it demonstrated a general solution to the problem of computer understanding of natural language. In my paper, I had tried to say that no general solution to that problem was possible, i.e., that language is understood only in contextual frameworks, that even these can be shared by people to only a limited extent, and that consequently even people are not embodiments of any such general solution. But these conclusions were often ignored.
Deeply correct, and AI still proves it all the time. NSA and Google have been working hard on this problem for 40 years, and it's still not solved.

Here's the strongest and most prophetic part.
...the question of whether or not human thought is entirely computable. That question has, in one form or another, engaged thinkers in all ages. Man has always striven for principles that could organize and give sense and meaning to his existence. But before modern science fathered the technologies that reified and concretized its otherwise abstract systems, the systems of thought that defined man's place in the universe were fundamentally juridicial. They served to define man's obligations to his fellow men and to nature. The Judaic tradition, for example, rests on the idea of a contractual relationship between God and man. This relationship must and does leave room for autonomy for both God and man, for a contract is an agreement willingly entered into by parties who are free not to agree. Man's autonomy and his corresponding responsibility is a central issue of all religious systems.
See the loss of two-way obligations.
The spiritual cosmologies engendered by modern science, on the other hand, are infected with the germ of LOGICAL NECESSITY. They no longer content themselves with explanations of appearances, but claim to say how things actually are and must necessarily be. In short, they convert truth to provability.

Surely, much of what we today regard as good and useful, as well as much of what we would call knowledge and wisdom, we owe to science.

But science may also be seen as an addictive drug.

Not only has our unbounded feeding on science caused us to become dependent on it, but, as happens with many other drugs taken in increasing dosages, science has been gradually converted into a slow-acting poison.

Beginning perhaps with Francis Bacon's misreading of the genuine promise of science, man has been seduced into wishing and working for the establishment of an age of rationality, but with his vision of rationality tragically twisted so as to equate it with logicality. Thus have we very nearly come to the point where almost every genuine human dilemma is seen as a mere paradox, as a merely apparent contradiction that could be untangled by judicious applications of cold logic derived from a higher standpoint.

Even murderous wars have come to be perceived as mere problems to be solved by hordes of professional problemsolvers.

As Hannah Arendt said about recent makers and executors of policy in the Pentagon:

"They were not just intelligent, but prided themselves on being 'rational'. They were eager to find formulas, preferably expressed in a pseudo-mathematical language, that would unify the most disparate phenomena with which reality presented them; that is, they were eager to discover laws by which to explain and predict political and historical facts as though they were as necessary, and thus as reliable, as the physicists once believed natural phenomena to be. An utterly irrational confidence in the calculability of reality became the leitmotif of the decision making."
And now we're back in the territory of Conelrad and CDC. Predictive models control everything, and the psychopaths decide the variables for the predictive models. Science has passed beyond slow-acting poison. It's now a full-fledged HOLOCAUST, obliterating life and soul and universe FAST.

The demons are simply hiding behind the computer. Most people STILL don't understand the mechanical nature of computing. Most STILL believe that computers think rationally. The latest New Superstitionist STILL says computers will achieve true thought with more power and more speed. (Translation: More grants.)

Only programmers know how powerful the programmer is.

In '76 these trends were just starting to show up, and only a few prophets saw them. At that point we could have tamed the trends, could have applied the brakes with some strong negative feedback mechanisms keyed on empathy. But we didn't. Nobody listened.

= = = = =

Irrelevant personal sidenote: I've never humanized a computer, perhaps because as a programmer I know it's just a machine. I do humanize my house to some extent, and fully humanize the air conditioner. This odd choice started from a charming Japanese ad for Hitachi ACs. The word Kura stuck in my head, and I started calling the AC Kura. When Kura fights through an especially hard day, I pat it and praise it. When I pull Kura out of the window in October, I lay it down on a soft towel and cover it with a soft towel, and give it a sendoff to its annual long vacation. Six months of rest and dreams, with no exposure to sun and wind and rain and snow.

Labels: ,


<< Home

blogger hit counter
My Photo
Location: Spokane

The current icon shows Polistra using a Personal Equation Machine.

My graphics products:

Free stuff at ShareCG

And some leftovers here.

March 2005 / April 2005 / May 2005 / June 2005 / July 2005 / August 2005 / September 2005 / October 2005 / November 2005 / December 2005 / January 2006 / February 2006 / March 2006 / April 2006 / May 2006 / June 2006 / July 2006 / August 2006 / September 2006 / October 2006 / November 2006 / December 2006 / January 2007 / February 2007 / March 2007 / April 2007 / May 2007 / June 2007 / July 2007 / August 2007 / September 2007 / October 2007 / November 2007 / December 2007 / January 2008 / February 2008 / March 2008 / April 2008 / May 2008 / June 2008 / July 2008 / August 2008 / September 2008 / October 2008 / November 2008 / December 2008 / January 2009 / February 2009 / March 2009 / April 2009 / May 2009 / June 2009 / July 2009 / August 2009 / September 2009 / October 2009 / November 2009 / December 2009 / January 2010 / February 2010 / March 2010 / April 2010 / May 2010 / June 2010 / July 2010 / August 2010 / September 2010 / October 2010 / November 2010 / December 2010 / January 2011 / February 2011 / March 2011 / April 2011 / May 2011 / June 2011 / July 2011 / August 2011 / September 2011 / October 2011 / November 2011 / December 2011 / January 2012 / February 2012 / March 2012 / April 2012 / May 2012 / June 2012 / July 2012 / August 2012 / September 2012 / October 2012 / November 2012 / December 2012 / January 2013 / February 2013 / March 2013 / April 2013 / May 2013 / June 2013 / July 2013 / August 2013 / September 2013 / October 2013 / November 2013 / December 2013 / January 2014 / February 2014 / March 2014 / April 2014 / May 2014 / June 2014 / July 2014 / August 2014 / September 2014 / October 2014 / November 2014 / December 2014 / January 2015 / February 2015 / March 2015 / April 2015 / May 2015 / June 2015 / July 2015 / August 2015 / September 2015 / October 2015 / November 2015 / December 2015 / January 2016 / February 2016 / March 2016 / April 2016 / May 2016 / June 2016 / July 2016 / August 2016 / September 2016 / October 2016 / November 2016 / December 2016 / January 2017 / February 2017 / March 2017 / April 2017 / May 2017 / June 2017 / July 2017 / August 2017 / September 2017 / October 2017 / November 2017 / December 2017 / January 2018 / February 2018 / March 2018 / April 2018 / May 2018 / June 2018 / July 2018 / August 2018 / September 2018 / October 2018 / November 2018 / December 2018 / January 2019 / February 2019 / March 2019 / April 2019 / May 2019 / June 2019 / July 2019 / August 2019 / September 2019 / October 2019 / November 2019 / December 2019 / January 2020 / February 2020 / March 2020 / April 2020 / May 2020 / June 2020 / July 2020 / August 2020 / September 2020 / October 2020 / November 2020 / December 2020 / January 2021 / February 2021 / March 2021 / April 2021 / May 2021 / June 2021 / July 2021 / August 2021 / September 2021 / October 2021 / November 2021 /

Major tags or subjects:

2000 = 1000
Carbon Cult
Constants and variables
Defensible Cases
Defensible Times
Defensible Spaces
Experiential education
From rights to duties
Grand Blueprint
Natural law = Sharia law
Natural law = Soviet law
Shared Lie
Trinity House

Powered by Blogger