In a few sentences, and with his characteristic penetrating trenchance, Chastek demolishes the Turing Test, and for that matter all arguments from similarity of causal effects; I post here without apology his entire argument, on account of its brevity, precision, and devastation:
One principle of (strong/sci-fi) AI seems to be that what can replicate the effects of intelligence is intelligence, e.g. the Turing test, or the present claim by some philosophers that a Chinese room would be intelligence.
So imagine you rig up a track and trolley to accelerate at 9.8 m/s2. This perfectly replicates the effects of falling, and so is artificial falling. It deserves the name too: you could strap a helmet to the front of your train and drive at a wall 10 feet away, and it will tell you what the helmet would look like if dropped from 10 feet. But for all that the helmet at the front of your train is obviously being pushed and not falling – falling is something bodies do by themselves and being pushed isn’t. The difference is relevant to AI, for just as falling is to being pushed so thinking for oneself is to being a tool, instrument or machine. Both latter are acted on by others, and have the form by which they act in a transient way and not as a principal agent.
Arguments from similarity of causal effect are all species of affirmations of the consequent; a basic and stupid logical fallacy. The fallacious character of the Turing Test should be apparent to every student of logic. That Turing – himself no slouch at logic, forsooth – proposed it in the first place attests to the absurdity of the physicalist – or, at least, the tendentiously agnostic – proposal that his other metaphysical commitments could not excuse.
The same devastating critique lands upon Utilitarianism, and other consequentialist deontologies. To think that a thing is good because it happens to (seem to) work out well is to put the cart before the horse. Indeed, it is to mistake the cart for the horse.
Post Scriptum: a sodden, sad and anxious thought: is not empiricism per se a sort of consequentialism, and thus epistemologically unreliable? An experiment, however well or badly designed, turns out with a certain result. Sure, OK. Induction and all that, I get it. But is it not the case that *reasoning from results is per se affirmation of the logical consequent, and thus logically fallacious?* I’m sure this topic has been exhaustively explored by the specialists. But I’m not a specialist in the topic, so …