In a few sentences, and with his characteristic penetrating trenchance, Chastek demolishes the Turing Test, and for that matter all arguments from similarity of causal effects; I post here without apology his entire argument, on account of its brevity, precision, and devastation:
One principle of (strong/sci-fi) AI seems to be that what can replicate the effects of intelligence is intelligence, e.g. the Turing test, or the present claim by some philosophers that a Chinese room would be intelligence.
So imagine you rig up a track and trolley to accelerate at 9.8 m/s2. This perfectly replicates the effects of falling, and so is artificial falling. It deserves the name too: you could strap a helmet to the front of your train and drive at a wall 10 feet away, and it will tell you what the helmet would look like if dropped from 10 feet. But for all that the helmet at the front of your train is obviously being pushed and not falling – falling is something bodies do by themselves and being pushed isn’t. The difference is relevant to AI, for just as falling is to being pushed so thinking for oneself is to being a tool, instrument or machine. Both latter are acted on by others, and have the form by which they act in a transient way and not as a principal agent.