Consciousness is what permits intelligent responses to unpredictable and unknown situations. A truly safe and functioning self-driving car would have to be conscious. At the moment computers regard a car filled with water, suspended 20 feet in the air on the end of two prongs of an earth moving machine as “parked.” It can’t tell the difference between a statue and a person. Snow and other weather conditions that obscure the sides of the road mean it grinds to a halt.
George Dyson, the science historian, son of Freeman Dyson, in an interview with Sam Harris pointed out, in response to Harris’ concern that if AGI is ever created it needs to be “controlled,” that, by definition, AGI will not be controllable. It will have a mind of its own. If we ever did develop an intelligence capable of safely driving cars in the actually variable conditions encountered in real life, i.e., outside the bland environment of southern California, there is no reason for thinking it would be our slave and drive our cars for us. More likely, we would be its slaves and drive cars for it.