r - Remove legend ggplot 2.2

I'm trying to keep the legend of one layer (smooth) and remove the legend of the other (point). I have tried shutting off the legends with guides(colour = FALSE) and geom_point(aes(color = vs), show.legend = FALSE).

Edit: As this question and its answers are popular, a reproducible example seems in order:

library(ggplot2)
ggplot(data = mtcars, aes(x = mpg, y = disp, group = gear)) +
geom_point(aes(color = vs)) +
geom_point(aes(shape = factor(cyl))) +
geom_line(aes(linetype = factor(gear))) +
geom_smooth(aes(fill = factor(gear), color = gear)) + 
theme_bw()