r - ggplot2 changes estimate of smoothed conditional mean when I change axis limits

I'm trying to smooth points from a scatterplot. There are two groups, and I want to do the smoothing on each of them separately.

In order to illustrate the problem that is happening, I generate data according to

set.seed(400)
x=seq(0,2,0.01)
Data=data.frame(x=rep(x,2),y=c(rnorm(5*x,0,0.05),rnorm(5*x+0.5,0,0.05)),z=c(rep("1",length(x)),rep("2",length(x))))

Here, z represents the groups.

If I plot the smoothed curves using

ggplot(Data, aes(x,y)) + geom_smooth(aes(color=z,linetype=z),method="gam",size=2,se=FALSE)+scale_y_continuous(limits=c(-0.01,0.01))

I get

However, if I change the y axis limits, I get a different result:

ggplot(Data, aes(x,y)) + geom_smooth(aes(color=z,linetype=z),method="gam",size=2,se=FALSE)+scale_y_continuous(limits=c(-0.005,0.005))

In other words: the only thing I've changed are the y-axis limits, however I get different regression estimates.

Any ideas of why this happens and how I can correct it?

Thanks!