R: converting some levels of a factor to numeric

I am new to R. I have a factor object with >10000 levels. Most of these levels are numbers which I would like to convert to numeric, but there are several levels that contain descriptions which I do not wish to touch.

Is there a good way for me to isolate the relevant levels of the factor to convert into numeric? Would as.numeric() be a good choice?

Many thanks!