This is not true at all. What hurts your site is building links from poor (spam) sites. It's actually good to balance your backlink profile with follow and nofollow links. If you are building only (or at most) follow links, that sends a signal to Google on your optimization motives. Remember, over optimization should be avoided. It's impossible to manipulate Google with that. There's no way. No way. You'd get hit.