Google has the unlucky task of trying to police the legions of hackers, spammers and bad-hats trying to game (trick) them into granting high rankings.
Google can and does penalize websites for violations.
It also can (and does) blacklist sites so that they are completely removed from their index - meaning they won't show up in searches. Period.
How do you know if you are suffering a penalty or have been blacklisted?
The short answer is, you probably don't.
Unless you read about it on the front page of the New York Times (as happened to several sites in the last year, including J.C. Penney's website).
Nevertheless, there is a direction to go if you suspect there is a problem.
Here's a new article - an interview with one of the Google staff that work in the area. Google isn't always very talkative about things. This provides more information on the subject than we've ever had before:
New Clarity on Reconsideration Requests