One of the trickiest aspects of evangelism is explaining to unbelievers who have been harmed by people they've met in various churches that they shouldn't assume that church folks are necessarily genuine followers of Christ. That's not to say that true Christians don't ever sin against other people, but that isn't always the explanation for such things.
A lot of folks consider themselves to be Christians just because they "grew up in the church".
In some cases, no one ever even bothered to present the salvation message to them, because they grew up believing that anyone who's been baptized as an infant is automatically a Christian (which is just one of the reasons I oppose the practice of infant baptism, although I know that the folks who do that usually mean well).
We even have pastors and other spiritual "leaders" who have never repented of their sins or accepted Christ as lord and savior! Maybe they went to Bible school or seminary just to please their parents. Practically every church-related school can tell stories about certain students who were known for partying and for blatantly disobeying God when their parents weren't around.
That's one reason I am disinclined to treat pastors as if they are somehow infallible on account of their job titles. When a pastor or someone else from a particular church blatantly claims that church leaders are beyond criticism (as I've seen at multiple churches here in the Chicago area), I take that as my cue to continue my search for a church led by a truly godly person.
Some would call me a "church hopper" on account of that fact, but I'd rather be a church hopper than a mindless syncophant who has abdicated my biblical responsibility to test the spirits.
A position of church authority is a stewardship, not a blank check for a pastor to do whatever the pastor wishes to do. The Bible says (Luke 12:42-48) that people who abuse positions of church authority will be cut in half and assigned a place with the unbelievers!
People who fail to speak out against abusive pastors are enablers, as surely as they would be if they failed to speak out against other types of sin. They will be held accountable for their silence on Judgment Day. So frankly, I don't care if someone chooses to slander me on account of the person's inability to recognize a prophetic voice. What God thinks about me on that day is the only thing which ultimately matters.