I know this is a pretty big idea and implementation may be impossible.
But I had a thought.....
You need 3k Rep to cast close votes. So, if you're casting those votes, you've got some rep.
What if each vote to close cost a user 100 Rep (or some amount to be determined). When you cast a close vote 100 rep is deducted from your account.
If the question actually does get closed by a mod or by other vote accumulation you get your rep refunded. Otherwise, you've lost the rep.
This may cause users to think twice about what they vote to close.
Mods would be exempt.... (and possibly dup-hammer users but only for duplicates, non-dup close votes would still cost dup-hammer users.)
I realize it's of-the-wall and probably an issue with the core structure of SE on the whole. But I was curious if anyone else here though this, or something similar, would offer any value.
Currently, as far as I'm aware, the only way to "spend" rep is via bounties. So adding additional rep-spending items doesn't seem horrible to me.
The primary hurdle I foresee would be poor questions not getting enough close votes and less user-moderation where closing is concerned.
This is really just a thought. I don't honestly expect it to be implemented.