There's more "ego downvotes" than "mistake downvotes"; usually the ego ones offer no comments or a competing answers for the reason of the downvote. They can stay anonymous but they're obviously not working for the community. Personally, I upvote everything I know is right, even though I'm not learning anything new from that answer or question.
You can have a look at the Data, you'll notice some users simply prefer to spend more energy downvoting than voting up/down in a balanced way.
If these people are very active and you don't have a lot of answers posted yet, yes you might notice a pattern of downvotes, especially when you decide to become an active member; there's a little "tribe" attitude at the start as anywhere else, you might get judge more harshly. But as you'll post, this ratio should get smaller and the impact on your reputation will also be less visible. And the OP you helped will recognize this and usually be thankful for this. So forget about the "tribe" mentality and keep going!
I know you said you don't really care about the reputation but it still a bit frustrating when you're new or start being more active to get downvoted for no apparent reasons and as you said, you can't even improve your answer or see what's wrong with it. Or expand your answer to explain stuff you know very well but others don't.
As you'll see some users simply ignore the basics of Stack Exchange.
It's not very complicated and it shouldn't be about "liking" or "enjoying" an answers (or a user), but only about the answer being right or wrong.
When should I vote up?
Whenever you encounter a question, answer or comment that you feel is
especially useful, vote it up!
When should I vote down?
Use your downvotes whenever you encounter an egregiously sloppy,
no-effort-expended post, or an answer that is clearly and perhaps
On a personal note:
I got downvoted on hardcore prepress principles and when I post something about it, I even got a bit anxious and started to expect some resistance from people who have never worked in a print place even though they like calling themselves prepress experts because they output PDF with an overprint trapping. It felt like I shattered their illusion of perfection sometimes; they often don't know what's going on backstage at the print shop and probably have no clue prepress specialists sometimes roll their eyes when they see their files.... and just fix them because it's faster than educating some designers; when there's 4 press operators waiting for the plates and their team is worth $700/hr, you don't have time to wait for the designer's revised files. That's the reality in print places. As much as designers laugh at a rasterized images in Illustrator or Word layouts, prepress people laugh at the clumsy dielines, improvised trapping, hex rich black, missing fonts, psd or ai files as final, and especially the color profile designers send them. So now you all know the truth (haha!)
It's unfortunate because the answers such as "ask your printer" are not very useful to someone working on a project at 1am that needs to be sent to print at 6am, while I consider my "printer's answers" are actually helpful if one needs to make a decision without the printer's input or when they're sent oversea ;)
It's alright, at some point some prepress people will upvote my answers because they are right and it's going to be the same for you as I think you are very experienced in photography (I'm not!)