these spells increase your faction with whatever faction the npc is on that you cast it on. depending on your faction, it might be a noticeable gain, or not. And if you agro anything from that faction, it negates the alliance buff you casted as if it never happened, back to square one.
Collaboration is marginally useful, sort of. I cast it on the animals in Caverns, for example, and they are no longer KOS. Until I wrecked my faction in Kael, I was able to bump myself up high enough to where they would talk to me and give quests and such. Before I did The Missing scroll quest, I'd illusion DE and do collaboration and run around non-KOS everywhere in N4th. It's not something that's used very often, but it's one more tool in the toolbox.
Original question not answered. Does the spell ever cause agro (like when resisted)? Also does it work on merchants?
No; Yes. Questions answered. The spell cannot be resisted. If the mob can't be allianced you will just get a "will not take hold" message.
Either intentional blocking by developers (such as in PoWar because they don't want you cheating the faction) or sometimes some mobs just can't be cast on period. Both of those are pretty rare and I doubt you'll run into many mobs that you can't Alliance/Benev.
Usually the typical "here's a spell, now good luck using it" mentality from developers. FWIW, I run into a fair number of mobs that throw the message at you, but there's usually some mob floating around nearby on the same faction that can be cast upon, so not typically a big deal.