I did the Castleview Hamlet: Cloister of Purity zone once before and seem to remember having to do something else rather than attack non-aggro elves. ZAM says that after destroying the posters they become aggro throughout the zone but they aren't now.
