They were the ones doing the attacking.

*but fine, fiiiine, he'll put the sword away. It's not like he can't take on an ordinary human--and probably a couple of small monsters or whatever they are too--without a weapon anyway. His expression and stance don't relax, though; he clearly wouldn't be surprised if he were attacked at any moment*

Then you're claiming you had nothing to do with bringing me here?