Please tell me is this a betrayal?

He is being very childish. If nothing happened between you and your colleague, then no it is not betrayal. He should think nothing of it. But just him saying he watched because you betrayed him, sounds like something a mentally disturbed 13 y/o would say. He needs to grow up a bit.