The husband shared his wife with a colleague