It's mostly that he's a gladiator, he's not leaving the arena for the most part.

Lan Xichen, Lan Wangji, Lan Sizhui, and probably Wen Kexing, even Wen Qing could likely point towards where he is outside of the arena, offhand, or handwave he spoke about having the head in some fashion, then Wei Wuxian could tell him how they could meet