“Incest” in this case refers to sexual relationships between close friends in a group.

The word “incestuous” is often used to describe when people in friend groups end up sleeping with (or even just dating, I guess) each other within the group which will almost always lead to complications.