his wife, his maid, the female emperor, her concubine(as a front) and the one lady he fought when he's transporting a prisoner to the other empire. I think the last one doesn't belong to the harem but I included her for some reasons. 3 children > 1 from his wife, 1 from his maid and 1 from the female emperor.
si li was just a front to cover up that the emperor is a female, and I kinda think there's a yuri vibe between them. both of them belongs to the harem, but can't be with mc physically because of politics.