In this paper, multilevel coded asymmetric modulation with multistage decoding and unequal error protection (UEP) is discussed. These results further emphasize the fact that unconventional signal set partitionings are more promising than traditional (Ungerboeck-type) partitionings, to achieve UEP capabilities with multilevel coding and multistage decoding. Three types of unconventional partitionings are analyzed for asymmetric 8-PSK and 16-QAM constellations over the additive white Gaussian noise channel to introduce design guidelines. Generalizations to other PSK and QAM type constellations follow the same lines. Upper bounds on the bit-error probability based on union bound arguments are first derived. In some cases, these bounds become loose due to the large overlappings of decision regions associated with asymmetric constellations and unconventional partitionings. To overcome this problem, simpler and tighter approximated bounds are derived. Based on these bounds, it is shown that additional refinements can be achieved in the construction of multilevel UEP codes, by introducing asymmetries in PSK and QAM signal constellations.
Motohiko Isaka, Marc P. C. Fossorier, Robert H. Morelos-Zaragoza, Shu Lin, and Hideki Imai. "Multilevel Coded Modulation for Unequal Error Protection and Multistage Decoding—Part II: Asymmetric Constellations" Faculty Publications (2000): 204-213. doi:10.1109/26.823553