We identify and discuss important design criteria in 80Gb/s DPSK-ASK and 120Gb/s DQPSK-ASK fiber communication systems by only using 40Gb/s equipment.For RZ-DPSK-ASK and RZ-DQPSK-ASK, the extinction ratio of the ASK signal is a tradeoff between good eye opening for the amplitude signal and good eye opening for the demodulated phase signal, so we fist find the best extinction ratio that can result in equal receiver sensitivities for both amplitude and phase modulated signals, hence resulting in the optimum performance for the system as a whole.After the investigation of the performance of 80Gb/s DPSK-ASK and 120Gb/s DQPSK-ASK modulated optical fiber communication system, we find the frequency offset between laser and MZI is still the most serious degradation factor in receiver, but due to the ASK receiver is insensitive to the frequency offset, the tolerance ability for 80Gb/s DPSK-ASK system is almost the same for 40Gb/s DPSK, even if the bit rate is doubled. For 120Gb/s DQPSK-ASK system, due to the bit rate is tripled, the frequency offset should be limited to 2.5%.We investigate the effect when multiple factors are present at the same time in 120Gb/s DQPSK-ASK fiber communication system. According to the simulation result, the degradation of the receiver is not the linear combine, it can lead to much worse degradation even the individual factor itself can only induce little, there is also a chance one factor can suppress another factor’s effect. This conclusion is also valid in 40Gb/s DPSK fiber communication system.