A theoretical analysis of the overall mean squared error (MSE) in hybrid vi
deo coding is presented for the case of error prone transmission. Our model
covers the complete transmission system including the rate-distortion perf
ormance of the,ideo encoder, forward error correction, interleaving, and th
e effect of error concealment and interframe error propagation at the video
decoder. The channel model used is a 2-state Markov model describing burst
errors on the symbol level. Reed-Solomon codes are used for forward error
correction. Extensive simulation results using an H.263 video codec are pro
vided for verification. Using the model, the optimal tradeoff between INTRA
and INTER coding as well as the optimal channel code rate can be determine
d for given channel parameters by minimizing the expected MSE at the decode
r. The main focus of this paper is to show the accuracy of the derived anal
ytical model and its applicability to the analysis and optimization of an e
ntire video transmission system.