Recently, outfit compatibility modeling, which aims to evaluate the compatibility of a given outfit that comprises a set of fashion items, has gained growing research attention. Although existing studies have achieved prominent progress, most of them overlook the essential global outfit representation learning, and the hidden complementary factors behind the outfit compatibility uncovering. Towards this end, we propose an Outfit Compatibility Modeling scheme via Complementary Factorization, termed as OCM-CF. In particular, OCM-CF consists of two key components: context-aware outfit representation modeling and hidden complementary factors modeling. The former works on adaptively learning the global outfit representation with graph convolutional networks and the multi-head attention mechanism, where the item context is fully explored. The latter targets at uncovering the latent complementary factors with multiple parallel networks, each of which corresponds to a factor-oriented context-aware outfit representation modeling. In this part, a new orthogonality-based complementarity regularization is proposed to encourage the learned factors to complement each other and better characterize the outfit compatibility. Finally, the outfit compatibility is obtained by summing all the hidden complementary factor-oriented outfit compatibility scores, each of which is derived from the corresponding outfit representation. Extensive experiments on two real-world datasets demonstrate the superiority of our OCM-CF over the state-of-the-art methods.