The purpose of this study was to develop a system to utilize the successful experiences and help the beginners of garment pattern design (GPD) through optimization methods. Size design of fit garments is a basic and difficult problem in GPD for inexperienced designers. We have proposed a hybrid system (NN-ICEA) based on neural network (NN) and immune co-evolutionary algorithm (ICEA) to predict the fit of the garments and search optimal sizes. ICEA takes NN as fitness function and procedures including clonal proliferation, hypermutation and co-evolution search the optimal size values. A series of experiments with a dataset of 450 pieces of pants was conducted to demonstrate the prediction and optimization capabilities of NN-ICEA. In the comparative studies, NN-ICEA was compared with NN-genetic algorithm to show the value of immune-inspired operators. Four types of GPD method have been summarized and compared. Moreover, the hybrid system for general features of garment has been discussed. The research is a feasible and effective attempt aiming at a valuable problem and provides key algorithms for fit prediction and size optimization. The algorithms can be incorporated into garment computer-aided design system.