The topic of this thesis is related to testing of very large scale integration circuits. The thesis presents the idea of optimizing mixed-mode built-in self-test (BIST) scheme. Mixed-mode BIST consists of two phases. The first phase is pseudo-random testing or partial pseudo-exhaustive testing (P-PET). For the faults not detected by the first phase, deterministic test patterns are generated and applied in the second phase. Hence, the defect coverage of the first phase influences the number of patterns to be generated and stored. The advantages of P-PET in comparison with usual pseudo-random test are in obtaining higher fault coverage and reducing the number of deterministic patterns in the second phase of mixed-mode BIST. Test pattern generation for P-PET is achieved by selecting characteristic polynomials of multiple-polynomial linear feedback shift register (MP-LFSR). In this thesis, the mixed-mode BIST scheme with P-PET in the first phase is further improved in terms of the fault coverage of the first phase. This is achieved by optimization of polynomial selection of P-PET. In usual mixed-mode BIST, the set of undetected by the first phase faults is handled in the second phase by generating deterministic test patterns for them. The method in the thesis is based on consideration of these patterns during polynomial selection. In other words, we are embedding deterministic test patterns in P-PET. In order to solve the problem, the algorithm for the selection of characteristic polynomials covering the pre-generated patterns is developed. The advantages of the proposed approach in terms of the defect coverage and the number of faults left after the first phase are presented using contemporary industrial circuits. A comparison with usual pseudo-random testing is also performed. The results prove the benefits of P-PET with embedded test patterns in terms of the fault coverage, while maintaining comparable test length and time.