QuEnc only supports YV12 colourspace. Have you tested you clip? Perhaps there is an error? Also you could try NuEnc, it has different rate control and supports upto 10 passes. Although if you are using QuEnc as part of DVDRebuilder or DVD2SVCD then I guess NuEnc won't work.