A double post, and copying something on rental... not a great start, but Hi and welcome anyway...
Are you sure it's not converting to another format, like VCD or SVCD?
Try the tutorials here - DVD Deccrypter (for ripping) and DVDshrink (for fitting to single layer) - tools that do what YOU tell them to! - and in a two-step process, it's easier to figure out where it wen't wrong.
PS. This is general advice, as while we support "fair use" in making a backup of what you've purchased, copying a rental is NOT fair use unless you're only doing it for testing.