MPEG2, but at lower quality - just like MP3 can be at different bitrates.
I really don't understand the details of how they do it without fully re-encoding, but the method is faster, but does lose out to a full re-encode on quality at deeper shrink levels.
Instant Copy apparently uses a slower but better method