For full-size components, we just make color_buf[ci] point at the input buffer, and thus avoid copying any data. Note that this is safe only because sep_upsample doesn't declare the input row group "consumed" until we are done color converting and emitting it.