Changing the block size has absolutely nothing to do with block time.

Smaller blocks doesn’t mean faster blocks times.
Bigger blocks doesn’t mean slower block times. 

 Right that's difficulty 

 Yes that’s what makes the adjustments to keep ~10 min blocks  as hash rate changes.