One could say that "4k" textures just means highly detailed at this point. The bigger it is, the more detail it can hold. And with that, technically larger textures do affect performance. Textures aren't utilized straight from RAM/VRAM. They are cached (in texture cache), and the GPU utilizes them from there. Cache is much more limited in size, so it can only hold so many textures at any given time, and textures get swapped in and out. The bigger the texture, the less space there is for other textures to be cached, and more swapping will be required. As
@oldpuck mentioned, mipmaps also get utilized. If the scene doesn't necessarily need the most detailed of the texture levels at a point of rendering, it can cache a lower-quality version, thereby saving space.
I think I got that right, but corrections are appreciated.