So, if I only use one cable for my polycon vvx3 and my pc, my pc internet and network speed will probably slow down because of the limited bandwidth of the ethernet connector in the phone?
Correct. That it why it is better to always have the cable-crew pull two or more 'home-runs' to every drop location. For the same reason, it also makes sense to install a two-gang box even if you don't have all four jacks terminated.
The current cost for drops is about $100-$150 per drop when you are talking about having a facility professionally cabled. There is room to negotiate how many jacks are made "hot" at each drop.
In similar fashion, it is better to have two half-filled switches that one fully filled switch, as moving to the second switch in the event of a failure takes seconds.