You can restrict who has remote access by simply unchecking a box.

Actually right now, they could potentially access your computer the same way you do with the external IP.