Bert,
As I get older and wiser-ha!, it seems I call the specialists less, and this may also be due to the technology at my fingertips (up to date, medscape epocrates). However, generally am able to get the specialist on the phone within minutes, unless they are in a procedure, and usually call back within the day.
Also with the immediacy of dragon creating notes/updox and efaxing, phone calling is becoming less important for the transmittal of info.
But, like you if a physician calls, my nurse pulls me out of the room, I try to respect my colleagues time, and my patients never complain when I excuse myself (at least to my face).