So most electrical devices are grounded, meaning that all the non-energized metal of the device is connected to the grounding system, through the third prong of the plug. So if there is a failure in the device, which brings voltage to the housing or a case of the device, it will immediately flow through the grounding system, and presumably trip the circuit breaker, protecting the user from shock.
Here's the issue. Not all are grounded to the same potential. Like I mentioned earlier, in some devices, the chassis ground is not the same as the Earth ground. In some devices which are highly susceptible to interference, the chassis ground is kept at a different potential. That's where the term floating ground comes from. It's kept above the potential of the Earth ground. And if you have a devices with different grounds, that leads to interference.
An example is a stereo system that "hums". This is a result of the chassis ground and the Earth ground being at a different potential. Or components having different ground reference points. It's also referred to as a ground loop. Assuming you had plugged such a device into your UPS and then pulled the power, the results would not be pretty if you were to unplug the UPS to test it.
Stereo humming is one of the reasons that certain devices have a chassis ground ("fixed" ground) that is above the Earth ground to prevent issues from things like ground loops. There's a good chance in the medical profession that you encounter such devices which need protection from this type of interference.
Similarly, a device running strictly on a battery backup (UPS) would have no connection to the earth, so grounding would not be necessary.
That is correct, but the issue is not after it's on battery power or even before. The issue is the transition. During that instant you unplug the UPS is where the problem is. The ground was expected to be there by the UPS so that current can leave via the ground wire during the switch. Once you remove that path to the ground, that could leave through you or one of your devices. This is why you hear random stories of people's UPS blowing up when they do the unplugging test.