I personally would prefer the toggle to jumping the cursor to the close button, so if i chose to I could loop through my guys using 1-8 and clicking the button to see their inv/stats.
You can click down near the health and what not if you want, but at that point, you might as well just click 'close'.