The best way to describe the difference between primary and secondary weapons in-game is "Does it fit the holster?" but also "Is it one handed?" With the exception of the SMG (and micro-shotgun, I think?), all the secondary weapons are one-handed use. It would make reasonable sense if the SMG (or both) were usable in a single hand, as secondary weapons allow you to carry something in your soldier's other hand, like a grenade or stun-rod or medikit, while still permitting your soldier to defend or attack.
Secondary weapons are just that - secondary. They make good weapons at the start of the game, but they fall into second place the moment armor appears on your opponents. A 7.62mm pistol just does not have the stopping power of, for example, an assault rifle or flechette loaded riot shotgun, although it does get more shots per TUs than either of the mentioned weapons. It is still possible to design a strategy around their use for extensive combat use, though there will likely be better weapons at this point that renders "pistol" weapons obsolete.
I am not sure how it is in other countries' armed forces, but American parlance refers to secondary weapons as a "side-arms"; they are worn at the side in a holster and used when the primary weapon (usually a rifle) runs out of ammunition or is otherwise disabled. In general, a side-arm(secondary) weapon is a pistol-type weapon that can be wielded with a single hand - this is by design: in case a fighter breaks or loses their arm, they can still defend themselves.
In UFO: AI, the most logical use of a secondary weapon is for ammo shortages (such as your stores running out of clips for your primaries and for whatever reason it goes unnoticed), rocket launchers, and laser pistols (they are very accurate and pack quite a punch when you first get them!). In personal experience, I use rocket launchers as one-shot-drop weapons - they're too inaccurate for my enjoyment and the reload time is, appropriately, expensive. Grenade launchers are more accurate, though sometimes impractical and my soldier switches to his pistol for close-quarters fighting. Once I get laser rifles, my guys generally don't carry sidearms anymore - the holster space gets used for grenades - gas or plasma, depending on how many prisoners I've got. Or just for more DF-cartridges - it never hurts to have spare ammo!
