This is by design, AFAIU, but I know that many people cannot work well with it.
To mitigate the issue for yourself you can:
- use the 'Objects' dialog for selecting the object (Object > Objects)
- hold down the Alt key when the object is selected, then use the mouse to move it (Alt allows you to click anywhere, and then drag, and still move only the selected object)