It sounds like you are doing everything correctly, unless I misunderstand. Here is a demo showing editing 1 audio track without editing the other tracks and it seems like that is exactly what your are doing.
I just tried it using the latest version (20211231) and it worked for me. I disarmed all tracks except for one audio track and could freely make adjustments on that track without affecting the other one.
Re-reading your post, are you trying to select and remove one audio segment? If so, when you highlight the segment to remove, you will highlight everything above and below that segment, but you will only remove the segment on the armed track.