1981, Boreas River (ADK) I was in prime physical condition, having wrestled, boxed, and been a competitive runner, solid swimming skills.
I was in a AppleLine kayak, and had a reliable roll.
Don’t ask me how, but I dislocated my right shoulder.
Regardless of how fit I was, how close the shore was, how mild the water was, if not for my PFD, I would have much more trouble than I had with the dislocation. I could not swim, nor recover my equipment without aid.
So not wearing your PFD is obviously a choice, but it’s cheap insurance for an unexpected event.