My opinion is simple: if you are out on the boat by yourself then you need to be wearing the jacket, whistle, etc., and the strobe or some kind of light if you are out at night alone. If you are out on a boat with someone who can stop the boat and help you get reattached then I would say leave it up to you.
I hate wearing a life jacket, but I promised my wife that I would never get into the Finn without it and so far I haven't broken my promise.