San Luis pass isn't the monster people make it into, but you do have to realize there is ALWAYS rip currents there, and they are powerful when the tides are at work or the wind is pushing the bay water. The bottom countours change constantly. If you swim very well, don't panic, and are familiar with swimming in currents and can tread water for a long time if you get in trouble, you are ok, but if you don't want to worry about it, in the pass itself, you want to have a PFD. It isn't that you get sucked under or anything so much as the current can really be moving and might carry you out of the pass a quarter mile or so before it puts you back on the beach. A weak swimmer usually can't deal with something like that.
Once you're outside or well inside the pass itself, you don't need to be quite so cautious.