Thanks for the tips, advice and opinions. It certainly did get worse later in the afternoon. I hit a bit of a "lull" when I returned and was able to time the swells and make it behind the breakwater.
The most nerve wracking part was that the wind was blowing almost directly on shore for me so I had a very narrow window of both space and time to get docked and tied up before I drifted too close to the rocky shore. I deemed getting in unscathed a success. The experience gained was priceless though and today the conditions are much much better.