Stay away.
I think you should do just the opposite. I think you should get every friend you have that has a boat and have them bring other friends that have a boat and all meet at Timber Island for a great big giant floating block party. These people are trying to bully you out of what you are legally entitled to do.
Except for the people on the tire swing, it sounds like the rest of you were just enjoying the day. If you have young children just explain to them that the people on the shore are unhappy and their using bad words. The kids won't think anything of it.
If you all meet up you'll have a blast. Maybe it could be the "First Annual Timber Island Floating Block Party". A kinda end of the summer party.
Besides it will give the shore residents something to think about all winter.
Give it some thought