Here is the law in question. It indicates snorkelers must have a flag but doesn't say swimmers can't.
http://www.boat-ed.com/nh/course/p4-...requipment.htm
Can you tell us hoytglp if there were swimmers in the water when you saw this? If there were no swimmers then using the flag is just plain wrong