Brightlingsea took their decisive lead after 27 minutes from the spot, Jake Turner despatching the penalty.
Swifts had a chance to equalise with a penalty of their own eight minutes from time but the effort was fired wide of the goal.
It got tougher for Swifts a minute later as they were reduced to ten-men when Liam Thomas was dismissed for a second bookable offence.
Ryman League
Follow us: @rymanleaguenews on Twitter | Ryman League group on Facebook
Keep up-to-date with our exclusive email newsletters.
Subscribe