Fraser McKenzie had given Edinburgh the lead with a try but that was cancelled out by Ulster centre Ian Whitten's dart over the line.
The visitors led 10-8 at the break but Rob Moffat's side were a different outfit after the interval.
Two Ulster penalties from Paddy Wallace and Talei's try set up a nervy ending before David Blair's drop goal sealed the win.
Coach Moffat said: "It wasn't the prettiest game of rugby but we got a win against an Ulster side that is pretty confident and has a strong pack.
"You'd have thought the kind of night it was would have suited them a bit.
"We showed control at times but I was frustrated because our discipline let us down, giving away silly penalties.
"All it does is let them control the pace of the game."