If you leave a fixed rate early, the lender may charge a break cost. It applies when the lender's cost of funding has fallen since you fixed, because they lose interest they expected to earn.
A rough guide is the difference between your rate and today's wholesale rate for the remaining term, multiplied by the balance and the time left. If wholesale rates have risen since you fixed, the break cost is often nothing.
Always ask your lender for a written break-cost quote on the day you plan to switch. It changes daily.