When a mortgage is refinanced with a different lender or paid off, the balance of any points being amortized becomes fully deductible in the year the loan is retired.     

IRS Pub 936 (2021), Page 8, indicates that if a mortgage is refinanced with the same lender the remaining unamortized balance of the points is not currently deductible. Instead, the remaining balance must be amortized over the life of the new loan. This position by the IRS is based upon the fact that a taxpayer must have unrestricted control over the funds in order to deduct the interest, which is very rarely the case. (Ferris v Commissioner, 1939 879 TC 751)

The IRS has issued a reminder to taxpayers that they may be eligible to deduct some costs associated with their home mortgages. Points paid solely to refinance a home mortgage usually must be deducted over the life of the loan. Other closing costs, such as appraisal fees and other non-interest fees, are generally not deductible. If refinancing for a second time, the unamortized balance of points paid for the first refinance may be deductible upon pay-off of the first refinanced loan. Taxpayers also should note that the amount of their adjusted gross income can affect the amount of deductions they can take because of the overall itemized deductions limitation (the overall limitation applies to years before 2018 and after 2025). (IR-2002-114)

California conforms to the Federal treatment of points.