Escrow accounts are set up based upon the previous year's actual taxes paid (unexempt amounts in the case of a purchase) or based upon the tax rates and appraised/purchase value (in the case of a new build). The lender does not fight you to hold onto your money, they can't. They are legally obligated to re-analyze your escrow account for a surplus or shortage once per year, and that's usually done after that year's tax payments are due (in Texas lender's pay taxes in December and assess between March and April).
There is no "fighting" to get your money back. Your friends that say this are generally not understanding the process, arguing over appraised values that reduce projected taxes, but not yet assessed taxes (true estimates are only used on new builds, actual assessed taxes are used otherwise), or asking for an off cycle analysis to be run before its scheduled (the lender is only obligated to run an analysis once per year and if you request an off cycle it means you will definitely have two). Errors can happen, but they're not malicious and they're not that common.
When a lender runs the analysis all they are doing is taking the previous year's disbursements, projecting that they will stay the same amount and paid for the same due dates. Then taking your starting balance, adding 1/12th of the total disbursements to that balance, finding the projected low point for the year, comparing it to the 2 month (1/6th of the total disbursements) cushion and determine if you're projected to have more than that (a surplus, returned to you within 30 days, by law) or a shortage (paid back over 12 equal payments over 12 months, or up front to avoid an additional increase in your payment, this is not necessarily required, but it's standard).
Notice of escrow changes must be done 30 days prior to the change.
When you have an escrow account the lender is legally obligated to pay your taxes at the most discounted rate (before penalties, or interest is charged depending on the terminology your taxing authority uses).
Using my escrow account as an example:
Taxes: 7800 (December due date)
Insurance:1700 (May due date)
Starting Balance: 3500
Projected escrow payment: 790
Projected balance:
April: 4290 (3500+790)
May: 3380 (4290+790-1700 insurance payment)
June: 4170 (3380+790)
July: 4960 (4170+790)
August: 5750 (4960+790)
September: 6540 (5750+790)
October: 7330 (6540+790)
November: 8120 (7330+790)
December: 1110 (8120+790-7800 tax payment) <- Low Balance
January: 1900 (1110+790)
February: 2690 (1900+790)
March: 3480 (2690+790)
My low balance needs to be at least 790x2=1580. That means my shortage will be my required minimum balance of 1580 minus the projected low balance of 1110, which is 470. I'll have the option to pay that in March to keep my escrow payment at 790, or I can opt to pay it over 12 months and have an escrow payment of ~829.
It is all highly regulated, lenders cannot "screw" you with your escrow account. The only thing you miss out on by having one is the minor interest gains you could get by keeping your balance in a low risk interest yielding account/investment.
.