Borrowing against your policy for a down payment works, and the failure here is almost never the policy. It's the mortgage underwriting timeline.
Underwriters source and season down payment funds. They want to see where the money came from and they want it sitting in your account, documented, usually for at least 60 days before closing. Money that appears in your checking account three weeks before closing generates a letter, and you have to explain it in writing with paperwork attached.
A policy loan explains fine. It's a loan secured by your own asset, the carrier will issue a statement showing the loan and the collateral, and that satisfies the sourcing question. What it doesn't do is disappear from your debt picture. Depending on the lender and how the loan is structured, the required payment can land in your debt-to-income calculation, and DTI is where marginal applications die. Ask your loan officer specifically how they treat a policy loan before you take it, not after.
Timing is the other piece. Carriers take anywhere from a few days to a couple of weeks to process a loan request and get funds out, and some still send a paper check. Start the request early enough that the money seasons in your account rather than landing during underwriting.
On the amount, don't drain it. Borrowing to the point where the loan sits near the cash value leaves no cushion for loan interest to accrue against, and that's how policies drift toward the lapse problem. Leave real headroom, and set the payback schedule the day the money moves rather than after you've unpacked the boxes.
One more thing to weigh. A policy loan reduces both your cash value and your death benefit until it's repaid, so a house purchase funded this way temporarily shrinks the protection over the family living in that house. That's usually fine, and it's a decision that deserves five minutes of thought instead of zero. If the loan is going to sit for a decade, the coverage gap sits for a decade with it.