"Hill people" are anyone who doesn't live in or near your urban paradise.
I think you're taking things a bit too literally... "Hill people" can come from, and reside, anywhere.
I live in probably a 400 house neighborhood where the houses are $200k-$300k and I'm outright surrounded by "hill people" regardless. One of the biggest offenders just took delivery of a brand new C8.
In regard to your exact circumstance, when I drive through the country side and see people with nice chunks of property I usually think the opposite of "hill people". It takes a lot of scratch to setup one of those places and most of the sub 80IQ folks we see in these videos have no prayer of ever coming up with it. Hell, most of you guys have lawn mowers that cost more than any road going vehicle I've ever owned.
Your neighbor just strikes me as a dip**** for assuming otherwise.
Anyhow...