The World IS a Strange Place, but The Weird is Where I Come From

I left the town I grew up in a long time ago. The name isn't important. You wouldn't be able to find it, anyway. No matter how you tried to get there. Not unless you fell into it by pure dumb luck. Which very likely would be bad luck.
Let's call my hometown, Little Valley. It fits. It is in a little valley, sitting right between several hills, with a  river running through it. Very picturesque as I remember it. The population has always been around 3,000, without much fluctuation as far back as I can recall. And everyone appears to be quite normal to those passing through, of which there are not very many. 
It's difficult to explain, but those who travel the roads that would, and should, carry them into Little Valley, suddenly find themselves on the other side of the small town that they didn't see. It might strike them as a little strange that they seem to be a little further along than they thought, kind of like there's a tiny blank spot in their perception. But the brief memory of something being out of place, quickly slips away, into the depths of their subconscious without even much of a ripple. You see, if you're not from Little Valley, it's simply not there for you to find. Oh, a few people stumble upon the town every once in awhile. But for them, that's not necessarily a good thing.
When I was growing up, there were stories about a very old tree that was very difficult to find, even if you'd been there before. And at the base of this tree was a hole. Well, more like a cave that was underground. And as the story went, this cave let to a kind of door, or portal, between worlds. Or, at least to someplace else. Depending on who was telling the story, it might be a wonderous and magical place, or, it might be a dark and scary place. I always thought it was just a story, until I was 12 years old. That's when I came across that tree myself. It was while I was hiking through the woods one day with a couple of my friends. That's also when I lost one of those friends. And never saw her again.
Enough. This is not a short tale, and I am wrestling with some strong emotions, and memories I haven't thought about in a long while. So, enough for now. You might find this a hard tale to believe. But I assure you, it's true. At least as true as I can remember it. And as difficult as you might find it to believe what I tell you, I guarantee, it is a much harder tale to tell.