I like the idea. I am sure my own thoughts/interests in this stem from being the definite shortest in class in grades 2 through 7, and then, the loss of that status by high-school, when I shot up to 6'4".
I love stories when a guy is thought to be younger, and have since I was that age... although it grew as I got older.
I can for sure see a story in a kid going 'into the world' and, with a pretend parent, becoming an uncatchable thief/assassin. I can also see it going another way for another apparent-kid from this village; an outside foster parent or entity who feels this seeming-children should be 'brought down' to a proper level, when they notice how unnaturally smart they are. Imagine rich, idle young wives who don't want the hassle of actually raising a child, but just want a cute munchkin to show off; what would they do to force the behavior they wanted from mentally capable adults who just happen to look like children?