Is everyone forgetting about the "all mangaka can only draw 6 faces" thing? I suppose it's fun to theory craft over, but if you look for it, you'll soon start to see everyone being relatable to someone else. Thus, there are tons of people who could be family because they look similar. So, for me, I never put much stock in trying to pin people together as family, because it is just the artist having a case of being able to only draw a certain way. You end up with similar looking people a LOT.
Edit: I suppose I should add that, if you removed the hair from everyone (so they're bald), and removed eye color, you'd have a damn tough time picking out who is who. In fact, the crazy hair colors that we know as standard today, we're basically done in order to make it easy for the viewer/reader to tell them apart. Eye color was also done for a similar reason, but it's not as obvious and flashy as a hair style.
So, hell, if we're going the eye route, then people should be mistaking Einhart for Nanoha's child, due to having one blue eye. We should be looking for someone with purple eyes to determine who the father is.
Yeah, I know it's all in fun, but it just feels a bit silly to me. And I did a Yuuno/Fate fic myself, that focused on what their child REALLY is!
On further thought, if we're referring to in-universe people thinking it, that makes me scratch my head. They'd have to conclude that Fate had Vivio when she was 13, and thus got pregnant at 12. Technically possible, but I don't really know of people who tend to assume someone else had a child that young. It's considered very odd. Also, Yuuno is a relative unknown. I could name a dozen of the top researchers and smartest people in the world, and I bet no one here would have any idea what they looked like. You can't assume someone is the child of someone else via visual traits, if you don't know what the father looks like.