Simply put: I think Victor is full of malarkey.
I feel like a few things could be going on here. 1. He's telling this story to Walton after his family has already died, he's got some serious grief googles on and there may be some desire for Walton to view Victor and his family favorably. 2. Victor has a strong, and to be quite honest well earned, bias against the Creature and the more saintly Victor makes his family look the worse the Creature comes off. I'm not saying Victor didn't love his family but that doesn't mean there was no toxicity there. I often wonder what we'd hear if the account was being told from the perspective of Ernest or Elizabeth because Victor is frequently shown through out the novel to either excessively idealize or excessively demonize.
We see his unreliability all the time with the Creature. He even ascribes supernatural abilities to his creation despite there being no textual evidence that the Creature has such powers and even when it's clear enough in the book that the Creature is leaving Victor food and provisions on their chase Victor insists it's a "good spirit" and at one point after the Creature outright tells Victor "I will be with you on your wedding night" Victor does a surprised pikachu face when Elizabeth is killed and goes "It was as if he blinded me to his intentions by magic!"
Victor, he told you. He told you in plain words to your face what his intentions were. He didn't obfuscate shit. You're just an idiot.
As Victor tells his story it becomes kind of apparent that while he's being truthful about the events he's not really an objective or entirely reliable narrator and that includes talking about his family.
Victor- "My mother was a saint and an angel! Gosh I love my fiance so much! She's also a saint and an angel!"
Also Victor- "So anyway I had a really disturbing nightmare that I was trying to kiss Elizabeth and she turned into this horrifying vision of my mother's rotting corpse."
Also Victor- "Btw if I create a new race of people they'll worship me as a god and love me and no one will ever be able to criticize me or my interests again."
Yeah...you don't have any issues at all, I'm sure you're the product of a great home life.