Those unfamiliar with the novel will miss lots of context; a subpar experience, not too confusing.
A minor example: Oriane tells Swann "life is a dreadful business", to which Swann replies "dreadful". In the novel we learn what caused Swann (once an optimist who was now filled with jealousy and pessimism) to agree with Oriane, is that her remark comforted him as if she had a shared understanding of Swann's troubles with Odette. This omniscient narrator is missing from the film and fails to convey, so what else is left to see and experience? A stream of unenjoyable references to a work which I admire.