After Oliver leaves they have a telephone conversation, but nothing revelatory is said in it, then six months later Oliver returns for a visit. It's during this visit that Oliver tells the family about his engagement, and Elio is naturally a little upset - though he hides it - so he ends up going to bed early. Oliver comes up later and goes into Elio's room to see him, Elio is still awake and they talk for a while. It's mentioned that the parents know about the relationship, and they talk about the engagement. Then Elio tries to get Oliver to join him in bed, but Oliver says that he can't and refuses. Elio isn't pleased and they end up falling out.
Twenty years pass. Elio's father his died, his mother seems to have some sort of dementia, Oliver is working as a professor at his old university, and Elio has a job also in the States. The two haven't kept in contact, but Elio's job has him passing through the town where Oliver is, so he steps in to say hello. They catch up together first in Oliver's office, where Elio notices that Oliver has on his wall a postcard which he stole from Elio's room that Summer, which depicts the spot in Italy where they first kissed. It transpires that Oliver has kept track of Elio's career over the years and knows a lot about what he's been doing. Oliver asks Elio to come to his house for dinner to meet his family, but Elio refuses as he can't bear the idea, so they go to the bar in Elio's hotel instead to have a drink.
They talk some more, about how Oliver feels like he's been living a parallel life since that Summer, when their paths diverged and he went away to start his new family, as opposed to the life he could have had with Elio. Elio is still very much single and seems to be as fixated on that Summer as Oliver. The writing all seems to suggest that the love they shared that Summer was special and unique, and nothing else could possibly come close to it, almost like they were soulmates torn apart. There's descriptions of how old the two are now, and what time has done to them physically.
Oliver goes to leave and Elio walks him to his car, and on the way Oliver persuades Elio to come back for dinner with his family. Oliver's oldest son is only two years younger than Elio was when they first met.
A little time later and Elio is back in Italy and Oliver comes to visit him, and they revisit familiar places and mull over old times. Elio still has one of Oliver's old shirts hanging in the wardrobe of his bedroom there.
The book concludes around that point, and they never rekindle their romance.