My grandmother (1922-2015) was from Königsberg, and while this doesn't make me an expert nor give my views any particular credibility, I can relay what was passed on to me. Please note that I am refraining from any political and/or historical judgment pertaining to these historic events; if you - incorrectly - feel such commentary is implied, please forgive my poor English skills.
My grandmother was the oldest daughter of the headmaster of a Latin grammar school in Königsberg. When the war was coming to a close, she was about to finish her training as a medical doctor. Many centuries later, she took the time to compose the family chronicles for both her and her husband's family, comprising - inter alia - the story of how she left Königsberg in 1945. From what I can gather, most of the people she knew had left Königsberg by the end of WWII. Only one uncle who was a country doctor outside of Königsberg stayed behind with his wife, and only left in 1949.
According to her chronicles, people started mass leaving Königsberg as early as 1944, increasingly so after the RAF bombed the historic center of Königsberg in August 1944. Up until January 1945, there were regular refugee trains out of Königsberg that brought Germans from the Eastern provinces westwards. My grandmother didn't get to go on one of these trains. Instead, they left Königsberg on Jan 27th 1945 by foot in order to reach the sea side harbor city of Pillau from where the German navy was shipping out refugees. She always had a very vivid recollection of that refugee trek (-26 degree celsius, masses of snow, and the many wounded and weak who didn't make it). Unlike many, she did make it - though her leg was severely frozen - and accompanied one of the refugee ships as an acting physician to Swinemünde. From there, she fled onwards to Berlin where the family reunited, she finished her training as a doctor, and she eventually met my grandfather.
They ended up settling in the Rhineland - on the west bank of the Rhine to be precise. They felt that if there were to be an escalation of the cold war, surely the Eastern troops would be halted at the Rhine river. My grandmother didn't return to what had now become Kaliningrad for nearly 5 decades. But when the wall came down, and the borders were wide open for a brief historical period, she chartered a camper and drove all the way to Kaliningrad to visit her old home city. She even found her parents' house still standing. In later years, she visited Kaliningrad infrequently with her children an grandchildren on several trips. Through a Russian taxi driver who she befriended on her second trip, she kept in contact with the city until she was too old to uphold the correspondence.
Hope that helps.
BTW: my grandmother's childhood home still exists. My friends Olga and Arseniy visited Kaliningrad this winter, and found the house still standing.
As far as I know (I lived in Kaliningrad for 4-5 years), they were violently made to move in Germany in year 1947. A couple thousands were still there up to 1949, but only because they were some important specialists (their help was necessary to rebuild the city). These people couldn't get a soviet citizenship, which is why they were also sent to Germany later.
Last questioning in year 2010 has showed, that there are still 1676 Germans living there.