I never understood what the different colors in the timeline supposed to convey:
There’s no legend as far as I know, and I noticed that the same place can appear in different colors at different times 
What does it mean?
The colors are for the different modes of transport (walking = green, airplane = purple, Car = black etc), and each place is colored the same as the trip which leads there.
Yep, what @Palmin said.
It’s basically just a design decision rather than a conveying of any useful information. Colouring the visits based on the colour of the activity type of the previous item ended up looking better than trying to colour the visits based on any other factor. Using special colours for visits ended up looking messy and more confusing. So I went with just using the previous path item’s colour, to keep the visual overload down.