How do you get from Beijing Summer Palace to Fragrant Hills Park?

You can take public bus no.563 from North Gate of Summer Palace to East Gate of Xiangshan Park. It takes around 40 mins on the way. After arriving, walk to the destination directly.
Besides, about the return way, what do you mean by the downtown? Take Wangfujing Street as an example, take bus no.698 from Xiangshan to Gongzhufen. Then transfer to bus no.99 to Wangfujing Bus Stop directly. Generally speaking, it takes around 1.5 hours on the way.

I meant near forbidden city as downtown. further question - how long should I allow for Fragrant Hills? I was thinking of seeing the old summer palace, new summer palace & fragrant hills all in 1 full day - is that doable?