If you want to see the winter scenery such as the snow covered wall or so, you can go there at that time. It is also beautiful for the wall in winter.
But generally speaking, it will be cold and sometimes with heavy wind during that time. I went there last winter. The scenery is very beautiful but it is so cold. If you are not afraid of the cold days, you can go.