I keep the house cold when Mrs. E. isn’t in residence. Which makes layering a necessity when the temperatures dip below freezing outside. When I was living in a very drafty house in Indiana, I learned that layering knits and woolen shirts was an easy and comfortable way to ward off the chill. Here I am wearing a grey long sleeved merino polo, and a darker grey chunky cardigan. (Tan and cream herringbone flannel trousers complete the total comfort package.)
The neckerchief is controversial, but I like them. Especially since the silk keeps my beard from sticking in the knit collars.
Try it, if only indoors.