It's very difficult to get the balance right. You can think of a "pointless" day out with your family/friends as costing you £xxx, but then think back you just missed a day out that will now never happen.
I've missed days out because I was too eager to trade, and I've taken a day off which turned out to be one of my most enjoyable days of the year.
I think it's best not to overthink it and just think it could be worse, like my old job where I was stuck inside at a desk 40 fixed hours a week. You can take days off and still earn more than most fulltime jobs at the end of the month.