So I've noticed that all the spiritualist armors have higher energy defense than physical defense. Yet hardly anything attacks you with energy attacks. Should Spiritualists actually go ahead and wear armor that ups physical defense over energy?

Its best to get 2 sets of amour for humans. One with physical defence, and one with energy defence. The best thing would be to keep switching in and switching out amour so then you will be prepared for any thing.

or combine / mix it.. like me

For the lazy people like me... best physical defense :P