I think it also depends on what kinds of recreation you do and will this be your primary car.
I do a LOT of out door recreation and even after having an '85 Toyota pickup which had serious off road capability I realized that I could get anywhere I needed to go with an AWD vehicle instead of 4x4, and if I couldn't I would walk.
Advantages of the FJ Cruiser would be it's towing capability, (the E only can legally tow 1500 lbs), it can carry 5 people (the E can only carry 4), it's true off road capability (the E can handle very nasty roads though) and more creature comforts can be found on the FJ (but the E is very comfortable and has all of the comforts you'd want or need.
All that said, I really like the idea of the FJ Cruiser, especially since I come from a Landcruiser family. But with the usability, practicality, comfort and fun of the E I have never once regretted getting it.
Just remember, the newness and style will wear off of any car but you will be constantly reminded of the car's comfort, quality, ability and utility.