I bought a 2007 EX with 232k miles for under $5500 back in December.
I don't think there really was much changed except maybe how the dash looks, interior, and the front with each "refresh (I believe the refreshes were 2007 and 2009 since that's when the headlights/front changed a bit).
Based on my research, 2007 is where I wanted to start when I was looking. I would have gotten something older if proximity and price were good, but the 2007 added 6 hp (up from 160 from previous years), electronic stability control standard, and side airbags standard (so I'd argue it's a little safer than older models).
I'm personally not a fan of the SC and never was. I needed something with height, 4WD, and storage space, not an SUV someone tried to pass off as a sedan.
An Element is an Element. Get whatever feels right. When you see a few you like, compare what each has the other doesn't and decide what you want based on what they have.
2016 Honda CR-Z LX (Aegean Blue)
2007 Honda Element EX (Atomic Blue Metallic)
2005 Honda Civic Hybrid (Opal Silver Blue Metallic) - Retired, pending sale