I think the simple answer has to be no, it is not a great family car. I have only had the E for a month, so haven't had much experience. Our other car is a minivan, and there are lots of reasons minivans make great family cars. My kids are 9 and 10 now, and I wanted something bigger for them for skiing and fun, but it is not the primary kids car.
You have a soon to be five year old. That means they start school next fall. And pretty soon you will be "dropping off" at school and other events. You will want to drop off and pick up on the passenger side for safety, and there really isn't any good way for the E do that. I can reach over and open the passenger door, and my kids are old enough and big enough to get the door open and get themselves out and get the doors closed again, but I think younger kids will struggle there. For pickup: there is no way to pick up on the passenger side: they can't close the door once they get in the back.
The suicide doors are also interesting. I didn't appreciate this until I bought mine, but I live in the country and rarely park in lots where this may matter. The drivers door is huge and swings wide. Once the back door is open, if you are near another car, you are penned in to that little space, and so can't let your kids in or out of that pen, without doing a sort of shuffle around.