I lift weights and kick/punch around on a heavy bag here at my house... as one of the aforementioned office workers, it's not so much a question of needing big muscles, it's a matter of not turning into a pile of shit. I go more for reps, and honestly, as a pack-a-day smoker I can skate all day until I puke, which happens sometimes. I can't always supplement my ass in the chair time with skating (like Sleazy mentioned) but I can always go outside onto my back porch, rain or shine.
Not to mention, having had a ruptured disc, fucked up ankles, and knee surgery, working out whatever muscles I have in those areas helps. Also, when I look at the other 35 year old geeks I work with, I'm confident that I can take a good slam better than just about anyone in my office (or pretty much every mid 30's non skater I know). Most of that's because of skating, but being active when I can't roll helps. All that shit helps.
I very much want to be a skater when I'm much older than I am... I don't see the big deal with working out and trying to improve yourself physically if you can.