If the overall diameter is much larger or smaller, then yes, your speedometer/odometer will be off.
If the overall mass of the wheel is increased a lot then yes, it can take more effort to accellerate or decellerate. BUT, aluminum is lighter than steel belted rubber, and so the tire often weighs more than the wheel itself. So a shorter sidewall can actually end up with LESS weight/mass at the outer end of the rotating mass, offsetting the slightly higher weight of the wheel (though, as was mentioned, going from a 14" or 15" to a 19" or 20+" wheel will be even more negative). There exists the possibility that a 17" wheel with a tall sidewall tire (like the stock setup on a big SUV) will weigh more, and have more mass farther from the hub than a 20-22" wheel with a thin sidewall performance tire.
A shorter sidewall WILL ride rougher, but will have ore precise response to steering input. Your turning radius MIGHT change if your tires can no longer turn the full amount, but for most cars that can handle large wheels, that's not an issue.
In fact, the performance issues shouldn't be an issue on street cars anyhow as you aren't using all your car's performance to start with. If you HAVE to use all your car's performance to keep up with traffic, then you have more problems than wheels.