But again, even with your caveat you're talking about optimizations that need to be programmed towards. If the application you're running doesn't specifically take advantage of them, you see no increase. Using multicore as an example, MANY people saw performance drops when they went from a P4 based computer to a C2D based machine because even though the C2D was multicore, each core was clocked lower (as was the mainboard's FSB). When combined with the fact that the application didn't specifically take advantage of multicore hardware, performance suffered.
And when you add to that the fact that Vista is overall less efficient than XP when it comes to most tasks, you end up with a net loss in performance.
Of course, if you're running a DAW that is written to take advantage of a 64bit OS and with Windows audio in mind (like Sonar), then the benefits are clear. But the promise of audio engine optimizations in Vista (like WaveRT) still haven't been adopted by many developers.
If you are able to pick only programs that were written for a 64bit OS, run hardware with 64bit drivers (digitally signed), and turn of all the eye candy and unnecessary crap (like Aero), then it's possible to see a net gain in performance using a 64bit OS.