I’ve got a post over at Digital Kiwis dispelling some of the myths I hear a lot surrounding WordPress and speed.
The tl;dr version is that it can be very fast, but ideally you need to have a professional sort that out for you.
The longer, full version however, is much more interesting and also a bit sweary.