Sure. Start with stage 2 or stage 3 binaries and than you can work on the recompiles at your leisure. Makes installation a little easier.
I liked it but it exaggerated my worst habits from Linux. Too much time spent optimizing my tools and not enough time working on stuff.
I am always amused when the power of the compiler comes up. If someone truly believes it, Gentoo is for you. Assuming you don't mind your computer churning away at night on the latest updates.