Damn Small Linux looks like it may be for you. Do you know much about Linux? If you have some experience, a debian net install may be for you.
http://www.debian.org/distrib/netinst
That way you only get the stuff you want, and get apt-get for more stuff later if you want that.