That's pretty much how I have my DSL connection set up. The only thing I had to change was to move from a static IP address on my boxes to obtaining an IP address automatically from the router.
The connection setup will be using your router. Most routers have a web browser setup page that you access by typing the url into your browser's address box. For example on Linksys routers you type
http://192.168.1.1. From there you can enter the info required by your provider to obtain a connection.