You should be ok
There are a few vulnerabilites where the person can attack your machine and it will use vulnerabilities in IE because explorer is always running (for your benefit!) but those are fewer than the ones where you go to a bad page.
Just make sure you keep up to date on the gazillions of security holes for IE with the patches.