With hex, you can't still think in decimal. You can convert like you were asking, but use groups of four. However, you have to think hex to do it:
0000 = 0x
0001 = 1x
0010 = 2x
0011 = 3x
0100 = 4x
0101 = 5x
0110 = 6x
0111 = 7x
1000 = 8x
1001 = 9x
1010 = Ax
1011 = Bx
1100 = Cx
1101 = Dx
1110 = Ex
1111 = Fx
So using your example:
11100110101
(0)111 0011 0101
7 3 5
735x
You group binary this way because of the numbers' base. Octal is Base-8, so you grab the most binary digits you can to fit in 8 (0-7). That's 3 binary places (111=7). Same goes for hex (Base-16, or 0-15) and that's four places (1111=15).
The easy way to do it is using the Windows Calulator in scientific mode. (as recommended to me by my MCSE instructor for the exams) Once you understand binary, you should understand subnet masks quite easily!
Lots of fun stuff!!