Ok, what you are missing here is how linux USES memory.
Linux is very aggressive about placing unused memory into a cache so that it can access it quicker and use it better. If you look at that picture you will see that under 30% of your actual memory is used, the rest is cached for use, still free for programs to use, just ready to be accessed in an efficient manner.
It will not touch your swap partition until you are using a good amount of physical memory, which makes sense as disk access is significantly more costly than main memory access.
That isn't a memory problem, that is efficient use of memory by a kernel