The reason is that AGP transfers only ever happen when the card is running low on its own local memory - it then starts borrowing system memory.
Now since system memory is by a huge amount slower than graphics card memory, performance IS going to stink anyhow, no matter what AGP mode is being used - and on the other hand, as long as the card has plenty of RAM, AGP transfers aren't going to happen at all.
So I'd rather be worried about whether the card has plenty of its own memory. 128MB will be fine for a while.