I have a VPS. It hosts a video website.

I recently added 128mb ram so it now sits at 320mb RAM.

I want my videos to buffer extremely fast, should I add more RAM, or is there something else I should be looking at? I cannot afford a dedicated server.