Programmable Video Signal Processors (VSPs) play an important role in multimedia applications due to their high performance and flexibility. In order to exploit the huge amount of parallelism inherent in the applications, VSPs employ aggressive parallel architectures, among which Very Long Instruction Word (VLIW) is becoming increasingly popular. For video signal processing, a carefully designed memory system is of particular importance, as video-rate applications have generated an unusual demand for high-bandwidth and low-latency memory access. Although many papers have addressed this issue for systems consisting of general-purpose microprocessors, few of them have considered environments containing VSPs, especially those based on VLIW VSPs. In this paper, we outline the problems involved in this specific context and compare five memory architectures for shared memory based VSPs. Our simulation results of six video applications show that combining caches with stream buffers in the proper way provides the highest performance.