From our study of the datapath and pipelining, we see that the time set aside to access memory is usually very small. In fact, the time for a memory access may be orders of magnitude faster than most forms of memory available. Most forms of memory such as DRAMs (e.g. SIMMs or DIMMs) or magnetic disks (e.g. hard drives) are much to slow - if we were to try and integrate them into our pipeline, there would be several cycles of delay (even 100s or millions of stalled cycles) every time a load or store instruction was used (to say nothing of instruction fetches). There are available forms of memory that are sufficiently fast to perform a memory operation in one clock cycle but they are correspondingly more expensive.
Instead of using this extremely expensive memory to implement main memory, we use the cheaper RAM for main memory while using a small amount of the more expensive memory as a cache holding copies of memory that will probably be accessed. While we cannot always avoid the slow access time of the cheaper RAM, we can generally achieve a very fast average access time by exploiting some common memory access patterns and using the cache to fill most of them. Conceptually, this is similar to the use of registers to avoid repeated loads and stores except that register use is specified by the programmer or compiler while caching happens behind the scenes (it is just an implementation optimization for the exact same machine language).
We can not predict exactly what the memory access patterns will be for arbitrary programs, but most programs exhibit patterns called locality, i.e. they generally work with a small portion of the entire memory space at a time. Specifically, programs often exhibit temporal locality (the same data is repeatedly accessed, e.g. a variable) and spatial locality (data accessed is usually close to other data that has been recently accessed, e.g. the stack and arrays).