1. Governments interfere with it - from making it the legal base of their currency, as in the US Constitution, to making it illegal to have any, as in the US in 1933; from guaranteeing the exchange rate of gold against the dollar (post WWII) to the Nixon Shock of 1971, when the gold window was closed.
2. Central banks claim to hold it, then (it is widely suspected) lend or sell it surreptitiously.
3. There is so little of it, that speculators can have a significant effect on the price, especially if (as appears to have happened in recent years), the speculation has been powered by vast amounts of borrowed money.
Below, I give three graphs, all comparing the price of gold per ounce in US dollars with inflation as measured by the Consumer Price Index (and that's another can of worms). It's clear that gold has a very volatile relationship with inflation and can spend a very long time above or below trend.
In the fourth graph I divide the Dow by the price of gold. It seems obvious that gold is a contrarian position for equity investors, rather than a simple hedge against inflation.
Currently, the Dow has come back to something like a normal ratio to gold, but past history suggests there will be an overshoot. And gold itself seems above trend over all three periods chosen; which suggests that both still have a way to fall in nominal terms, but the Dow more so.
(N.B. gold prices to the end of 1967 are annual averages, then monthly averages to the end of 1974, then the price is as on the first trading day of the month; all gold price figures from Kitco).