NOT true.
Just some facts on copper, silver and gold:
Copper: electrical resistivity 1.7E-8 Ohm.m, thermal conductivity 400 W/m/K.
Silver: electrical resistivity 1.6E-8 Ohm.m, thermal conductivity 430 W/m/K.
Gold: electrical resistivity 2.2E-8 Ohm.m, thermal conductivity 320 W/m/K.
(All data courtesy of webelements.com)
As you can see, silver has the lowest electrical resistivity, hence the highest electrical conductivity. It also has the highest thermal conductivity. On the other hand, gold is the last on the list both in terms of electrical and thermal conductivity.
So, why is gold used on audio connectors? Because it LOOKS good and it doesn't oxidise as easily as silver or copper (it has the highest electron affinity amongst the three). Silver is used for internal electrical contacts such as in switches and relays, because you cannot show them off, so the best material is used rather than the prettiest.