i would say the cores in a intercooler are too big to allow efficent cooling for oil.... if u look at oil coolers they have fairly thin cores which spread the oil out over a wider area and allow better cooling.....
i may be wrong but thats how i see it at the moment ,ive also thought similarly......