I live n Torquay and work in Anglesea, the traffic will slow down a bit by Saturday, most people will have got to their destination by then.
Every fine weekend on the GOR is busy and the po po never rest, make sure your doing the speedlimit as your driving into Lorne, they are notorious for setting up there.