Armie Hammer will have conversations with his kids about his controversies ‘when it’s time’