How to get throw is you want to go to the numbers: 1, 4, 1, 0. This is after you walk north into the Four Way Maze. Then on your way back you want to go: up, right, down, right, down. This is all after you walk north into the Four Way Maze.
They're fighting about two numbers; if you touch one then he will ask you a question and you want to answer so they are both happy the number that makes them happy is the one you want to put the keys into on the starting board.