Boris Johnson’s speech was out of step with reality, but in the end he knew his time was up