Brexit turmoil causes Labour, Tory election losses