How can an employer use its recruitment and selection process to minimize the problem of hiring people who don’t fit?
In my experience, interview is most crucial and it is good to have an open conversation with the potential candidates. I understand that most companies have a list of scripted questions that must be covered, however these tend to be over prepared by candidates. Once you have ticked these off, talk openly to them! I find candidates loosen up and become more true to themselves. If you are concerned about cultural fit, chat to them more about their previous employers and roles.
More generally, if you don’t already it helps to anonymise CVs and applications at the beginning of the process when screening. Removing natural bias really helps delve in to a candidates experience and understanding and then you can continue to assess them during the interview.
I hope this helps as a starting point!