Using machine learning to flag issues with police forces sooner

With all the talk of artificial intelligence in the media after the AI Safety Summit in Bletchley Park, it is timely to look at how one method – machine learning – is being tested out on police force data to see how it could enhance inspection. I spoke to HMICFRS’s Insight Portfolio Director, Jacquie Hayes to find out more.

It is no surprise that His Majesty’s Inspectorate of Constabulary and Fire and Rescue Services (HMICFRS) is awash with data. Collecting it from police forces for decades and from fire and rescue services since 2017, making sense of it is a huge task and there are many analysts employed to do just that. They contribute to the inspection process and have a critical role to play in the development of judgements about how well police and fire are doing as they spend public money to keep us safe.

I recently came across the annual report of the Accelerated Capability Environment (ACE). In the introduction, Richard Alcock, who is the Director of Data for the Homeland Security Group in the Home Office wrote,

“Leveraging technologies such as artificial intelligence and data science can accelerate and automate processes, reduce duplication of effort and better equip decision makers and the operational front line to be their most effective.”  

This may explain why HMICFRS worked with ACE to find a supplier to help them solve a problem and how machine learning became part of the solution.

Framing the problem

Jacquie Hayes said that the inspectorate wanted to find out if it was possible to estimate what the PEEL inspection grade for police forces would be before the inspection is carried out. “We wanted to identify the turning point in the data: when did it start to go up or down.” She is keen to emphasise that data is just one part of the inspection. “Quite often the data raises questions rather than answering them,” she adds.

Developing a predictive tool requires technical expertise and a lot of data. Jacquie explains,

“Policing has an enormously rich source of data, but the issue is how to get the best of value out of it. Our aim is to prevent forces moving into ‘engage’ because we have identified the issues earlier and have been able to put the support in place to prevent it.”

Engage is the part of the force monitoring process that no force wants to find itself in – often referred to as special measures, it’s the trigger point for a lot of extra scrutiny and resource to help turn things around and get a force back on track. It makes sense that the inspectorate would want to see if they can better predict when a force is going in that direction.

Developing a proof of concept

ACE supported HMICFRS to find a contractor to do the technical work and create a proof of concept model. ACE is a kind of procurement dating agency, it did the contract tendering and appointed a small tech start up called the London Data Company to do the work. And true to the A part of ACE, the project was completed in just eight weeks.

HMICFRS wanted to focus the proof of concept on question five in the PEEL dataset which is about how well forces investigate crime. The London Data Company created a machine learning based model to predict what grade each force would have received for this question. It then crunched through eight years’ worth of publicly available data from a range of sources linked to criminal investigation.

Accuracy in prediction

The results are interesting. In about 60 per cent of cases, the model correctly predicted the grade on the latest PEEL assessment and in about 90 per cent of cases the model was one grade above or below the actual grade.

I ask Jacquie if the model could predict what the force grade could be say five years in the future? She says not, “I don’t think it will ever be able to predict that far ahead.”

It’s worth noting that HMICFRS collects data from forces on a regular basis throughout the year irrespective of where a force might be in its inspection cycle. This data is analysed and the analysis shared between forces so that they can see how they compare against their peers. As they continue to build up data, the model may be able to determine trigger points, the things that change in the data to indicate that they may be a sign that a force is heading towards engage.

Understanding public sentiment

A separate but complementary part of the work undertaken by the London Data Company focused on gaining insight into public sentiment of policing by analysing social media. Jacquie explains,

“We looked at tweets sent by the public to police forces over a period of eight years. This amounted to around five million tweets. All were given a positive, neutral or negative rating through a set of rules developed with the London Data Company. There was a reasonable correlation between our PEEL judgements and public sentiment.  It reinforces what we found in our inspection.”

The correlation between the public sentiment and major events or incidents is less interesting for Jacquie, she wants to see the direction of travel and changes in trends, the subtle triggers. “The lower-level noise is more interesting than the big set pieces,” she adds, saying that it could be useful when looking at community engagement for example.

Where next?

Jacquie says that machine learning based tools like this proof of concept wouldn’t be used in isolation and that the inspection would take account of it along with other sources.

“We are thinking about how we move all the London Data Company work into our systems and how this works with the inspection teams and the HMIs.”

Part of the next steps is to assess the impacts on our inspection methodology. Part of that will focus on ethics and discussion about bias in the data influencing outcomes, which is a common challenge for any artificial intelligence application. Jacquie says they will involve the inspectorate’s ethics panel going forward.

It is encouraging to see the inspectorate involved in innovation using machine learning to explore how to get more out of its data and enrich its work. Insights have already been shared with forces, the NPCC and other stakeholders.  It’s a small step in the right direction and a long way off becoming embedded; what will be interesting is to apply the same approach to fire and rescue service data and see the differences. Definitely one to watch.