Why investigate? Part 13. Questioning questionable science: finding cures rather than treating symptoms

Summary

Continuing the 'Why investigate' series, in this blog, Martin Langham looks at collecting data, introduces the idea of measurement, and asks what published science is there for testing it ‘beyond reasonable doubt’.

Content

In the 'Why Investigate' series we have considered the why, the who, the when, and the what, all related to the incident investigation process. We have looked at some of the technical aspects of Human Factors, and you have been upgraded to better writers than me – and some only wearing their underpants ('we have a situation blog'). Big congratulations to Lara ('ethics in research' blog) for submitting her PhD thesis and Alex ('making wrong decisions blog') now president of the Chartered Institute of Ergonomics and Human Factors. A welcome to the new MSc study types who will be contributing to the series: Afiah (rail) and Livia (marine).

Normally by now we get into the interesting stuff about how to interview witnesses, limits of memory, general aspects of perception and action. However, this is healthcare – where I’ve said not many human factors types have ventured (sorry). This void or ‘vacuum of science’ appears to have been filled by ‘others’ who have what appears to be rather strange ideas, methods of investigation and data collection concepts. Philosophers (and writers that Alex’s blog highlighted) have noted that where a vacuum of science exists mankind returns to superstition and the like.

Human Factors is often referred to by mainstream psychology people as ‘psychology in the damp, the cold and the wet’, or in my areas ‘psychology with the chance of getting shot’. In essence Human Factors is a postgraduate qualification where the willing participant spends a long time learning the science and the practices of the industries where they will work.

Data from humans is difficult to obtain, time consuming and often only near the end of the study do you find that you asked the wrong question. Human Factors and psychology students spend a good portion of their course learning about statistics and research methods. Statistics is the greater part of this endeavour because when you collect data the natural variation within humans is often so vast you need statistics to understand how anything you did to the environment, the human, the equipment or method of working may have contributed to what you observed and the data you collected. I always recall the gasp of horror psychology undergraduates would make on their first research methods lecture as I wrote Greek letters on the board and talked about statistical variance. “I’ve come here to find myself, not to study symbols – I’m here to find out why I don’t sleep well”.

If something is difficult, or is cold and involves wearing scrubs, body armour, or you go home smelling of diesel or jet fuel, its only to be expected some will think of avoiding those nasty things. 

Quick fixes typically involve a training course, and often watching peoples’ behaviours and then saying there are markers that mean something. Go on a course and you get a certificate to show you attended. Congratulations! You can detect the innermost workings of someone’s mind by observing them for a short amount of time. Yes, its psychobabble and it takes a while for science people – once they have showered and put their scrubs into the correct bin – to say hold on that’s wrong and dangerous.

These next blogs will enable you to detect psychobabble and give you, dear reader (remember Mrs Trellis of North Wales from the questions and answers blog?), some questions to ask before you engage with this questionable endeavour, Let’s start with NTS, or non-technical skills. This is sometimes called the dirty dozen, or sometimes behavioural training.

NTS originates in the aircraft maintenance world. The question is if two engineers operating on an aircraft is similar to a team operating on a person in theatre?

Here I’ll cover a bit of history, some sources of reference that you can use and set the scene, get you thinking about collecting data and introduce the idea of measurement. Avid readers may recall I left you with an idea of some images that had clever maths associated with them (Why investigate, part 8). The three ideas I’m going to advance therefore are:

1. Is there any face validity in the idea, and possible questions?
2. What precisely do you measure, what is the scale and the units of measurement?
3. What published science is there for testing it ‘beyond reasonable doubt’?

Face validity and a simple test

I was invited to a nice lunch, the sort of place without laminated menus. I was there after many years as a government science auditor to help other bits of government. The sales team pitching to government said that with training you can detect liars by observing if they looked up or down, and with extra training – purple belt associate grand wizard – you could get perfect memory, and with the green and purple belt and full wizard pass you could get other people to recall 100% of the memory with 100% accuracy. Impressive I thought, as I only recalled at the last moment that I was supposed to be there. Anyone who says 100% about humans – walk away.

Just after arrival and seated with the ‘wizards’ I made an excuse to visit the toilets and went to the maître d and said – I’m from this bit of government and can you help run an experiment. I asked our waiter to remove his tie.

At the end of the meal, hearing from them how human memory really works (and how traditional science gets it wrong), and how with their training you can see behavioural markers and this means you have 100% recall, I said: “That’s very impressive can you all do that 100% stuff?” They confirmed they could, so I asked: “What colour is the tie of our waiter”. Black was the answer, and I said, “is it a bow tie or a cravat?” All said they were 100% sure black bow tie. I beckoned our waiter over who held a tray over his front. Are you sure I asked? “Yes” they said. The waiter then removed the tray and showed he was not wearing a tie, and he confirmed that from the moment he came to the table he never wore a tie, unlike every other waiter in the place. 

So the moral is test any claims simply. It’s not science but it’s a starting place.   

Measurement

Medicine is all about measurement, but when it comes to forensics in healthcare that’s often forgotten. I did a conference a few weeks ago and asked a colleague to name 12 different types of measurement. Micrograms, milligrams, pressure, beats per minute, centigrade, all sorts, were mentioned.

NTS fans often reflect that a ‘lack of assertiveness’ is a causal factor of incidents. I’m going to use this example to demonstrate why measurement is essential in forensic investigation, proper conclusions and implementable recommendations.

So, the question is, “what is the scale”, and if there is a lack of it, there must be the correct amount, and by implication too much. Perhaps it’s like the three bears and the porridge. Well not really, as porridge temperature can be measured in Fahrenheit or Celsius (editor points out in K). So what could the assertiveness scale be?

Can you measure assertiveness or anything that’s a bit unusual? Well, if you visit those blurred images in my last blog I can describe them on a scale to three decimal places, and cite the metric, about each one and how blurred they were compared to the original. If you can measure and calibrate blur, then the idea is that science is about measurement. If you can’t measure it – does it exist? You are now thinking were all those images of the drug trolley really blurred – or was it just a long shift!

How do you measure things you can’t see? Well wind is on the Beaufort scale. 1–12 with a light breeze (3), violent storm (11) and hurricane (12) as the labels. This is the inspiration of the assertiveness scale – there is a lot of hot air around.

Force 5 assured assertiveness, gale 9 disapproval and violent storm stroppiness 11.

NTS in this context makes little sense – think firstly about measure and scale. More in the other blogs. If you hear it – ask the proponent – what scale?

I’ve chosen assertiveness as a medic was sacked for not having enough of it. They were not assertive to the surgeon, as the surgeon made the mistake. Well, how do you know they were looking at the surgeon at the time?  And from our questions – was not the use of non-standard equipment, the 13-hour shift, and trainee scrub nurse asking questions more relevant? Simply, the incident will occur again. Sadly it did, 8 months later. Remember you investigate to stop the incident reoccurring; If you don’t get to the proximate cause, then it occurs again.

What appears in court – investigations allowing 'beyond reasonable doubt' decisions

I said that the idea of observing behaviour gets into forensics every now and then, but it’s dangerous in terms of why we do the investigation. It’s prevention – nothing else matters. We also do investigations for the court. Although these blogs look only at prevention, as there is a Duty of Candour, it might be worth a chat about investigations and metrics for court. 

Psychology used to be – in the 1940s – all about behaviour, but we have moved on. During the 1990s and early noughties it was the idea that you can work out intentionality, drug abuse, alcohol consumption, just by watching how someone walked. Forensic gait analysis it was called. Essentially, any aspect of human behaviour can be assessed by seeing someone walk. 

A case I dealt with was blessed by people who said, from viewing degraded video images (a frame every 2 seconds), that you can tell that a driver was fit but not happy by the way they walked. You can make this assumption by never seeing any comparative degraded video of the same person or any other data sources. Forensic gait analysis was starting to get into court. 

Rightly the lovely judiciary questioned the science. The Royal Society and the UK top judges started a series of guides as to science in court. Remember, in court the measure is ‘beyond reasonable doubt’. 

The Royal Society and the judges asked for a panel of writers to comment upon science. The writers are typically world leaders and their work is reviewed by many top science types. Gradually the ‘primers’ are being published. Useful source of forensics here for investigators.

Gait analysis report – well you have predicted the findings, and judges are now cognisant of the facts and the danger of watching behaviours as evidence to be used in court. Of course, it’s still up to the judge to decide what happens in their court and personally I defend that right absolutely.  

On the Royal Society Science and the law page, it shows that statistics, etc., are also subject to guidance. The latest one is about incident investigation in transport and Human Factors and sets the scene about Human Factors very well. It’s a good starting place for understanding Human Factors.

Summary

There is a lot of non-science out there. Data are collected in ways that make science types shudder. Behavioural markers are not robust and reliable on their own. Humans do not have the skill to generate good data and prevent incidents re-occurring just by watching each other.

Remember we investigate to stop the incident occurring again. To get my classic literature quote in to conclude:

"The life of the dead is placed in the memory of the living."

[Marcus Tullius Cicero, 106 – 43 BC]

Finally

To answer a number of questions about my last blog saying there is a regulator in this domain and comments received saying I don’t know what I’m talking about because it’s all done by institutes and there definitely is no government regulator, absolutely not and no way there is a regulator. Here is the government regulator website: www.gov.uk/government/organisations/forensic-science-regulator. This maybe useful for those thinking about investigation methods.

Alex has covered the main papers from the regulator on biases on reasoning (making wrong decisions blog) and I’ll do the issues of statistical fallacy. Those in Scotland who have read ahead – yep correct, it’s also known as the prosecutor's fallacy, and yet another issue with root cause analysis.

Post script...

We are very aware that we are pointing out lots of problems without offering any solutions. Shortly Alex, Graham and myself will be offering some courses. We have avoided virtual learning as all three of us have thought about its problems. However, we believe that if we teach Human Factors and forensics in the three domains – the environment, the equipment, and the human – then we need to have those things present and to hand. We wish to avoid the phrase, ”Imagine there’s defib machine, imagine there is a ward, imagine the person is fatigued…".

Read the other blogs in this series

• Why investigate? Part 1
• Why investigate? Part 2: Where do facts come from (mummy)?
• Who should investigate? Part 3
• Human factors – the scientific study of man in her built environment. Part 4
• When to investigate? Part 5.
• How or Why. Part 6
• Why investigate? Part 7 – The questions and answers
• Why investigate? Part 8 – Why an ‘It’s an error trap conclusion’ is an error trap
• Why investigate? Part 9 – Making wrong decisions when we think they are the right decisions
• Why investigate? Part 10. Fatigue – Enter the Sandman
• Why investigate? Part 11: We have a situation
• Why investigate? Part 12: Ethics in research

About the Author

Martin is a topic leader for the hub.

He founded the Human Factors group at the University of Sussex (1999), which became User Perspective Ltd in 2003. Martin, User Perspective MD and Chief Scientist, aided by his team, has undertaken almost 600 research and forensic investigation projects. He is interested in human error and human factors.

Martin is a research auditor for the UK government, EU academic networks and many governments worldwide. Within healthcare he has investigated matters as diverse as neonatal safety in transport, unexplained injuries in the hospital mortuary, sepsis diagnosis and retained instruments. Martin co-authored the very first Healthcare Safety Investigation Branch (HSIB) report that investigated orthopaedic surgery in the UK and Europe. His interest in the law and justice extends to his voluntary role as a justice of the peace (JP) in the Magistrate and Crown courts.