Remember: whatever you do, DON’T PANIC.
With roughly four in 10 GPs now using AI-powered scribes in the consult room, the medicolegal warning to “check all notes” has become ubiquitous. But how, exactly, does one check AI notes?
GP Dr Darran Foo, the medical director at Healthdirect Australia and co-deputy chair of the RACGP digital health specific interest group, has been asking himself exactly the same question.
“No one’s ever started to talk about the best way to review AI generated notes,” Dr Foo told delegates at the RACGP’s annual conference on Friday.
“Like, do I just read it from top to bottom? Is that really an efficient use of my time? Does it cancel my time savings?
“And it was an interesting question I’d ask myself as I start to use [AI] more and start to teach students and registrars.”
After having a look through the literature on where large language models and speech recognition models tend to make the most errors in terms of generating notes, Dr Foo came up with a “cool” (his words) mnemonic: DON’T PANIC.
D stands for drugs and doses; specifics like medication names and doses.
O is for orders and outcomes, as in the names of any tests ordered or patient management recommendations.
N is for negations (yes/no statements). A common mistake in AI transcription is to miss words like ‘not’ or ‘didn’t’, completely changing the meaning of the sentence.
T is for temporal order and timestamps.
“Did the joint pain come after the febrile illness, or did the febrile illness come before the joint pain, for example,” Dr Foo said.
P is for problems and provisional diagnoses, or what generally sits in the differential diagnosis section of a note.
“[This is] ensuring that the AI scribes don’t overstep and put things in there which were not considered or were not said,” Dr Foo told delegates.
A is for attribution – as in, making sure that words are not being put in someone’s mouth. Dr Foo said errors in this respect commonly occurred when there were multiple people (say, a child or a partner) in the appointment room.
N stands for numbers and units, another area where transcription AI has historically struggled.
“[Scribes] are language models, not medical models, so it’s always important to check numbers and units, especially if you’re discussing test results or renal function,” Dr Foo said.
I is for invented exam and ensuring that there are no ‘hallucinations’ in the notes where something has been added that did not occur.
C is for completeness – or making sure that there are no significant omissions to the content of the consult.
A poll conducted by the RACGP’s media arm last year found that roughly 22% of GPs were using AI scribes; a repeat of the same poll this year saw that figure increase to 40%.
Another speaker at the same session, WA GP Dr Sean Stevens, predicted that the next 12 months would see a similar explosion in the popularity of AI voice agents.
His practice, Grove Medical, has been using an AI receptionist to help patients schedule appointments via phone over the last year.
Still, it has not been without its hiccups.
“It’s really interesting talking to the tech heads who have never worked in general practice,” Dr Stevens said.
“They’ll tell you how easy it will be for them to have AI doing everything that your receptionist can do – I can assure you, the front desk job is about 100 times more complicated than most people realise.”
The RACGP’s annual conference, GP25, was held at the Brisbane Convention and Exhibition Centre between 14 and 16 November.
