Abstract: In this article, you will learn what a clinical note is, why understanding clinical notes may be useful, what are some special characteristics of such notes, what is negation and how may it be useful, what are ways in which information in a clinical note may be retrieved, how the same information may be present in different ways in a clinical note and some ideas to explore further once we have a basic understanding.
The mission
Clinical notes contain valuable information that helps develop a more complete understanding of the patient.
Our mission is to find out how we can extract what is commonly used in a clinical note – Negations.
(If you are already aware of what this is and would like to jump right into the code, click here).
Let’s dive in!
What’s a clinical note?
A clinical note is written by a clinician or a healthcare professional when providing clinical care for patients. It describes patients, their history be it medical, family, socio-economic, dietary, hereditary etc., their present illness or condition, any findings from the physical exam that was carried out (if patient history taking interests you, check out this transcript of the You+AI podcast episode – the doctor-patient relationship).
It may also have details of lab tests that have been ordered, their results, the details of any procedures done, the medicines being taken by the patient and any dosage information.
What does a clinical note look like?
Here is a sample of a clinical note that describes the condition of a patient when he showed up to seek care.
Many more may be found at https:/mtsamples.com/.
What is the point of parsing a clinical note and looking for interesting information?
Clinical notes can yield a wealth of information when examined. Some uses are as follows.
- It adds to the primary diagnosis that has been done for the patient.
- They act as references for the clinical team helping them in clinical decision support.
- The patient history may be valuable in understanding what kind of patients present what types of conditions.
- Mining clinical notes may help in the identification of new patterns of symptoms and disease.
- Clinical notes may help add billing codes to the patient’s medical record.
In what ways is a clinical note different from regular language?
A clinical note is written as part of medical care that is given. It may not be grammatically correct, it will most likely have spelling errors, it may have short phrases and a number of abbreviations and acronyms and there is hardly any standardization i.e. it will vary from one hospital or clinic to another and even from one clinician to another.
Here’s the same clinical note that was presented above, in a different format.
Some more key characteristics of clinical notes
Negation – A typical clinical note in addition to reporting the conditions that the patient has, also reports the conditions that a patient does not have. This is important to have a better medical understanding of the patient. This is not something that you would see in a pronounced manner in regular language.
Examples of Negation
- Not diabetic.
- No chest pain.
- No weight loss or episodes of stomach pain.
- Hypertension absent.
Context – This refers to a condition that a patient had previously or a relevant condition that the patient’s family member had.
Examples of Context
- Patient’s mother and father developed Diabetes in their 50s.
- Patient – long history of common cold.
What is our objective here?
Our objective is to identify the negations in the clinical note presented above.
As stated earlier, negation allows for a more complete understanding of the patient.
Where do we start?
Many of the NLP frameworks available today from the key providers say Amazon, Microsoft and Google have APIs that will accept English language text and identify many aspects in that text.
So, we will pass the clinical note through the Google NLP API and see what it yields. For contrast, we will also pass in text that is not a clinical note to see what comes out.
Non-clinical text
Clinical text
Analysis
When comparing the results above, the following is evident.
- The API identifies a lot more entities in non-clinical English language text.
- The API does not classify medical terms well i.e. as diseases, medicines, treatments etc.
- The API classifies some clinical terms in an unexpected manner.
- Negation is not at all detected.
So, this API may not be the best suited for clinical note parsing. This is also the reason why Google (and it’s competitors) have a separate, specific Healthcare NLP API. There is a set of steps to get access to that which I am working through which is why I don’t have the results of that API yet.
At this time, our discussion will continue to explore Open Source Software (OSS) options to meet our objective.
What do we proceed with?
We use the following libraries and continue the work in Python.
- spaCy – An open source NLP library in Python.
- Negspacy : A spaCy NLP pipeline object for identifying negation in text. Based on the NegEx algorithm.
- NegEx : — NegEx is an algorithm that locates trigger terms indicating a clinical condition is negated or possible and determines which text falls within the scope.
Setting up the basics
The first step is to install all these libraries in case you already don’t have them. Note that the model being used is a Spacy NER model trained on the BC5CDR corpus.
The next step is to import the relevant modules into your python script.
Specify the model and the clinical note
Extraction of Negation Entities
This comprises two parts – adding a Negex component to the NLP pipeline and then using that on every sentence of the clinical note to identify the negation entities. Once all of them have identified, they are printed in a list.
Expected vs. Actual findings
From this table, we see that ‘Hypertension’ was not found as a negation entity. Note that the ‘absent’ keyword was used after ‘Hypertension’ in the clinical note.
Experiments to improve findings
A couple of ways that may be tried out to improve the findings are listed out below.
- Change the clinical note
Change
“Hypertension absent.”
To
“No Hypertension.”
Result: Hypertension is recognized as a Negation Entity.
Note: It is impractical to change the clinical note when considering a production use case. Therefore this experiment is only to show when a negation entity is easily recognized vs. when it isn’t.
- Use custom patterns in Negspacy
Certain keywords or phrases that follow an entity and indicate negation (e.g. ‘absent’) may be added to Negex. In our example, this means changing the negation_model() function as follows.
Result: Hypertension is recognized as a Negation Entity.
Note: This is more practical to implement but the challenge may be to find such preceding or following phrase patterns in all of the clinical text corpus under consideration.
What could be other negation entity patterns that may be challenging to find?
- Termination patterns — Example: Diabetes denied, Stomach ache not present, Congestion none
- Use of prefixes — Example: Patient is non-diabetic, non-hypertensive.
- Double negatives — Example: The patient is not abnormal.
- Symbols usage — Example: Hypertension +, Diabetes –
- Positive/Negative usage — Example: Hypertension positive, Diabetes negative.
- Yes/No phrases — Example: Hypertension – No, Diabetes – Yes, Urinary problems – No.
- Use of more generic language than clinical terminology — Example – No complaint of high sugar (rather than No Diabetes).
This only means we need to test our solution more and more with a rich dataset of clinical notes to iron out as many cases as possible.
Closing thoughts
In this article, we started with what a clinical note is, explored what kinds of information may be available in such a note, pondered about how does negation play a role in understanding the patient’s condition better, tinkered with ways to extract the negation entities and experimented with ways to improve our solution.
This investigation project is one of many ideas that I would like to explore around healthcare data mining. I believe there’s huge potential in the application of Machine Learning in Healthcare but we need to peel one layer at a time.
Some more ideas to take this further may be –
- Identify further interesting negation patterns in clinical notes – crowdsourcing will certainly help.
- Try out the Healthcare NLP API from one of the big 3 cloud vendors – Amazon, Microsoft and Google. Identify what more does that achieve over what OSS does.
- Compare the Healthcare NLP API of the big 3 cloud vendors.
I will be exploring these and more in this area. If you’re interested in such endeavors, please join along my journey by following me on Twitter and subscribing to the youplusai YouTube channel.
Big picture ideas!
Mining negations in clinical notes is a way of mining useful information from unstructured data. Besides the use cases that we have delved into above, I see a few big areas this may have a significant impact in.
- Building disease correlations
As an example, when reviewing clinical cases in a specific area, say infectious diseases as part of an Mortality & Morbidity (M & M) Review, a diagnosis of typhoid but with a negation entity of fever being documented in the clinical note may either point to an incorrect diagnosis or to an erroneous note.
So, analysing a large volume of clinical notes may provide us the correlations that exist between diseases and symptoms. This may especially help for rare diseases.
- Discovering best practices
Clinical notes taken within various medical departments of a hospital may be compared or clinical notes taken by the same specialty department, say gastroenterology may be compared across hospitals to discover the typical patterns of patient history taking or discharge summaries. The best practices in clinical note taking may surface from this and this may help inform clinical management.
- Teaching
Taking down patient conditions when they get admitted and noting down patient history is an important aspect that is taught to medical students. In addition to having a repository of digital clinical notes, an understanding of correlations mined from these notes, best practices gleaned and the extent to which information from clinical notes may be mined to discover new insights may drive home the importance of taking down such notes in adequate and appropriate measure.
Many thanks to Dr. T. R. Gopalan, Madhusoodhana Chari and Umashankar S for their feedback on this blog.
Feature photo by David Travis on Unsplash