In this article, we explain:
- De-duplication with CLIScan
- How automated CLIScan de-duplication works.
- The different de-duplication strategies available in CLIScan.
- How CLIScan processes and identifies potential duplicates.
Introduction
Literature screening is an important part of pharmacovigilance. Every day, safety teams search multiple databases to identify published information about adverse events, medicinal products, and potential safety concerns.
However, there is one common problem: the same research article may appear in more than one literature database. This means that reviewers often receive several records referring to the same publication.
Checking and removing these duplicates manually can take considerable time, particularly when dealing with large volumes of literature.
This is where automated de-duplication becomes useful.
Introducing CLIScan by Drugvigil
CLIScan is a literature screening tool developed by Drugvigil to support pharmacovigilance literature monitoring and screening activities.
One of its important features is its multi-level de-duplication functionality.
Instead of depending on a single matching method, CLIScan provides different strategies to identify duplicate publications.
Users can select a suitable strategy depending on their literature data, the quality of the metadata, and how conservatively they want the system to identify potential duplicates.
The objective is simple: reduce repetitive manual work while keeping the screening process reliable and traceable.
“Effective de-duplication is not just about removing repeated records; it is about identifying genuine duplicates while ensuring that no relevant safety information is overlooked.”
De-duplication strategies with CLIScan
CLIScan provides multi-level four de-duplication strategies. Our advanced strategies of trained models does have potential to identify and minimise duplicates in more than 95 percent a guaranteed.
Each strategy works differently and is intended for different situations. Let’s discover one by one.
DOI Only
This is the most common and conservative approach most of the tools that use for literature de-dupication process.
It is very direct method the system identifies duplicate records when they have the same DOI or PubMed ID.
Because these identifiers are generally unique to a publication, this method offers a high level of confidence when a match is found.
However, it has a limitation: not every publication has a DOI or PMID. Therefore, duplicate records without these identifiers may remain undetected.
When to use it: When you want to identify duplicates using exact publication identifiers and avoid relying on title similarity.
Conservative
The Conservative strategy uses exact identifiers along with title and author matching.
It identifies records with the same DOI or PMID. It can also identify records when their titles are highly similar and their author information supports the match.
The default title similarity thresholds are:
- At least 90% similarity when author information is missing or at least half of the authors match.
- At least 85% similarity when at least half of the authors match.
This approach provides a balance between identifying duplicates and avoiding unnecessary false matches.
When to use it: For general pharmacovigilance literature screening where a balanced approach is preferred.
Aggressive
The Aggressive strategy uses broader matching criteria to identify more potential duplicates.
In addition to exact identifiers and title-author comparisons, it applies additional title and abstract similarity checks.
Its thresholds include:
- At least 85% title similarity when author information is missing or at least half of the authors match.
- At least 80% title similarity when at least half of the authors match.
- At least 94% title similarity using the Levenshtein method.
- At least 90% abstract similarity when the title similarity is at least 80%.
This approach can identify records that may be missed by more conservative settings.
However, broader matching may also identify different publications as potential duplicates. Therefore, reviewers may need to examine more matched pairs manually.
When to use it: When your priority is to identify as many potential duplicates as possible and you have sufficient reviewer capacity for verification.
Content Similarity
The Content Similarity strategy is useful when publication metadata contain inconsistencies, formatting problems, or minor text differences.
It compares title similarity and, optionally, abstract similarity.
The default thresholds are:
- Title similarity: 94%.
- Abstract similarity: 90%.
Users can adjust these thresholds using the available sliders. Abstract matching can also be switched off.
This gives reviewers more control over how closely two records must match before the system identifies them as potential duplicates.
When to use it: When records contain inconsistent metadata, spelling differences, or formatting issues that make exact matching difficult.
Conclusion
De-duplication is more than simply removing repeated records. It is an important quality-control step in pharmacovigilance literature screening.
When literature searches involve multiple databases, duplicate records can increase the workload and make the screening process less efficient.
CLIScan addresses this challenge through its multi-level de-duplication functionality, offering four strategies: DOI Only, Conservative, Aggressive, and Content Similarity.
By combining identifier matching, title comparison, author matching, and optional abstract similarity checks, the tool helps reviewers identify potential duplicates and organise the screening workflow.
Most importantly, CLIScan keeps the reviewer involved in the final decision through its pending-review process.
The goal is simple: spend less time checking the same publication repeatedly and more time reviewing literature that may contain relevant safety information.
CLIScan is a literature screening tool developed by Drugvigil. Explore its features and see how it can support your pharmacovigilance literature screening workflow.