Screening alone or with a colleague

If you are doing a review on your own, perhaps for a thesis, you may be wondering whether one person is enough, and if not, how a second person fits in. It is a fair question. The answer depends on the kind of review, and on who will read it.

This guide sets out what each standard asks, what the evidence says about screening alone, and three ways to work with a colleague without sharing a login. It ends with a methods sentence for each way of working.

Before you start

• Know your review type: an intervention review, a qualitative evidence synthesis, a scoping review or a rapid review.
• Read what your programme or journal asks. Their rule can be stricter than the standards below.
• Have your criteria written down, numbered in order of importance, with a version number and a date.

Check what your review type asks

Know what one screener can miss

A methodological review of 4 evaluations (23 single screenings) found a median of 5% of relevant studies missed (range 0–58%); 3% for 6 experienced reviewers and 13% for 3 less experienced ones (Waffenschmidt et al. 2019). For 7 of those screenings the effect on meta-analyses was reported, and the 3 that would have changed the findings substantially were all by the same less experienced reviewer.

The authors conclude that single screening “is not equivalent to double screening, as substantially more studies are missed”. In their opinion it could still be an appropriate shortcut in rapid reviews, as long as an experienced reviewer does it. The evidence base is small, and they call the evidence on the impact of missed studies insufficient. None of this is a judgement on you. It is a reason to share the reading if you can.

Find a second screener

A second screener need not know your topic: the Cochrane Handbook notes that one who is not a content expert can be an advantage (section 4.6.4). For a thesis, four routes are open. Agree yours with your supervisor before you start, and report it as it was done.

1. A peer screens everything with you. Two independent screeners at both stages. You might offer to screen for their review in return.
2. A second screener takes a proportion. Two of you screen the same share of records and compare, and you continue alone if agreement is high. This is the rapid-review route in Table 1, and JBI names a similar check for scoping reviews; in other reviews, say why you chose it.
3. A second screener checks your exclusions. JBI names this for scoping reviews, and the PRISMA 2020 explanation and elaboration gives it as an example to report: “records screened by one reviewer and exclusions verified by another”.
4. Your supervisor joins at full text. Where two people are required only for full texts, as in a Cochrane intervention review, the second reader has far fewer records.

Keep the two sets of decisions apart

Independent means that neither of you sees the other’s decisions until both have finished. The PRISMA 2020 explanation and elaboration puts it the same way: reviewers who “were unaware of each other’s decisions”.

Build it in, so that it does not rest on anyone’s care. If the file you send holds none of your decisions, there is nothing for your colleague to see. Compare only when both sets are complete.

Work together without a shared login

Three ways work. Each suits a different team.

Two spreadsheets, compared by formula

The formulas assume the layout in “How to screen titles and abstracts”: one record to a row on a sheet named Records, with IDs in column A and decisions in column F. Give your colleague a copy of your file with the Decision, Reason and Note columns empty, and each of you screens in your own copy.

When both have finished, paste your colleague’s Records sheet into your file as a sheet named Colleague. Add a sheet named Compare with the record IDs from your Records sheet in column A, then these formulas in row 2, copied down:

Column F marks every record the two of you decided differently, Include against Maybe included. A cell showing #N/A means that record’s ID is not in your colleague’s copy. For the 2×2 table, with Maybe counted as Include, put these in any empty cells to the right:

These are the a, b, c and d of the guide “Agreement between screeners”, which turns them into percent agreement, kappa and PABAK. This way needs nothing but a spreadsheet, and anyone can open it. It asks for care: the same record IDs in both copies, and one spelling for each decision, which a drop-down list gives you.

Separate copies of a tool, exchanging files

Each of you screens in your own copy of a reference manager or screening tool, and decisions travel as files. Bramer, Milic and Mast (2017) describe this in EndNote: each reviewer works in their own copy of the library, and afterwards one reviewer’s includes are copied into the other’s library to compare them. Check what a file carries before you send it. Independence holds only if it carries none of your decisions.

One shared online tool

Both of you sign in to one review in an online screening tool. Everything sits in one place, you can screen at the same time, and the tool can compare your decisions. Each person usually needs an account. Check that each of you sees only your own decisions until both have finished, and whether anyone can change that.

Hand over what your colleague needs

Send your colleague:

• the records, each with its ID, title and abstract;
• your criteria, numbered, with their version and date;
• your numbered reasons, if you use them;
• the date you would like the decisions back;
• at full text, the reports themselves. Check what your library’s licences allow before you send PDFs.

Ask for three things back: a decision on each record, the reason for each exclusion, and any notes. Nothing else needs to travel. The hand-over note puts all of this on one page.

Settle the differences, and write each down

When both sets are in, work out your agreement first, then go through the differences one at a time. The Cochrane Handbook says disagreements can usually be settled by discussion, with another person to arbitrate if needed (section 4.6.4).

For each difference, write down what was decided, how (by discussion, or by a third person) and why. That record is what your methods paragraph rests on. The disagreement log gives each one a line.

Keep notes about the paper

Notes travel with the decisions, and may end up in an appendix. Keep them to what the paper says and why you decided as you did, with nothing personal about anyone.

Check your work

• The file you sent held none of your decisions.
• Every record has both decisions, or the share you agreed on has them.
• Records marked “same” plus records marked “look together” equal the records both of you decided.
• Every difference has a line in the log, with what was decided and how.

Writing it up

PRISMA 2020 item 8 asks how many people screened each record and each report, whether they worked independently, and about any automation tools. Sentences to adapt, one set for each way of working:

Two screeners throughout

Two reviewers ([initials]) independently screened all titles and abstracts and all full-text reports against the eligibility criteria, each unaware of the other’s decisions until both had finished. Disagreements were resolved by discussion, or by a third reviewer ([initials]) where needed. Agreement at title and abstract was [x]% (Cohen’s κ = [k]), with Maybe counted as Include.

Two screeners on a proportion

Two reviewers ([initials]) independently screened a random [20]% of titles and abstracts ([n] records). Agreement was [x]% (Cohen’s κ = [k]), with Maybe counted as Include, and one reviewer ([initials]) screened the remaining records. Two reviewers independently assessed all full-text reports. Disagreements were resolved by discussion.

One screener, with exclusions checked

One reviewer ([initials]) screened all titles and abstracts, and a second reviewer ([initials]) checked every record excluded at this stage. Two reviewers independently assessed all full-text reports. Disagreements were resolved by discussion.

One screener at title and abstract, two at full text

One reviewer ([initials]) screened all titles and abstracts. Two reviewers ([initials]) independently assessed all full-text reports. Disagreements were resolved by discussion.

If you used an automation tool at any point, say how: item 8 asks for it. If none excluded records, add: “No automation tools were used to exclude records.”

Reading on

• Agreement between screeners: percent, Cohen’s kappa and PABAK
• How to screen titles and abstracts
• PRISMA 2020 explanation and elaboration (2021)
• Waffenschmidt et al. (2019), single versus double screening