Annotation guidelines template

Last reviewed: 2026-07-07.

A task-agnostic starting point for a new corpus's annotation guidelines document, modelled on the structure MasakhaNER 2 (Adelani et al., 2022) and AfriSenti (Muhammad et al., 2023) use. Fork this file, rename it, drop your task in, and adapt the sections. The section structure is what matters: it is the same structure the playbook's Case Studies chapter compares projects against.

How to use this template

1. Copy everything below the divider into a new file (docs/annotation-guidelines.md in your project repo).
2. Replace every [BRACKETED FIELD] with a real answer.
3. Delete task-inappropriate sections and add a one-line Not applicable: [why] note in place of what you removed.
4. Version this document. Annotation guidelines drift; a guidelines document without a version and a change log is untrustworthy.
5. Store the guidelines alongside the corpus so future readers can verify what an annotator was actually told.

Motivated by the Annotation Design chapter and by every corpus retrospective in the Case Studies chapter that names annotation-guideline drift as a source of quality problems.

---

[PROJECT NAME] annotation guidelines

Version: [X.Y] · Date: [YYYY-MM-DD] · Language(s): [LANGUAGES] · Task: [TASK]

1. What we are annotating and why

• The task in one sentence: [E.g., "Classify each Twitter post in Hausa, Igbo, or Yoruba as positive, negative, or neutral sentiment."]
• What the annotated corpus will be used for: [Training a specific model, benchmark release, downstream deployment. Be specific; annotators annotate differently when they know the use case.]
• What the corpus is NOT for: [Avoid uses the community has agreed on. Explicit here helps the annotator make judgement calls at the margin.]
• Reference project(s) this is modelled on: [E.g., "MasakhaNER 2 guidelines, adapted for [DOMAIN]."]

2. The team

• Lead annotator: [NAME + short bio]
• Native-speaker annotators: [NAMES, or "N annotators, native speakers of [LANGUAGE], recruited via [CHANNEL]."]
• Adjudicator: [The person who resolves disagreements. Usually the lead annotator; can be someone else.]
• Linguistic consultant, if any: [NAME + role]
• Reviewer / auditor: [The person who periodically checks annotation quality without doing annotation themselves.]
• How annotators were compensated: [Rate + benchmark. Fair-rate community-anchored rates are part of the core principles; state what you paid.]

3. Definitions

Every term the annotator uses is defined here, in one place, in plain language. This is the most-important section. Guideline drift usually starts with two annotators reading the same term and understanding it differently.

• [TERM ONE]: [Definition + one positive example + one negative example ("this is NOT [TERM]") for contrast.]
• [TERM TWO]: [Same shape.]
• [TERM THREE]: [Same shape.]

Add every term the annotator will encounter. Cross-reference to definitions used in the reference project (MasakhaNER 2, AfriSenti, LAFAND-MT) explicitly, noting any differences.

4. The annotation itself: decision procedure

Written as a step-by-step procedure the annotator can follow for a single example, from opening the labelling tool to submitting the annotation.

1. Read the example fully before assigning any label. [Especially important for classification tasks, since snap judgements based on the first few words are the main source of low-quality labels.]
2. Check the context. [If the example is a tweet, what is the thread? If the example is a sentence in a document, what is the surrounding paragraph? Instructions on how much context to consider.]
3. Assign the label using the decision tree below.
4. If uncertain, mark the example as UNCERTAIN with a short note. [Uncertain examples go to the adjudicator; do not force a label when unsure.]
5. Submit and move on. Do not revise earlier annotations after the fact unless a guideline change requires re-annotation.

4.a Decision tree

Is [DIAGNOSTIC QUESTION 1]?
├── Yes → label = [LABEL 1]
└── No: is [DIAGNOSTIC QUESTION 2]?
    ├── Yes → label = [LABEL 2]
    └── No: mark UNCERTAIN with a note explaining the ambiguity.

Replace with your task's diagnostic questions. The tree should be depth ≤ 4; deeper trees indicate the label set is not well-defined.

4.b Corner cases

The specific patterns that break the decision tree. Every case listed here is a case at least one annotator got wrong in the pilot round; write them down after the pilot, not before.

• [CORNER CASE 1]: [Example + correct label + reasoning.]
• [CORNER CASE 2]: [Same shape.]

5. Label set

If the task has more than three labels, name them here explicitly with examples.

Label	Definition	One positive example	One negative example
[LABEL 1]	[DEFINITION]	[EXAMPLE]	[WHAT LOOKS LIKE BUT ISN'T]
[LABEL 2]	[DEFINITION]	[EXAMPLE]	[WHAT LOOKS LIKE BUT ISN'T]
UNCERTAIN	Ambiguous; adjudicator will resolve.	n/a	n/a

For sequence-labelling tasks (NER, POS), replace with the tag inventory and the span-boundary rules; see the specific per-tag guidance in MasakhaNER 2 as the reference.

6. Orthography, script, and diacritics

Load-bearing for African languages. Guidelines that skip this section produce corpora that cannot be evaluated consistently.

• Script(s) used: [E.g., "Latin only. Ajami content is out of scope for this corpus."]
• Orthographic convention: [Which spelling standard applies. If the language has more than one, name the one the corpus follows and cite the reference.]
• Diacritic handling: [Preserve fully, normalise to NFC, strip, with the reasoning.]
• Handling of code-switched or mixed-script examples: [Include, exclude, mark separately.]

7. Inter-annotator agreement

• Metric used: [Cohen's kappa for two annotators, Krippendorff's alpha for more, span-level F1 for sequence tagging. See the evaluation-script template for the metric implementations.]
• Target level: [E.g., "κ ≥ 0.7 by the end of the second round"; state what "acceptable" means for this project.]
• Cadence: [How often IAA is measured: after every N examples per annotator, every round, etc.]
• What happens when IAA is below target: [Additional training, guideline revision, annotator recalibration. The response is a design choice; state it.]

8. Adjudication

• Who adjudicates: [Named person / role.]
• When adjudication runs: [Every UNCERTAIN mark? Every annotator disagreement? Sampled review?]
• Decision authority: [Final say on labels; state whether adjudicator decisions can be over-ruled by the lead annotator or the community reviewer.]
• Guideline updates from adjudication: [If adjudication reveals a systematic issue, when and how the guidelines are revised.]

9. Change log

Every version of these guidelines is recorded, dated, and briefly annotated. Guideline changes downstream of pilot annotation are normal; unrecorded changes are the problem.

Version	Date	Change	Impact on prior annotations
1.0	[YYYY-MM-DD]	Initial.	None
[NEXT]	[YYYY-MM-DD]	[CHANGE]	[Re-annotate? Discard? Keep as-is?]

10. Contact and community

• Guidelines maintainer: [NAME + contact]
• How annotators raise concerns: [Channel + response time commitment. See the ombudsperson role in the consent template; the same principle applies for guideline issues.]
• Community reviewer, if any: [NAME + role]

---

Contributor's note. If you have run an African-language NLP annotation project and have a version of this document that survived contact with real annotators, the highest-value contribution is a redlined comparison against this template with your project-specific additions clearly marked. Real annotation-guideline documents from real projects are more useful than generic prose.