How to calculate Syntactic Context IAA in Knowtator.

These instructions can be adapted for calculating other IAAs.

You need two projects – one for each annotator. They can be in the same directory and share the same Knowtator files, e.g.

cell_donnie.pprj

cell_donnie.pont

cell_donnie.pins

knowtator.pprj

knowtator.pont

knowtator.pins

cell_miriam.pprj

cell_miriam.pont

cell_miriam.pins

  1. Assign annotator name in batch.

Load each project separately and assign the annotator name in batch to all the annotations. This is to ensure that there are no annotations marked as labeled by the preprocessor or some other annotator.

  • Load project for first annotator.
  • Knowtator menu -> Reassign annotator value in batch
  • Click on “Knowtator human annotator” and select the annotator’s name from the list.
  • A “Choose an annotation filter” window will appear. Select “Show all”. Hit “Ok”.
  • Save and close project.
  • Repeat process for second annotator.
  1. Merge the two projects.
  • Open first project and do “save as … merged.pprj”. Do NOT close it.
  • Knowtator menu -> Merge annotations
  • Open second project when prompted.
  • Hit “Ok” to proceed with merging when prompted.
  • Select “show all” filter when prompted.
  • When prompted, select the text sources you wish to calculate the agreement over. This can be all or a subset of the text files in the project. To select multiple files, click on the first file name, then hold the shift button while clicking on the last file name in your set.
  • When merging is complete, you should get a “merging completed without error” message.
  • Go to “Instances” tab in Knowtator.
  • Expand “Knowtator support class” in left panel (by clicking on the triangle).
  • Select “Knowtator filter”.
  • Click on “syntactic contexts only” in the Instance Browser panel.
  • Click on the copy symbol above (double diamonds).
  • In the Instance Editor panel, change the name “syntactic context only” to “syntactic context IAA”.
  • In the Instance Editor panel, add the two annotators to the “annotator” box. Do this by clicking on the diamond symbol with the plus sign and then selecting the first annotator and hitting “ok”. Then do it for the second annotator.

3.Calculate IAA

  • Knowtator menu -> Calculate IAA.
  • The wizard will prompt you for an output directory, which you will have to create in advance (e.g. SyntacticContextIAA).
  • The wizard will prompt you to “Choose Filter”. Click on the “Choose Filter” button and select “Syntactic contexts IAA”.
  • The wizard will prompt you to “Add text sources”. Click “add”. You must again choose the files you wish to calculate the IAA on from the list. Your selected files should then appear in the box.
  • The “IAA configuration” box will appear. You should select “span matcher” (i.e. deselect all other options).
  • Hit “Go!”.
  1. View the IAA metrics files

In the output directory you selected, you should now see various html files that you can open in a browser.

Span matcher.hmtl : This contains the actual IAA results.

Span mather.matches.annotator1.hmtl : This lists the agreements between the two annotators (very long and boring, don’t bother looking a this, as for syntactic contexts we expect a very high agreement).

Span matcher.nonmatches.annotator1.html : This lists the disagreements between the annotators by showing the differing text spans.