Page Text: Associate professor, Hussman School of Journalism and Media, UNC-Chapel Hill
ReCal2: Reliability for 2 Coders
ReCal2 (“Reliability Calculator for 2 coders”) is an online utility that computes intercoder/interrater reliability coefficients for nominal data coded by two coders. (Versions for 3 or more coders working on nominal data and for any number of coders working on ordinal, interval, and ratio data are also available.) Here is a brief feature list:
Calculates four of the most popular reliability coefficients for nominal data: percent agreement, Scott’s Pi, Cohen’s Kappa, and Krippendorff’s Alpha.
Can calculate reliability for multiple variables at a time
Accepts any range of possible variable values
Results should be valid for nominal data coded by two coders (other uses are not endorsed, and accurate results are not guaranteed in any case — trust but verify!)
If you have used ReCal2 before, you may submit your data file for calculation via the form below. If you are a first-time user, please read the documentation first. (Note: failure to format data files properly may produce incorrect results!) You should also read ReCal’s very short license agreement before use.
Documentation
ReCal2 operates under the following requirements and assumptions:
Data should be nominal
Data for each variable should represent two coders’ judgments on the same units of analysis
All codes must be represented numerically
Input file must be formatted properly
If you are calculating reliability for multiple variables at once, all variables must contain the same number of units of analysis (see below)
To format your data for ReCal2 analysis, follow these instructions:
For each variable, make sure that each of your content analysis code values is represented by a unique number. E.g. 0 = absent, 1 = present, 99 = N/A. Your file must contain no characters other than numeric digits—no letters, no dashes, no decimal points, only digits.
In Excel, SPSS, or another spreadsheet-like program, create a new file.
Into the first two columns of your new file, enter the first and second coder’s data respectively, one unit of analysis per row, ensuring that each row represents the same unit of analysis. If you wish to calculate reliability for multiple variables, insert the second data pair into the third and fourth columns, the third pair into the fifth and sixth, etc. Because ReCal2 requires paired data for each variable, the number of columns in each data file submitted to it must be even. The screenshot below shows a file containing three variables, each with a corresponding pair of code data—A-B, C-D, and E-F.
Header information may be included in the first row of the file. If ReCal detects any letters on this row, it will discard the entire row and begin its analysis on the second row.
There should be no missing data; ReCal2 will generate an error otherwise.
If you are calculating reliability for multiple variables at once, all variables must contain the same number of units of analysis (i.e. all variables must end on the same row in your spreadsheet). If you need to calculate reliabilities for variables with different numbers of units of analysis, they must be entered into ReCal2 separately.
Save this file in comma-Separated values (.csv), semicolon-separated values (also .csv), or tab-separated values format (.tsv)—simply changing the file extension to “.csv” or “.tsv” will not work; the file needs to be “saved as” CSV in whatever spreadsheet or stat program you’re using—and choose “comma” as the column or field delimiter (if applicable). Click through any warning messages that may pop up. The file should have a “.csv” or “.tsv” extension. Your file is now ready for analysis; use the file selection box above to locate it on your hard drive.
Here is the full example file from which the screenshot above was taken. It contains three variables across six columns (two columns per variable) and 20 units of analysis. In a spreadsheet program it will look like a normal spreadsheet, but a web browser or text editor will display it as a series of comma-separated numbers.
If you’re having trouble getting ReCal2 to work with your data, first check the FAQ/troubleshooting page , and if you don’t find the answer to your question there, send me an email . Feel free also to leave any general questions or comments regarding ReCal2 below in comments.