From an initial qualitative evaluation, a draft codebook with 96 questions was created by the project director. After extensive review and pre-testing by ICMPA's team, the codebook was revised, and the final version consisted of 74 questions dealing with several aspects of the coverage, including coverage of victims, coverage of leaders, framing of the war and of each side, sources used.
Twenty Five news outlets from around the world were chosen, but only 14 remained after a search on LexisNexis returned less than five articles per week for some sources. Up to three articles per day, published between July 12 and August 15, were randomly selected from the front pages and news sections of each publication. More weight was given to articles that came earlier in the newspaper. The final sample (over 1000 articles) was downloaded from LexisNexis and uploaded to a web site easily accessible to the research team. Each article was given a unique ID number and randomly assigned for coding.
Jad Melki, the instructor of the research methods class at Towson University, trained his student-coders extensively at the beginning of each semester on quantitative and qualitative content analysis methodology. Furthermore, two full sessions were dedicated to explaining the topic of the study and to introducing the codebook. A third session was dedicated for a test run of the codebook, where all students coded the same news article, compared their results and received feedback. Then, every four students were given a news article to code individually, and their results were compared and evaluated. Groups that received low consistency rates were given further guidance and instructions. The actual coding didn't start until after the middle of the semester. In addition, the first 10-15 minutes of each session, throughout the semester, were allocated for questions and feedback for the coding.
In order to ensure reliability, the coding process was divided into several phases. First, each news article was analyzed by at least two coders. Results were then downloaded into an excel sheet, and each pair of codings was compared for consistency. Articles that received an overall consistency rate of 74 percent or less were sent back for recoding by a different pair of coders. If on a second attempt they still received the low rate, they were thrown out. Those that achieved a consistency rate of 75 percent or over were given to "senior coders" who checked and recoded the inconsistent answers. "Senior coders" are a small group of the best and most reliable coders who were chosen from among the student coders. Those were the top one or two students in their class who showed deep understanding of the topic and a high commitment to research. Finally, the project director evaluated the work of the senior coders, prepared the data set and uploaded it to SPSS for analysis.