New Jersey Institute of Technology (NJIT)
Computer and Information Science Department (CIS)
CIS677:
Information System Principles, Spring 2000
Professors: Michael
Bieber and Catherine
Campbell
We have included the grading criteria in these guidelines.
Unless otherwise noted, the categories are:
Grading the length of the article review:
-5 points for each full page missing (i.e., less than 4 full pages)
and -3 for every half page missing. Note that large gaps between
sections will not count towards the full page length.
The purpose of research articles is to present new or refine conceptual ideas, or to present new evidence for conceptual ideas. A research article review results from critically examining a research article. You will have to read your article several times to understand it fully enough to review properly.
Often, comparing your article to others will help you determine its quality.
Also, think about the article and its research ideas in terms of each of the different week's concepts and frameworks we study in class. Do the ideas in the article fit all the categories of appropriate class concepts, models and frameworks? This often will help you see things the authors missed, think about things the authors write in a new light, or see that the authors indeed covered a topic thoroughly.
Please include the following categories in your article review.
State the article's full bibliographic reference (authors, title, journal name, volume, issue, year, page numbers, etc.)
Grading: -3 if missing
Paragraph 1: State the objectives (goals or purpose) of the article. What is the article's domain (topic area)?
Paragraph 2: State whether the article is "conceptual" or "empirical", and why you believe it is conceptual or empirical. Empirical articles and conceptual articles have a similar objective: to substantiate an argument proposed by the author. While a conceptual article supports such an argument based on logical and persuasive reasoning, an empirical article offers empirical evidence to support the argument. Empirical articles, therefore, include hypotheses (or propostions), detailed research results, and (statistical) analyses of this empirical evidence. Empirical research includes experiments, surveys, questionnaires, field studies, etc, and to limited degree, case studies. Conceptual articles may refer to such empirical evidence, but do not provide the detailed analysis of that evidence. Of course, both types of articles can use real life examples to back up their points.
Grading: Objectives: great - 5; ok - 3; poor - 1
Grading: Conceptual vs. empirical: great - 2; ok/poor - 1
Very briefly summarize the article (2-3 paragraphs).
If relevant, follow this format, but remaining very brief:
If this is an empirical article, be sure to briefly describe what kind of empirical study was done as part of the evidence.
Grading: great - 4; ok - 2; poor - 1
Very briefly summarize the important points (observations, conclusions, findings) and "take home messages" in the article.
Please do not repeat lists of items in the articles - just summarize the essence of these if you feel they are necessary to include.
Grading: great - 8; ok - 5; poor - 2
1. Does this article directly cite any of the class readings, i.e., does any class reading appear explicitly in its bibliography or reference section? If not, state this explicitly. If so, clearly decribe how the authors use the cited article. How does the article you are reviewing relate to and/or build upon the class article it cites?
2. Do any of the class readings cite your article (besides the textbook)? If so, clearly describe how.
If the article does not cite class readings and vice versa, then state this.
Be sure to add all references you cite to the bibliography.
Grading: great - 4; ok - 2; poor - 1 {If none, then score 4 by default.}
An article makes a "contribution" by adding to the knowledge of researchers in a research field. An article can make a contribution to the research field in many ways. Does it provide a new way to look at a problem? Does it bring together or "synthesize" several concepts (or frameworks, models, etc.) together in an insightful way that has not been done before? Does it provide new solutions? Does it provide new results? Does it identify new issues? Does it provide a comprehensive survey or review of a domain? Does it provide new insights?
Also, is it salient (relevant and current) to a particular scientific issue or managerial problem? Are the issues addressed introduced in a way that their relevance to practice is evident? Would answers to the questions raised in the article likely to be useful to researchers and managers?
Novelty: Does the article make an original contribution? Are its take-home messages new?
Place each contribution in a separate paragraph or bullet point.
Alternatively, if you believe the article makes no contributions, explain why clearly.
Grading: great - 8; ok - 5; poor - 2
Which theoretical foundations does this article and research build on, if any? In what ways? Include references/citations of the foundation work. (You can determine this in part from the works the article cites. Note, however, that most works cited are not core foundational work, but rather just support certain aspects of the article.)
If the article does not build upon key pieces of prior research, then state this.
Grading: great - 5; ok - 3; poor -1 {If none, then score 5 by default.}
The synthesis section should be at least one full page.
Synthesis means analyzing a particular topic by comparing and contrasting it with, and thinking about it from the viewpoint of, the class materials from across the semester. These materials include the articles, models, frameworks, guidelines and other concepts we've covered. (Of course only certain materials will be relevant for any given article.)
Discuss the article's research ideas and results in terms of any relevant materials covered in class or which you have found in the readings. Cite these readings explicitly, including their source in the bibliography and a bibliographic marker in the text (e.g., [Turoff et al., 1999]).
Analyze the approach the author took to the article's analysis and discussion. Discuss the article's approach and results in terms of one or more of the frameworks, etc., from the text or readings, or any you find elsewhere. For example, if the authors discuss any type of information system, you could use Alter's WCA analysis to examine how they approached that information system. Try to do this for all the models and frameworks, etc., which apply to your article.
As part of this analysis, reference other articles you've read, when appropriate. Compare the approach, results and contribution with all articles about similar topics or with a similar approach. For example, if your article develops a new framework, compare it with Bandyopadyhah's framework criteria (and vice versa - whoever does Bandyopadyhah's article could test his criteria on frameworks from the other readings). Include any articles you cite in the bibliography and use bibliographic markers in the text.
Grading: four items up to 16 points total - for each item: great - 4; ok - 2; poor - 1
What has changed since the article was written? How do it's lessons, ideas and theories still apply? To what extent has its issues been resolved?
Grading: great - 4; ok - 2; poor - 1
Optionally, try applying the article's models, frameworks and guidelines, etc. yourself. Do you find them useful?
In addition, you may optionally add your own additional analysis in a separate subsection. (Do not repeat the author's analysis in the paper - you could summarize this as part of the results section.)
Grading: this section is extra credit only: great - 8; ok - 5; poor - 2
In this section you should state your opinions of how well (or poorly) the authors did their research and presented the research results in the article. Your critique can contain both positive and negative comments.
Explain and justify each of your critique points in at least 3-4 sentences.
Questions to ask yourself when justifying your critique points:
Grading: four items up to 12 points total (10 points plus 2 points extra credit) - for each item: great - 3; ok - 2; poor - 1
*** only for conceptual articles
{adapted from guidelines from Dr. Dan Robey, Georgia State University}
A critique of a conceptual article examines the logic of the arguments made by the authors. Both strengths and weaknesses should be identified in a critique. Explain and justify each of your critique points in at least 3-4 sentences. Give examples whenever possible.
Discuss each of the following categories in a separate paragraph:
1. LOGICAL CONSISTENCY: Do any parts of the article or research contradict or invalidate other parts? If so, have the authors acknowledged and explained this adequately?
2. COHERENCE: Does the article make sense? Did the authors approach this article (and this research) sensibly? Does the article develop an argument that follows a coherent line of reasoning? Are the boundaries of the argument reasonably well defined? Does the argument anticipate most, if not all, rival arguments?
Does the article flow in a logical sequence? Do later parts build logically upon earlier parts?
3. SUBSTANCE: Does the article provide an argument or a line of reasoning that offers insight into important issues, or does it merely summarize previous studies in a shallow way that does not reflect depth of analysis? Does the article provide ways (a model, framework, guidelines, etc.) to guide future thinking about the issue(s) the author is addressing?
4. FOCUS:Is there a clear audience that the authors address? Was the article written at the appropriate level for this audience?
Grading: for each: great - 3 ok - 2; poor - 1
*** only for empirical articles
{adapted from guidelines from Dr. Dan Robey, Georgia State University}
A critique of an empirical article examines the strength of the empirical evidence supporting the author's argument. Both strengths and weaknesses should be identified in a critique. Explain and justify each of your critique points in at least 3-4 sentences.
Discuss each of the following categories in a separate paragraph:
1. CLARITY: Is the article's purpose and argument clear? Do the researchers clearly develop a major research question, proposition, or hypothesis that is to be evaluated in the empirical study and discussed in this article? If the study is exploratory (preliminary), is sufficient justification for an exploratory strategy given?
2. THEORETICAL GROUNDING: Is the researcher's argument grounded in more basic theory? Is it clear whether the structure of the empirical study (i.e., what they do) was derived from theory, or just made up? In theory-building articles, is the need for new theory adequately established?
3. DESIGN OF RESEARCH INVESTIGATION: Is it clear exactly how the empirical study was carried out? Is the design of the research approach (field study, experiments, questionnaires, etc. &emdash; both contents and how they will be used) adequate to address the common threats to internal and external validity? Have appropriate controls been established, and is the selection of research sites justified? Are the hypotheses and experiments, etc., significant?
4. MEASUREMENT: Empirical studies can have quantitative measurements (i.e., numeric results) and qualitative or subjective measurements. Are the measures used adequately described (i.e., what is measured in the study and how)? Are data on the reliability and validity of these measures reported? Does the article feel anecdotal or solidly supported with evidence? For example, in case or field studies, are the results well documented? Is it clear who the subjects were, and with whom interviews were carried out? Were important results cross-checked, i.e., determined across a range of subjects or just gotten from one or two subjects?
5. ANALYSIS: Is the analysis of empirical data conducted properly? Do the data conform to the requirements of any statistical tests used? Are qualitative data adequately described and presented?
6. DISCUSSION AND CONCLUSIONS: In discussing the results of the empirical study, do the authors remain true to the actual findings of the study? Are the claims made in the conclusion of the article actually supported by the empirical data? If the study is exploratory, do the authors offer research questions or hypotheses for future research?
7. BIASES: Do the biases of the authors affect the design of the research or the interpretation of the results? Are the authors aware of potential biases and the affect on the study?
Grading: for each: great - 2 ok/poor - 1
What open questions or issues has the author stated remain unresolved? Discuss each in a separate paragraph of 3-5 sentences. The paragraph should take the following format:
Grading: 3 items up to 6 points total - for each item: great - 2; ok/poor - 1
List several open questions or issues which remain unresolved in your opinion? Discuss each in a separate paragraph using the format immediately above.
Grading: 4 items up to 8 points total - for each item: great - 2; ok/poor - 1
To determine how much impact this article has had, do a citation analysis. Include this with your review. Note, if your article has more than 20 citations, you only need to include a selection of them:
Don't just list the citations. Discuss what this citation analysis shows, and why.
If the article has no citations, state this, and explain why.
Grading - citation analysis: 3 points in the regular indexes, 3 points for the internet search; if the article is from 1997 or later then: 1 point for the regular indexes, 5 points for the internet search;
Grading - impact discussion: great - 3; ok - 2; poor - 1
List three insightful questions of your own, arising from this article. Do not ask definitions, but rather questions that really make one think.
Grading: 3 questions, up to 6 points total - for each question: great/ok - 2; poor - 1
List the full bibliographic references (authors, title, journal name, volume, issue, year, page numbers, etc.) for anything you have cited in your review.
Ensure you have included a bibliographic marker to each (such as [Bieber & Campbell, 2000]) in the text of your review.
Grading: -5 if missing references; -3 if you mention the authors explicitly in your text and put the references in this bibliography section, but forget to explicitly place citation markers in your text.
This page: http://www.cis.njit.edu/~bieber/CIS677S00/guidelines.html