Chapter 3 QUANTITATIVE Workbook for a Comparative/Ex Post Facto
Study
Introduction (no heading is used for this section, as per APA)
A. Background Reading for This Section
None.
B. Critical Points to Address for this Section
Begin by stating the purpose of the chapter, how it fits in the dissertation,
and the organization of the chapter. Briefly preview the Chapter III focus. Identify
the major topical areas to be covered in the Chapter.
Restate the purpose of the study, consistent with information provided in
Chapter I. Provide a brief synopsis of the research purpose using present tense.
For example, “The purpose of the proposed quantitative ex post facto study is to
examine differences in GPAs and retention between male and female first
generation, second generation, and third or higher generation college students
using institutional archival data.” “The purpose of the proposed quantitative
descriptive comparative study is to examine differences in math anxiety, as
measured by scores on the Mathematics Self Efficacy and Anxiety Scale, between
incoming male and female first generation and second or third generation college
students.”
Briefly describe the method of inquiry used (quantitative), the specific
research methodology chosen (quantitative), and how it derives logically from the
problem, the purpose, and the Research Questions. End with a transitional sentence
to the Research Method and Research Design sections.
C. Considerations for Alignment
• The research purpose should match the statement of the purpose in the
Chapter I Purpose Statement.
• The method of inquiry and the specific research design (quantitative
descriptive comparative or quantitative ex post facto) must match the
information in Chapter I.
D. Suggested Resources for Enrichment
None.
In the box below, please write the section on the introduction.
Introduction
State your introduction, including the purpose of the Chapter and an
overview of the contents, the purpose of the study, and the research method
and research design.
Note to Student: Ensure that your entry is cohesive. The ideas should flow
logically and with appropriate transitions between sentences.
E. Self Check Your Work to the Dissertation Handbook
Introduction
- Begins with the purpose of the chapter, how it fits in the dissertation, and
the organization of the chapter
- Describes the method of inquiry used (quantitative)
- Describes the specific research methodology chosen and how it derives
logical from the Statement of the Problem and the Research Questions
F. Tips from the Research Review Board
None.
Methodology Selected
A. Background Reading for This Section
Architecture Map (Quant)
Dissertation Field Guide
Chapter 7, Sections 7.1-7.2, pp. 102-109; Section 7.4, pp. 111-115, Section 7.6,
pp. 121-122
Chapter 11, Section 11.1, p. 185
F. Critical Points to Address for this Section
In this section, you will describe and justify the quantitative research method
and either (a) a descriptive comparative or (b) an ex post facto design. First you
will demonstrate your understanding of the nature of quantitative research. Rather
than providing an extensive tutorial, you should demonstrate your understanding of
how the quantitative research method supports measuring variables using numeric
data, describing variables statistically, and examining statistical relationships and
comparisons using analytic techniques. Avoid describing what quantitative
researchers do, instead focusing on the nature of quantitative research. Back all
statements with major and foundational sources in quantitative research.
Describe how the problem that drives your research indicates the need for a
quantitative investigation in order to generate information to resolve the problem.
Explain why a quantitative study is most appropriate.
Describe how the quantitative method is congruent with the purpose of your
study. Explain why conducting a quantitative study will support achieving the
research purpose. Avoid circular reasoning. Instead, explain how the nature of
quantitative research, as described in Section 11.1 of the Dissertation Field Guide,
aligns with the information needed to achieve the purpose of your study. Draw
connections between the aspects of quantitative research described in the table and
the aspects of your study purpose. Do not provide detail on other methodological
decisions, which you will cover in later sections of the chapter.
Next, you will provide an elaborate description and justification of either (a)
a descriptive comparative design, or (b) an ex post facto design, and describe the
nature of non-experimental research and how it differs from experimental research.
Next, describe the nature of comparative or ex post facto research. Explain how
comparative or ex post facto research differs from each other and from
correlational designs.
Read and cite sources in quantitative research to support your design choice
and develop deep understanding of descriptive comparative or ex post facto
research and its application in a research study. Do not rely on general research
textbooks.
Align your decision to choose either (a) a descriptive comparative design or
(b) an ex post facto design with the specific problem that drives your research
focus and the purpose of the study. First, describe how the problem that drives
your research indicates the need to examine statistical differences between groups
formed by one or more naturally occurring independent variables in order to
generate information to resolve the problem. Explain why a descriptive
comparative design or an ex post facto design is most appropriate to address the
problem.
Next, describe how either (a) a descriptive comparative design or (b) an ex
post facto design is congruent with the purpose of your study. Explain why
conducting a descriptive comparative or ex post facto study will support achieving
the research purpose. Avoid circular reasoning. Instead, explain how the nature of
comparative or ex post facto research, as described in Table 7.6 of the Dissertation
Field Guide and in the sources you’ve been reading, aligns with the information
needed to achieve the purpose of your study.
Draw connections between the purpose of comparative or ex post facto
research and your study purpose. Do not provide detail on other methodological
decisions, such as the data analysis plan, which you will cover in later sections of
the chapter. Next, explain the congruence between your comparative research
questions and hypotheses (see DDD) and the nature of comparative or ex post facto
research.
As you write other sections of Chapter III, keep in mind that the
methodological decisions you will make about your particular study must reflect
the nature of quantitative descriptive comparative or ex post facto research. The
decisions must align with the application of either (a) a descriptive comparative
design or (b) an ex post facto design you have selected and with your research
problem, your research objectives, and your research questions. Use the sources
identified above to support methodological decisions later in the chapter.
Another thing you should keep in mind is that you should describe the
methodological plan – what will occur – rather than who will be implementing
each stage of the study. Students are often tempted to model their chapters after
research textbooks, which describe what researchers do. You should focus on
describing the methodology, not the actor. “The researcher” as actor is implicit.
C. Considerations for Alignment
• Alignment means congruence between and across all elements of a study,
from the problem and purpose to the research questions, research method,
research design, and research methodology.
• Good alignment is critical.
• The quantitative method must be appropriate to address the specific
problem that drives the research focus.
• The descriptive comparative design or the ex post facto design must be
appropriate to address the specific problem that drives the research focus.
• A quantitative method and either (a) a descriptive comparative or (b) an
ex post facto design must be congruent with the research purpose and the
form of the research questions and hypotheses.
• An ex post facto design does not involve data collection from human
subjects; archival data only are used in ex post facto studies.
D. Suggested Resources for Enrichment
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Cozby, P. C., & Bates, S. (2015). Methods in behavioral research (12th ed.).
Boston, MA: McGraw-Hill.
Fisher, R. A. (1970). Statistical methods for research workers (14th ed.).
Edinburgh, Scotland: Oliver & Boyd.
Fisher, R. A. (1974). Statistical methods and scientific inference (3rd ed.). New
York, NY: Hafner Press.
Kerlinger, F., & Lee, H. (2000). Foundations of behavioral research (4th ed.). Fort
Worth, TX: Harcourt College Pub.
Vogt, W. P. (2007). Quantitative research methods for professionals. Boston, MA:
Pearson.
Sage Little Green Books available through the Sage Research Methods Collection
in the University of the Rockies library.
In the box below, please write the section on the Methodology Selected
Methodology Selected
Identify and describe the research method in detail.
Justify the research method.
Identify and describe the research design in detail.
Justify the research design.
Demonstrate that the research method and the research design derive
logically from the problem statement and are appropriate to achieve the
purpose and answer the research questions.
Support your methodological decisions with major and foundational sources.
Note to Student: Ensure that your entry is cohesive. The ideas should flow
logically and with appropriate transitions between sentences.
E. Self Check Your Work to the Dissertation Handbook
- Includes a description of the research design and approach
- Provides justification for using the research design and approach
- Demonstrates that the research design and approach derive logically from
the problem or issue statement
- Supports the research design and its application with major and foundational
sources
F. Tips from the Research Review Board
Describes and justifies the research method and design as appropriate to the
problem, purpose, and research questions/hypotheses.
The methodology description in Chapter 3 aligns fully and is consistent with
methodology description in Chapter 1.
Methodology description is aligned across chapters, consistent, clear, and detailed.
Research Questions and Hypotheses
A. Background Reading for This Section
Dissertation Field Guide, Chapter 5, Sections 5.1 -5.8, pp. 55-72 (focus on
relational comparative questions); Chapter 6, Sections 6.1-6.8, pp. 73-100 (focus
on relational-comparative hypotheses)
B. Critical Points to Address for this Section
In a descriptive study, you may have one or more purely descriptive research
questions. Whether your study is comparative or ex post facto, you will definitely
have one or more comparative research questions pertaining to between-group
differences, along with a null and alternative hypothesis pair for each comparative
research question.
You first developed and proposed comparative research questions and
hypotheses in your Letter of Intent. In Chapter 1, you described and justified the
form and focus of your research questions. In Chapter III, you will restate your
comparative research questions and your hypotheses, which must match your
research questions and hypotheses as stated in Chapter I. A common mistake
students make is to change the research questions in one chapter but not revise
them in the other places they appear in the proposal. For example, your research
questions and hypotheses must reflect a consistent set of variables. Thus, if you
Commented [Anon1]: IMO, the checklist
should match the RRB re: research method
and research design.
revise your research questions and hypotheses in Chapter III, you should go back
to Chapter I and update the research questions and hypotheses accordingly.
Briefly explain how answering the research questions will address the
problem, research objectives, and purpose of the study. Then briefly explain how
answering the research questions will address the knowledge gap in the literature.
The knowledge gap should match the knowledge gap you identified in Chapter I
and in the literature review in Chapter II.
C. Considerations for Alignment
• Research questions must inform the gaps identified in the literature
• The potential answers to the research questions must contribute toward
addressing the problem and achieving the research objectives.
• The focus of the research questions must address the purpose
• The research questions and hypotheses must pertain to differences between
two or more groups of subjects or data
• Research questions and hypotheses must be consistent across chapters
D. Suggested Resources for Enrichment
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Cozby, P. C., & Bates, S. (2015). Methods in behavioral research (12th ed.).
Boston, MA: McGraw-Hill.
In the box below, please write the section on Research Questions and Hypotheses
E. Self Check Your Work to the Dissertation Handbook
Research Questions and Hypotheses
Restate your research questions and your hypotheses, which must match
your research questions and hypotheses as stated in Chapter I.
Briefly justify the alignment of the research questions and hypotheses to
the research objectives, purpose, and the knowledge gap.
Research Questions/Hypotheses
Describes specific research questions and hypotheses (where appropriate) that
_____ Are clear and succinct
_____ Are congruent with the Statement of Problem
_____ Are answerable/testable
_____ Correspond to the number of variables of interest
_____ Have hypotheses that correspond to research questions
_____ Are clearly stated
_____ Are open-ended (not yes/no questions)
F. Tips from the Research Review Board
If hypotheses are included, they align with the research questions. Hypotheses are
testable.
Population and Sampling
A. Background Reading for This Section
Dissertation Field Guide, Section 11.5, pp. 195; Chapter 10, Sections 10.1 – 10.10,
pp. 172-178; Chapter 11, Section 11.5
B. Critical Points to Address for this Section
The specific population is the body of human subjects, organizations, or
archival data from which you will select a sample or conduct a census. Ex post
facto studies do not involve any human subjects, only archival data, whereas
descriptive comparative studies involve selecting human subjects from the
population. In this section, you will identify and describe the specific population
for your study, which must match the population you identified in Chapter I. For
example, the specific population may be mid-level managers in the manufacturing
sector in Detroit, Michigan, or archival data about substance abuse treatment
outcomes housed by the Substance Abuse and Mental Health Services
Administration (SAMHSA). Describe the population characteristics, such as
known demographic information or other descriptive traits of the population.
Some populations are publicly accessible, such as parents encountered in a
public park or data posted on a website for public access, such as school district
data posted on a state education website. Other populations, such as employees,
teachers, students, association members, social media group members,
organizational data, etc., are accessible via organizational permission. Indicate
whether the population is publicly accessible, or if the population will be accessed
with permission. You should also identify the size of the available population.
Once you have described the population and the plan to access it, you will
identify and describe the major type of sampling approach – probability or
nonprobability - you plan to use to select your sample from the population. Then
identify and describe the specific sampling technique, such as simple random
sampling (probability) or purposive sampling (nonprobability). If you propose
random sampling, the characteristics of the sample will match the characteristics of
the specific population. Explain how participants or data will be selected, such as
every 10
th
manager on the list or every 5
th
patient record in an archival database. If
you propose purposive sampling, identify and describe the sample characteristics.
Next, justify the sampling approach and technique appropriateness to your study,
keeping in mind that while probability sampling is the gold standard in quantitative
research, its use is not always feasible or appropriate.
You will need to identify and justify the anticipated sample size(s) for each
type of participants or data source. For example, if you plan to collect data from
managers and non-management employees, you will need to identify the estimated
sample size for both categories of participants. If you plan to collect institutional
data from nursing homes and acute care facilities, you will also need to identify the
estimated sample size for both categories. Explain the sufficiency of the sample
size relative to the size of the available population. If you are proposing a type of
probability sampling, identify the confidence interval(s).
You will also identify the power and effect size for the proposed analyses.
Justify the sample size as sufficient to support the proposed comparative analyses,
such as an independent t-test or Analysis of Variance, with adequate statistical
power. Download and use a free sample and power calculator, such as G*Power 3.
[http://www.gpower.hhu.de/]
C. Considerations for Alignment
• The specific population must match the population identified in Chapter I.
• Ex post facto studies do not involve human subjects, only archival data.
• The population must be appropriate to answer the research questions.
• The sampling criteria should not exceed the characteristics of the population.
• The justification of the sample size must address the specific comparative
analyses proposed.
D. Suggested Resources for Enrichment
SMARTLab Unit 1. Samples and Populations
Daniel, J. (2012). Chapter 2: Choosing between taking a census and sampling. In
Sampling essentials: Practical guidelines for making sampling choices.
Thousand Oaks, CA: Sage.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible
statistical power analysis program for the social, behavioral, and biomedical
sciences. Behavior Research Methods, 39, 175-191. Download PDF
Mayr, S., Buchner, A., Erdfelder, E., & Faul, F. (2007). A short tutorial of
GPower. Tutorials in Quantitative Methods for Psychology, 3(2), 51-59.
Retrieved from
http://www.gpower.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-
Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerShortT
utorial.pdf
Mugo, F. W. (n.d.). Sampling in research. Retrieved from
https://www.uonbi.ac.ke/fridah_mugo/files/mugo02sampling.pdf
In the box below, please write the section on Population and Sample
E. Self Check Your Work to the Dissertation Handbook
Population and Sample
Identify and describe the specific population and its source. Identify
permission to access the population, if needed. Describe and justify the
sampling technique(s), appropriateness, and sampling characteristics.
Identify and justify the sufficiency of the sample to represent the
population and to support the proposed analyses.
- Identifies and describes the source of the specific population and the
population’s characteristics
- Identifies that permission has been obtained to access the population, if
relevant
- Describes the sampling method, its appropriateness, and the sampling frame
- Identifies the sample size as sufficient in terms of statistical power to
support the analyses
- Identifies the criteria for selecting subjects
- Describes the sample characteristics
F. Tips from the Research Review Board
Clearly describes the population and how it is being accessed.
The sampling approach is clearly identified and the sample size is justified and
sufficient to either have adequate power
Ethical Concerns
A. Background Reading for This Section
Dissertation Field Guide
Chapter 11, Section 11.6 Informed Consent, Anonymity, and Confidentiality Issues
in Quantitative Research, pp. 198-199.
B. Critical Points to Address for this Section
In this section, you will describe processes and procedures for protection of
human subjects and confidentiality and privacy. First, identify whether or not the
study involves human subjects. If so, identify and describe procedures proposed for
their protection and ethical treatment. If participants belong to a protected class, or
are vulnerable, or if the study involves collecting any personal or sensitive
information, explain and justify the risks in light of the potential benefits. Explain
how the level of risk will be minimized to the extent possible.
Describe the two-part informed consent process – (a) how potential
participants will learn about the nature of the proposed study and their involvement
in it, and (b) how they will indicate their active informed consent, such as via an
ink signature or an online survey portal. Do not use a passive consent process,
such as returning a completed survey, as the IRB will not approve it. Keep in mind
that no data collection may occur prior to receipt of informed consent, including
data for screening purposes.
Whether your study involves collecting archival data or data from human
subjects, you will need to address confidentiality and privacy concerns.
Describe all procedures to protect confidentiality and privacy during data
collection and analysis. Explain whether the data will be collected anonymously or
if identifiers will be assigned to participants and their data. If archival data will be
used, explain whether personal identifiers are contained in the dataset or if the
identifiers will be stripped prior to accessing the data. If the dataset will contain
identifiers, you must describe procedures to protect confidentiality and privacy.
Explain how privacy and confidentiality will be protected when reporting
the results. Explain who will have access to the data, for what purpose, and how
access will be limited. Describe the plans for data storage.
C. Considerations for Alignment
- The descriptions of the nature of the study, the risks and benefits, and the
processes and procedures match the types of data to be collected and the data
collection techniques described throughout the proposal.
- The risks should not outweigh the benefits
- If you are proposing an ex post facto study, you will not have any human
subjects involvement; informed consent is not pertinent
- Protecting privacy and confidentiality is important regardless of whether the
study involves archival data or human subjects
D. Suggested Resources for Enrichment
Dissertation Field Guide, Chapter 13
CITI Training Modules
In the box below, please write the section on Ethical Concerns
E. Self Check Your Work to the Dissertation Handbook
Ethical Concerns
- Identifies procedures for protection and ethical treatment of human subjects
- Describes the informed consent process in detail, including how human
subjects will provide their informed consent, where applicable.
- Includes Informed Consent Letter in Appendix
- Describes procedures to protect confidentiality and anonymity in data
collection, analysis, reporting, and storage.
- Where applicable, describes in detail any treatment or intervention to which
human subjects will be exposed
- Identifies concealment or deception as part of the treatment, if used
- Identifies the process for debriefing human subjects following use of
concealment or deception
- Identifies if a control group will be used, if the group will receive a
standard intervention, and the nature of that intervention
F. Tips from the Research Review Board
All processes and procedures are clearly described.
Identify whether or not the study involves human subjects. If so, identify
and describe procedures for their protection and ethical treatment.
Describe the two-part informed consent process – information then
consent.
Describe all procedures to protect confidentiality and privacy during data
collection and analysis and in reporting and storage. If archival data will
be used, explain whether personal identifiers are contained in the dataset.
If so, address procedures to protect confidentiality and privacy.
Note to Student: Ensure that your entry is cohesive. The ideas should
flow logically and with appropriate transitions between sentences.
Commented [Anon2]: Should be privacy;
anonymity is not always possible.
Commented [Anon3]: This item and sub-
items are not applicable to descriptive
comparative or ex post facto studies
Instrumentation
A. Background Reading for This Section
Dissertation Field Guide
Chapter 11, Section 11.7 Instrumentation, pp. 200-203
B. Critical Points to Address for this Section
In this section, you will describe and justify the instrumentation for your
study and the variables to be measured using these instruments. You will also
address evidence of reliability and validity of instrumentation.
Identify and describe the instruments and data collection forms to be used to
collect data from human subjects. If you are going to use an existing instrument(s),
identify each instrument by name and cite the author or developer. Identify the
specific type of instrument(s), such as a survey questionnaire, psychological
assessment, or test.
If you plan to access archival data, such as test scores or survey data,
identify and describe the instrument(s) that was used to collect those data. If
published evidence of reliability or validity exists, present and cite the evidence.
If you are planning on using or modifying an existing instrument to collect
data, identify that permission is granted. If you are creating a new instrument,
explain why no existing instruments are suitable to measure the variables in the
study. If you are modifying an existing instrument, explain and justify how.
Justify the appropriateness of these instruments to the variables to be measured.
Whether you are planning to collect data or use archival data originally
collected with a given instrument, explain the types of questions on the instrument.
Describe the scoring of each instrument – such as subscales, scales, typologies.
Explain what each instrument measures – which constructs, concepts, or variables.
Explain in detail how the value of each variable will be derived from a given
instrument. Operationally define each variable as it will be measured, and identify
its level of measurement (nominal, ordinal, interval, or ratio).
If you are going to collect data, identify, describe, and cite published
evidence of reliability and validity for each existing instrument if such evidence
exists. If no evidence exists, if you plan on modifying an existing instrument, or if
you are planning on creating a new instrument, present a detailed plan to examine
the modified or new instrument for evidence of reliability and validity.
D. Suggested Resources for Enrichment
SMARTLab Unit 2. Variables and Scales of Measurement
American Psychological Association. (2015). PsycTESTS® database. Available at
http://www.apa.org/pubs/databases/psyctests/index.aspx
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Carlson, J. F., Geisinger, K. F., & Jonson, J. L. (Eds.). (2014). The nineteenth
mental measurements yearbook. Lincoln, NE: University of Nebraska-
Lincoln Buros Center for Testing. Retrieved from http://buros.org /mental-
measurements-yearbook
Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting
Cronbach’s alpha reliability coefficient for Likert-type scales. Paper
presented at the Midwest Research-to-Practice Conference in Adult,
Continuing, and Community Education, The Ohio State University,
Columbus, OH, October 8-10, 2003. Retrieved from
http://pioneer.netserv.chula.ac.th/~ppongsa/2013605/Cronbach.pdf
Keyser, D. J. (Ed.). (2005). Test critiques (Vol. 11). Austin, TX: Pro-Ed.
Maddox, T. (2008). Tests: A comprehensive reference for assessments in
psychology, education, and business (6th ed.). Austin, TX: Pro-Ed.
Retrieved from http://www.proedinc.com/customer/ProductView
.aspx?ID=4244&sSearchWord=
Murphy, L. L., Geisinger, K. F., Carlson, J. F., &. Spies, R. A. (2011). Tests in
print VIII: An index to tests, test reviews, and the literature on specific tests.
Lincoln, NE: University of Nebraska-Lincoln Buros Center for Testing.
Retrieved from http://buros.org/tests-print
In the box below, please write the section on Instrumentation
E. Self Check Your Work to the Dissertation Handbook
Instrumentation
List and describe the survey, research, or testing instruments and data collection
forms to be used in gathering data for the study. The reason for the use of these
instruments should also be given. Defend the choice of instruments and address
reliability, validity, and utility of the instruments.
- Describes in detail all instrumentation or data collection tools, including:
- Instrument name, if using an existing instrument
- Specific type of instrument
- Instrument’s measures in terms of constructs, concepts, or variables
- Scoring of scales, subscales, typologies and their interpretation
- Whether a new instrument is developed or an existing instrument is
modified and describes its use
- Published reliability and validity statistics for existing instruments OR
a detailed plan for statistically assessing the reliability and validity of
new or modified instrument(s) or existing instruments without
established reliability or validity
- Includes a copy of any instruments used, in an appendix
- Includes a detailed description of data that comprises each variable in the
study
Identify and describe the survey, assessment, questionnaire, or test
instruments and data collection forms to be used to collect data or with
which archival data were collected. Justify the appropriateness of these
instruments to the variables to be measured.
Identify, describe, and cite published evidence of reliability and validity
for each instrument or a plan to examine the instrument to be used to
collect data for evidence of reliability and validity.
Note to Student: Ensure that your entry is cohesive. The ideas should
Commented [Anon4]: Another criterion
below is redundant to this one.
- Includes operational definitions of variables
- Provides a clear explanation of how the value of each variable will be
derived from the instrument
- Identifies the level of measurement (NOIR) for each variable
- Either describes established (published) reliability and validity statistics or a
clear plan to statistically validate the instrument (for modified or created
instruments
F. Tips from the Research Review Board
All instrumentation is described clearly. Pilot test is proposed for any
instrumentation lacking prior validation. Face validation will occur for observation
protocols.
All quantitative variables are operationally defined.
Data Collection
A. Background Reading for This Section
Dissertation Field Guide
Chapter 11, Section 11.8, Data Collection Techniques, Logistics, and Format, pp.
203-205.
B. Critical Points to Address for this Section
In this section, you will describe and justify the types of data you will collect
and the techniques you propose to collect these data. First, describe the specific
types of data to be collected. For example, types of data include archival survey
data, scaled response data, archival financial data, test scores, psychological
assessment data, etc.
Describe the plan to collect these data, step by step, from beginning to end.
Describe and justify the specific data collection techniques proposed, such as an
online or pencil and paper survey questionnaire or assessment. Describe how the
technique aligns with the instrument(s) to be used to collect the data. If you will
collect archival data, explain how, such as via downloading an Excel file from a
government website. Use sufficient detail so that another researcher could
replicate the data collection plan. For example, subjects might click on an email to
access a URL with a web-based survey hosted on SurveyMonkey. Participants will
indicate their informed consent to access the survey, the survey questions will load,
Commented [Anon5]: Already stated
above
and the participant will select responses. Upon completion of the survey, the
participant will click “submit.” Once data collection is complete, the data will be
accessed in Excel format from the web host.
C. Considerations for Alignment
• The types of data identified must match what the instruments are intended to
measure
• The data collection techniques must be appropriate to the type(s) of data to
be collected
• The data collection techniques must be consistent across the proposal
D. Suggested Resources for Enrichment
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Cozby, P. C., & Bates, S. (2015). Methods in behavioral research (12th ed.).
Boston, MA: McGraw-Hill.
In the box below, please write the section on Data Collection
E. Self Check Your Work to the Dissertation Handbook
Data Collection
- Describes any pilot study results, if applicable.
- States the sequence of steps followed in conducting the research from
development of the research instrument(s) to data analysis
Data Collection
Describe the types of data to be collected. Describe the plan to collect
these data, step by step. Describe and justify the specific data collection
techniques proposed, explaining how you will use the instrument(s) to
collect the data. Use sufficient detail so that another researcher could
replicate the data collection plan.
Commented [Anon6]: A pilot study
before IRB approval is an ethical violation.
This criterion should be removed.
Commented [Anon7]: The second
criterion does not pertain to describing or
justifying the data collection technique or its
appropriateness.
- Details all steps such that another researcher could follow the steps to
reproduce the study
F. Tips from the Research Review Board
Technique is described and is justified.
Data Analysis
A. Background Reading for This Section
Dissertation Field Guide, Chapter 11, 11.9: Selecting Appropriate Data Analysis
Techniques, pp. 205-209.
B. Critical Points to Address for this Section
First, in an introductory paragraph, preview the section focus and content.
Then identify which specific descriptive statistics (e.g., frequency counts/
percentages, cross-tabulations, specific measures of central tendency and
variability) will be used to describe the sample and the variables, as appropriate to
their level of measurement. See Dissertation Field Guide, Table 11.5, for an
example.
Describe how the distribution of the data will be assessed statistically to
determine if the distribution of the variables in the sample is normal. Normality is
one of the assumptions you must meet in order to justify using a parametric
statistical test. See http://rt.uits.iu.edu/visualization/analytics/docs/normality-
docs/normality.pdf
Next, identify the specific comparative test to be used to test each of your
hypotheses. Use full names. For example, instead of stating intent to use a t-test or
ANOVA, refer to an independent t-test or a two-way ANOVA.
The type of test must be congruent with bivariate or multivariate hypotheses,
a focus on between-group comparisons, and the level of measurement of the
variables. You will need to justify the appropriateness of the analyses proposed for
the specific null hypotheses to be tested and the number and levels of measurement
of the variables in a given null hypothesis.
You will address the appropriateness of using parametric tests or provide a
rationale for using a nonparametric equivalent. You will either (a) justify the use
of a parametric test based on the assumptions that must be met – interval or ratio
Commented [Anon8]: This one should
apply to the entire study
dependent variables and normally distributed data; or (b) if both of these
assumptions cannot be met, propose a nonparametric alternative comparative
analysis. You will identify the levels of measurement of your variables in the
Chapter III variables section. You will explain how you will assess the distribution
of the data above.
Identify the alpha level for the statistical tests proposed. You will identify
the number of tails for your tests. Alpha of .05 is the standard maximum alpha
level used for hypothesis testing. You will identify the number of tails for each
analysis. The number of tails, one or two, will depend on whether a given
hypothesis pair is directional or non-directional. Directional hypotheses are one-
tailed, whereas non-directional hypotheses are two-tailed.
While you won’t report the results in Chapter III, keep in mind the following
caveats. When reporting the results of statistical tests, you must conform to APA
style. Rather than copying and pasting SPSS outputs in the body of the paper or
the appendix, you need to report and write up results in-text and in tables that
conform to APA. Review resources that may help you to learn how to understand
and report the results of your statistical analyses in APA style.
C. Considerations for Alignment
• The descriptive and inferential statistical procedures proposed must be
appropriate to the level of measurement of the variables
• Comparative analyses must be appropriate to the relational-comparative
hypotheses and the number of independent and dependent variables in a
given hypothesis
• In order to use parametric tests, the dependent variables must be interval or
ratio and the distribution of the data must be normal
• The sample must be sufficient to use a given test with adequate power and
effect size
D. Suggested Resources for Enrichment
SMARTLab Unit 3. Charts and Graphs in Statistics
SMARTLab Unit 4. Measures of Central Tendency
SMARTLab Unit 5. Measures of Variability
SMARTLab Unit 6. Probability
SMARTLab Unit 7. Normal Distributions and z Scores
SMARTLab Unit 8. Hypothesis Testing
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Henry, G. T. (1995). Graphical complements for T - tests and Analysis of
Variance. In Graphing data. Thousand Oaks, CA: Sage Publications.
Jaccard, J. (1998). Interaction effects in factorial analysis of variance (No. 118).
Thousand Oaks, CA: Sage Publications.
Lee, E. S., & Forthofer, R. N. (2006). Analyzing complex survey data. Thousand
Oaks, CA: Sage Publications.
Lee, E. S., Forthofer, R. N., & Lorimor, R. J. (1986). Analysis of complex sample
survey data: Problems and strategies. Sociological Methods &
Research, 15(1-2), 69-100.
Johnson, N. J. (1978). Modified t tests and confidence intervals for asymmetrical
populations. Journal of the American Statistical Association, 73(363), 536-
544.
Sirkin, R. M. (2006). Statistics for the social sciences (3rd ed.). Thousand Oaks,
CA: Sage Publications.
Steinberg, W. J. (2011). Statistics alive! (2nd ed.). Thousand Oaks, CA: Sage
Publications.
Turner, J. R. (2001). Introduction to Analysis of Variance. Thousand Oaks, CA:
Sage Publications.
Vacha-Haase, T., Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B.
(2000). Reporting practices and APA editorial policies regarding statistical
significance and effect size. Theory & Psychology, 10(3), 413-425.
Wendorf, C. A. (2014). Statistics for psychologists: An online textbook. Retrieved
from http://www4.uwsp.edu/psych/cw/statistics/textbook.htm
Wilkinson, L. Task Force on Statistical Inference, American Psychological
Association, Science Directorate. (1999). Statistical methods in psychology
journals: Guidelines and explanations. American Psychologist, 54(8), 594-
604.
Williams, V. S. L., Jones, L. V., & Tukey, J. W. (1999). Controlling error in
multiple comparisons, with examples from state-to-state differences in
educational achievement. Journal of Educational and Behavioral Statistics,
24(1), 4269. doi:10.3102/10769986024001042
Sage Little Green Books (focus on between-group comparisons):
In the box below, please write the section on Research Questions and Hypotheses
E. Self Check Your Work to the Dissertation Handbook
Data Analysis
Addresses how the data will be analyzed. What statistical methods will be used?
- Includes a detailed analysis plan that explains the descriptive and/or
inferential analyses proposed
- Descriptive statistical procedures
- Whether the data are normally distributed or skewed
- Parametric and/or nonparametric statistical procedures
- Statistical power
F. Tips from the Research Review Board
Data analysis plan is clear, appropriate, and aligned with the specific research
design, research questions, and hypotheses.
Validity
Describe and justify your data analysis plan in terms of descriptive and/or
inferential statistical analyses to describe the sample, test hypotheses, and
answer research questions.
Address the distribution of the data, the use of parametric vs. non-
parametric tests, power and effect size for the analyses, the number of
tails, and the alpha level for each test.
Commented [Anon9]: The Dissertation
Handbook checklist has past tense, which is
not appropriate. I used only future tense here.
A. Background Reading for This Section
Dissertation Field Guide, Chapter 11, Section 11.10, pp. 210-212.
B. Critical Points to Address for this Section
In this section, you will identify and describe threats to internal, conclusion,
and external validity as they apply to non-experimental research. Unlike
experimental studies, which involve applying a treatment and examining its effect,
descriptive comparative studies do not involve examining or inferring cause and
effect, thus internal validity concerns pertinent to experimental studies do not
apply in descriptive comparative studies. Unlike experimental studies, ex post
facto studies involve searching backwards in time to explain current differences
between groups based on a naturally occurring independent variable using archival
data; the independent variable is not manipulated and no variables are under the
researcher’s control. Thus, internal validity concerns pertinent to experimental
studies do not apply to ex post facto studies. Other threats to validity may apply to
descriptive comparative studies and ex post facto studies involving archival survey
data, such as self-selection and non-response bias (those persons who agree to
participate may differ in unknown ways from those who do not), response bias
(how people answer questions), and attrition (loss of subjects between rounds of
data collection). Describe threats to validity in the areas above as they pertain to
your study, and what approaches you will use to minimize their impact.
Additional threats in descriptive comparative and ex post facto research include
conclusion validity, statistical conclusion validity, and external validity
(generalizability). Conclusion validity pertains to the extent to which the
conclusions you draw about any differences between groups are reasonable based
on the available data. Statistical conclusion validity, a subset of conclusion
validity, pertains to the extent to which the conclusions you draw about these
differences, based on the results of your data analyses, are reasonable. Consider
alternative explanations for potential significant results, such as other variables that
may contribute to the differences found. Explain how you might minimize the
contribution of these variables, if feasible.
External validity pertains to the extent that the results in a given study are
generalizable, i.e., are applicable, to other persons in other settings and at other
times. Describe how your specific population and the type of sampling technique
you propose using may affect the external validity of your study, and how you
might increase generalizability.
C. Considerations for Alignment
- Keep in mind the inability to measure or infer causality (cause and effect) in
a non-experimental study
- Threats to validity are specific to the nature of the study – the population,
sampling, the variables, the instrumentation, etc.
- Be explicit in connecting threats to validity to the nature of the proposed
study
D. Suggested Resources for Enrichment
Black, T. R. (1999). Doing quantitative research in the social sciences: An
integrated approach to research design, measurement and statistics.
London, England: Sage Publications.
Statistical conclusion validity (n.d.) Retrieved from
http://www.indiana.edu/~clcl/Q560/Lecture_3/Validity.pdf
Stone-Romerol, E. F., & Rosopa, P. J. (2008). The relative validity of inferences
about mediation as a function of research design characteristics.
Organizational Research Methods, 11(2), 326-352.
doi:10.1177/1094428107300342
Trochim, W. M. K. (2006). Conclusion validity. Retrieved from
http://www.socialresearchmethods.net/kb/concval.php
In the box below, please write the section on Validity
E. Self Check Your Work to the Dissertation Handbook
Validity
- Addresses internal and external validity threats and how they will be
addressed.
F. Tips from the Research Review Board
Student describes threats to internal and external validity and how they will be
addressed.
Summary
A. Background Reading for This Section
None.
B. Critical Points to Address for this Section
In this section, you will summarize key points in Chapter III. Summarize the
proposed methodological plan for implementing the proposed research study to
achieve the research purpose and generate answers to the research questions. Do
not present any new information. Instead, briefly summarize the research method,
the research design, and their appropriateness to the problem, purpose, and
research questions.
Briefly summarize the population, the sample, the type(s) of data to be
collected to measure the variables, and the data collection techniques. Identify the
instrumentation. Briefly summarize the proposed analyses. End with a transition to
Chapter IV.
C. Considerations for Alignment
Validity
Describe threats to internal validity, including conclusion validity and
statistical conclusion validity, and external validity.
Describe methodological steps to address these threats.
Note to Student: Ensure that your entry is cohesive. The ideas should
flow logically and with appropriate transitions between sentences.
• All information in the summary must be consistent with methodological
decisions identified and described in the chapter.
D. Suggested Resources for Enrichment
None.
In the box below, please write the section on Summary
E. Self Check Your Work to the Dissertation Handbook
Summary
- Summarizes key points in the chapter
- Transitions to Chapter IV
- Conforms to the recommended length of 10-25 pages for Chapter III.
F. Tips from the Research Review Board
None.
Summarize key points from Chapter III pertaining to the proposed
methodological plan for implementing the study. End with a transition to
Chapter IV.
Note to Student: Ensure that your entry is cohesive. The ideas should
flow logically and with appropriate transitions between sentences.