Opinion Pieces‎ > ‎

Research Design

Designing good research studies is an important part of becoming a researcher, no matter what your field is. The exercises on this page are aimed at junior researchers who are designing their first studies in education research. If you've already done one or two projects, these exercises will help you get better at seeking funding and developing more projects. If you've never done research before, these exercises will help your first project be more successful. 

What does my project need?
Every project in education research needs to address four areas. While the details of these areas can be (should be) emergent, well-formed and successful research projects identify as much as possible ahead of time.

Every project needs:
  • Research question: what do you want to study?
  • Access: what populations can you study, and how much time / which modalities are available?
  • Theory: what theoretical frameworks guide your work?
  • Methods: how will you generate observations and interpret them to become data? How many observations?
Additionally, when you present your work for publication or funding, you will need to consider two more areas:
  • Relevance: what intellectual merit or broader impacts will this project have? Why is this important or interesting work?
  • Audience: where are you planning to publish your work? What counts as novel and important to them?
We're going to leave these two aside for now because they reference a broader sense of where the research community is, what societal needs are, and how your project fits into a much larger narrative. Those considerations are outside the scope of this guide, though you might consider reading ahead to other guides on writing. Let's work on the four primary areas.

In a good research project, the four areas are all tightly related and supportive of each other. You should develop them in concert with each other. The exercises on this page will help you design a research study, and they will also help you develop your design skills in general.

Details of the four areas
Research Questions:
Your research question(s) tie together your theoretical frameworks, methods, and access. They give purpose to your data collection and analysis. Answering them generates new knowledge about human behavior.  In the ordinary process of doing research or thinking about the world, you will ask lots of questions. As you pursue some of them, you'll develop follow-up questions and related lines of inquiry.

Research Question templates
If you're in the very beginning stages of thinking about your project, you might need help brainstorming some possible research questions.  Here are some templates to get you started.  It's not an exhaustive list.
  • Theory X says A, but theory Y says B. How can they be made commensurate?
  • This paper used population A, but I have population B. How can I apply their findings to my population?
  • Surveys shows that students can do X. What is the actual process of learning to do X?
  • What are the moderating factors which control success at task X?
  • Our previous work shows X happens sometimes. Why does X occur?
  • What's better at teaching X, curriculum A or curriculum B? 
  • How do teachers make sense of X in light of Y?
Making your research question better
When you have an idea about what you'd like to investigate, you need to refine your ideas into a research question that suggests how you will answer it and how you will know when it is answered. This exercise helps you refine your ideas into a research question. 
  1. Write your question in the form of a question. 
  2. Your research question needs to be answerable in principle, and your research design needs to have a high likelihood of answering it. To make it more answerable, make it more specific. 
  3. If your research question has a binary answer ("does X happen?"), revise it to permit a more subtle answer ("to what extent does X happen?"; "how much does Y mediate X happening?"; "under what conditions can X be optimized to happen?")
  4. If your research question uses comparison language, what are you comparing? For example, if your research question is about whether a new curriculum is "better" or if students are learning "more", what will you be comparing it to? Do you need to collect baseline data? Will you be able to run a treatment group and a control group at the same time?
  5. If your research question uses development language (e.g. "learning"), over what time are your subjects changing? An hour? Four years? Their lives? How will you know if change is durable? how will you know if it occurs at all?
  6. Your research question should include technical language and reserved words that mean something specific to the research project. Define each reserved word and link it to specific theoretical frameworks, methods, or data streams. 
A good research question is a living question. As you interact with theories and data, it will necessarily change. The more specific you can make it in the beginning, the better you will be able to see it change and adjust your future work in an intentional way. 

Access:
The Access area is about practical constraints on your project: what populations do you have access to, and in which modalities? how much time do you have, and which analysis resources can you marshal? Of all the areas, Access is the one which is usually fixed earliest in the project, because the kinds and amounts of data you have access to are usually determined before you can collect any data at all, and the scope of your project is usually outside your control.

Questions that detail your access to data:
  • What kinds of people will you measure? Some examples: introductory students, pre-service teachers, graduate teaching assistants, third graders in a specific elementary school.
  • In what modalities can you collect data from them? Some examples: I can talk to them individually in interviews once per person, I can video them in class every day, I can put a problem on their final exam, I have three years of archival data but cannot collect new data, etc.
  • How many people / how much data? One or two significant figures are ok here: about 10 students, about 300 students, about 3 hours of video, about 100 matched pre-post tests, etc
You probably can't answer all of these questions alone. Get specific guidance from your collaborators, advisor, and people who control your access to research subjects (their instructors, their principals, the registrar, the data librarian, etc). At early design stages, you don't need to seek IRB approval yet, and you don't need written permission from every stakeholder. When your study is more fully designed, you will talk to these people again to firm up the details of your access and adjust your research questions and methods.  

Questions that detail your access to resources:
  • How long can you spend collecting data? How long analyzing it?
  • How many researchers will be involved in data collection and analysis? What are their skill levels?
  • How much data (and what kinds) can you reasonably expect to collect / analyze in the amount of time and effort that are available to you?
It is entirely possible that you have access to more data and analysis resources than you will need or use in your project. That's great! You don't need to collect (or use) everything. Alternately, you might not have enough access (or the right kind of access) to do the study you really want to do. That's disappointing. You will need to adjust your research questions and methods in light of how much (and what kinds) of data you can collect or analyze with your resources.  

On rare occasions, you can use your research questions to argue for access to more resources or different modalities. For example, suppose your research question is about student epistemology and retention, and you already have access to students' CLASS scores. You might be able to ask the registrar for students' demographics and final grades to enhance your analysis. 

Theoretical frameworks:
The role of your theoretical frameworks is to tell you why your observations are meaningful and in what ways your analyses generate new knowledge. Without a theoretical framework, your observations are meaningless and your work is unpublishable.

The best theoretical frameworks are a) explicit; b) well-matched to your research question and methods; and c) intentionally chosen. There isn't a "best framework" for everyone, or even every research question. Within a family of questions, researchers may disagree about which frameworks to use. You might find that you need to use multiple frameworks to best connect all the parts of your research question with your data. That's ok. You might find that combining multiple frameworks suggests research questions. That's awesome.

A quick caveat: I'm using "Theoretical framework" in a loose sense to include things like knowledge-in-pieces, communities of practice, speech genres, models of institutional change, error-based learning, etc. (I've used all of these in the last 3 years, but there are a lot more out there.) Some people use the phrase "theoretical-methodological framework" to acknowledge that good frameworks must tie theory, methods, and data together. In this article, I'm not going to explore those subtleties.

Methodology
The role of your methodology is to tell you how to generate observations to answer your research question, how to convert those observations into data, and how to analyze that data. While theoretical frameworks are mostly concerned with why those observations and analyses are meaningful or interesting, methodologies are mostly concerned with the practicality of converting observations into analyses.

It is becoming a lot more common in PER to be explicit about the methods that you choose and why. While it used to be sufficient in papers to outline what you did, now you also need to discuss why you did it and how it fits into a broader research tradition.

Many projects -- especially large projects, -- coordinate multiple kinds of data and multiple kinds of analyses in order to make robust conclusions. This is (broadly) called "mixed-methods" or "multi-methods" design. There are lots of ways to mix methods well (and some ways to do it badly). If your research questions demand multi- or mixed-methods, you will need to write sub-research questions and choose theoretical frameworks for each method, and you will need to think about how the analyses from each method will interact to generate new knowledge. Before you jump into a mixed-methods design, ask yourself carefully if your research questions really warrant it, and if your access really allows it.

Sources for theoretical frameworks and methodologies:
  • There are books and papers written on this subject. Some of them are textbook-style for students; others are monograph style for researchers. To find them, you will have to step outside PER and look at the broader educational research literature, the learning sciences, or psychology (depending on your research questions).
    • The Journal of the Learning Sciences has an excellent series on methodology and many beautiful papers on theory.
    • Getting Started in PER has a few papers with brief overviews of some kinds of methods and theories.
    • Probably the most highly-cited book on methods is Creswell's work. It is not comprehensive, but it is extensive.
  • Talk to your advisor or collaborators about what they would use (or require you to use). 
  • When you read papers, make note of their frameworks and methods (and their citations!). 
  • Write a one-page prospectus that outlines what you want to do and why you think it's interesting or important, and send it to someone who does similar work. Ask them (nicely) for suggestions.

Develop the four areas in concert for a specific research project
In this exercise, you're going to iteratively refine each of the four areas so that they are tightly integrated with each other.  
  1. On a whiteboard, write down a preliminary research question. If you don't have a preliminary research question, start with one of the research question templates or do the exercise on making better research questions.
  2. Write down what kind of access you have. Be specific about what populations, what kind of resources you have to undertake this research and how long it will take, and what kind of data modalities are available to you.
  3. If you're structurally constrained (by your funder, or your advisor, or your equipment) to use particular methods or theories, write them down as well. 
  4. Return to your research question, and update it so that it is constrained to the populations you have access to (as well as other structural constraints). 
  5. Which theories support your research question? Write them down. Amend your research question to explicitly reference at least one theoretical framework. If your question is about how individuals develop, you might look at the Resource Framework. If it's about how communities form, try Communities of Practice. If you don't know any theories, what have you read that makes you think this would be an interesting research question? You might need to use two or three frameworks in concert with each other to fully answer your research question.
  6. What kind of observations will you collect? Surveys, interviews, artifacts (like homework) and observations (e.g. of classrooms) are the most common in PER. Make sure that your access permits this kind of observation, that your theoretical framework will be able to use the data from it, and that it will be able to answer your research question. Amend your research question and theoretical framework(s) to reflect the kind of observations you will collect. You might triangulate across several different kinds of observations: preliminary surveys will identify participants for in depth interviews, and you ask them for their homework, for example. 
    1. If you are using surveys, are you generating them yourself (how validated?), using an off-the-shelf survey (why that one?), or modifying an existing one (why and how?) 
    2. If you are using interviews, will they be individual or group? what kind of protocol will you use, and how will you develop it?
    3. If you are using classroom observations, how will you position your camera? what kind of interactions do you want to capture, and how will you know if you have captured them?
    4. If you are using archival data, what was the original intent of the person who collected it? Any quality research group will have a large archive of data that isn't fully analyzed, or hasn't been analyzed for your research question. Always look to old data before you go to collect new data! You could save yourself a lot of time and expense in data collection and processing.
  7. How much data will you need to collect or analyze to show the effects you are looking for? Part of the answer to this question is about where you plan to publish your results at the end of your study: if you want to exhaustively prove your solution, you need a lot of evidence, but if you are only looking to prove its existence, you don't need as much. Even a thoroughly theory-driven, theory-generating project needs something data-like (reinterpretation of old data, for example).
    1. If your project is based on finding patterns of human behavior, there are formalized methods for estimating effect sizes. A quick-n-dirty rule of thumb is that your error bars will go like 1/sqrt(N). If you can estimate differences in your treatments based on the literature, you can guess about how many subjects you will need. If your estimates suggest you will need many more subjects than you have access to, you need to revise your research question.
    2. If your project is based on finding cases of human behavior, you will need to think carefully about episode selection. How many episodes will you need to prove your point substantially? A good rule of thumb is 3-5, most of which should be similar and one of which should be contrasting. More or fewer are possible.
  8. Adjust your research question and methods in light of how much data you will be able to generate.
  9. Write down a preliminary data collection and analysis plan. You may find that drawing a logic model is helpful. You may find that a narrative of what you're planning to do and how is helpful. Compare your plan with your chosen theories and research question. Does your plan make use of your theories? Is it likely to answer your research question? Is it possible with the time and resources you have allotted?
  10. Imagine that everything in data collection goes swimmingly and all of your data are fantastic. What does the answer to your research question look like? To what extent can you answer it with your methods and access? If course, you won't know exactly what the answer will be -- if you already knew, it wouldn't be research -- but you should be able to guess at an approximate shape to the answer. If you think you'll need additional kinds of data to better triangulate an answer your question, amend your access and methods. If you think you'll need a lot more data than you can get, amend your research question. 

Exercises to develop your skill with designing research projects
These exercises will develop your skill in designing research projects. If you do them a lot, then designing research studies will become a habit for you.
  • When you read papers, imagine using their theory and methods with a different population, or using their access with different theory and methods, or their research question with different access and methods. Make notes about your choices, so that later you can cite these papers in your own work. This exercise also makes you a better reader of papers.
  • Read through the abstracts of NSF's recent awards for either IUSE or ECR. For every project, imagine that you have been given a supplement to do some research related to that project. What would be interesting? What would be possible, but not personally interesting? What would be exciting, but you don't know very much about? You should be able to find something personally interesting or exciting in almost all of the projects. Design a study for each. This exercise also makes you a better citizen of the broader DBER community, because you will know a lot more about the shape of current work in the community.
  • Read through the NSF's upcoming deadlines and design a study for every program sponsored by Education and Human Resources (EHR), particularly the DUE and DRL divisions. This exercise also makes you a better researcher, because you will be more knowledgable about how to frame your work to get funding.
  • Read this delightful piece by the former editor of Sociology of Education.
Comments