Data and Access

Image: St. Thomas, US Virgin Islands

New education researchers are often curious about how much data they need, or whether their small classes are big enough to collect enough data.

Let's talk about common data types and quantities in education research, and how to match your data types to your research questions. There are tons of resources which go into a lot more depth than this quick introduction, and there are a lot of details omitted and subtleties elided here. It's great to use this resource as a starting point to help you frame your work. You'll want to go more in depth for the particular data types which interest you.

What kind of data should I collect?

You should collect data which helps you answer your research question and which helps you generate new research questions to ask. There isn't a single "best" kind of data to collect, only data which matches research questions well (or poorly). However, most education research projects tend to settle into the following major data types (even though other types are possible).

For small projects or exploratory work, you might only use one data stream to get you started. As projects grow in scope and complexity, many projects use multiple data streams to help triangulate answers to their research questions. Combining different data streams together (of different data types or multiple streams of the same type) is standard practice.

What kinds of data are frequently used in education research?

Surveys

Surveys are usually administered on paper or electronically. They are formed of a series of predetermined questions, often free-response or multiple choice. Some surveys have branching logic, (e.g. if participant answer X, then show question A; if they answer Y, show question B), but many surveys do not.

Surveys are a great choice ...

  • to bring prevalence information to rich qualitative data.

  • for screening potential interview participants.

  • to replicate other studies or compare across groups.

Surveys are a poor choice ...

  • when you have few participants

  • if you don't validate them

  • as a sole source of information in a project.

Other considerations about surveys

Some people use a survey to screen participants for some other research participation. For example, you might survey all students in a class to identify students with particularly interesting ideas for follow-up interviews (or just interest and availability for interviews), or you might conduct followup surveys of all your interview participants one year later, to see how their ideas have developed.

Other people use surveys as a primary data source about student ideas. For example, you might administer a concept inventory to all students before and after instruction, so that you can see how their ideas are changing. Alternately, you might want to look for differences in student beliefs about science as a function of student demographics or institutional characteristics.

  • Where can I get them?

    • If you know which one you want, write to the author

    • If you want to know what's available, do a literature review

  • Can I make my own?

    • This is very common if the purpose of your survey is very local, or if you're using the survey information primarily to screen participants for some other research participation. You should check that participants are responding to your questions in ways that you understand, but you don't need to go through an extensive validation process.

    • Alternately, some researchers want to develop a new concept inventory or generalizable research survey, for use beyond their particular local context. Validating a new survey about student ideas can be a really extensive process without much payoff at the end, especially because new survey developers often overestimate the desire of other faculty to use their developed surveys. While building a new research survey is an appealing idea to many new researchers, it is rarely actually a good choice. It's often better to start with qualitative analysis of written / verbal information, and use your initial research to motivate other extensions or generalizations of your work.

  • If you use surveys, you might need to use statistics to understand your results quantitatively.

    • Some people are really excited about this. Some people are intimidated.

    • If your research design might require the use of statistics, then you need to conduct a power analysis (or similar quantitative estimation) to guess at how many participants you will need in order to see the differences you expect. If your power analysis suggests that you will not be able to detect differences with the group of participants you have, then surveys are unlikely to be a good use of your time (or the time of your participants).

    • (of course, if you're using a survey to collect screening or scheduling information about participants, statistics might be excessive or nonsensical).

Interviews

Interviews are characterized by real-time human interactions, usually between an interviewer (who asks specific questions from an interview protocol) and an interviewee. Some interviews have more structure to the questions, while others flow more like a conversation.

Interviews are a great choice ...

  • to probe how individuals/groups think about a concept or procedure

  • to probe how people reflect on a situation or experience

  • for validating potential survey questions

Interviews are a poor choice ...

  • when you aren't quite sure what you're looking for

  • if you need to see what happens in classrooms

  • if there's a big power differential between you and your participants

Other considerations about interviews:

  • There are lots of different kinds of interviews.

    • Think-aloud, problem solving, focus group....

  • Practice your protocol!

    • You should plan to try it out on 2-3 people, then revise the protocol, then try it out again.

    • Your IRB will want to see the final protocol after revision, before data.

  • How will I know how many people to interview?

    • Exploratory interviews to see what kinds of ideas are available: 3-5

    • Saturation to find all the ideas: emergent process, but usually 10-15

    • Population: all the people in your population (unusual, but possible)

  • Generally, you want to overestimate how many you'll need

Classroom artifacts

Classroom artifacts include anything the students are doing as part of their work in the class, for example copies of their homework or exams, or pictures of their whiteboards. It also includes artifacts that instructors generate, like slides or problem sets, as part of their work in the class. It doesn't include anything that researchers specifically ask students to do for research purposes only.

Classroom artifacts are a great choice...

  • because they are cheap and easy to collect

  • when you are curious about what happens in real classrooms

  • because they usually permit reanalysis for multiple projects

  • as an additional source of data for a larger project

  • when you want to explore new ideas

Classroom artifacts are a poor choice...

  • when you need to find out why students do something

  • if you want to test a new idea or strategy

Other considerations about artifacts:

  • IRB for these can be tricky to navigate

    • IRBs are very different in their interpretations of the regulations

    • Post-fact permission for classroom artifacts is ethically fraught.

  • You will see awesome things ...

    • ... and then not be able to follow-up with those students

  • Artifacts are common in archival data

    • Meta-data is very important in understanding artifacts.

    • Always put in more meta-data than you think you need.

Classroom observations

Classroom observations are a great choice for when you want to see what real students and teachers are really doing in real classrooms. Broadly speaking, collecting observational classroom data means recording what happens in a classroom. Video is the gold standard for recording, but it's also possible (and sometimes desirable) to collect only audio data, or only field notes from observers. In online or computer-based contexts, classroom observations usually take the form of screen recordings or zoom recordings; things like their written work are more often considered artifacts.

Classroom observations are a great choice...

  • because the data are rich and multifaceted.

  • when you need to see what "really happens"

  • if you're interested in how people interact with each other or with equipment

  • early in a research project, to help you generate new ideas

Classroom observations are a poor choice...

  • when you want to be able to probe participants' ideas in the moment

  • if you cannot also collect classroom artifacts to understand what they're doing

  • for research questions that depend on controlled conditions or individuals acting alone.

Other considerations with classroom observations

  • Some observations are highly unstructured and emergent

    • This is more common in the early stages of a research project, to help you generate new ideas and questions

    • Video data is commonly used to support repeated viewings of the same interaction to gain insight.

  • Some observations are highly structured and deterministic

    • There are observation protocols which can be used in live classrooms or with classroom video.

    • They are more common in situations where video recordings are difficult to obtain (e.g. with children or in the EU), or where large quantities of classroom time need to be analyzed to find patterns.

  • Archival videos of classroom observations are super common, but the ethical considerations of how to analyze the video are complicated.


Reflections

Reflections include writing that researchers generate to systematically document or understand a setting or the people in it. They can include your generative writing or reflective memoing. The central feature of this data type is that you, the researcher, generate this data.

In contrast, analyzing reflections written by your research participants (either as part of a free-response survey or as assignments for their classes) is another data type.

Reflections are a great choice...

  • If you want to describe a curriculum or the implementation

  • If you have a dual role as an instructor and as a researcher, and your research is about your own growth in teaching.

  • as one of several data streams, to help contextualize them or tie them together.

Reflections are a poor choice...

  • as a sole stream of data

Archival data

Archival data are research data which already exist. This is not the same thing as classroom records which are sitting around in the back of the file cabinet! For a data set to be archival research data, it must have been collected as research data and (still) covered under a relevant ethics approval.

Archival data are a great choice...

  • when you want to explore ideas and possibilities

  • when you need to write a paper quickly

  • if your research question is about development over a long time

  • if you can't access new data

  • if you want to make arguments about baseline vs. change

Archival data are a poor choice...

  • when your research questions are very rigid or specific

  • when they are very poor quality or missing crucial metadata

  • if the IRB which covered it has expired

Other considerations with archival data:

  • Can I use old exams / homeworks / etc as data?

    • This is an IRB question.

    • If they aren't already research data, then probably not.

  • You get what you get.

    • sometimes, the data are not well-aligned to your research questions

    • you cannot ask follow-up questions of the same participants

  • Is it ok to write new papers if parts of the data have already been analyzed?

    • Emphatically: yes.

    • Articulate difference & cite other papers.

  • Who has some archival data that I can work with?

    • Lots of people! Start by emailing the authors of recent studies that you find interesting.

  • The older the data are, the harder it is to remember what's going on.

    • Meta-data is enormously important when building a catalog of archival data or maintaining a data library.

    • You're going to lose data because files get corrupted or whatever, but it turns out that's not the worst. The worst is when you still have some data files and you have no idea what's in them or what the students are doing or when the data were taken or what the answers correspond to.


Other considerations: Small sample sizes

Some research is great for small sample sizes

  • Very rich data / many observations of the same people

  • case studies / exemplars

  • claims about existence

Some research is not

  • Impoverished data (e.g. surveys, homework responses)

  • claims about prevalence

My papers include a study of 50k students over 20 years, and a close analysis of 37s of video in one group where only two people talk. Sample size matters less than matching data with research question.

Examples

These example exercises and discussion prompts might help you think about how to coordinate research questions and data streams.

Emily and alternate classes

Emily thinks that teaching her students explicitly how to craft scientific arguments will help them understand popular press articles in science. She already has an in-class activity to support this, but she wants to develop an online project and quiz.

To show that her new assignment is working, she plans to implement it her 10:30 section, but not in her 9:30 section. Both sections will take the quiz.

  • What kinds of data could Emily collect? Be specific, e.g.:

    • not just "surveys": from whom? how frequently? on which topics?

    • not just "registrar data": what pieces of information?

    • not just "classroom artifacts": which ones? on which topics?

  • Oftentimes, in considering this scenario, people have an implicit research question in mind for Emily to pursue. What research question is Emily pursuing? For this research question, is she likely to be able to collect enough data? If not, it's time to develop a different research question that matches the amount of data she can collect.

There are some ethical issues with what Emily wants to do, and when you're designing a research project you need to attend to ethical design issues as well as ethical data collection. As you consider Emily's project, think about what ethical issues arise in data collection. Think ahead to how Emily will need to apply for ethics approval (IRB) to collect this data. How might she anticipate the kinds of questions her ethics board should ask about this study?

Arthur and baseline data

Arthur's department teaches an introductory class with 8 different instructors and 10 different GTAs. The department is contemplating making big changes to the format and coverage of the class, and has tasked Arthur with collecting baseline data so that they will know if the changes are working.

  • What kinds of data could Arthur collect? Be specific, e.g.:

    • not just "surveys": from whom? how frequently? on which topics?

    • not just "registrar data": what pieces of information?

    • not just "classroom artifacts": which ones? on which topics?

  • Some baseline data will come from the upcoming year, before the department makes changes to their teaching. Other baseline data concerns semesters which have already passed. What kinds of retrospective data might be available ethically?

  • What do you think Arthur's department (or Arthur), means by "if the changes are working"? Working for whom, and in what ways? Think about what kinds of research questions Arthur could pursue. and how those are supported (or not) by each of the data types he could collect.

For each of these scenarios, working on which kinds of data to collect or how much data will be possible suggests that you should adjust your research questions to better match your access to data. This research design process of iteratively refining your research question and data collection plans is important and central to my work as a researcher.