Survey Guidelines

Institutional Research, Innovation, & Strategy (IRIS) is responsible for administering major institutional surveys as well as overseeing survey activity on campus to ensure projects are in alignment with University priorities. IRIS serves as the primary steward of survey data on University of Notre Dame populations. As part of its survey oversight responsibilities, IRIS will work with researchers and campus partners in an effort to:

Protect the rights and privacy of potential survey respondents
Facilitate the development and administration of high-quality survey tools
Minimize over-surveying of University groups
Maximize accuracy and effectiveness of survey methodology
Eliminate the collection of duplicate information
Ensure effective dissemination of survey results within the Notre Dame community
Promote the use of Qualtrics, the University’s online survey tool

Note that these guidelines apply to any surveys of the Notre Dame community, including students, faculty, staff, alumni, and parents and friends. For all other surveys, faculty should contact Notre Dame Research, and students should consult their faculty advisors.

TERM	DEFINITION
Research Question	The primary inquiry you are trying to address through conducting an investigation
Survey Instrument	A document that contains the full body of survey questions, survey directions and respective response options as well as the language for the introduction page, including instructions and informed consent, and text for the submission confirmation/thank you page. This document should include headers to identify sections of questions and indicate question types (e.g., Likert-scale, rank order, dropdowns, open-text response, “select all that apply”).
Communications	Messages used to solicit survey participation, including the initial invitation, reminders, and prenotification announcements. The document containing the text for the messages should also include subject lines, the dates and times emails are to be sent, who the emails come from, where questions or comments should be directed, and mention the voluntary nature of survey participation. If using incentives, details should be provided in the invitation email.

I. OVERVIEW

In general, the survey process includes 7 steps from start to finish. These steps include:

Survey Preparation
Identifying Target Survey Population
Project Timeline
Survey Design
Obtaining Appropriate Approvals
Data Collection: Survey Administration Options
Data Analysis

Please select the tabs below to learn more about each step in the process of designing and implementing a survey at the University of Notre Dame.

II. SURVEY PREPARATION

Before launching a new survey project that involves Notre Dame population(s), all researchers should discuss their needs for survey data with IRIS consultants. IRIS serves the University of Notre Dame community by providing guidance on how to best move projects forward and on best survey practices.

As a first step, clearly define the purpose of your project and your guiding research question(s). In addition, identify the target population (who will be taking the survey) as well as the scope or timing of the project (e.g., in the Fall or Spring semester). Not only will these factors serve in drafting a focused instrument, they will also help determine whether a survey is the best method to collect the information you need. For example, the data you seek may already be available, either through existing surveys or through corporate data (University records). In addition, you may consider collecting information in another format, such as focus groups or interviews.

As you develop your research question(s), be sure to consult colleagues and relevant decision-makers for their insights. You should also review the literature on your topic to see how other researchers have structured their studies and framed questions. In addition to utilizing University databases to research your topic, professional associations in your field of practice may also feature current literature.

Once you have defined your inquiry, IRIS will help answer your questions based on already available data or will provide guidance on the best ways to obtain such information. Furthermore, some projects may be accepted for implementation by IRIS.

Examples of projects that IRIS could accept for implementation include:

Surveys that have a University-level institutional impact (e.g., senior outcomes, campus climate)
Surveys conducted to meet the University’s legal reporting requirements (e.g., Title IX, placement of graduates)
Large-scale program assessment (e.g., First Year of Studies)
Survey projects that require multi-unit coordination (e.g., ImproveND)

Note that IRIS may also recommend adding your questions to an existing survey rather than administering a separate survey.

III. IDENTIFYING TARGET SURVEY POPULATION

Identifying who will be participating in your survey, as well as the target sample size, is an important part of the survey process. Please consider the following:

Requesting Population Samples

If a survey or focus group project requires access to a population of students, faculty, or staff, administrative units, faculty, and student groups can submit requests to IRIS for random samples. For student samples, please review the Guidelines for Authorizing Access to Notre Dame Students for Research Purposes. For faculty and staff samples, IRIS will provide guidance on how to route the request for access authorization. Requests involving off-campus populations (e.g., alumni) should be submitted to University Relations.

In order to generate samples and provide email addresses for survey and focus group studies, IRIS requires the following:

Description of the purpose of the study, target population, proposed timing, and administration method (web-based, paper, face-to-face, etc.)
Copy of the survey instrument or focus group discussion plan
Survey/focus group approvals as described in the above table (provide documentation).

As a general rule, it can take IRIS up to two weeks from the time of approval to produce a simple random sample. Complex stratified samples may take longer as IRIS capacity permits.

IV. PROJECT TIMELINE

In order to guarantee the success of your project, the timing of the project, from start to finish, as it coincides with other surveys and the work of IRIS and other affected units should be considered.

Timing of Survey Administration

Consulting with IRIS will help determine the best timing for your survey. IRIS maintains a calendar of planned surveys for the year which should be consulted before scheduling your survey. Surveys that target the same population can lower response rates and overburden respondents if they have overlapping or back-to-back administration windows. IRIS will suggest a survey window and will also add your administration to our calendar in an effort to support your survey campaign.

V. SURVEY DESIGN CONSIDERATIONS

Writing Survey Questions

Survey instruments should be kept as short and simple as possible and stay focused on the given topic. Length and complexity can deter respondents from completing a survey. Furthermore, a negative experience may deter people from participating in other projects as well, so it is important that all survey instruments at the University adhere to well-established design principles. There are many resources for writing survey questions including these simple tips from Qualtrics:

In general, use language that your audience will understand and that will resonate with them; be wary of any leading or biased wording; make sure response options are mutually exclusive and cover the full spectrum of experiences; and keep the survey as short as possible (under 15 minutes is ideal). You may also consider looking up other survey instruments on your topic to get ideas for questions to ask. Be mindful of copyrighted materials if you take this approach.

Remember that the University may already hold some of the information you are interested in collecting (e.g., participants’ sex or class standing). If you are interested in asking for this information in your survey, you should be mindful of Notre Dame’s data definitions for the sake of consistent messaging and clarity for your audience.

If you are consulting with IRIS on survey design, you may share an outline or list of question ideas, and IRIS can provide a first draft of the survey instrument for your review, or review the questions you prepare. It is important to recognize that IRIS does not utilize experimental methods with surveys. That is, IRIS never implements conditions—such as deception or blindness—on participants. All surveys are administered with transparency.

Plan Reports In Advance

Early in the survey design process, you should also develop a plan for how you will report the results to different audiences, namely key stakeholders. This plan should include how the data will be used, what decisions it will inform, and the format of sharing the results (e.g., presentation, executive summary, electronic reports).

Once you have digested the key findings with relevant stakeholders, you should also report results in a way that is accessible to the population that you surveyed. The greater transparency researchers practice, the more our community will find value in participating in all of our survey projects.

Distribution Method

There are different ways that you can administer a survey to your participants. It is important to select a method that is the best fit for your project and audience. Note that the first three methods below can be set up using Qualtrics. You may request support from IRIS or from OIT if you need help or advice in selecting or setting up the survey distribution. Note that help from OIT should only involve navigating Qualtrics; not creating a survey sample (i.e., only IRIS should provide help with contact lists).

Email: Send each participant an email invitation with a personalized link to the survey. It is important to let participants know not to share their unique links or forward their emails to others; each survey link is meant for one individual and can only be used to complete the survey once. Reminders are automatically only sent to those who have not completed the survey.
General Link and Authentication: Invited participants can access the survey using a general link that can be posted on a website, embedded in a QR code, or shared through a general announcement, whether in print or on social media. Authentication must be set up in the survey flow so that respondents will either gain access through their Okta login (Single Sign On), or they can use a unique password, such as their email address and/or ID number, to access the survey. Participants can only take the survey once through the authentication process.
General Link: No authentication process is required to access and complete the survey. This means that anyone with the link can complete the survey, not necessarily only the intended audience. It also means that people can complete the survey more than once. Finally, with general links, reminders will go to all members of the target population, not just to those who have not yet completed the survey. This type of distribution method is usually employed only under special circumstances due to numerous possible confounds associated with data collection in this manner.
Paper Survey: Print out copies of your survey and distribute to your participants. Responses will then need to be entered into a digital format (e.g., electronic survey platform, Excel spreadsheet) for data analysis.

Testing Your Survey

Prior to launching your survey, you should conduct pretests. This may include several colleagues outside of the design process; ideally, it would involve a small sample of the intended audience for the survey. Pretests will give you an opportunity to see how long the survey takes, whether there are any functionality issues, and if there is any confusion in answering certain questions. See 7 Ways to pretest your survey before you send it, available on Qualtrics, for more details.

The Rights of Respondents

IRIS adheres to the Association of Institutional Research’s Statement of Ethical Principles. Survey invitations and/or introduction language should provide the facts of the study and informed consent. This should clearly identify the group or person who is conducting the survey and provide contact information (e.g., name, email address, and telephone number) for respondents who have questions about the survey or how the survey results will be used. Below is language that IRIS has used in the surveys it has administered on behalf of our University partners:

“This survey is being administered in partnership with the Office of Strategic Planning & Institutional Research (IRIS). All responses will be held in strict confidence, and your name will not be associated with your responses. Survey results will only be reported to [the Committee] in a summary format, and no attempt will be made to identify any individual respondents. All comments will be reported to [the Committee] unedited.”

“Please be assured that all individual answers will be held in the strictest confidence. The Office of Strategic Planning & Institutional Research is responsible for data management and reporting. No one will examine your individual answers to any questions, and survey results will not be reported in a manner that could identify individuals. Aggregated results will be shared with the university community.”

Survey participation must not be presented as mandatory. Those invited to complete a survey should be able to decline to participate for any reason or, if they are in the process of participating, to stop at any time. They may also refuse to answer any individual questions. Even after signing a consent form, they should be permitted to stop. Should they decide to decline to participate or stop participating, this decision must not in any way influence any services or benefits to which they are otherwise entitled. Sample language that can be included in communications and/or the introduction page is below:

“Your participation is completely voluntary. You may stop taking the survey at any time, and you may also choose to skip individual questions. There is no penalty should you decide not to participate in the survey, to exit the survey, or to skip any questions.”

If you plan to connect your survey results to education records (and personally identifiable information therein) for Notre Dame students, you may be subject to Federal Family Educational Rights and Privacy Act (FERPA) regulations. Those designated within the University and acting with a legitimate educational interest are permitted access to student education records as “school officials.” A person must be designated as a school official and must have a legitimate educational interest in an education record before accessing and obtaining such education records. Furthermore, education records must not be reported in a way that would identify individual students. For more information on FERPA, please visit the website of the Office of the Registrar.

Confidentiality vs Anonymity

All surveys of Notre Dame populations must guarantee confidentiality but do not necessarily need to ensure anonymity (differences explained below).

If your survey data will be connected to names or any other unique identifier (e.g., email addresses, NDID, netID), you must pledge confidentiality to your respondents. This ensures that even the researcher(s), who will be able to connect identities to survey responses, will not reveal individuals’ responses when sharing the results with others. Rather, they should inform respondents that data will only be analyzed and reported at an aggregate level. Surveys implemented by IRIS promise confidentiality, excluding cases when disclosure is required by university policy.

If you prefer to conduct the survey such that respondents need not be identified to the researcher(s), your survey is anonymous. Note, however, that anonymity is not ethically necessary. It is also methodologically challenging if not impossible. Anonymity can be particularly challenging if you are collecting longitudinal data as you will need a unique identifier to link a participant’s responses from one point in time with their responses at other time points. Rather than using a personal identifier (e.g., name, netID), you can assign a code to each participant in the dataset, keep a code key in a separate file, then destroy the key at the conclusion of your longitudinal study (i.e., use a strategy that leaves the data the least vulnerable possible). For more on anonymity and other survey research methods, see Learning Research Methods: A Social Studio Approach.

To administer a one-time anonymous survey, turn on the Anonymize Response feature in the Qualtrics survey settings so that respondents’ personal information will not be recorded. To further ensure anonymity, use a general link with no login required rather than emailing unique survey links to each invited participant or requiring authentication on the survey landing page. Again, as noted in the Distribution Method section, a general link without authentication should only be employed under special circumstances due to the various confounds associated with this practice.

Incentives

You may consider offering incentives to survey respondents under certain conditions.

Negative incentives (i.e., penalties for not participating) are not permissible.
You can offer an incentive that is not necessarily based on participation, such as providing a small token in advance when inviting people to participate.
You can also offer group-level rewards based on participation levels, such as rewards for the group with the highest response rate or if a certain threshold is met.
Finally, you may offer rewards on the individual level to all or randomly selected participants.

Note that incentives should be small enough so as not to be coercive (e.g., gift cards to the bookstore, nearby eatery, or Amazon; Domer Dollars; campus gear such as T-shirts or water bottles). It is also important to be mindful of your audience and survey subject matter in selecting what type of incentive, if any, would be most appropriate.

Finally, incentives above a certain dollar value can be construed as gifts and potentially interfere with a student’s financial aid or violate NCAA rules. Raffles or lotteries can also be construed as gambling and may require special documentation. Specific guidance can be provided by IRB and The Office of General Counsel or Athletics Compliance.

Please consult with IRIS if you have questions about incentives.

Sample vs. Census

In designing a survey administration, IRIS will help determine whether the survey should be distributed to an entire population (e.g., all Notre Dame first-year students) or a sample of the population (e.g., 500 randomly selected first-year students). Using a larger population (census or larger sample) allows for a more nuanced analysis of the results; using a smaller sample, however, means fewer people need to be contacted. In most cases, IRIS will recommend and identify the largest-possible sample of the population to use in order to avoid over-surveying the Notre Dame community while also allowing for a more robust analysis.

Note that the number of survey responses you analyze also needs to be large enough to (a) protect the privacy of respondents and (b) be considered representative of the relevant population. To help determine what your minimum sample size should be in order to be considered representative within a given margin of error, see Calculating sample size: a quick guide (calculator included), available on Qualtrics. Note that your sample size will also depend on your expected response rate.

Response Rates

IRIS-implemented surveys generally have a response rate of 45% to 90% for students, 60% to 90% for employees, and 30% to 45% for alumni (note that all of these rates are on the high end for online surveys conducted at most colleges and universities). IRIS recommends using the following strategies to maximize your survey response rate:

Prenotification: Let your population know when to expect a survey invitation or when survey access will be available. This is also your opportunity to let invited participants know the details of how to take the survey and to convey the importance of the survey effort.
Recognizable Sender: Communications should come from a name invited participants are familiar with and respect so that they are inclined to view the message and respond to the request.
Customized Salutation: Like messages coming from a recognizable name, using a personalized salutation (e.g., Dear Sally) in an email invitation can evoke pressure to comply with the request to take the survey. However, using a general greeting (e.g., Dear Students) can have the benefit of conveying a layer of anonymity if that is desirable for the subject matter of the survey.
Reminders: Send reminders to invited participants who have not yet completed the survey. Remember that timing reminders is important—avoid holidays, weekends, academic-year breaks, and other times when your audience may not be checking their emails. If possible, send reminders when you know there will be a captive audience (e.g., immediately after the survey is referenced in a general announcement).
Incentives: Successful incentives depend on what your population would find rewarding. It may be as simple as promised improvement on the issues addressed in the survey. In terms of material prizes, a small token of appreciation for all participants (or even in advance for all invited participants), often works better than a raffle or drawing for a few large prizes. Knowing your audience and what they value is key (see the Incentives section above).

When designing your survey, it is recommended that you take into consideration the importance of maximizing response rates. Individuals or even sub-populations that do not respond to the entire survey itself or even specific questions within the survey (missing responses to certain questions) may influence the results of your study. This may generate biased findings and could potentially lead to inaccurate conclusions. In this way, the handling of both non-responders, as well as item missingness (limited responses to specific survey items), should be considered at the start of the design process, making efforts to reduce potential issues.

For more tips, see How to increase survey response rates, available on Qualtrics.

VI. OBTAINING APPROPRIATE APPROVALS

Before administering a survey to members of the Notre Dame community, and therefore contacting all or a large share of a certain population, it is important to notify and get approval for your project from those responsible for that population (see first table below). Who you need approval from also depends on the purpose of your survey (see the second table below). Note that scholarly research projects will need to go through the Institutional Review Board for approval or exemption.

Population	Approval Needed
Faculty	Provost if across Colleges/Schools; OR relevant Dean if specific to one College/School
Staff	Human Resources if across divisions OR relevant Vice President or Dean if specific to one College/School/Division
Students	a. For students defined as members of a specific academic unit: dean, department chair, etc. of that academic unit b. For varsity student-athletes: Director of Athletics c. For all other student body subgroups: VP for Student Affairs and VP and Associate Provost for Undergraduate Education
Alumni	University Relations

Note: Depending upon the specific nature of the survey, approval from other parties may be needed in addition to the groups listed in the table above.

Purpose	IRB Approval Needed
Internal Program Assessment	No
Research, Intended for Publication	Yes

Institutional Review Board Approval

You and the University may incur legal liability if the treatment of survey participants is unethical if data resulting from your survey are misused, or if any part of the survey violates certain protected rights of individuals. Survey researchers should be aware of their responsibilities and make every effort to protect the rights of survey respondents.

The Institutional Review Board’s (IRB) major role is to safeguard the rights and welfare of all human subjects who participate in research. Therefore, in compliance with federal law and institutional policy, research projects involving human subjects or materials collected from humans (data, samples, etc.) must be reviewed and approved by the IRB.

There are certain cases, however, such as studies collecting data for educational assessment or administrative purposes, when survey research may be exempt from a full formal review by the IRB. Note that it is still necessary to submit your proposal to the IRB for them to determine whether an exempt status applies. IRIS manages the exemption process for those surveys it accepts for implementation.

Any research where data could potentially be used for a publication or conference presentation will require Institutional Review Board approval. Please note that IRB approval cannot be granted retroactively and has to be obtained prior to data collection.

Please consult Notre Dame IRB for further guidelines.

VII. DATA COLLECTION: SURVEY ADMINISTRATION OPTIONS

Once you have determined that a survey is the best method to collect the information you need, you can then consider your administration options:

Facilitator	Your Role	IRIS Role	When to use this option
You	Draft survey instrument Use your Notre Dame Qualtrics account to build and administer the survey Define population if requesting contact list from IRIS Analyze results	Consultation on survey design Provide population file if needed Archive data upon request	Faculty and student scholarly projects (including theses) or other investigations intended for publication Assessment projects that your office has the capacity to facilitate
Third-party vendor	Identify vendors that offer instruments covering the topic of interest; explore options Serve as a liaison between selected vendor and University Analyze and interpret data and reports provided by vendor	Provide population file if needed Consult on survey customizations (e.g., additional questions) Serve as a liaison between vendor and University if needed Archive data upon request	Having benchmarking data (comparisons to other institutions) is preferable Specific questions and types of analyses you are interested in are offered and possibly copyrighted by outside group Assessment projects IRIS cannot accommodate
IRIS	Provide summary of survey purpose Draft or revise IRIS’s draft of survey instrument and communications Provide or define population for IRIS	Survey design Survey implementation in IRIS-specific Qualtrics domain Basic reports plus further analysis as requested Archive data	Administrative- and assessment-related research in alignment with the University’s mission and priorities (see further details below).

As a general rule, proposals for new survey projects should be submitted to IRIS at least four months ahead of the proposed implementation. IRIS manages recurring surveys throughout the year, and this advanced notice allows us to organize our work to accommodate the new request. Furthermore, the process of drafting, revising, and acquiring input from relevant stakeholders on the instrument and communications may take many weeks of coordination. Finally, it is necessary to build in time to implement the survey in Qualtrics, review and pilot the prototype, and make final revisions before distributing the survey to invited participants.

For studies that involve recurring survey administrations (e.g., annual or end-of-term surveys), or longitudinal studies, the client should contact IRIS and confirm their plan to continue the project at least a month in advance of each survey launch if they expect no changes to the existing instrument and administration process, or at least three months in advance if the client proposes modifications. IRIS will conduct periodic reviews of these ongoing projects to ensure that the instrument, frequency of administration, allocation of resources, and how the results are used continues to meet the needs of the client and the University. IRIS will review projects involving recurring survey administrations after three years of administration to determine if there is still a need to collect the given information or if modifications should be made going forward.

VIII. DATA ANALYSIS

Analyzing and Reporting Results

After your survey responses have been collected, you can conduct simple analyses by using Excel or Google Sheets, and showing results in tables and graphs (e.g., frequencies for each question, cross-tabulations of responses across subsets of respondents). More advanced visualizations can be done in Tableau, while more advanced statistical analysis will require software such as R, SPSS, SAS, or Stata. All of these tools are available to the Notre Dame community (download software), but require some training. Please note that OIT provides training opportunities for Excel, Google Sheets, and Tableau. IRIS primarily uses R and SPSS. If IRIS has administered the survey for you, it can provide reports, starting with overall frequencies and means, as well as relevant breakdowns and comparisons. As a further step, IRIS can conduct more sophisticated analyses, such as significance testing, correlations, or regression modeling upon request. Depending on schedule and availability, IRIS may be able to analyze surveys that are hosted independently.

Small Cell Size

In order to protect the privacy of your survey participants, you should not report results for very small groups of people. For example, reporting responses for two students who have a given minor would both compromise their identity and would mean drawing conclusions from too small of a sample. On the other hand, if you have a low response rate for a certain group, such as 25 faculty members out of a group of 500, it would be deceptive to extrapolate findings from those few responses. IRIS only reports summary results and uses the standard practice of only reporting results for group sizes of 5 or greater.

Archiving Data

IRIS will archive several components of the project including the instrument, raw data, and reports for completed client projects. IRIS maintains a secure server for University data and reports. It is also possible to archive your data with IRIS, even if IRIS did not administer the survey on your behalf. Having us archive your data and reports can be useful so that results can be accessible for future discussions and decision-making at the University especially if relevant parties change or take leave. Data from such projects, in consultation with the originating unit, may be used for strategic analysis and/or internal presentations by IRIS.