Guidelines for designing a research project

Author: Dr Simon Moss

This document specifies a series of tasks and activities that students or researchers should complete to optimise their research projects. Although relevant to researchers, the document is primarily directed to fourth year and postgraduate psychology students. These principles should be regarded merely as guidelines, not inflexible rules.

Part 1 - Clarify your interests

Task 1. Have you collected enough information from previous literature?

According to recent research, the most effective, progressive, and informative projects integrate ideas and constructs from several related, but distinct, topics. In other words, you should:

Initially, glean a series of interesting arguments and findings from the literature.
Then, attempt to integrate these ideas and findings into a unified framework.

Typically, not all of these ideas and findings can be integrated. Nevertheless, attempts to integrate many diverse concepts tend to stimulate innovation and thus progress. Therefore, before you finalise your topic, you should collect a broad gamut of ideas and fact. For example:

You should collate suggestions for future research from articles you perceive as interesting.
In addition, you should uncover some counterintuitive, illuminating empirical findings in these fields. That is, you should collect findings in which the opposite outcome would have been equally, if not more, plausible. To illustrate, some studies have revealed that pressing up on a table enhances creativity, a finding that is clearly unexpected rather than inevitable.
Finally, you should collect the explanations that were utilised to accommodate these unexpected findings.
You should, however, focus on explanations that are theoretical. In other words, you should collect explanations that could apply to many contexts and settings. You should not focus on methodological explanations that seem specific to one study but do not apply to other contexts, settings, or approaches.
In summary, you should:
Collect between 5 and 10 suggestions for future research in literature that corresponds or relates to your interests
Collect between 10 and 20 counterintuitive, illuminating empirical findings.
Collect theoretical explanations that were proposed to reconcile these findings.

Task 2. Have you specified a range of independent and dependent variables?

Once you have completed the first task, you should consider the independent and dependent variables that you would like to explore. For example, you might want to examine the effect of body odour on the extent to which individuals are perceived as attractive by onlookers. In this context, body odour can be regarded as the independent variable, or cause, and attraction can be regarded as the dependent variable, or outcome.

You could select independent and dependent variables that have either:

Seldom--if ever--been studied simultaneously, but nevertheless could be related.
Often been studied simultaneously, but have spawned conflicting findings.

Often been studied simultaneously, but the sequence of events that underpin their association has not been established definitively.

The number of independent and dependent variables varies appreciably across research projects. Most large theses, such as doctoral projects, comprise between 2 and 5 sets of independent variables. For example, the independent variables might include personality, which might comprise 5 distinct traits, and coping, which might comprise 3 distinct styles. The total number of independent variables might thus range from 6 to 30 depending on the number of subscales associated with each set. Most large theses comprise between 1 and 4 dependent variables, such as self esteem and wellbeing.

Most smaller theses, such as fourth year projects, comprise between 1 and 3 sets of independent variables. The total number of independent variables might thus range from 4 to 18. These projects also comprise between 1 and 3 dependent variables.

In summary, you should:

Ensure the study entails an appropriate number of primary independent and dependent variables.
Specify the independent and dependent variables you plan to explore in the space provided.

Task 3. Have you considered variables that might mediate these relationships?

Suppose you were interested in whether or not body odour influences the extent to which individuals are perceived as attractive. In addition, suppose a series of studies have shown that individuals seem more attractive after smothered in dencorub. This finding, alone, however is not informative. In particular:

This finding does not ascertain why dencorub enhances attraction.
As a consequence, we cannot determine whether or not this finding would persist in other settings.
Indeed, we cannot even determine whether or not this finding would persist in the same setting tomorrow.

Somebody might retort, "The day of the week should not influence this finding. Therefore, this finding should persist tomorrow". This retort, however, assumes that day of the week is irrelevant. This assumption, however, assumes some understanding of why dencorub enhances attraction--an understanding that cannot be gleaned from the finding alone. The optimal method to develop this understanding is to explore the sequence of events that intervenes between the independent and dependent variable.

First, you need to uncover possible explanations or mechanisms that could intervene between the independent and dependent variable.

In this example, perhaps when individuals apply dencorub, they are perceived by onlookers as active and thus attractive.

Second, you need to identify the mediators this explanation entails.

In this example, the explanation implies that perceived level of activity mediates the relationship between dencorub and attraction, as illustrated below.

Finally, you need to either measure or control this mediator. For instance, you could create a scale that measures the extent to which participants are perceived as active. Alternatively, you could control this mediator methodologically. The individuals who are rated might be athletes only, whose level of activity will be perceived as high. If this variable is indeed a mediator, controlling that variable, either statistically or methodologically, should eliminate the relationship between the independent and dependent variables. This finding would thus provide an insight into the pathway that links the independent and dependent variables.

In summary, you should, if possible:

Identify several explanations that describe why the independent variables influence the dependent variables
Each explanation implies the independent and dependent variables will be mediated by specific variables. These variables are called mediators..

Part 2 - Assessing alternative explanations

Task 4. Have you considered spurious variables that influence both your independent and dependent variables?

Suppose you discover that your independent variables correlate with your dependent variables. For example, individuals who wear dencorub might be perceived as more attractive. This finding does not necessarily indicate that dencorub enhances attraction. Instead, a spurious variable, which is a factor that influences both the independent and dependent variable, might underpin this finding. This concept is illustrated below.

To illustrate, suppose that young individuals are more likely to both apply dencorub and appear attractive. However, suppose that dencorub itself does not influence attraction. The table below provides some hypothetical results. The first column specifies the age group of participants. The second column specifies the number of times each year they apply dencorub. The third column specifies the extent to which they are perceived as attractive, on a scale from 0 to 20. The top set of rows comprises young individuals, who will often tend to apply dencorub and seem attractive. The bottom set of rows comprises older individuals, who seldom apply dencorub or seem attractive.

Taken together, these findings suggest that dencorub correlates with level of attraction. That is, individuals who apply dencorub tend to be perceived as more attractive. Nevertheless, when each age group is considered separately, dencorub does not correlate with level of attraction. Thus, to understand the effect of dencorub on attraction, age should be controlled, either statistically or methodologically. Alternatively, participants should be randomly assigned to various conditions, called an experimental design, which tends to ensure these spurious variables are controlled. In short, spurious variables should ideally be controlled. Spurious variables that cannot be controlled should be conceded in the limitations section of your thesis.

In summary, if you do not plan to randomly assign participants to conditions, you should:

Identify variables the literature suggests could influence one or more of your dependent variables, apart from the independent variables and mediators you specified earlier
Identify which of these variables could also influence one or more of your independent variables or mediators. Only consider the independent variables that you are not manipulating yourself.
Variables that influence both your independent and dependent variables are designated as spurious.

Task 5. Have you considered whether or not relationships between your independent and dependent variables could be ascribed to common method variance?

A prevalent, almost ubiquitous, variant of a spurious variable is called common method variance. Specifically, each participant demonstrates a particular tendency or strategy when completing surveys or other measures. For example, some participants always respond favourably or leniently. In contrast, some participants always provide neutral responses. Nevertheless, some participants always respond unfavourably or harshly. The following table presents some hypothetical data. In this example, the first three rows represent lenient participants, the next three rows represent neutral participants, and the final three rows represent harsh participants. Again, the two variables appear to be related& high scores on one variable correspond to high scores on the other. This relationship, however, is spurious and merely reflects the observation that each participant applies the same strategy to each question--a problem denoted as common method variance.

Several strategies can be undertaken to minimize common method variance. These strategies are described by Podsakoff, MacKenzie, Lee, and Podsakoff (2003). Specifically:

First, common method variance diminishes if the questions are concrete and objective. For example, a question such as, "How often do you attend parties alone--once a week, month, year, or less than once a year" is relatively objective. Spies who had watched these participants over many years would provide the same responses as the participants themselves& hence, these responses are objective.

Second, if possible, the questions should refer to current, rather than retrospective, attitudes, perceptions, or behaviours. Retrospective questions, such as "At primary school, did you like the smell of dencorub", are more susceptible to various biases such as attempts to maintain consistency across responses.

Third, you could apply different response scales to measure each variable. A series of 'Yes' and 'No' scales could be utilised to gauge the dependent variables. A series of Likert scales could be utilised to gauge the independent variables.

Fourth, you could measure the variables from the perspective of different individuals. For example, if employees complete the scales that gauge the independent variables, their managers could complete the scales that gauge the dependent variables.

Fifth, you could vary the time and location at which each variable is measured. For example, the independent variables could be measured one day after the dependent variables.

Sixth, you should ensure the scales seem superficially dissimilar.

Seventh, you could encourage individuals to respond honestly. You could introduce procedures that emphasise the anonymity of these measures or express statements such as, "There are no right and wrong answers" or "Please ensure your responses are accurate rather than lenient or harsh". These tactics reduce the likelihood that participants will exhibit leniency towards themselves or individuals they like.

Eighth, you could measure positive affect, negative affect, and mood in participants, perhaps using the PANAS or POMS. You could then include these measures as control variables. In other words, these measures could be designated as covariates - in the context of ANOVAs - or additional predictors - in the context of regression analyses. Hence, affect and mood - which can bias responses - is controlled.

Ninth, you could apply other sophisticated statistical techniques. For example, some researchers undertake an exploratory factor analysis, save factor scores, and control the first factor in all subsequent analyses. Other researchers apply techniques called MTMM or correlated uniqueness models to control common method variance.

In summary, you should:

Identify whether or not the common method variance could contaminate your findings.

This problem is rife if the measures that are applied to gauge the independent, mediator, and dependent variables utilise a similar format. For example, do measures of both the independent and dependent variables entail a Likert scale in which participants specify their level of agreement to a series of statements? If so, common method variance is likely.

This problem is also rife if the questions are particularly subjective and abstract or the answers on one scale might influence the responses to a subsequent scale.

Consider strategies to reduce common method variance. For example, you might need to adapt the items of established scales to ensure they are more concrete and objective, but not retrospective. You could utilise different individuals, formats, times, or locations to measure the independent and dependent variables. Finally, you could apply statistical procedures to nullify the effect of common method variance.

Task 6. Have you considered suppressor variables that might obscure relationships between your independent and dependent variables?

Sometimes, a suppressor variable might obscure the relationship between an independent and dependent variable. To illustrate, consider the model below. According to this model, dencorub directly reduces attraction, perhaps because the odour is offensive. On the other hand, dencorub facilitates fitness, which in turn enhances attraction. These two pathways nullify one another.

As a consequence, the correlation between dencorub and attraction will not reach significance. Nevertheless, if fitness is controlled, perhaps through the inclusion of this variable in a regression, the bottom pathway is obstructed. Hence, the top pathway will prevail. That is, dencorub will be inversely related to attraction. In other words, relationships that become significant only after some variable is controlled confers vital information. This outcome can arise if :

The independent and dependent variables are negatively related to one another, but a mediator or spurious factor is positively related to both variables or negatively related to both factors, as depicted in the previous diagram.

The independent and dependent variables are positively related to one another, but a mediator or spurious factor is positively related to one of these variables and negatively related to the other. In each instance, the top and bottom pathways nullify one another

In summary, you should:

If the independent and dependent variables are positively related, identify mediators or spurious variables that you uncovered earlier that positively correlate with the independent variables but negatively correlate with the dependent variable, or vice versa. These variables are potential suppressors.

If the independent and dependent variables are negatively related, identify mediators or spurious variables that you uncovered earlier that positively correlate with both the independent and dependent variables or negatively correlate with both the independent and dependent variables. These variables are also potential suppressors.

Specify the suppressors you plan to measure or control as well as the suppressors you do not plan to measure or control.

Task 7. Have you considered the possibility that perhaps the direction of causation in relation to your independent, mediator, and dependent variables is the opposite to your hypotheses?

Suppose the researcher hypothesises that dencorub enhances attraction, perhaps because this product fosters fitness. Furthermore, suppose that elevated use of dencorub does indeed correlate with attraction. Contrary to this hypothesis, this finding might arise because level of attraction might influence application of dencorub. Specifically, unattractive individuals might become unconfident and thus decide to refrain from dencorub to enhance their popularity. In other words, the direction of causality might oppose the hypothesis. Accordingly the finding does not support the theory that dencorub fosters fitness and thus attraction.

Researchers can undertake a variety of measures to discredit this alternative direction of causality. First, they could identify and assess mediators that are compatible with one direction but not the other. For example, suppose that fitness level - but not confidence mediates - the relationship between dencorub and attraction. This finding would support the hypothesis that dencorub enhances attraction rather than vice versa.

Likewise, researchers could identify and assess moderators - that is variables that influence the magnitude and direction of some relationship - that are compatible with one direction but not the other. For instance:

Suppose that justification of use moderates the relationship between dencorub and attraction.

In particular, suppose the relationship between dencorub and attraction diminishes in participants who utilise dencorub merely because the odour fosters relaxation.

In these participants, dencorub is unlikely to enhance fitness.

Accordingly, the finding that justification of use moderates the relationship between dencorub and attraction is compatible with one direction only.

In particular, this finding is compatible with the hypothesis that dencorub facilitates fitness and thus attraction.

Alternatively, researchers can refine the research design to distinguish between the two directions. First, researchers could manipulate the independent variable--that is, randomly assign participants to each condition, called an experimental design. To illustrate, half the participants could be asked to apply dencorub and the remaining participants could be asked to refrain from dencorub. Suppose the application of dencorub still correlates with attraction. Clearly, this finding cannot be ascribed to the hypothesis that attraction influences the use of dencorub. Finally, a longitudinal design could be considered. That is:

Suppose that dencorub at time 1 correlates with attraction at time 2 after controlling attraction at time 1.

Nevertheless, suppose that attraction at time 1 does not correlate with dencorub at time 2 after controlling dencorub at time 1.

This pattern of findings supports the hypothesis that dencorub influences attraction and not vice versa.

This argument invokes the rationale that causes tend to precede effects. In summary, you should:

Identify several explanations that describe why the dependent variable could influence the independent variable.

These explanations could suggest the possibility of other mediators.

These explanations could also suggest the possibility of other moderators--variables that influence the magnitude and direction of the relationship between the independent and dependent variables.

If these mediators and moderators are compatible with both directions of causality, consider another avenue to assess the direction of causality. Either manipulate the independent variables or consider a longitudinal design.

Part 3 - Generalizing the findings

Task 8. Do you believe the relationships between the independent, mediator, and dependent variables will generalise to other contexts and populations

Most theories are intended to apply to a particular population or context. For example, the theory that dencorub enhances fitness and thus attraction is probably intended to apply to adults - and not children. Furthermore, this theory is probably intended to apply both inside and outside buildings. Admittedly, these boundaries are implicit rather than explicit, but are nevertheless obvious. Of course, your sample and methodology will not necessarily be representative of this entire population and context. That is:

Your study is unlikely to be representative of all adults, but merely representative, or even unrepresentative, of adults in one city or community.

Your study is unlikely to be representative of all contexts. For example, your study might be conducted inside a University building.

Researchers should determine whether or not their sample and methodology differs from the population and context to which this theory should apply on several variables. For example, perhaps the adults on this sample are less familiar with rank odours than is the population. Alternatively, perhaps the odour of dencorub inside a University building is less salient that is the odour of denorub in most contexts. In other words, of course, the findings of this study might not apply to all relevant populations and contexts. To overcome this problem, researchers can pursue two possible options. First:

Researchers can ensure their sample and context is representative on these variables.

In this instance, the researcher would need to ensure the distribution of 'familiarity with rank odours' in the sample mirrors the population.

Likewise, the researcher would need to ensure the distribution of 'odour salience' in this study also mirrors the range of possible of contexts.

Unfortunately, this option is often impractical.

Second, the researcher could ensure these variables, familiarity with rank odours and odour salience, vary to some extent across participants. The researcher could then measure these variables. Finally, the researcher could assess whether or not the relationships between the independent, mediator, and dependent variables are moderated by these variables. This approach, called a moderated model and depicted below, determines the extent to which the relationships between the independent, mediator, and dependent variables apply to a broad range of individuals and contexts.

In summary, you should:

Identify the contexts and populations in which the theoretical explanations you would like to assess should apply, such as all non-clinical settings, parents, and so on.

Identify the context and populations to which you have restricted your sample, such as large organisations and mothers

Consider variables differ between the context of your study and the breadth of contexts in which the theoretical explanations apply.

Are any of these variables likely to moderate the relationships between your independent, mediator, and dependent variables.

Can you measure or vary these variables and thus examine whether or not they moderate the relationships between your independent, mediator, and dependent variables.

Part 4 - Preventing confounded treatments

For between-subject manipulations only. If your study does not comprise any between-subject manipulations, in which each condition comprises separate participants, proceed to Task 11.

Task 9. Have you considered whether or not compensation, demoralisation, demand characteristics, Hawthorne effects, or some other confounding variable could explain the observed relationships between independent and dependent variables.

Between-subject treatments, in which each condition comprises separate participants, can present several drawbacks. Specifically, these treatments might not manipulate only the independent variable. Instead, these treatments might also manipulate some other confounding variable. This confounding variable could, in turn, influence the dependent variable. Thus, treatments that affect the dependent variable cannot necessarily be ascribed to the independent variable. This sequence of events is illustrated in the figure below.

To illustrate some common confounding variables, suppose some participants are asked to apply dencorub, whereas other participants are asked to refrain from the application of dencorub. Participants asked to refrain from dencorub might feel unfit. To compensate, they might undertake more exercise, which in turn could influence the dependent variable, the extent to which they are perceived as attractive. Although this example might seem contrived, compensation could explain the effect of some manipulation, especially if the participants who receive the treatment could feel complacent (Campbell & Stanley, 1963).

Nevertheless, the participants who do not receive dencorub might feel demoralised. For instance, they might feel upset they had not received the treatment. As a consequence, their motivation wanes or their frustration heightens, which in turn could influence the dependent variable (Campbell & Stanley, 1963).

Along similar lines, participants might decipher the hypotheses. Participants who receive the dencorub might believe they are expected to be fit and attractive, and this belief could impinge upon their behaviour. Likewise participants who do not receive the dencorub might believe they are expected to be unfit and unattractive. This effect of perceived expectations on performance is often denoted as demand characteristics.

Finally, novel experiences tend to foster hope and motivation. Accordingly, the application of dencorub might promote this sense of inspiration, which in turn could influence the dependent variable--a sequence of events referred to as the Hawthorne effect. This sense of inspiration might not persist, however. As a consequence, the finding that dencorub enhances inspiration and thus attraction might represent only a transient effect.

In summary, you should:

Determine whether or not participants in one condition might feel more complacent or unmotivated than participants in another condition. Identify measures that you have considered to minimise this complacency if needed

Determine whether or not participants might decipher the hypotheses. In the space provided, specify measures that you have considered to obscure the hypotheses, if necessary. Alternatively, justify why their recognition of these hypotheses would not influence their scores on the dependent variables.

Determine whether or not one condition involves a more pronounced sense of novelty than another. Identify measures that you have considered to equate this sense of novelty& otherwise, you will need to concede this shortcoming in the limitations section.

Task 10. Could experimenters vary the inadvertent cues they exhibit, or the procedures they use to measure variables, across the conditions.

Sometimes, experimenters attempt to conceal the treatment. For instance, consider the previous study in which only half the participants receive the dencorub. Perhaps, to conceal this manipulation:

Some product is applied to all participants.

Half the participants receive dencorub.

The other participants receive some other product that confers no effect.

Participants wear a gasmask to ensure they cannot decipher the difference.

This approach would preclude many confounding variables.

Nevertheless, the researcher might be aware of which participants receive the treatment. For example, they might exhibit an expression of disgust only when they apply dencorub. Participants might decipher this subtle hint, which could influence their behaviour. Rather than exhibit these subtle hints, researchers who are aware of which participants receive the treatment might not measure attraction fairly. In other words, this awareness could somehow bias the measure. To illustrate:

Suppose the researchers ask onlookers to verbalise whether or not each participant seems attractive.

The researcher who is aware that some participant received the treatment, and thus assumes this individual will be perceived as attractive, might misconstrue the response of onlookers.

Again, although this example is contrived, the possibility that biases could influence subjective measures is genuine. In summary, you should:

Identify measures you will introduce to ensure that participants do not decipher the treatment they have received from the behaviour of experimenters. Perhaps these experimenters who interact with participants are unaware of which treatment each individual receives, which is called a double-blind procedure.

In addition, specify tactics you will introduce to ensure that measures of the dependent variable are not unbiased. This problem is confined to observational data.

Task 11. If the conditions are not counterbalanced, have you considered whether or not time influences the dependent variables (see Between-subject versus repeated-measures designs).

If your study does not comprise any within-subject or repeated-measures manipulations, in which each condition utilises the same participants, proceed to Task 14. Within-subject treatments, in which each condition utilises the same participants, can also be contaminated by confounding variables. For example:

Suppose that onlookers rate the extent to which they perceive participants as attractive on two occasions.

First, onlookers rate the participants before they receive dencorub.

Second, they rate the participants after they receive dencorub.

Suppose that participants are rated as less attractive after they receive dencorub. This finding might not indicate that dencorub tarnishes the appearance of participants. Instead, this finding could be ascribed to other factors that vary across time.

Perhaps the participants aged marginally across time.

Perhaps the participants began to fatigue.

Counterbalancing circumvents this drawback (see Between-subject versus repeated-measures designs). To counterbalance the conditions in this example, onlookers could rate the extent to which they perceive participants as attractive on two days. On the first day, only half the participants will receive dencorub. The remaining participants receive no dencorub. The participants who received dencorub on the previous day would receive no dencorub on the second day, and vice versa, as depicted in the table below.

This approach, however, cannot be applied when the design comprises one participant only, often called single-subject designs. In these instances, the researcher must select one of three options. First, an ABAB design could be applied:

That is, the researcher could repeat the treatment and control conditions on many occasions.

To illustrate, on every second day they could ask participants to apply dencorub.

On every other day, they could ask participants to refrain from dencorub.

Hence, any effects of dencorub cannot be ascribed to time.

Unfortunately, to undertake this approach, researchers must withdraw the treatment during every second session, which is sometimes unethical. Hence, a multiple baseline design could be applied instead. That is:

The researcher could apply two treatments, which are hypothesised to influence separate dependent measures
To illustrate, on the first day, no treatment is applied
On the second day, dencorub, which is hypothesised to enhance attraction, is applied
On the third day, both dencorub and vegemite, which is hypothesised to enhance mood, is applied
If the hypotheses are supported, attraction should be elevated on the second and third day. Mood should be elevated on the third day only.
This rationale can be extended to more than two treatments

Unfortunately, this approach can be applied only when the researcher wants to explore several treatments that influence distinct outcomes. If this condition is not fulfilled, an incremental approach can be adopted instead. For example:

On the first day, no treatment is applied
On the second day, 5 mg of dencorub is applied
On the third day, 10 mg of dencorub is applied
On the fourth day, 15 mg of dencorub is applied
If the hypothesis is true, attraction should increase gradually over time.

These single-subject designs can be applied when the project comprises more than one participant. For example, this design is applicable when the project comprises five participants, but each individual effectively represents a separate study.

In summary, you should:

Determine whether or not scores on the dependent variable could vary across time - perhaps as a consequence of practice, fatigue, or aging. Justify your response in the space provided.
If the dependent variable could vary across time, you should counterbalance the order in which the conditions are presented. Alternatively, you should somehow minimise the effect of practice, fatigue, an aging on the dependent variables.

Task 12. If participants withdraw, have you considered the possibility that later conditions might be biased?

If the conditions are not counterbalanced, another factor that could vary across time is the response rate. For example, suppose again that onlookers rate the extent to which they perceive participants as attractive on two occasions: before and after the application of dencorub. After the first set of ratings, some of the participants might withdraw from the study. These participants might have received unfavourable ratings, and thus might have withdrawn because their confidence had been decimated. As a consequence, the individuals who participated in the second condition thus tended to be more attractive.

The finding that participants in the second condition were more attractive than were participants in the first condition cannot necessarily be attributed to dencorub. Instead, this finding might arise because individuals in the second condition did not represent an unbiased, representative sample.

In summary, you should:

Estimate the number of participants who might withdraw during the study and thus complete only some of the conditions.
Determine whether or not these participants are likely to yield especially high or low scores on the dependent variables.
If so, you will need to counterbalance the conditions. This approach will be effective, provided the likelihood of withdrawal does not depend on the order in which participants complete the conditions.

Task 14. If the conditions are counterbalanced, have you considered the possibility of asymmetric transfer

Counterbalancing, unfortunately, does not eradicate all confounding variables. Instead, a subtle but consequential problem often arises, typically called asymmetric transfer. Consider the method that is represented in the table below.

First day	Second day
First group of participants	Receive dencorub	No dencorub
Second group of participants	No dencorub	Receive dencorub

To explicate asymmetric transfer, consider the participants who applied dencorub on the first day but not the second. The application of dencorub on the first day could alleviate any injuries that participants had recently endured. On the second day, these participants thus feel healthier and thus more attractive. In other words, participation in the condition that involves the application of dencorub might subsequently enhance performance in the condition that involves no dencorub, as represented by the asterisks in the table below.

First day	Second day
First group of participants	Receive dencorub	No dencorub ****
Second group of participants	No dencorub	Receive dencorub

In contrast, consider the participants who did not apply dencorub on the first day, but did apply dencorub on the second day. Participation in the condition that involves no dencorub will not subsequently affect performance in the condition that involves dencorub. Taken together, in the first group of participants, performance in the condition that involves no dencorub will receive an unfair benefit: a problem referred to as asymmetric transfer.

In summary, you should:

Estimate the extent to which participation in Condition A could enhance or impair performance in Condition B
Estimate the extent to which participation in Condition B could enhance or impair performance in Condition A. Extend this rationale if your design entails more than 2 conditions.
If asymmetric transfer is likely, consider a between-subject design instead.

Part 5 - Optimizing measures

Task 14. Have you collected evidence to suggest that your measures are reliable and valid?

Specifically, you should specify some evidence that your measures generate reasonable levels of alpha reliability or test-retest reliability as well as convergent and discriminant validity in populations that are relevant to your study (See also Properties of excellent measures).

Task 15. Have you selected measures that are not too susceptible to social desirability?

Some measures are invalid because the responses of participants align with social expectations or desirability. For example, if asked the extent to which they, "Cooperate with colleagues", most participants recognise that such cooperation is socially desirable. Hence, participants will overestimate the extent to which they exhibit this behaviour. This bias presents complications only if both the independent and independent variables are susceptible to social desirability. In this instance, social desirability is tantamount to a spurious variable (see Socially desirable responding).

Several options are available to prevent this shortcoming. First, scales have been created that gauge the extent to which the responses of individuals align with social desirability. In essence:

Participants are asked to specify whether or not they have engaged in some undesirable, but prevalent, act such as overdue library books.
Participants that deny they have engaged in these acts are assumed to align with social expectations.
This trait can then be included in analyses to control the effect of social desirability.
Unfortunately, these measures are obviously not a pure measure of alignment to social expectations, especially because this construct is probably multifaceted.

Second, some measures have been demonstrated to be uncorrelated with scales of social desirability. This finding suggests these measures are less susceptible to social expectations.

Third, in some instances, the desirable response is unclear. For example, if asked to specify the extent to which they, "Worry about the health of your parents", participants are uncertain which response is desirable. Agreement suggests they might be neurotic whereas disagreement suggests they might be heartless.

Finally, participants are less likely to be swayed by social expectations when they recognise their responses are entirely anonymous. Nevertheless, anonymity might not completely eradicate this bias. In summary, you should:

Demonstrate the measures are not unduly susceptible to social desirability.
For example, cite studies that reveal that such measures are not highly correlated with scales that gauge social desirability.
Alternatively, argue the scale does not measure some ideal trait or construct.
Finally, you could argue the responses are sufficiently anonymous to preclude social desirability.

Task 16. Have you considered the sensitivity of your measures to this population, with particular reference to floor and ceiling effects?

Sometimes, most of the participants will receive the same score on some measure. For example, participants might all specify the maximum score, called a ceiling effect, or the minimum score, called a floor effect. This limited variability reduces power and thus obscures significant findings. Past studies that have applied these measures to similar populations, or even pilot studies, should be considered to preclude this problem.

References

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Podsakoff, P. M., MacKenzie, S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903.

Academic Scholar?
Join our team of writers.
Write a new opinion article,
a new Psyhclopedia article review
or update a current article.
Get recognition for it.