PURCHASING PROCEDURES - University of Central Lancashire

Samples used in surveys should be representative of the population defined in the
brief. Large samples can still be unrepresentative. The sampling method should be
clearly defined and will normally be randomly determined (involving stratification
where appropriate). Where there is weighting of samples by key population variables
to yield better population estimates, this should be clearly documented and justified.
Where appropriate and feasible, new or bespoke question sets should be piloted prior
to full survey.
2.5 All statistical results reported in surveys should be accurate, comprehensible to
the intelligent lay reader and should not be misleading or overstated.
This means that statistical findings should be presented clearly and rooted in the
actual data. Comparative statistical judgments should reflect normal standards
associated with inferential statistics. Survey response rates and any significant issues
in validity arising from unsatisfactorily low response rates or potentially skewed
response rates must be foregrounded.
Specifically this means that:
Where comparisons are the focus of a sample survey, differences should normally
only be highlighted where they are statistically significant (normally at 5% level of
significance but there will be exceptions).
Tables which report means across groups (with or without claims of statistically
significant differences) should also report measures of dispersion such as standard
deviation.
Nonparametric statistics should be used where more appropriate than parametric
statistics. Parametric statistics should not be used where normal parametric
assumptions are violated, with particular care and attention to Likert study data.
Sub sample analyses should be reported with caution particularly where sub sample
sizes are small.
Statements which are only true of a sub sample should not be presented as being true
of the sample as a whole. Equally, statements which are true of the sample as a whole
should not be presented as true only of a sub sample.
2.6 All surveys should be auditable and reproducible
This means that raw data, appropriately anonymised, should be available for any other
party nominated by the commissioning manager to check statistical assumptions,
calculations and inferential assessments as appropriate. The University will normally
carry out an „evidence trail‟ for a small number of selected specific conclusions in
each final report (see below).
Purchasing Procedures December 2011