Quantitative Market Research
Quantitative market research poses structured questions to people and businesses so as to guage their opinions in a way that generates hard facts. Typical outputs are market research tables showing percentage reponses for everone interviewed (e.g 29% of people like cream cheese) or for different sub-sets of individuals (eg 35% of men like cream cheese and 25% of women do). Quantitiative research is the most common form of market research undertaken and it is important to understand how to interprete the results it generates when presented in tabular form.
Always look for and include the Base
In market research surveys, the number of respondents who answer a question is known as the Base. It is always important to show the Base in a market research table as it influences the validity of the results shown. Assuming the respondents in a survey were selected randomly for interview, then a base size of between 50 and 100 should ensure that the results are sufficiently reliable for the majority of purposes.
Be wary of small bases
A small base will have a much larger margin of error in the results compared with a large base, meaning the accuracy of the results are lower with a small sample. Smaller bases also make it harder to make comparisons between groups. For example, are men more likely to like eggs than women? For results generated from a large base, the differences between the genders in the percentages liking eggs can be smaller and still significant, while with smaller Bases the percentage differences have to be larger before you can say with certainty that men or women like eggs more or less.
Always identify the Base
Not all bases are alike. Some questions will be asked of all the respondents participating in a survey, while others are asked of a sub-set. Always know and identify who has been asked a question. So if the “do you like eggs” question was asked only of individuals who indicated in a previous question they ate breakfast, make sure this is clear when presenting results.
Is the Base weighted or unweighted?
Why weighting is necessary
When undertaking interviews with a group of individuals who are expected to be representative of all individuals in the UK, by chance the resulting interviews might be slightly skewed or bias towards one group. For example, while men and women represent roughly 50% each of the UK population, a survey of 1,000 individuals might result in 550 female replies and 450 male replies. To ensure the results from the survey reflect the whole population, weights are attached to the responses. For example, in the simple example above, men would have a weight of 500/450 = 1.11 and women would have a weight of 500/550 =0.90. So replies from 9 men would represent around 10 weighted responses and replies from 11 women would also present 10 weighted responses.
In a real life survey, weighting is much more complex than in the above example because the weights have to reflect differences across a wide range of criteria, such as gender, age, economic status, location etc. As a consequence, the size of the resulting Base in the weighted sample often differs slightly from that of the unweighted sample.
Do you use weighted or unweighted data?
It is always important to know if the results presented are the weighted results or the unweighted results, because the differences can be significant. The Figure opposite, based on a very simple example, shows a weighted result of 60% of adults liking eggs compared with an unweighted result of 59% of adults liking eggs. In this case, the differences are small but they can be much larger in more complex surveys. If you are looking for, or reporting, nationally representative results you must use weighted data.
When only one specific sub-set of individuals are interviewed and no one else – e.g. only mortgage holders and no one else – weighting is sometimes not used because the aim is not to create a nationally representative sample of individuals but only a sample of individuals with one thing in common.
Frequency vs Contingency tables
There are two main types of tables used in market research surveys, frequency tables and contingency (or cross tabulation) tables.
Frequency tables are often used for top-level results, showing how many or the percentage of respondents that have something, have done something, or agree with a statement. The frequency table opposite shows that credit cards are by far the most popular form of unsecured debt owned by consumers in the UK and the only form of unsecured debt owned by more than half of unsecured debtors (i.e consumers who have any form of unsecured loans).
Contingency tables show how one or more groups compare in their actions or attitudes and they can also be used to show how the results from one question influences the results of another.
When handling cross-tabulations, the Bases must be shown for each separate group. So in the first figure opposite, the Bases for females, males and all adults are shown at the top of the table.
The first contingency table opposite also shows, for example, that females are more likely than men to have Student Loans, mail order debt and store cards, while men are more likely to own credit cards and car finance loans. Whether these differences are statistically significant is influenced by the Base sizes. Given the large Base sizes, we can say that there is a statistically significant difference in unsecured loan ownership among men and women with regards to: Credit cards; Mail order; Student loans; Store cards; and Car finance.
The second contingency table opposite shows a contingency table were the results from one question (about debt arrears) is cross tabulated with unsecured debt ownership levels. It shows, for example, that individuals who have gone into arrears are much more likely than those who haven’t to owe money on overdrafts, personal loans, mail order purchases, DSS social funds and payday loans. There is a statistically significant difference in the level of ownership between individuals who have and have not gone into arrears with respect to: Hire purchase; Personal loans; Mail order; DSS Loans; Overdraft; and Payday loans.
Differentiate single and multiple response tables
Market research studies tend to pose two main types of questions to respondents: Single response questions and Multiple response questions.
Single response questions allow only one option to be selected. For example,
Do you own a credit card?
- Don’t Know
The respondent can only choose one of the three possible responses: selecting “Yes” prevents the respondents from also answering “No” or “Don’t know”. Therefore, a table showing the results across all respondents must add up to 100%. The Bank of England survey, for example, showed that 58% of respondents have a credit card (answered “Yes”) and 42% do not (answered “No”), coming to a total or 100%.
Multiple response questions allow a respondent to select multiple options. For example, in the Bank of England study consumers who had arrears on their unsecured debt were asked why they had fallen behind in their loan payments. They were presented with 11 possible responses and could select as many of these as they liked. In this case responses will total more than 100%. See the table opposite.
Multiple response tables can be created from single response questions
It is possible to create multiple response tables from single response coded questions. For example, if respondents were asked
Which of the following types of debt product do you own?
and they were presented with 10 options (e.g. credit cards, HP loans, payday loans etc.) and the answers for each type of debt product were single coded (i.e. “Yes”, “No”, “Don’t Know“), then the resulting table showing only the “Yes” answers would be a multiple response table summing to more than 100%. This table would be identical to the frequency table shown above.
Apart from surveys with small response numbers, in most other cases, including counts on a table can make it harder to interpret the results as the Figure above shows. This Figure has the same data as the contingency table shown in the Frequency vs Contingency section but it has been simplified by showing only a few loan products and the refused/don’t know category has been removed. Even, so it is a much harder table to interpret than the earlier table.
Percentages or counts?
It is normal in market research surveys to show results in percentage terms rather than show the number of responses (or counts). However, sometimes it is useful to show the number of counts especially if the user of the results wants to conduct their own statistical tests on the results.
Moreover, if responses are small as can be the case for some surveys were businesses are interviewed, percentages can be misleading. Imagine a company that only has 40 business customers, then a survey of their customer base may only result in 30 completed interviews. Given the small base, even small variations between responses can result in large percentage changes. For example, if customers were asked to state which of the company’s products they like and 10 said Product 1 and 12 said Product 2, the percentages would be 33% for Product 1 and 40% for Product 2. A table showing 10 responses for Product 1 and 12 for Product 2 gives a better impression of the small differences in liking.
Columns or row, profile or penetration
The data in a market research table can be analysed from top to bottom (column) or from left to right (row). If the table is showing the characteristics of respondents (e.g. their age, gender, income etc.) then:
A column analysed table shows the PROFILE of the group analysed (e.g. the percentage who are aged between 18 and 24, for example).
A row analysed table shows the PENETRATION of the group (e.g. the percentage of 18-24 year olds who belong to a group, for example).
Typical column, profile and row, penetration tables are shown opposite. The profile table shows the percentage of individuals who do and do not have hire purchase loans who are female or male. So 56% of individuals without an HP loan are female while 55% of those with a loan are female. The penetration table shows the penetration of HP loans, with 5% of females and 5% of males having HP loans. Note that in the profile table the bases appear at the top of each column, while in the penetration table the bases appear in each row (because in the penetration table the percentages refer to each gender group).