The Musings Of An Opinionated Sod [Help Me Grow!]


Does This Prove Research Or Britons Are Flawed?
February 4, 2008, 2:09 pm
Filed under: Comment

Go here then come back and tell me what you think.


60 Comments

Flawed? Fucking stupid.

Comment by The Kaiser

Insane – he’s clearly a bloke!

Comment by John

It only proves that the media are stupid.

Not one of the major UK newspapers who ran with this story (Daily Mail, Telegraph, Herald, Scotsman) have reported any details about the survey apart from the sample size (3,000) and some didn’t even bother with that.

One of the world’s biggest press agencies, AFP, didn’t even see fit to describe the sample (teenagers). The Daily Mail went as far as to describe the results ans representative of “the population” and the Yahoo! story you linked to describes the sample representative of “Brits”. Even BBC Radio 4 picked up on the survey on the Today programme this morning without questioning it’s validity.

No details seem to be available about the methodology on the web at all (which instantly arouses suspicion). The sample size and composition is very unusual and would not have been recommended by a research agency. Therefore my hunch is that it was an opt-in pop-up survey on their own websites – i.e. not at all representative of even the teenagers it purports to sample.

Can anyone find the press release?

Comment by Lee McEwan

The New York Times is one of the only newspapers that reports polls with sufficient detail to determine the adequacy of their sampling methods.

e.g. http://tinyurl.com/3x24yy

Clearly this level of (anal)ysis is going way too far for the average PR survey reported in print but that’s why you use a reputable research agency and then then provide the press release online on both their site and your own.

e.g. this piece conducted by MORI for The Sun can be found easily and provides the basics albeit using jargon like “omnibus”:

http://www.ipsos-mori.com/polls/2007/xmasshopping.shtml

Comment by Lee McEwan

Lee. Who buys this shit anyway? Oh, the great British public.

Comment by The Kaiser

Before I pass comment, does anyone know who did this survey and what were the reasons for it. Yes the media should detail the facts, but I still find this a poorer reflection on the research industry and if they’re blameless, then they should fight against it, not sit there hoping it will go away.

Comment by Lee Hill

In that case Lee, this is the most effective campaign on the planet because for what I can only assume as a small outlay – they’ve managed to get in the pages of the World press.

Maybe P&G should hire whoever was behind it for their media planning ๐Ÿ™‚

In all seriousness, I appreciate what you’re saying but I don’t think the blame can be layed entirely at the media’s door – unless they completely fabricated this story, which is also possible. Sadly.

Comment by Rob

Does anyone know WHY UKGold did this survey and what was the brief.

I’m just trying to work out why teenagers were asked their opinion on Churchill.

I am guessing it is for some shite talkshow about British History / Values, but I could be wrong. It could be part of an impending UK immigration test like the one they have in Oz – where people need to know fucking sporting facts to be allowed in the country.

Not that Australia is prejudice, oh no …

Comment by Rob

I’m guessing that no research agency was involved as I said above. I could go into detail as to how I know that but I’m sure you’d all fall asleep.

The irony is that I know that UKTV has a decent research manager but I doubt he was even consulted by the PR agency or department which created this.

Rob – I gather from one of the reports that UKTV did the survey to raise publicity for Robin Hood. Given that very few of the newspapers chose to mention this fact, I wouldn’t say that it was *that* successful. However, they certainly got plenty of coverage for a tiny outlay if they did this survey online themselves as I suspect.

I blame the media because they didn’t have to run it. You’ll notice that many media outlets chose not to, presumably based on their judgement of the veracity of the findings.

As Nick Davies mentions in todays Guardian (http://tinyurl.com/2vlazp) only a minority of what gets published in newspapers today is produced by journalists. Most of it is PR-generated tosh like this. Nick estimates based on a content analysis study he conducted with Cardiff University’s school of journalism for his new book that 80% of newspaper content is “mainly or partially constructed from second-hand material, provided by news agencies and by the public relations industry”.

I’ll get off my high horse in a minute.

Comment by Lee McEwan

I like you on your high horse Lee – and this is definitely the blog to do it on – so please carry on, ha!

I still don’t think you can totally blame the media for printing this farcical story because someone out there actually conducted the ‘research’ – however what you can rant on is the standard of journalism, that’s another thing altogether.

Comment by Rob

Everything I wanted to say on this has already been said above…

Comment by Rob Mortimer

That never stops anyone Mr M … go on, have a rant.

[PS: Thank you for helping me see sense last week too!]

Comment by Rob

Thats ok. I’ve seen a few jokes about, which always happens. Some are funny in a respectful way, I just felt that one went too far!

Comment by Rob Mortimer

I don’t give a flying fuck who did the research or which newspaper printed the story or if a journalist is involved. What worries the shit out of me is that some answered the questions – and what their answers where.

That’s the problem.No surprising – but a problem never the less. That’s a huge fucking problem.

Comment by The Kaiser

and of course, if it’s the media circus. then rob, i’m sorry to say, but you have successfully contributed to it ๐Ÿ™‚ well done.

oh, and rob, you’ll be pleased to know that Howard’s citizenship test is a complete fucking failure and Kevin07’s government is reviewing it, looking to phase it out. which means that you may just be allowed back in the country… ๐Ÿ˜‰

Comment by lauren

Lee H – I totally agree with your comment that if the research industry is blameless then “they should fight against it”.

Sadly the research industry doesn’t do enough to educate the media who in turn are not helping their readers to critique the research that is thrown at them every day.

Sadly, some so-called research agencies are merely fronts for PR agencies these days. This research entity (http://www.onepoll.com/) is part of a PR agency (http://www.72point.com/) who claim to have “an 80% success rate of national press coverage”.

You may say “so what?” but the difference is independence. Ben Goldacre describes the potential for bias introduced by a PR agenda quite well in this piece for the Guardian last September … http://www.badscience.net/?p=520#more-520

Comment by Lee McEwan

@ Rob @ 6.04pm You say “I still donโ€™t think you can totally blame the media for printing this farcical story because someone out there actually conducted the โ€˜researchโ€™.”

My contention is that this was not conducted by a reputable research agency. It was either conducted in house (e.g. a pop up on their own website) or by their pr agency (the unusual sample and sample size are a give away that is is not well with this research). Neither UKTV or their pr agency will have a reputation to defend as a provider of bona fide research.

The real issue here for me is that we have no way of telling if this is “good” research or whether or not is has just been made up. As the Telegraph & Top Gear motoring journo James May wrote recently, he suspects the latter is often the case: http://tinyurl.com/2txr89

As a planner who has to use research daily it is galling that trust in another bit of our toolkit is being systematically eroded by these poxy PR surveys.

Right, I really have to go and lie down now.

Comment by Lee McEwan

Kaiser – We don’t really know that anyone actually answered these questions and even if they did how do you know that these were the results? Really, how do you know? [Promptly disappears into vortex of his own making]

Comment by Lee McEwan

You do know that it was a PR-network that REALLY killed John F. Kennedy don’t you Lee?

Comment by The Kaiser

It was, I swear. But why should you trust me?

Comment by Lee McEwan

Lee – I’m sorry about that last comment. That was silly. Now then to answer your question….

I don’t know. I simply don’t know. But I’ve decided to take a leaf out of your book and wait until the Guardian tells me how is really “went down”.

Comment by The Kaiser

Doesn’t the fact this research was possibly conducted by an unqualified organisation demonstrate the need for the industry to take a stand against this sort of thing?

I absolutely agree with Robert that this is a sad indicment on the industry because regardless of who conducted the research, someone somewhere got paid for it and are now claiming it is valid.

To stop this once and for all, does anyone know just what the brief was, who conducted it and what were the parameters they worked to?

Comment by Lee Hill

Brief: We need to show that Britons are stupid

Answer: Ask teenagers about people who died years before they were born and have rarely been mentioned in their education.

Comment by Rob Mortimer

Given planners have to understand the attitudes and perceptions of society to help form the foundation for their ads, could we not say they are as guilty in spreading the fabrication of “fact” as the media?

Comment by Lee Hill

You can pay me for that comment later Andy.

I really must get back to work, I’m becoming as bad as Robert for procrastinating.

Comment by Lee Hill

wow. you english are lucky. your media is up in arms about teenagers who don’t know important historical figures from important fictional ones. our media are still going on about a kid in dumb sunnies who threw a party while his folks were away.

Comment by lauren

At least thats a real story…

Comment by Rob Mortimer

excuse me for making a sensible comment for once but all fucking research is flawed, its just some is less bollocks than others. i like research, it helps me to great stuff but to take everything as gospel is like believing there’s unconditional love in a marriage.

off to do some fucking work now. my holiday is officially over. tit, piss, wank.

as an aside, i bet every little teenage fucker knows who jenny jameson is. mind you shes done more for “standing proud ” than churchill ๐Ÿ™‚

Comment by andy@cynic

Andy makes a sensible intelligent comment, and tops it off by talking about Jenna Jameson in a string of expletives.

Good to have him back.

Comment by Rob Mortimer

That’s my boss, all intelligence, swearing and porno ๐Ÿ™‚

Comment by Billy Whizz

People who think Churchill is myth = 750 people out of 3000 people who self-select to answer a survey (and yes it is self-selection – regardless of what the industry would claim).

Even if the survey sample reflected the demography of the UK that would produce probably 1000 respondents born after Churchill died and a majority of them might be argued to have no great interest in and certainly no large education via the national curriculam in history. Being a survey for a TV company, I can imagine it being skewed to a younger 16-49 group and so that effect would be exacerbated further. These people havent heard of Churchill – so whn asked who of these people are mythical, they answer myth to anyone they’ve never heard of. They’re not saying he’s a mythical character.

To people who know who Churchill was, the result is truly a shock – an incomprehensible one, but if you stand back a little, yoiu can rationalise it easily. And Marcus i do mean rationalise – I don’t mean excuse it. It is an indictment of the education system but look around and you see those every minute.

Comment by John

Indeed the reactions of people to simple surveys seen without any reference or evidence is another.

As Homer says: Facts Schmacts, you can use facts to prove anything thats even remotely true…

Comment by Rob Mortimer

Andy, what is going on – that is a very sensible answer – are you suffering with the same ‘illness’ as Charles over on the IBM post?

Comment by Robert

I’m late, but I’m very angry about this. You just know it was shoddy methodology, leading questions and wonky analysis.
Bollocks.
If it’s correct, then I really do give up.
Humph

Comment by Busy northern

I just like to point out that this piece of research turned up in a German newspaper today.

Comment by The Kaiser

As I said yesterday – regardless how inaccurate the information/methodolgy is – the fact the research industry doesn’t come out in arms over this sort of thing means they have to accept some of the blame because that is how this sort of shoddy shit still gets publicity.

Comment by Rob

marcus, it was in der bild zeitung! that’s not shocking. or a newspaper, for that matter.

Comment by lauren

Erm, it happens to be Europe’s biggest selling newspaper Lauren. You may not like the fact, I may not like the fact…but it is.

Comment by The Kaiser

true, opi. even so, it’s hardly shocking that it featured this crap in the headlines. in fact, if the research company is a PR company, then they’re actually doing a good job, because as ‘europe’s biggest selling newspaper’ if it’s featuring this garbage on the cover (or at least on the front page of the site) then they’ve got that corner of the mass hysteria market sorted.

Comment by lauren

wasn’t on the cover Lauren. I had to look really hard to find it.

And who says this is garbage? I think that what this research is saying is totally plausable – but then, I grew up in Southampton. Remember, contrary to what most of the “plannersphere” would have you believe only a very small percentage of the world actually read the guardian.

Comment by The Kaiser

well fucking said marcus. i hate how so many middle class wannabes treat the guardian like its the fucking messiah of journalism because its not.

sure its better than the rest of the shit out there but that doesnt mean its the best, its just the best of a bad lot.

i cant fucking believe were still banging on about some innocuous alleged research when theres countless examples of real shit going down that no one bats a fucking eyelid at, like how government “research” showed it was immigrants who were the ones fucking england up with crime etc.

what a fucking load of wank but no one goes on about that and yet the consequences are a fuck load worse than whether some snotty kid knows who fucking churchill is or not.

if research companies want respect then they should fight for it and not just whine when some chancer gets mass media coverage with total bollocks. and if theyre any good, theyd understand why not knowing who churchill was would be a big thing for british and kraut press.

fuck im making another serious comment, time to put an end to that. back to work.

Comment by andy@cynic

Agreed. Is it not really the fault of the media (and thereby the public) who constantly put headlines like this in their shitty publications; inspiring further rubbish research to generate said headlines.

Comment by Rob Mortimer

If it didnt sell they wouldnt do it.

Comment by John

and you would be out of a job John.

Comment by The Kaiser

this is boring, i almost miss campbells blog bollocks but ill survive ๐Ÿ™‚

Comment by andy@cynic

Bored. Bored. Bored. Bored. Bored. Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.Bored. Bored. Bored. Bored. Bored.

Comment by The Kaiser

Tonight Andy Boucher will be Gloria Gaynor!

Comment by John

im younger than you dodds.

Comment by andy@cynic

and wittier too

Comment by John

morning. We are nothing without him. OK, that’s not strictly true, but we’ve got nowhere else to go. OK that also not strictly true, we could go to Adscam but that would only cheer George up (and we don’t want that to happen) but…

jesus I’m bored.

Comment by The Kaiser

are you fucking mad making complimentary comments about campbell like that. take it back immediately. not for my sake, but yours. what the fuck is going on here? im scared marcus, im fucking scared ๐Ÿ™‚

Comment by andy@cynic

i’m going to unashamedly boast that it was my birthday today (well, officially yesterday now that it’s after midnight here) and i had a fucking ripper of a day. my colleagues sang happy birthday to me, not just in key, but WITH HARMONIES!!! they gave me presents, cake and lovely words in a shit hot card – i’ve only been there two weeks!

i’m sure it’s all downhill from here, but that’s what you get kids, when you don’t work in advertising. song and presents! woo hoo!

and marcus – may i suggest knitting to alleviate your boredom too…

Comment by lauren

Belated harmonious happy birthday with descant choral accompaniment Lauren. But no present. And Marcus – pull yourself together man. Remember the sum is greater than the hole and just because the hole is off on vacation, it doesn’t mean that wailing and fawning will be tolerated.

Comment by John

happy late birthday my swearing lovely. ive got something huge for you when i finally see you ๐Ÿ™‚

marcus, you still should be hanging your head in shame following your campbell fawning.

Comment by andy@cynic

it better be huge andy, for all the publicity you’re giving it.. ๐Ÿ™‚

Comment by lauren

You are only encouraging him Lauren – and trust me, if it’s what I think he is alluding to, you don’t want it … hell, even his wife doesn’t want it.

Happy Birthday for whenever it was my lovely!

Comment by Rob

Ahem. Sorry, I just couldn’t let it lie.

This is the shorthand list that Bob Worcester (founder of MORI) recommends that journalists use to decide whether or not polls are nonsense:

1. When were the fieldwork dates?
2. Was the sample representative and large enough?
3. The more sampling points the better (face to face polls)
4. Make sure of where the sample was taken
5. Is it an internet, face-to-face or a telephone poll?
6. Are the questions unbiased?
7. How are the โ€œDonโ€™t knowsโ€ re-allocated? (voting intention)
8. Are differences statistically significant?
9. Look for full question wording, and full answer wording
10.Who paid for the poll?

From a comment on this rant against so-called “voodoo polls”:

http://ukpollingreport.co.uk/blog/archives/1117

Yes, I know this is boring. But sadly it is important.

Comment by Lee McEwan

i like you lee. i like people who dont let things lie. infact if you werent a planner i would buy you a drink ๐Ÿ™‚

so youve commented on a proposed set of rules journalists should follow when writing/commenting on research findings (though to me it seems very long and doesn’t accept the research companies have a role to make sure information is clear, concise and “media friendly”) but what set of “rules” do you think the research companies should follow to make sure their value, independence and insight isnt slowly undemined to the point where what they say is about as trustworthy as the words from a politicians mouth. its not just the media at fault here, its also the passive research bods who let this shit find an audience then dont fight against it for their own integrity.

Comment by andy@cynic

While you’re at it, what set of rules should the client impose on the research people? All the statistical rules add to the cost of conducting the research and clients have to stop being cheap.

Comment by John

given those wpp fuckers make the most of their cash from research companies, maybe they should stop being so expensive as well dodds.

Comment by andy@cynic




Comments are closed.