English First (EF) have just released their EPI or English Proficiency Index. Essentially it is an analysis of English language ability in countries around the world. But how truthful is it? After just a quick look there are some real concerns that what they’ve done is incredibly unreliable.
They’ve produced this report for the past few years and many newspapers take the findings and repeat them verbatim as news stories without really looking at the details. If they did, they might begin to question what is being said.
In this blog I’m going to look at the report and raise a few simple points which will show you how unreliable it is and how the findings should really be taken with a gigantic pinch of salt.
So let’s ask a first question: how reliable is the EF report? It is touted as “the world’s most comprehensive ranking of English ability” and although this may be true, there are certainly some major problems with it and being the world’s most comprehensive ranking does in no way make it reliable or accurate!
Where are All the Countries?
As you might guess, ranking top are several Scandinavian and Baltic countries with Sweden coming out at Number 1. At the very bottom at Number 60 is Iraq, preceded by a couple of other Middle Eastern countries.
Fair enough. But one thing which struck me immediately was the number of countries included in the survey: 60 in all.
The problem is that there are just under 200 countries in the world; if you take away the English speaking ones (about 90 according to Wikipedia) that leaves about 110.
Which means that this report covers just over half the non-English speaking countries in the world.
To take a few surprising omissions Greece, Cyprus and many other Balkan countries aren’t in the report. Nor is Malta which, according to an EU report (see below), has better perceived knowledge of English than Sweden which ranks first in the EF report!
Not to mention that virtually all of Africa is absent from the report.
Of course EF justify this by saying there isn’t enough data, however just because EF don’t know the level of English in some countries (presumably because they don’t have schools there) does not mean that they are free to tout their report as reliable! Quite simply it means that the report is missing half the useful data.
To put this into perspective, there are 20 teams in the UK Premier League football rankings. Imagine if we could only see how 10 of them compared. This is how the EF report looks to me; as a tool for comparison it is not really useful at all!
Who is Being Surveyed?
The second major problem I have is with the people who were surveyed.
According to EF the report was taken from test data submitted by 750,000 adults who took EF tests. I can’t find any details of which countries these students came from which immediately raises my suspicions.
However, let’s look at the likely type of person who took this test because they are from a very specific group of people: adults who want to improve their English and who have a certain level of education and income.
As I see it there are a few problems with this.
- People who already speak (and know they speak) good English already won’t appear in the statistics because they won’t have taken the test.
- People who aren’t bothered about improving their English won’t appear in the statistics because they won’t be interested in checking out EF and their tests.
- People from lower income brackets who either don’t have the internet or can’t afford to attend a school won’t be included in the statistics.
- People who attend other schools than EF won’t appear in the statistics.
To stretch the comparison with premier league football again, it’s like Nike compiling some statistics about all the footballers in the world but… leaving out the best players, leaving out the underpaid players, leaving out the players who wear any other boot than Nike.
In other words, this renders the data almost worthless!
Is it Worthless?
The report is a start. It raises some questions and it’s an interesting, if flawed, read. But that’s about as far as it goes as far as I am concerned. I cannot use the findings in any serious way I’m afraid.
On the other hand, as an advertising outlet for EF it’s almost priceless!
Links
This guest blog written by John Plum, the pseudonym of a TEFL teacher who is working in Japan right now for a major school. Plum describes himself as “annoyed by an unquestioning attitude of the press” and tries live by Bertrand Russell’s Liberal Decalogue.
0 Comments