Thames Valley Speed Cameras

An independent report        

Collisions occurred at around the mean rate during all the periods that were outside the SSP, therefore the rates during the PreSSP and ASBiC were similar.

 

RTM is observed as a sudden (vertical) change in collision rates immediately after the SSP ends (labelled).

 

At the start of the SSP, the collision rate suddenly changed from the mean rate up to the much higher rate during the SSP.

 

A crucial observation is that RTM occurs before speed cameras start operating, whereas speed cameras can only cause an effect after they start operating. RTM can therefore be separated from any speed camera effect by evaluating timing.

 

4.7 The FTP method

 

The FTP method is capable of measuring RTM effects at speed camera sites and is also capable of fully excluding RTM effects from the final results. I believe it is the only method capable of doing either of these given the data available.

 

In order to evaluate the effects of speed cameras, we need to determine how many collisions would have occurred at the speed camera sites had speed cameras not been deployed (so that this can be compared to the number that did occur). The best method would be to perform RCTs but, in the absence of RCTs, the next best method would be to establish the mean ("normal or expected") collision rate from the collision data prior to speed cameras because this is the rate that would be expected to have continued had speed cameras not been deployed. This is what the FTP method does.

 

Essentially, the FTP method reverse engineers the site-selection process. To apply the method, the SSP needs to be identified. Having done this, the mean collision rate prior to speed cameras is established using the collision data that was outside the SSP (during the PreSSP and ASBiC). The mean collision rate prior to speed cameras can then be compared to the collision rate after speed cameras were deployed and the results will include the effect of the speed cameras, but will not include any of the RTM effect.

 

Although this is fairly straightforward, the mean collision rate may change over time due to trends in road safety (the sum total of all the general influences on road safety, such as rising traffic volumes, safer vehicle design, cost of fuel etc). This factor also needs to be compensated for.

 

To apply the FTP method to collision data at speed camera sites, 3 initial steps are performed:

 

4.7.1 Express collision numbers at speed camera sites as relative collision rates (to compensate for trend)

4.7.2 Identify the SSP and separate the data into the 4 time periods: PreSSP, SSP, ASBiC and after speed cameras start

4.7.3 Use the PreSSP and the ASBiC to determine the "before" relative mean collision rate


Then, to measure RTM at speed camera sites:


4.7.4 Compare the relative collision rate during the SSP with the "before" relative mean collision rate (4.7.3)


And, to exclude RTM effects from the final results:


4.7.5 Compare the relative collision rate after speed cameras started with the "before" relative mean collision rate (4.7.3)

 

The 1st step (4.7.1) is to compensate for trends in road safety. Trend is well understood. The standard method is to calculate the average area-wide change in the number of collisions during the years within the analysis, and then adjust results at speed camera sites by this average amount. This method works well when trends are fairly consistent but they often aren't (and certainly weren't in Thames Valley).


I therefore developed a new method that compensates for trend by analysing relative collision rates at speed camera sites. A new database is created by taking each item of collision data at each speed camera site, dividing by the number of collisions that occurred in that calendar year across the whole area, and then multiplying by 100. By analysing this new database of relative collision rates, trend is automatically compensated for no matter how fast or slow the trend changed, or even if it varied. Although this method may seem obvious, I'm not aware of any report using it prior to my Feb 2012 report. I used it again in my Nov 2012 report and Professor Allsop also used it in his RAC Foundation report (2013).

 

The 2nd step (4.7.2) is crucial. The full potential of the FTP method is only achieved if the SSP is correctly identified therefore it is important to define exactly what the SSP is. Individual speed cameras cannot be evaluated because collision rates generally have too much random variation therefore speed cameras can only be evaluated in groups. The SSP used in the analysis of a group of speed camera sites must fully include the SSP for each site within the analysis. In other words, if some or all of the SSP of any individual site extends outside the SSP used in the group analysis, then RTM may not be accurately measured or fully excluded from the final results.

 

For most sites, the only known date is the commission date (the first day speed cameras could be in operation), the start and end dates of the SSP are not known. This means that the SSP may have to be identified primarily from collision data. To do this, the sites in an analysis are aligned using the commission dates, and the start and end of the SSP is identified relative to the commission dates. This will always be a judgement (and may not always be possible) but the combination of collision data analysis and other information about the site-selection process does make it possible, under certain circumstances, to prove when the SSP was beyond reasonable doubt. If this can be done, the full potential of the FTP method can be achieved.

 

Note: I would hope it is obvious that, if the SSPs for each site within an analysis cannot be lined up (as may often be the case for fixed speed cameras, or speed cameras used before the national guidelines etc), then the SSP used for the group analysis may have to be longer than any individual site's SSP. The SSP for a group of sites may therefore include some of the PreSSP and/or ASBiC periods of individual sites but this is not necessarily too much of a problem. Final results can still be obtained that do not include any RTM effects but the amount of data that is available to establish the mean will be reduced. Also, the real magnitude of RTM could actually be higher than the analysis would obtain. Furthermore, RTM would not be expected to be a sudden change, but a change over a duration equal to the greatest difference in ASBiC durations of the included sites.

 

After I developed and applied the FTP method, I was informed that elements of the method had been used before. Professor Mike Maher (University of Leeds) (co-author of the RTM estimate in the government’s 4YE) informed me that he and others had previously used the ASBiC (they called it the "gap" period). Also, Professor Richard Allsop of UCL (University College London) (author of 2 speed camera reports for the RAC Foundation) informed me that the PreSSP had been used in at least one Danish report (where it was referred to as the "before-before" period). Therefore the basic concept behind the FTP method (that collisions may not occur at their mean rate during the SSP, but should do so outside the SSP) was already known. The FTP method combines both the PreSSP and the ASBiC (for the 1st time) along with several new features.

 

When developing the FTP method I realised that, if the RTM effect is as large as estimated in official reports, it should be possible to identify the start and end of the SSP by analysing the collision data at speed camera sites. If the SSP could be identified, the collision data could be separated into the 4 time periods: PreSSP, SSP, ASBiC and after speed cameras started. The two "before" periods that were outside the SSP (the PreSSP and ASBiC) should have collision rates that were around the mean and should therefore be similar (when trend adjusted). If they are similar, that should confirm both that the SSP has been correctly identified and that the true "mean" collision rate prior to speed cameras can be established. If the start and end of the SSP for every site within the analysis can be aligned, a graph should show a sudden (vertical) change in the collision rate at the start of the SSP, and another sudden (vertical) change at the end. This should prove (as far as is possible) that this was indeed RTM and not something else. The graphs should also make obvious, even to those who don't understand RTM, that significant changes may already have occurred long before speed cameras could have had any effect.

 

Previously, the EB (Empirical Bayes) method was considered to be the 2nd best method to evaluate the effect of speed cameras (2nd best to RCTs). The EB method uses a prediction model involving complicated formulae and data derived from a group of sites thought to be similar to speed camera sites. To apply the EB method, large amounts of extra data has to be collected at the speed camera sites to allow the prediction model to produce an estimate of the collision rate that should have occurred at the speed camera sites. This is then compared to the collision rate that did occur after speed cameras were deployed. The EB method is far from ideal, though, because the data required is not available for any speed camera sites (even for the few sites where it was collected), the calculations are very complicated and it only produces estimates.

 

A further serious problem with the EB method was revealed in it's practical application in the 4YE (the government's largest report on speed cameras). The EB model estimate was considered so inaccurate that the estimate was increased by factoring in an amount of the baseline data. The explanation for doing this was: (p149) “because the high observed collision frequency (in the baseline) will in part tend to arise because of site features which are not included in the model". But this is not necessarily true and no evidence is presented in the 4YE to support this. Obviously the EB model will not be perfect but any errors are just as likely to be positive as negative. Simply increasing the EB model estimate will not correct for inaccuracies in the EB model but what it will do is introduce some of the RTM effect into the final estimate, despite the fact that the EB model was applied precisely to exclude the RTM effect (see also 3.5 in government reports).

 

The FTP method presents substantial improvements over the EB method. Firstly, the FTP method uses data that is routinely collected and available. Secondly, it is far simpler to apply and understand (the analysis does not require complex mathematics and graphs can show when collision rate changes occurred). Thirdly, and most importantly, accuracy. The FTP method calculates the changes in collision rates that actually occurred at the sites in the analysis before any speed cameras were deployed. This can produce measurements (rather than estimates) of the RTM effect. Furthermore, by establishing the true mean "before" collision rate for the sites within the analysis, and comparing that to the collision rate after speed cameras were deployed, the final results of the FTP method can include the effect of the speed cameras, but without including any of the RTM effect. In other words, with the exception of RCTs, a more accurate method may not exist.

 

4.8 Achieving the full potential of the FTP method

 

The full potential of the FTP method is that it is capable of measuring RTM effects at speed camera sites and is also capable of fully excluding RTM effects from the final results. To achieve this, the single most important criteria is that the SSP must be correctly identified but this requires other criteria to be met and there are also other considerations.

 

To achieve the highest confidence in the results produced using the FTP method, these criteria should be met:

 

4.8.1 There are a large number of sites in the analysis (to reduce the effects of random variations in collision rates)

4.8.2 The collision data available should be sufficiently detailed to allow timings of changes to be analysed

4.8.3 All sites in the analysis were selected by the same people using the same site-selection criteria

4.8.3a Each site in the analysis had a consistent SSP duration

4.8.3b Each site in the analysis had a consistent delay after the SSP before being commisioned (consistent ASBiC durations)

4.8.4 The ASBiC was a long period (to reduce the effects of random and seasonal variations in collision rates)

4.8.5 There is data for a long PreSSP duration (to reduce the effects of random and seasonal variations in collision rates)

4.8.6 No other collision remedial work was done at the sites

4.8.7 The relative mean collision rate remained fairly constant thoughout (other than the effect of the speed cameras)

4.8.8 After the original SSP, the data was not subject to any further selection process that might affect the results

 

4.9 Practical application of the FTP method

 

4.9.1 The FTP method has been used in 2 reports so far.

 

The first was my report "The effects of mobile speed cameras on road safety". The mobile speed camera sites in Thames Valley satisfied all 8 criteria above therefore it was possible to achieve the full potential of the FTP method. The final results included the effects of the mobile speed cameras, but did not include any RTM effects. There was found to be a 31% increase in the number of people killed or seriously injured after the mobile speed cameras were deployed. Because this result does not include any of the RTM effect, and because it is the first report to have achieved this, the final results should therefore be the most accurate evaluation of the effect of speed cameras to date.

 

The second report to use the FTP method was "Guidance on Use of Speed Camera Transparency Databy Professor Richard Allsop of University College London, published by the RAC Foundation. In his report, Allsop advises his readers to use the FTP method and he also applies the FTP method to collision data at fixed speed camera sites. He assumes when the SSP might have been and separates the collision data into the 4 time periods. Having done this he finds there was a 27% reduction in fatal and serious collisions at the fixed speed camera sites.

 

4.9.1a Why are the results using the FTP method in these 2 reports so completely different?

 

There are many differences between the 2 reports but the primary reason for finding completely different results was that there were no RTM effects in the final results of my report but there may have been substantial RTM effects in Allsop's results. This is because of how accurately the FTP method was applied.

 

For the FTP method to fully exclude all RTM effects from the final results, the SSP must be identified but, unfortunately, Allsop didn't do this. Allsop (p30) "...assumed (the SSP) to be the last three full years before camera establishment." Had the SSP for every site within Allsop's analysis been fully within the SSP Allsop had assumed, then his results would not have included any of the RTM effect, but this was not the case. Many of the fixed speed cameras in Allsop's analysis had been installed by LAs (Local Authorities) prior to the national guidelines. This means that different LAs may have used different SSP durations. Also, fixed speed cameras had inconsistent delays before being installed (from 6 months to over 3 years). For both these reasons the start of the SSP for many sites in Allsop's analysis may have been in what Allsop assumed was the PreSSP.

 

Allsop was further hindered in his analysis because the data he used did not meet at least 4 of the 8 criteria: 4.8.2, 4.8.3 (and therefore 4.8.3a), 4.8.5 and 4.8.6. The effect of this is that, while some of the RTM effect may have been excluded, his final results may simply be due to the effects of RTM that remained, rather than the effects of the speed cameras. Allsop was not able, therefore, to establish what effect the actual speed cameras may have had.

 

For my report, on the other hand, the data I used met all 8 criteria (4.8.1 to 4.8.8 inclusive) and I was able to positively identify the SSP for the group of 75 mobile speed camera sites in my analysis. This meant that RTM effects were fully excluded from the results thereby establishing the effects of the actual speed cameras.

 

This demonstrates that, while the FTP method is potentially very powerful, it needs to be applied with great care and should only be considered as the 2nd best method. The best method is, of course, to run scientific trials (RCTs).

 

4.9.2 Fixed speed camera sites in Thames Valley

 

It is not possible to use the full FTP method to evaluate fixed speed camera sites in Thames Valley due to the following criteria not being met: 4.8.3 (and therefore 4.8.3a), 4.8.5 and 4.8.6. Very few sites have sufficient collision data during the PreSSP (some have none) therefore criteria 4.8.5 is the main problem. It is possible, though, to use the ASBiC to evaluate these fixed speed cameras and this will be done at a future date.

 

4.9.3 Does reducing traffic speed using speed cameras reduce the number of collisions?

 

It is claimed that the collision rate should reduce by 5% for every 1mph the average speed of traffic is slowed down, but is this true when speed cameras are used to slow traffic speeds? Again, the full FTP method cannot be used but the ASBiC part of the method can be. The analysis has been done and results are here: report

 

4.10 Investigate site-selection effects in Oxbube for yourself

 

With computers and the internet this is very easy to do. Download my spreadsheet (where I assembled the data and calculated the numbers for figure 4.1) here: rtm_oxbube.xls.

 

The spreadsheet has 300 roads where it is possible to place imaginary speed cameras and each road has 0, 1 or 2 collisions per year, determined entirely randomly.

 

The criteria for choosing speed camera sites can be selected on the results page in cell G5 (in yellow). In figure 4.1 this was set to 4. In other words, if there were 4 or more collisions in the SSP (that’s the years 3, 4, 5), then a speed camera was installed. Cell G6 shows that 121 sites matched that criteria in this example (therefore 121 speed cameras were installed in Oxbube), and that they happened to have a 31% reduction in the number of collisions.

 

Select your own criteria (cell G5) and see what happens.

 

You can also generate your own data. Go to http://www.random.org/integers/ and select:

 

1)               “Generate 2700 random integer”

2)               “Each integer should have a value between 0 and 2”

3)               “Format in 9 column(s)”

 

Then copy the data field generated and paste it over the data on the page “database”.

 

Go back to the front page “Results” and see what happens as you select different criteria.

 

Obviously, each sample of random data may give different results, but you should find a pattern very similar to figure 4.1 emerges most of the time.

 

Obviously you can also try different integer values.

RTM (Regression to the mean)

and the

FTP (Four Time Periods) Method

 

The BIG effect at speed camera sites: RTM (Regression to the mean)

 

RTM is actually a very simple effect but the authorities have not found any method that could either measure RTM, or fully exclude its effect from results. I therefore developed a method capable of both of those functions and I applied the method to speed camera site data. I then gave the method a name: the FTP (Four Time Periods) method.

 

Speed cameras are often deployed at sites following a higher than normal collision rate therefore the collision rate would tend to reduce back to normal, whether or not speed cameras had been deployed. This change (that would have occurred anyway) is called RTM (regression to the mean) and, importantly, its effect can be measured.

 

RTM is estimated to be the largest influence on fatal or serious collision rates at speed camera sites but, since no official report has managed to fully exclude the effect of RTM, the effect of speed cameras has never been officially established.

 


Contents:


4.1 How can the effects of speed cameras be determined directly (without having to consider RTM)?

4.2 How big is the RTM effect on KSI collisions at speed camera sites (KSI: Killed or Seriously Injured)?

4.3 What is RTM?

4.4 Why does RTM affect results at speed camera sites?

4.5 How can the site-selection process lead to RTM?

4.6 Features of RTM in Figure 4.1

4.7 The FTP method

4.8 Achieving the full potential of the FTP method
4.9 Practical application of the FTP method
4.10 Investigate site-selection effects in Oxbube for yourself


4.1 How can the effects of speed cameras be determined directly (without having to consider RTM)?


Demonstrating and quantifying the effects of speed cameras would be straight forward if speed cameras were operated within simple scientific trials known as RCTs (Randomised Controlled Trials). RCTs would give a direct answer to the question: “What effect do speed cameras have on road safety on the roads where they operate?If RCTs had been run, there would be no need for any of the detailed evaluations of RTM and other effects at speed camera sites and no need for the prolonged confusion and debate over what effect speed cameras actually have.


In the absence of RCTs, the factors that cause collision rates to change at speed camera sites must be evaluated, including:

 

4.1.1 The speed cameras (the effect we're trying to measure)

4.1.2 RTM (an effect of the site-selection process)

4.1.3 General influences (aka "trend”, such as increasing traffic flow, improved vehicle safety, cost of fuel etc)

4.1.4 Diversion of traffic


These 4 factors are mentioned in the government's largest report on speed cameras, the 4YE (Four Year Evaluation), but there are other factors as well such as other site-selection effects and co-intervention (other measures also installed).


4.2 How big is the RTM effect on KSI collisions at speed camera sites (KSI: Killed or Seriously Injured)?

 

4.2.1 The National Safety Camera Programme (4YE) estimates that RTM has a larger effect than all other factors combined

4.2.2 The RAC foundation report (2010) (3b) indicated that RTM has a larger effect than any other factor


These are not obscure reports. The 4YE is considered to be the foundation of the entire speed camera programme in Britain and the RAC foundation report is probably the most publicised and influential report of recent years. These reports suggest:


4.2.3 RTM is so huge that its effect should be obvious when evaluating collision data (it should stick out like a sore thumb)

4.2.4 RTM effects need to be fully excluded from the results in order to find the much smaller effect of the speed cameras


When I first started investigating, there appeared to be no method available that could fully exclude RTM effects therefore I developed, published (in 2008) and used (in 2012) what I now call (in 2013) “The FTP (Four Time Periods) method”. The FTP method is capable of measuring RTM effects at speed camera sites and is also capable of fully excluding RTM effects from the final results. I believe it is the only method capable of doing either of these given the data available.


4.3 What is RTM?


Regression is “to return towards” and the “mean” is the normal or expected value of a random variable.


Therefore “regression to the mean” can be thought of as “return towards normal”. If there had been an unusual number of events in the recent past, then the number of events will tend to move in the direction towards normal. Speed camera sites are often selected following a high number of collisions therefore collision numbers would tend to reduce back to normal afterwards anyway, even without speed cameras. This is known as RTM and it can be demonstrated using random numbers.


A pack of playing cards can generate random numbers from 1 (ace) to 13 (king) and the "mean" of those numbers is 7 (the average of 1 to 13). If a card chosen at random has a number greater than the mean (7), then the next card chosen at random is likely to have a number lower than the 1st. In other words, the numbers are likely to “regress downward towards the mean”. Likewise, if the 1st card has a number lower than the mean, the number on the next card is likely to “regress upward towards the mean”. Furthermore, the greater the difference between the number on the 1st card and the mean, the greater the chance that RTM will occur when the 2nd card is selected, and the greater the magnitude of change that is likely.

 

This is RTM but, when researchers discuss RTM, we don't mean all of the general RTM both upwards and downwards following random fluctuations because these will tend to even out over time. We use the term RTM to refer to the RTM that influences the results, and that occurs because the site-selection process can bias RTM toward a particular direction.

 

4.4 Why does RTM affect results at speed camera sites?

 

4.4.1 There would be no RTM effects in the results if collisions always occurred at a constant rate (no random variations)

4.4.2 There would be no RTM effects in the results if the people selecting the sites had no knowledge of the collision history

 

RTM therefore affects results at speed camera sites because of two factors. The first is that collision rates vary and the second is that collision histories influence site selection. In other words, RTM influences the results because of the interaction between the site-selection process and random variations in collision rates.

4.5 How can the site-selection process lead to RTM?

 

To investigate how RTM occurs, consider a theoretical county called “Oxbube” where collisions are represented by randomly generated numbers. A site-selection process is performed in order to select where to use imaginary speed cameras and the results are examined. The analysis uses this spreadsheet (rtm_oxbube.xls) which has 3 pages.

 

On the 3rd page is a database, see Page 4.3 (only shows the top of the page). There are 300 rows in the database, each of which represents a road. There are 9 columns in the database, each of which represents a year. Each cell in the database is populated with a 0, 1 or 2 generated randomly. This gives a mean rate of 1 collision at each site per year.

 

The years 3,4 and 5 are used as the SSP (Site Selection Period) and, if a road meets the criteria of 4 collisions during the SSP, that road is selected as a speed camera site.

 

Although the authorities may try to start speed camera operations at new sites as soon as possible, there will always be a delay after the end of the SSP. This delay is represented by year 6 and the imaginary speed cameras start operating at the start of year 7.

 

On the 2nd page is the same database but only showing the roads selected as speed camera sites, see Page 4.2 (only shows the top of the page).

 

The results are on the 1st page (Page 4.1). Table 4.1 shows the total number of collisions in each of the 9 years at all of the selected sites.

 

It is standard practice to compare a 3 year period after the speed cameras were deployed, to a 3 year "baseline" period before (often the SSP). When this was done at the Oxbube sites, there was found to be a 31% reduction in collisions. These speed cameras, though, are not real (and could not influence randomly generated numbers anyway) therefore something else caused the 31% reduction. This is the effect known as RTM.


The graph on Page 4.1 shows how the 31% RTM reduction came about. There was a significantly higher collision rate during the SSP than at any other time therefore, compared to the SSP, collisions were around 30% lower at all other times, including after speed cameras started operating.


This analysis shows how the site-selection process can lead to RTM. Because the choice of sites is influenced by the number of collisions during the SSP, sites are likely to be selected following collision rates that were different to the mean. Collision rates will then tend to return to the mean afterwards, even if nothing was done at the sites.

 

Also, RTM is not a gradual change. RTM is a sudden change that occurs immediately after the SSP ends. It is so sudden in fact that collision rates should, in theory, be back to their mean rate the day after the SSP ended.

 

The RTM effect cannot be observed as a sudden change at individual sites, though, (because collision numbers are too low) but RTM can be observed as a sudden change when a sufficient number of sites are evaluated as a group. If the end dates for each site's SSP can be aligned, a graph of a group of speed camera sites should show RTM to be a sudden (vertical) change in collision rates directly after the end of the SSP.

4.6 Features of RTM in Figure 4.1

 

Figure 4.1 shows the same data as the graph on Page 4.1 but it is labelled to show the relevant features. The data falls into 4 time periods which I have named, in chronological order:


4.6.1 PreSSP
4.6.2 SSP ... (Site Selection Period)
4.6.3 ASBiC ... (After SSP but Before Intervention Commences)
4.6.4 After speed cameras start operating

 

A total of 121 speed camera sites were selected in Oxbube and each had a mean rate of 1 collision per year. Therefore the mean rate for all of the selected sites was 121 collisions per year (labelled).

 

The collision rate was higher during the 3 years of our SSP than it was during any other period. This occurred because it was the collisions during our SSP that influenced the selection of these particular sites.

 

In other words, it was the site-selection process that led to collisions being higher during the SSP than they were during all other periods.