ARE STOCK ASSESSMENTS TOO OPTIMISTIC

 

A peer-reviewed stock assessment represents the gold standard for fisheries management, a document representing the best scientific information available, which will form a firm foundation for management measures.

Or so we always believed.

Admittedly, there have been doubters.  It is a virtual certainty that every time a new stock assessment is released, if that assessment suggests that the stock is in decline, and that more restrictive management measures are needed to conserve and rebuild the fish population, a motley assortment of commercial and recreational industry members will come out of the woodwork to cry, “The science is wrong!,” because any science, no matter how thoroughly peer-reviewed and how carefully done, will always be wrong if it threatens those folks’ short-term cash flows.

In such cases, we will be regaled with tales of fish being abundant somewhere, whether farther offshore than they typically travel, for farther north, or south, or anywhere else than where the scientists were looking, usually salted with comments about how “people who are out on the water every day” have a far better idea of the health of fish stocks than “scientists who don’t know where the fish are and spend their time sitting behind computers.”

But regardless of the precise nature of the comments, the underlying assumption that such people make is always that the assessment is wrong, because there are really more fish in the ocean than the assessment suggests.

Recently, a team of scientists published a paper which suggests that the truth lies in the opposite direction—that assessments have historically been somewhat biased toward optimism, and overstate the size of fish stocks, particularly those that are suffering from overfishing.

Titled “Stock assessment models overstate sustainability of the world’s fisheries,” and appearing in the August 22 issue of Science, the paper avers that

“For stocks that were overfished, low value, or located in regions with rising temperatures, historical biomass estimates were generally overstated compared with updated assessments.  Moreover, rising trends reported for overfished stocks were often inaccurate.  With consideration of bias recognized retrospectively, 85% more stocks than currently recognized have likely collapsed below 10% of maximum historical biomass.  The high uncertainty and bias in modelled stock assessments warrants much greater precaution by managers.”

That is certainly a startling finding.  As the editor of the article noted,

“Assessment of the status of fisheries stocks is a key component of their management.  Although there has been much debate around how to do fishery assessments, there has been a general belief that estimates are roughly accurate.”

If the article’s authors are correct about many assessments having an optimistic bias, their findings could place the efficacy of many fishery management programs in doubt.

The authors argue that

“Best practice methods for assessing fisheries involve complex models integrating past catch data with biological and other information.  Complex stock models can include more than 40 different parameters and setting related to fish life history (e.g., natural mortality, length and age at maturity, and growth rate), catch (e.g., landings, gear selectivity, and discards), effort (e.g., days fished and number of hooks), and management controls (e.g., fleet allocations and allowable catch).  The many estimated parameters and settings can lead to model overfitting, whereby uncertainty accumulates with each additional estimate.  Accuracy of simpler stock assessment approaches is typically evaluated relative to complex models; however, the accuracy of complex stock models remains unknown because the true fish biomass is not directly observed.”

In their study, the authors took advantage of the fact that the most recent year in any stock assessment is the year that tends to be plagued by the most uncertainty, as the estimates are only based on a single year of data.  As time passes and additional years of data are added to the available data set, the uncertainty surrounding earlier estimates is steadily reduced.  Thus, in evaluating the precision of earlier assessments, and determining whether such assessments’ conclusions tended to overestimate stock size, the researchers noted that

“In the absence of accurate biomass data, a retrospective analysis of differences in estimated stock biomass reported over time can indicate the magnitude of uncertainty and test for systematic bias.”

Thus, they wrote,

“we relate past stock assessments to the most recent assessment.  Our reasoning is that modeled output of the most recent assessment should, on average, be the most accurate because estimates are hindcast using the longest time series and with the most knowledge for defining model structure.”

Their goal was to determine

“whether systemic bias exists between past and most recent assessment estimates and how that bias varies with stock status,”

observing that

“Bias matters because overfished stocks might not be identified for recovery actions if stock size is overestimated, or recovery actions may have unnecessary economic consequences if stock size is underestimated.”

The study encompassed 128 species or species complexes, 230 stocks, and 986 stock assessments; on average, the  biomass estimates for each stock spanned 47 years.  It found that annual estimates of biomass often swung wildly from year to year, with such swings sometimes “greatly exceeding” the uncertainty levels calculated in the stock assessments.  The study also found that there was a general trend indicating a rebuilding/recovery of fish stocks, although

“The recent over-all rise was driven by sustainable stocks…whereas overfished stocks remained low in recent years on average.”

It also revealed that older stock assessments tended to be more optimistic, when compared to the most recent assessments of stock status, with the level of optimism growing as the researchers looked farther back in time.

“[P]lots for overfished stocks generally featured upward slopes in the final 2 years of older assessments—suggesting improving stock size—that were no longer evident in the most recent assessments.  Such phantom recoveries progressed across assessments as updated stock assessments were released.

“…Large downward revisions in stock biomass in later assessments, and a consistent tendency for phantom recoveries, typified overfished stocks.”

The researchers found that

“Uncertainty associated with stock assessments include interrelated process, observation, model, and estimation uncertainties.  These uncertainties led to high interannual variability and consistent bias evident in our analyses, raising doubts about the accuracy of integrated stock assessment models regardless of sophistication.  Bias erred toward stable stocks, with overestimation of biomass for overfished stocks and underestimation for sustainable stocks.”

They suggested that

“The tendency for bias to inaccurately imply stable stock trajectories for both increasing and declining stocks suggests systematic technical issues or confirmation bias, where overfitted models align with modeler’s expectations.  Some model parameters are particularly relevant in this context, including natural mortality and the steepness of the stock-recruitment relationship…Subjective decisions on such highly uncertain parameters provide a possible pathway for systematic bias and management failures.  In particular, poor parameter choices can delay recognition of collapsing stocks, which become obvious only with subsequent data and hindsight.”

And noted,

“Although some stock assessment reports candidly discuss parameter adjustments needed to avoid projections of stock collapse, parameter tweaking is rarely well communicated to managers and policy makers.  Modeled output presented in reports typically depicts uncertainty generated through randomization routines included in the model algorithms.  By contrast, the much higher uncertainty contributed by model adjustment, choice of model structure (including spatial dynamics), input parameters, and inadequacy of input data is frequently overlooked…Nevertheless, recognition of uncertainty by managers is not enough unless it also elicits precaution, such as reducing catch quotas to allow for assessment errors.  [emphasis added]”

The researchers also warned,

“Although inadequate precaution can generate short-term catch benefits, it erodes long-term societal interests through loss of species with immense economic, environmental, cultural, recreational, and spiritual value.  Considering just the economic value, an inability to reverse decline when fish numbers are decreasing ultimately has far-reaching negative impacts on the fisheries workforce, ecosystems and their stability, and the world’s capacity to provide protein to an increasing population.”

So will the recent paper spur fisheries managers to employ greater precaution when analyzing stock assessments and setting management measures?

Probably not.

As the New York Times reported soon after the paper was released, it experienced a very mixed reception.  The Times noted that Dr. Boris Worm of Dalhousie University in Nova Scotia, Canada,

“said that the study breaks new ground in revealing a bias toward optimism,”

while Dr. Ray Hilborn of the University of Washington was unimpressed.

“Many fishery managers, he explained, already look back at historical trends to correct for a possible tendency to over or undercount fish populations.”

And Dr. Steven Cadrin at the University of Massachusetts, Dartmouth went a step farther than that, calling the paper’s findings “invalid,” in part because the researchers assumed that the most recent stock assessments were the most accurate when, in his view, that was not necessarily true.

So at this point, the impact that the paper might have on fisheries management is not particularly clear, although given the institutional inertia found in any government body, the odds are probably against it inspiring managers to adopt a greater level of precaution when setting annual catch limits in United States fisheries.

Still, the very fact that the paper is out there will spur some discussion, and that alone would be a good thing.