The following article by Meg Kelly was posted on the Washington Post website April 27, 2018:
The president is fond of using data to prove the value of deregulation, but he should double check the numbers before calling them fact. (Meg Kelly/The Washington Post)
“According to a survey by the National Small Business Association, the average small business spends $83,000 to comply with regulations in just their first year of existence.”
— President Trump, remarks at the Latino Coalition Legislative Summit, March 7, 2018
President Trump often highlights the burdens of government regulations on small businesses. He regularly claims (without verification) that he has repealed more regulations than any other president. This survey folds nicely into the president’s talking points, but the reported cost caught our eye.
So, does this survey, released in 2017 by the National Small Business Association, justify Trump’s claim?<
The Facts
The number — $83,000 (or $83,019.23 as the survey so precisely says) — sounds impressive, but the data was drawn from a survey conducted among business owners who had willingly provided their email addresses to the National Small Business Association (NSBA) either because they are members, prospective members or previously signed up for updates from the association.
We shared this with The Washington Post pollsters, who were quick to point out that NSBA’s survey was not a random sample of all small businesses, but only those that were members or prospective members. Unlike a poll of small-business owners that used random sampling, this type of targeted poll cannot be used to make claims about all small businesses (“the average small business”); it only reflects the views of those who choose to participate.
Regular readers know that we have urged caution against relying on surveys or polls among self-selected groups that are not designed to generalize to broader the population. We’ve previously dinged the National Rifle Association, the U.S. Chamber of Commerce and former president Barack Obama for this very reason.
Looking at two other surveys from the NSBA help illustrate why this matters. Thirty-six percent of participants in a 2016 survey report having from one to five employees. In the 2017 survey, that number jumps to 48 percent, and dips to 44 percent in a 2018 survey. Of course membership levels fluctuate and businesses change, but these shifts are quite large, suggesting a significantly different picture of businesses that participate from year to year.
Regardless, let’s take a deeper look. As part of its advocacy mission, the association conducts several surveys each year to find quantitative support for issues relevant to its members. This particular survey was the first of its kind, apparently inspired by hearing from members that the daily cost of doing business was becoming burdensome. In other words, the survey began with an underlying assumption that regulations were costly.
The NSBA provided the survey to more than 45,000 business owners (or roughly one-eighth of 1 percent of all small businesses in the United States, according to the Small Business Administration) — and 998 responded, or about 2.2 percent. In this case, the report says 27 percent of respondents had gross revenues between $1 million and $5 million and nearly half employed between one and five people.
For the purposes of this survey, the association defines regulation as anything the government requires a business to do. That means questions varied dramatically — from ranking “how burdensome” the federal tax code is to the burden of the clean-water rules or state licensing requirements. It also combines current and proposed regulations into one speculative category. Molly Day, the NSBA’s vice president of public affairs, said the range of regulations included in the survey were decided by the association’s veteran staff members and member-driven oversight committees.
Participants were asked to self-report a “reasonable” estimate of the amount their business spent on regulations in its first year. Day said the organization compiled the data, removed any clear outliers (anything far exceeding $5 million) and then averaged the remaining numbers in order to determine the average amount a small business spends to comply with federal and state regulations. Before publication, the results were reviewed by a committee of veteran small-business owners.
The report also says that the “average” small business owner spends $12,000 per year on regulations. It is unclear what accounts for the $71,000 difference with the claim that first-year costs amount to an average of $83,000. Looking at the U.S. Small Business Administration’s calculator for start-up costs, only a few of the budget areas it lists — permits, legal fees, accounting, insurance and organizational dues — can generally be considered costs caused by complying to regulations.
We also asked the NSBA for a breakdown of how long respondent’s businesses had been open. Presumably, the cost to start a business 10 years ago differs from 10 months ago, and the more recently a company was started, the more accurate the owner’s estimate. But on this survey, the NSBA said it did not ask that question.
However, for a rough idea, they pointed us to two of the association’s other surveys with similar demographic breakdowns, which showed on average respondents had been in business for more than 12 years. It also showed that nearly half had been in business for over 20 years. Moreover, the estimates were not weighted or broken down by business type or size of business. After all, it’s unlikely that a small team of Realtors and a fine dining restaurant would have similar start-up costs.
Another oddity: 43 percent of respondents said the Affordable Care Act was “very burdensome,” and 23 percent said it was “somewhat burdensome.” That adds up to 76 percent of those responding. But 82 percent of those responding have fewer than 20 employees — and the ACA applies only if a business has 50 or more employees. That suggests that some of the regulatory burdens are more imaginary than real.
A White House official defended the president’s use of this survey, noting the survey previously had been cited by media outlets including CNBC and Investors Business Daily. “The President cited a survey from a nonpartisan organization; these results were widely featured by mainstream media outlets highlighting the burdensome regulations that face small businesses across the country,” he said. “President Trump’s economic policies have given hope to small-business owners and those who dream of starting their own businesses.”
Pinocchio Test
There’s little question that small businesses struggle with regulations. This survey, however, only drew its sample from a small pool of small-business owners who are members (or prospective members) of a group advocating for their interests. The Post’s pollster pointed out that it’s like going to a golf course and asking who likes to play golf.
Without a weighted analysis and a more representative sample, the average the NSBA calculated is so unspecific that it’s effectively useless. To the NSBA’s credit, when we pointed out the holes in its analysis, it said it was a clear oversight at the time when it conducted the survey.
As always, the burden of accuracy is on the speaker. President Trump (and his staff) should steer clear of this kind of survey — regardless of whether or not they support an argument. He cannot say that the results represent an “average small business” as the survey was not a random sample. We were torn between Two and Three Pinocchios, but ultimately the overly simple calculations and Trump’s eagerness to tout numbers that support his concern about regulations pushed us to Three.