ProBeat: Throw out all your pre-coronavirus surveys and start again

I got a pitch this week to cover an annual developer survey with tens of thousands of respondents across the world. It was pretty straightforward — reputable company, big sample size, tons of findings. I quickly noticed, however, that the survey was conducted from February 5 to February 28.

February. As in, three months ago. As in, long before the World Health Organization declared the coronavirus outbreak a pandemic on March 11. Before Italy overtook China as the country with the most deaths on March 19. Before the United States overtook China and Italy with the highest number of confirmed cases on March 26. Then we had two more months of lockdowns, social distancing, layoffs, work from home, and deaths.

The point is, those survey results are useless. And so are your results if your company conducted a survey about anything to do with technology, workers, or business in general earlier this year. I let the company know my thoughts and then asked for comment. The company agreed, to an extent:

Data around languages, frameworks, and technologies remains informative and helpful for developers, technologists, and companies that are interested in the latest trends in the space. However, given the current environment, any data around employment trends (e.g. compensation, looking for a new job, etc.) has most likely changed significantly since we surveyed our users and should be viewed through that lens.

No. None of it is helpful anymore. This is a global, life-changing event we are living through. The languages, frameworks, and technologies that developers are interested in and not interested in are going to change. In fact, they already have. COBOL developers are suddenly in extremely high demand because governments and banks distributing stimulus money still use systems written in the 60-year-old programming language. That’s a drastic example, but it’s easy enough to extrapolate to every corner of the world where technology exists.

I’m not naming the company in this case because I don’t want you to focus too much on this specific developer survey. This is about all tech surveys — internal, external, local, global, you name it.

A survey from earlier this year simply can’t factor in the major shifts underway in tech. For example, I looked back over the stories in this column since February and every one somehow ties back to the pandemic. ProBeat has covered permanent WFH, a cancelled smart city, surveillance drones, ecommerce exploding, contact tracing, invisible AI, Zoom’s mistakes, tech events, and collaboration tools.

The world is changing quickly. Even if you believe your industry isn’t directly impacted by the pandemic, it will be indirectly impacted by how people work, evolving technology, and the economic fallout. That will drastically influence your tech survey results right across the board.

So back up, throw out your results, and start again. You don’t even necessarily need to write a set of new questions or talk about COVID-19 specifically. You just need to make sure your tech survey wasn’t conducted before a historic global pandemic changed everything.

ProBeat is a column in which Emil rants about whatever crosses him that week.