I recently read an article in Fortune magazine titled “Humans are underrated.” Its premise is that technology is advancing rapidly, more and more jobs will become automated, and the need (and pay) for relational soft skills will increase. According to the article, in the future, people with empathy, social sensitivity, collaboration, storytelling, leadership, and relationship-building skills will be much more successful than those with technical skills. The article also noted that “as demand for empathy grows, supply shrinks.” For example, college students have shown a decrease in “empathy” over the last 25 years (roughly correlating with the increase in time spent staring at screens and the decrease in time spent communicating face-to-face). It appears that the workforce of the future will require more “big empathy” skills than Big Data technical skills. The ability to understand others, to build trusting relationships with them, and work well with them (regardless of cultural, social, or other environmental background differences) may become more important than Big Data computer skills. What do these changes mean for Big Data technologists? Will their jobs be automated? Will they become obsolete?
Most companies today are increasingly focusing on becoming more data driven in their decision making and are hiring more data scientists and technicians to implement Big Data solutions. These professionals are pushing to capture all data in a digitized format, integrate it across corporate silos, and augment it with external data from social media or other sources. By amalgamating and analyzing this Big Data, companies hope to be able to detect trends in the data that will lead to useful insights and better decision making. Ideally, many of these insights could be captured and used automatically in real time to create tailored marketing messages and engage customers at a point when they are most receptive to the message. Will Big Data eventually automate most of the decisions people are currently making? That would leave mostly those with “big empathy” skills to oversee the data engines and debate with each other about how to tweak the rules before calling on the technician to actually implement the change. I think the essence of the answer lies in understanding a few concepts.
DATA QUALITY
One of my first computer classes years ago introduced me to the concept of GIGO (Garbage In, Garbage Out). This is still one of the most challenging aspects of Big Data—ensuring we base our decisions on good data. For most organizations, this means hard work combining data from multiple internal sources and “cleansing” it to create a single version of the truth. While this is doable, it’s still tedious, painstaking, and time-consuming work. Once these cleansing processes are created, they can be reused. But new data streams will always come in, and old data streams will be phased out, which will require changes and adjustments based on the quality and structure of the new data. With the recent advent of better tools for handling unstructured data (e.g., NoSQL databases), it’s conceivable that these challenges will largely be overcome and automated at some point in the future.
CORRELATION VS. CAUSATION
One of the basic underlying tools for all Big Data analysis is data correlation analysis. For example, an increase in the sales of homes may lead to an increase in moving van rentals the next month. The premise of Big Data analytics is that, by examining the data in a systematic way, we can uncover statistical correlations that aren’t apparent to a human examining the data without Big Data technology tools. Unfortunately, we know that just because there’s a correlation between data at any point in time doesn’t mean that there’s an underlying cause and effect in the relationships between the data.
One of the most common examples of this is the “Super Bowl Indicator,” which states that if a team from the original NFL (National Football League) wins the Super Bowl, the stock market will end that year in positive territory, but if a team from the original AFL (American Football League) wins, the stock market will produce a year-over-year decline. While there’s very high correlation between these events, there are no underlying reasons to justify a cause-and-effect relationship between these two variables. If your Big Data analysis uncovers a relationship between the number of homes painted fuchsia and increased sales of highway construction equipment, you may want to be skeptical and decide not to base your sales forecast on this information. Enabling Big Data tools to identify causation instead of mere correlation is currently a primary area of research, and it may take several years before this research results in mature tools with this capability.
CORPORATE DECISION-MAKING CULTURE AND PROCESS
At one point in my career, I reported to a director who confessed to heavily relying on the “gut feel” approach for making decisions. Even today, with more data available than ever before, many experienced business professionals still tend to rely on intuition instead of hard facts. In a data-driven company, a culture shift needs to occur where senior business leaders no longer answer the hard questions—the data will do that for them. Instead, they will need to focus on asking the right questions. Changing the corporate culture to trust and support the answers provided by Big Data may be the most difficult transition of all. It will take both Big Data expertise and “big empathy” skills to explain and “sell” the new corporate culture needed in a true data-driven company.
BIG EMPATHY
After examining these three roles, it doesn’t appear that the need for technical acumen will become less important anytime in the near future. This isn’t to say that technologists shouldn’t also be working on their “big empathy” skills. Indeed, if they want to influence their corporate culture to support the adoption of a Big Data culture, their “big empathy” skills will be a key component of their success because creating trust in the data and decisions driven by that data requires more than just technical skills. Having both Big Data and “big empathy” skill sets will allow professionals to stand out now and in the future.
December 2015
