What makes the analysis unique are the three authors, all staff at Stanford University. Rob Reich is a philosopher, Mehran Sahami is a tech innovator, and Jeremy M. Weinstein is a former White House staffer, and their prismatic view includes the technology, ethics, and related policy for each issue discussed in the book.
Albert Einstein once warned about losing control of the products of our genius: “The hardly bought achievements of the machine age in the hands of our generation are as dangerous as a razor in the hands of a three-year-old child.” When you add the current obsession with speed expressed in Facebook’s motto, “Move fast and break things,” Einstein’s toddler becomes even more disturbing. To these, the authors add their own caution: “If we accept that technology is simply beyond our control, we cede our future to engineers, corporate leaders, and venture capitalists.”
INHERENT PROBLEMS
Part one of System Error explains how we got to where we are, from the perspectives of the engineers’ narrow focus on optimization, the marriage of the engineers and venture capitalists, and the obsessive pursuit of disruption without regulation. All three influences share a myopic neglect of other consequences beyond market share and notice—costs imposed on others.
To understand the current tech ecosphere, we need to begin with the mindset of an engineer. The almost exclusive emphasis on measuring and optimization begins early in the training of computer scientists. The terms optimize and optimization were unknown prior to 1950, but computer science has elevated them to where some see algorithmic insights as a form of wisdom. The authors explain, “What begins as a professional mindset for the technologist easily becomes a more general orientation to life.” But there are deficiencies of efficiency, especially where the optimization mindset is misapplied or consequences are ignored.
In tech, venture capital (VC) is replacing older means of funding—bootstrapping or grants. “The potential for gaining riches from a future IPO [initial public offering] adds more tinder to the fire that fuels the breakneck pace of technological development.” And as VC fund managers look for hockey-stick growth, “blitz-scaling” prioritizes speed, causing some to invest less in security, to write code that doesn’t scale, or to even wait for things to start breaking before building in tools and processes, all in order to get big fast.
As technology companies accumulate and turn economic power into political power, engineers/CEOs are now able to set the rules for how they are, or aren’t, regulated. The authors note that “In 2019 and 2020, Facebook and Amazon spent more on federal lobbying than any other company, besting even defense contractors such as Lockheed Martin.” And now, looking ahead, it appears likely that Big Tech won’t submit to regulation without serious pushback. The book section titled “Government is complicit in the absence of regulation” explains these trends with examples.
FOUR FRONTS
Chapters four through seven illustrate where Big Tech is already off the rails or is heavily listing. Chapter four asks whether AI’s algorithmic decision making can ever be fair. The bias can come from the programmers writing the algorithms because they decide what criterion to optimize. The quality of the data used in the learning models for the algorithms can be flawed, or just the wrong data might be used by the designers. And there’s the question of whether you can reduce fairness to a mathematical formula. The authors note that “fairness is not easily understood as a timeless and universal thing on which we all agree.” They suggest designers should pay attention to what fairness means in particular social contexts, and they offer practical advice on how to govern algorithms. The three other questions covered in this section are equally serious:
- What’s your privacy worth?
- Can humans flourish in a world of smart machines?
- Will free speech survive the internet?
CAN WE REBOOT BIG TECH?
In the final section, there’s specific advice about three alternate routes for a different future. The authors encourage developing a greater appreciation of ethical issues among technologists. They propose regulations to check corporate power and call for “empowering citizens and democratic institutions to govern technology instead of passively allowing technology and technologists to govern us.”
System Error offers a valuable historical overview, but more than that, it’s a sorely needed ethics text for the guardians of Einstein’s three-year-old.
November 2021
