The Cloud in your Pocket

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


I talked last week here about the arbitrary, if not blurry, line between the server and the app. Products are products, features are features. It is probably good that you don’t care whether it is the supercomputer in your pocket that enables that Instagram filter, or some server running in a data center in Nebraska. Through thousands of layers of magic, and some cool math, internet glosses over distances.

The general trend here is that more of the computation is slowly moving towards the cloud, and more devices and apps are just becoming windows into what’s running on the server (“the cloud”), as opposed to having any logic in them. In the industry jargon, more and more devices are just becoming thin clients.

You can view lots of businesses through this lens. Internet first killed the shrink-wrap software, and now it’s on its way to making it all some web apps running in your browser. Google (and Nvidia, and Sony, and Microsoft) even want to kill the gaming console by streaming you interactive video games. Whether that thin client, through which other things run, is a browser, or a VNC client, or some other shell is really an implementation detail.

Physics Strike Back

Yet, you can’t just avoid the physics entirely. Stadia might be right around the corner but its limited rollout points to how hard it is build not just a high-bandwidth network, but also a low-latency one. I once worked at a company whose entire premise was building a cloud-first filesystem (with local data as a mere cache), but I also found it somehow poetic that the previous tenants for our office was OnLive, the original “streaming gaming company”. We did a tad better, but not much better.

In fact, a good half of my time at Uber was really dealing with this problem. It’s hard to figure out whether things should live on the server or the client (i.e. “the app”). Here’s some context. My team worked on the authentication system all Uber apps worked on, which meant that we had to build and support various account management lifecycle flows, such as account recovery (“Forgot Password”). You could put the entire logic on the server, and serve the flow with a “Webview”, an in-app-browser, but you end up losing some of the benefits of native form handling, and possibly make too many network requests that slow down. But if you do put things on the client, you lose the ability to change things on the fly, or at least make it much more complicated. This stuff is hard!

Latency isn’t however the only reason why you’d consider moving more of the computation to the “edge”. Just ask Apple. The one-time world’s biggest corporation established itself as the privacy czar of the tech world, in the face of fierce competition from its data guzzling peers. Where Google makes billions by knowing more about you, Apple tries to do the same with knowing less.

Thermonuclear War on Privacy

It’s important to remember how we arrived here. For many years, tech companies like Google pushed the scientific edge in deriving insights from huge data sets, which allowed it to build even more engaging services that pulled in more services that provided both larger and higher fidelity data sets; a positive feedback loop.

Apple’s response to this has been multi-faceted. First prong in the approach is to change the narrative around privacy; that make people realize (or believe, depending on your initial stance) privacy is a human right. Yet, in a world where many people seem to be happy with the compromise of exchanging some personal data in return for free products, that can be a tall order. How do you even communicate to millions of people that their data doesn’t leave the device? For many that aren’t technologists, the connection between “data some company collects on me” and “features I get to enjoy” simply isn’t there.

The second prong in this approach is Apple moving more of the computation that generally happens in the cloud to the client. This started a few years ago, but seems to be picking up. Apple showcased how it can detect and identify faces solely on the device, and followed it by even more advanced recognition of objects and sceneries. The data never leaves your phone.

Yet, it’s hard to shake off the feeling building sophisticated ML models without enough data. Apple also invested deeply in what is called differential privacy. This grossly means that Apple can ship data from your device (“upload it to the cloud”), but scramble it enough it before to make it anonymous. In some way, Apple still does a bunch of “machine learning” in the cloud. A good example for this is the predictive text. I don’t need to know that you write “Manchester” and follow it with “United”, but if I collect enough even-scrambled data from people, I can tell that one follows the other often enough.

Uber is another firm that invests heavily in differential privacy, especially for sharing data internally. Most people are really interested in the patterns in data, not the individual pieces. What is the real use case for someone to look at my individual trips?

To me, it seems like there’s room for investment in privacy preserving or even privacy enhancing research areas like differential privacy. The field is still nascent, at least compared to the years of person-hours and billions of dollars spent on building models that demand more and more data. As I’ve written various times before, it’s likely that we have not only over-estimated the value of data (by undercounting its liabilities), but we’ve generally not been ambitious about what we can do without putting all the data in one place, waiting patiently to be hacked or use some nefarious purpose.

On its face, tide seems to be slowly turning, at least away from “more data is good” dogmatism. I am old enough to remember when it was not just “edgy” but generally accepted to say that “privacy is dead” in Silicon Valley. That was never really true, but it provided a good cover for those who never cared about other people’s privacy. It’s true that in America, where privacy is generally considered a commercial matter, as opposed to, say, Europe where it’s an enshrined civic right can’t be priced. As I discussed before, some of the liabilities around data are more widely recognized.

So, which one is better? Cloud or the edge? Server or the client? Datacenter or the app?

Outlook: Cloudy?

I know this is a cop out, as far as predictions go. But here’s the truth: we’ll probably end up somewhere in the middle where we use a combination of both. And in that new world, we’ll have new questions to grapple with. For example, differential privacy is great for shipping data around, and models that allow devices to make predictions can be downloaded to be run on device. Yet, devices, once they are shipped, are hard to update, unlike a server that can acquire truly new capabilities, and increase its performance. It’d be misleading to say an edge-computing heavy approach wouldn’t result in some casualties in terms of products and features.

On the flip side, even with technologies 5G rolling out slowly, there’ll still be huge performance benefits to doing some computing locally. The line between the server and the client will depend on many factors. Some connection technologies will be rolled out in some countries than others. Regulations such as data residency requirements will add further friction in some places, but they will also act as encouragement in others. Fears of balkanization will do the same. Higher costs of breaches, both regulatory and publicity wise and increased sensitivity towards privacy will make edge computing more attractive.

Service Oriented What Again?

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


People commonly ask me what I did at Uber. I wrote code, is one approximation. In many people’s mind, many “companies” exist as some app on their phones. Most people realize there’s a “backend” or “datacenter” or “servers” somewhere, but the lines are blurry. The most high resolution view of Uber, or any tech company, is the demarcation between the “app” and the “server”. Engineers are engineers, as I’ve written before:

[Uber] felt like a tech company in that people used an app to get where there were going, but a lot of the work initially was really about keeping the lights on, while we either wired together off-the-shelf tech, or should have.

That’s a good starting point, but it belies some of the real challenges of running a sizable technology stack like Uber. The fluid operation hides much of the complexity, but there are many parts in play. There are obvious ones, like “matching riders with drivers” and “maps” but that’s barely scratching the surface. Think of the payments, automated support, authentication, communications, promotions, and such.

There are many different ways to draw an org chart here. You can have functional teams, or cross functional teams, or a matrix organizations. More thoughtful companies realize companies in different stages in their evolutions need different organizations, less thoughtful ones generally go with what is on the cover of HBR or top of High Scalability that week.

How to Hire Thousands of Engineers

Regardless of your overall organizational design, on the technical side, you need these different “features” be developed at the same time, by different teams, ideally without those teams stepping on each other teams’ toes too much.

Like many other technical organizations of its size, Uber settled on a service oriented architecture, or SOA in short. This largely means that different services are managed by independent teams, and they operate on a protocol they agreed on.

For example, a single “service” can be responsible for calculating “ETA between point A and point B” while another could be responsible for “calculating the estimated fare between point A and point B”. I am stylizing things here a bit, but you see the point. One of them is really a routing problem, whereas the other has to take into account many things like promotions you are part of, the distance between A and B” etc, etc. Then, there’s obviously a different service that talks to the “app” that collects all that information and sends it over the internet.

There are hundreds of nuances here, of course. Where the boundaries of these services are a common source of tension in many tech orgs, as well as how these things are independently monitored (i.e. ignored), deprecated (i.e. really ignored) and decommissioned (i.e. ignored forever). The benefits, generally, overweigh the costs though.

That is a lot of services.

There are significant technical benefits to a service oriented architecture. One big win is that since services are relatively isolated from each other, you can have more graceful failures rather than “Uber is down” type of catastrophes. Some of that is more aspirational than realistic, but it still a good rule of thumb. If you have a service that just manages “images of trips”, and it’s down, that’s probably fine for a bit. Your receipts might look funky for a bit, but Uber overall would work.

Services All the Way Down

A less obvious benefit is the ability to combine and compose these services make building new features much easier. Instead of simultaneously wading through thousands of lines of code and tip-toeing around other people’s work, you can just see if there are “interfaces” that you can put together like bunch of LEGO pieces. Better yet, you can be inspired to build entirely new products by simply being inspired by all the features so clearly laid out in front of you. 

The productivity benefits of these abstractions are hard to overstate. I am obviously biased here. But besides the “forward-leaning” work culture, Uber’s high level and number of abstractions was one of its competitive advantage for a long time,. You could imagine a feature, and have it working in production (as in customers’ hands) in less than a couple weeks, even including the App Store review process. 

But of course, the flip side is also true. Sometimes things break in surprising or even amusing ways, and things can be hard to debug. There’s probably not a single person who can describe you in detail how Uber with all its services, it’s a living organism of its own. You can probably put enough people in a room, and they can draw you a systems diagram but probably, by the time they are finished, the technology would have changed significantly enough to make your diagram obsolete.

My co-host Ranjan likes to point at growing inaccuracies of Uber’s ETA as a sign of impending doom. Surely, there’s a point here about the ETA accuracy is an underrated proxy metric for people’s satisfaction with Uber in general, but fact of the matter is it’s just a single service, owned by a single team.

Of course it didn’t start out that way. Uber, like many other orgs, started out (way before my time) as a single, monolithic service, hilariously called api. For my first few months there, I worked on rationalizing that madness in small part. My favorite part of this process was seeing how this distributed architecture made it clear how we were “shipping our org chart”, as they say. Most teams happily played along, but there were of course holdouts who believed their privileged status deserved some preferential treatment. Engineering, in the end, is as political as any other function.

SOA is not DOA

SOA is not a panacea, but it can return major business benefits. Jeff Bezos now is a household name thanks to his wealth —and a few other things—, but he was first and foremost an engineer who saw the potential upside of systematically tearing apart a technical organization and reshaping it around functions. 

Here is a semi-famous rant by Steve Yegge, a former Amazon engineer who then went to Google from 2011:

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon’s vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon’s dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

It’s eerie to think Yegge wrote this in 2011, which is then a few years after Bezos sent his warring orders. Many consider AWS as something that Amazon has been working on for many years, but the reality is probably closer to (but not exactly!) Amazon slowly externalizing services that they have built for their internal services.

It’s easy to sound facetious now, as I’ve spent several years working on such services. Of course, “simply externalizing” a service is a major effort requires coordination of many different teams in a firm. And many of the benefits of SOA really become apparent at a certain size. Besides the obvious organizational challenges, the technical overheads in even the best implementations are real.

Yet as the software itself becomes more and more abstracted out, and more and more companies become “tech” companies in some vague sense, how your technical organization is built might end up returning huge yields. And as we software eats more of the world, and starts guiding, defining, ruling our lives, it could be useful to anyone working in technology, even in a business function, to understand how software really works.


What I’m Reading

Why High-Tech Commodization is Accelerating: HBS Professor Willy Shih writes about how abstraction of technology presents challenges to building competitive advantages in high-tech industries.

Sophisticated design and simulation tools are de rigueur for modern product design. Tool suites that allow companies to analyze structures, noise and vibration, acoustics, thermal behavior, fluid flow, motion, and dynamics have democratized design. They have lowered the entry barriers in engineering-intensive sectors, automated the process of cumulative innovation, and allowed new market entrants to stand on top of a pyramid of earlier innovations. In short, they have unleashed a powerful force that’s driving commoditization in globalized markets.

The Twilight of Combustion Comes for Germany’s Empire of EnginesElectrifying cars changes entire supply chains, which employ thousands of people across many industries. What does that mean if your industry might is building that complex supply chain, that’s not just a simple engine?

BMW hasn’t been willing to forecast how many jobs might be lost, but the company’s human resources department acknowledges that making an electric motor takes roughly 30 percent less time than a gasoline-powered engine. “The numbers of hours worked to make an electric motor are smaller than for a combustion engine,” said Carreiro-Andree, the BMW board member. “That’s a fact.”  

The Good, The Bad, and The Ugly of Startup Options

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


Suppose you are joining an early stage startup and you are faced with choosing an offer that is either heavy on the options or cash. Which one do you pick? Do you lean more decisively on the stock side, or prefer cold, hard cash? Obviously, this is not a common occurrence, but it’s not purely stylized. I’ve even gone through this ordeal twice.

There are several ways you could approach this problem. For example, you could be a Paul Graham retweeting, Hacker News reading believer in the tech ecosystem-cum-bubble. Presumably, you are well aware the chances of hitting a home run is tiny, but the potential is really big. If you are really nerdy, you can put this in to an Expected Value type of equation. Following this startup-forward approach, you clearly want an equity heavy package.

You can also follow a more capitalism-forward approach to arrive at a similar conclusion. You’ve read your microecon textbook, and realize production is a function of capital and labor. And given our capitalist setup, you then realize, the spoils flow to the capital, as opposed to labor. In other words, if you want to make some real money, you don’t want to just turn your labor into cash, but rather be on the capital side of productivity equation. Then, again, you lean towards equity. 

Yet, you could just be a cold, clinical realist and think that most venture backed startups fail. In this perfect storm of cheap capital and irrational fascination with anything startup-y, there are so many startups, you think, that come and go. Put differently, you again make an expected value calculation in your head (or Excel, I am not a cop), but your probability is much closer to zero than it is in the startup-forwardapproach. In this case, you just take your cash and go home. Let’s then call this the cynicism-forward school of thought. 

Of course, it is hard to separate realism and cynicism at times, but it’s not like you can pay your grocery bills with unbridled optimism and La Croixs you, ehem, borrow from the fridge.

Or, as my cohost Ranjan put it in the The Margins Executive Slack Lounge (it’s ok to be jealous), you might think this is a futile exercise. Most startup offers are really come in narrow bands, that are simply a function of your level of experience and the stage the company is in. Even if you convince yourself that you are making a good decision by taking more equity (or cash), it’s all a shot in the dark. No firm will realistically show you their cap table, or tell you the number of outstanding shares. 

Even if you magically got some of that and put them in a spreadsheet or some app, there would still be so much hidden from you in terms of dilution rights, clawbacks, ratchets, it’d be pointless to lose sleep over any of this. To be frank, I am not sure what you’d do in this case, or what you would call this philosophy. I’m going with gamble-forward. They don’t call it a startup lottery for nothing!

Let me say this outright: I fall somewhere in between the “capitalism-forward” and the “realist” approach, but really, lean towards the former. If you are accepting a startup offer, you should be better off with more options, rather than more cash, but only after covering your most essential needs. If you are really worried about immediate cash, then you should probably not join a startup, as there are much more surefire ways to make a lot more money. 

Of course, you might say as one of my favorite finance professors like to say, “Nothing Compares to You”, Can. That is true; every financial situation is unique and you should definitely not take financial advice from this newsletter. I happen to not have any debt, and have some savings to live off of for a while.

But then, even my personal preference are built on the assumption that the capital structures that were built for startup employees many decades ago still are relevant to the current macroeconomic conditions. What if they are not? What if the deck is so stacked against early stage employees that it’s really, really hard to tell people to work for equity these days? Would that be too cynical?

This is what Steve Blank has to say on Harvard Business Review, just last week:

Investors and founders have changed the model to their advantage, but no one has changed the model for employees. Moving the liquidity goal posts may have removed the incentive for non-founders to want to work in a startup versus a large company. Stock options with four-year vesting period are no longer a good match for employees when it may take 10 to 12 years for the company to go public or be acquired.

Blank’s argument largely builds on how the rise of growth capital has changed the mechanics of early employee stock options, much to the detriment of its recipients. As time to liquidity events go from a few years to more than ten, the early employees who forego cash see more uncertainty build up.

With each infusion of cash, the capital structure becomes more complicated, with dilutions, ratchets, preferences, clawbacks and various other complexities slowly eating away at any chance of meaningful financial outcome. This is all mechanical, but also, these firms are run by people who do have human flaws. As more startups, if you can call them that, get accustomed to such monstrous cash inlays, they become lax with financial discipline, further lowering the potential outcomes.

In such an environment, can you really blame potential employees to sour on the whole idea of startups? The exorbitant costs of living in places like San Francisco definitely colors people’s decision here, but I’ve heard first hand from CEOs that hiring has practically become impossible in places like Bay Area. Prospective employees simply discount their options to zero. Such a cynical and negative hiring environment is not conducive to long term health of an industry that is built on not just increasingly rare skills, but also optimism to a fault. Something has got to give.

I joked on Twitter that we always talk about Venture Capitalists in the tech ecosystem, but you rarely hear about the Venture Laborers. And I should know. I worked on not one, not two, but three firms as an early employee (well, sort of). They all got “acquired” but, as my financial advisor would testify, I definitely have not seen much (any?) of those spoils. Of course, that also has more to do with the why those quotation marks are there in the first place, but the point remains.

There’s definitely some hope at the end of the tunnel. Many companies such as Pinterest and Quora have extended their exercise windows to several years, allowing employees some flexibility. On the transparency side, Square engineer and writer Jackie Luo is attempting to bring some transparency into the startup windfalls by anonymously collecting “exit” outcomes. These are all good steps, but they are not enough. A systemic issue will not be solved by individual actors, whether they are persons or companies. 

And while I play a cynic on Twitter often, I still believe in the tech ecosystem to a force for good. Sure, it can be at times a bit hectic, or truly dark. Nonetheless, for the ecosystem to thrive as much as it did before, the doors of opportunity should be kept open for new entrants as well.


What I’m Reading

This week, it’s about cities. I’ll let you guess why.

San Francisco’s Slow Motion Suicide: Venture Capitalist Michael Gibson laments about San Francisco on The National Review:

Housing and zoning committees obscure responsibility for governance. But somewhere in the bureaucratic hierarchy faceless city functionaries administer labyrinthine regulations that benefit the rich over the poor, the old over the young, the here over those to come, the past over the future.

Is This the Neighborhood New York Deserves?: Moving to the East Coast, Hudson Yards gets The New York Times treatment:

With its focus on the buildings’ shiny envelopes, on the monotony of reflective blue glass and the sheen of polished wood, brass, leather, marble and stone, Hudson Yards glorifies a kind of surface spectacle — as if the peak ambitions of city life were consuming luxury goods and enjoying a smooth, seductive, mindless materialism.

this land is your land: Somewhere in between the coasts, Anne Helen Peterson talks about the coolification of small American towns:

Cue: the mid-size city migration. Our parents’ generation went to the suburbs. But many of us have internalized the notion that the suburbs aren’t cool — and lack the walkable culture we’ve become accustomed to in the big city. So we move to a “cool” affordable city and contribute to the cool-ification — and, in the process, make it less affordable for those who’ve lived there before.

On Global Accountability

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


Hi. This is your host Can Duruk speaking.

Earlier, this week Facebook CEO Mark Zuckerberg went on a surprise charm offensive, literally asking to be regulated. He first published an opinion piece on The Post, and now is rubbing elbows in Europe with the Eurocracts. What a turn of events!

But maybe it’s not too surprising. Just a few months ago, Facebook was seen as the main driver of a major genocidal violence in Myanmar. At the time, I had visited Myanmar as a tourist and awestruck by the presence of Facebook in the country.

Following, I wrote a piece about how Facebook, a US company with most of its users outside of the US, can be held accountable for a US national publication. Ironically, the piece never made it to print due to another Facebook crisis. The narrative moves on.

I’ve decided to publish it here instead for all The Margins readers.


Facebook’s own blog post about its role in the genocide in Myanmar is not particularly flattering. In it, the company admits that haven’t done enough to keep Facebook “from being used to foment division and incite offline violence.” That admission isn’t surprising: There’s a very long list of groups who blame Facebook for its involvement in Myanmar, including Burmese activists, the United States government, and the United Nations.

What is surprising, however, is the sleight of hand that follows. While Facebook hesitantly agrees that it hasn’t done enough, it claims that the situation on the ground in Myanmar was so dire, so complicated, so violent that it could only do so much. “Facebook alone,” the company announces in its blog post, “cannot bring about the broad changes needed to address the human rights situation in Myanmar.”

Digital is Real

Facebook’s deflection of responsibility is merely the latest instance common line of argument that social media companies like Facebook put forward is that their work exists on a different plane of reality. The digital realm ties into the analog, but the relationship is not a two-way street. Rather, they claim, it is a set of two one-way streets. One of these streets is from computers to the real world, where only the good stuff travels, enabling free speech, liberating the oppressed, democratizing the internet. The bad stuff, however, only goes the other way, where bad individuals misuse and abuse internet platforms. In other words, Facebook argues the good things happen on Facebook, but the bad things happen to Facebook.

The report commissioned by Facebook acknowledges a key fact about Facebook’s role in Myanmar’s crisis: that in Myanmar, there’s no internet, there’s only Facebook. There are an equal number of Facebook and internet users in the country, and even the government officials use Facebook without having a separate online presence. This didn’t happen by accident: as Myanmar transitioned from military rule to civilian governance, Facebook saw an opening and positioned itself as the country’s central internet platform.

Facebook dove right into Myanmar head first, throwing caution to the wind. Ever since 2014, activists have been warning the company about hate-speech on the platform, and social scientists like Zeynep Tufekci have been writing about the impending problems that Facebook will cause. Facebook completely ignored these warnings, and up until recently it had only a dozen support personnel working in Myanmar, a country of 20 million Facebook users.

This matters because one of the report’s key findings is that the lack of digital literacy in the country contributed to rife false news and rumors circulating on the platform: people got on Facebook so fast, they didn’t know how to use it. And Facebook did nothing to solve this problem: until recently, the help section of Facebook in Myanmar was largely inaccessible because of a font problem (most of the country uses a nonstandard font).

Admit There’s a Problem

Facebook’s online domination in Myanmar is a problem. But an even larger, more general problem is Facebook’s casual denial of its impact on the world. We’ve seen how Facebook’s popularity in Myanmar fueled what the United Nations said “bears the hallmarks of genocide” — do we really want to see what happens in the rest of the world as Facebook grows in popularity?

It almost goes without saying that we must criticize Facebook for the humanitarian crisis they’ve enabled, and we should demand further accountability for what has taken hold. That’s largely what the response has been, but, as we have seen, it’s not enough. Facebook does not seem to care about the negative role it has played in the world, and world leaders have been unable to hold Facebook and its founder, Mark Zuckerberg, accountable.

There’s a pragmatic problem that we have to solve before we can expect anything at Facebook to change: how do we make a product manager or software engineer sitting in sunny Menlo Park care about the millions of users over at Myanmar, or India, or Turkey? Employees at companies like Facebook should be made very well aware of their impact around the world, before, during, and after their work is deployed. Most other fields of engineering, like civil engineering, already have this built into their culture, but software engineering is lagging behind. Tech employees need to realize that their responsibility doesn’t end at the last line of code — that’s just where it starts.

Tech companies like Facebook are responsible for this shift in mentality, but the global news media also needs to shift its focus, and proportionately cover those who are most at risk. The “techlash” coverage has focused primarily on the problems of wealthy countries like the United States, ignoring smaller, poorer countries until it is too late. Covering remote regions like Myanmar requires more money and effort — and might command less attention initially than the latest privacy debacle — but the potential impact of such coverage is far greater than reporters might expect.

There is also the matter of truly holding Facebook accountable, and the answer lies in elevating this global problem to a global arena. The uncomfortable truth is that Facebook wields such power: already, elected officials and politicians from multiple countries who gather together are unable to get Mr. Zuckerberg to show up for a hearing. If Western democracies where Facebook makes most of its money cannot summon Mr. Zuckerberg and hold him accountable, then smaller, poorer countries don’t stand a chance.

Global Response to a Global Issue

The global community needs to elevate these issues to where they belong; the avenue of international politics. Facebook doesn’t just run a social media application, but is practically privatized infrastructures for politics, media, and commerce around the world. It is naïve to assume a company headquartered in California to be able wrap its head around all the complexities of such a task, even with the best of intentions, for all the countries it operates in, let alone deal with the complexity of a single, large country like US, as the 2016 elections showed. Many countries like India were already not on board with handing over the keys to their digital future to Facebook. Now the humanitarian crisis in Myanmar should act as a wake-up call for the global leaders to take the reins back.

It is time that we all realize tech companies do not operate on a different plane of reality. The many layers of abstractions that lie between the keypresses over in Silicon Valley and the rest of the world often doesn’t just blind the fresh-faced faces in California, but often obscures what’s happening to the rest of the world as well.

Facebook will only grow in size and impact around the world. For many years, we acted like such companies only bring good to the world, and the bad stuff happens to them. Yet, the good, the bad, and the ugly are all interconnected. We need to hold those accountable for their mistakes, and then plan as a global, interlinked world how and who should run our digital infrastructure. Myanmar deserved better and so does the rest of the world.

Singapore, November 2018

The Secret Liabilities of Data

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


I love stretching analogies to the point they break. Take the tired cliche that data is the new oil. It makes some sense, considering the biggest data hoarders seem to be doing great, like the oil companies once were and still are. But there are big flaws in the analogy. The point of oil is not just it is very valuable, which it is, but it’s also a physical good that’s consumable. Data, on the other hand, is exceedingly cheap to store, and more importantly free to copy. Try doing that with carbon molecules!

But a different way to look at data as the new oil debate is to consider whether more data is actually an asset, or a liability. True, you can derive an insane amount of value from many types of data, if you are Amazon. But what about if you are Facebook, or Google and store billions of people’s private data? Obviously, these companies’ quarterly earnings belie any notion that private data is simply a negative asset, but I’d argue there are plenty of unaccounted liabilities there too.

Some of it is probably a solely function of the amount of data stored. Somewhat ironically, the quantity takes an intangible quality when it crosses a certain threshold; you are exponentially a bigger target if you have a dossier on the entire population than on just one-quarter of it.

The other factor is the nature of the data stored. It’s one thing to remember what people like to watch as Netflix does (which is still surprisingly controversial), but what if you literally accidentally leak hundreds of millions people’s passwords? Millions of keys to millions of castles, like an apple pie by the window, waiting to be snatched.

So, Facebook did not exactly do that, but came scarily close. Brian Krebs, a well-known security reporter, reported that Facebook has been storing the passwords for a few hundred million people in plaintext, without being hashed or salted (more on this later), both accessible to anyone within the company but also ready to leak at a moment’s notice.

Before we go on any further, we need to discuss what hashing is and how it ties the whole analogy together.

What the hash is a hash?

In order to check whether someone entered their password correctly, you don’t necessarily need to know their password, but rather need to know what their password “could do”. Imagine the password as a physical object casting a shadow through a linen cloth; you could possibly tell that only that password could cast that shadow, without ever knowing what the password itself looked like. This way, you could possibly just remember what the casted shadow would look like and store that in your database. You would still look at the password (as it is entered), but not have a copy of the exact password with millions of other passwords lying next to it. Neat!

The key insight here is to never actually store the passwords. Rather, you store the hash of the password (or rather a salted hash, but let’s ignore that for now). Hashing is a one-way mathematical function where you can generate a hash (cast a shadow) from a given input, the password in this case, but you can’t go the other way in a reasonable time. Since you can’t ever regenerate the password from the hash itself, this means that the stored hashes are useless when they (inevitably) leak with a security breach.

This is only one of the big unaccounted for liabilities that come with storing so much data in one place, including but not limited to, passwords but also personal information. Facebook, in a single mishap, could expose passwords of more than 200 to 600 million people. Just the mere range, 400 million, is larger than the US population.

Password re-use is the obvious big problem here, and one that won’t be solved easily until we collectively find a better way to authenticate users. Currently, password managers such as 1Password, or those included with all major operating systems remain as the next best options. The fact remains, however, that overwhelming majority of people still use the same password to secure their latest cat food purchase and their finances. A single leaked password is all it takes, for most people, for their entire online identity be at risk.

Then why are we not losing our collective minds over what Facebook did?

Weeks since last Facebook Crisis: 0

There are couple reasons. First of all, Facebook claims the plaintext passwords were never leaked outside of Facebook, and while tens of thousands of people technically could access them, a much smaller number actually did look around where the passwords are, and . not necessarily at the passwords. And there’s no evidence of any intentional access, or misuse. It appears that damage was contained within Menlo Park. This time.

There’s a more salient point though, that it is in Facebook’s interest to keep your data as safe as possible. This is where our analogy comes back to the rescue. The end-game for Facebook is to be the broker of your identity, and in order to do that, they need to keep your data as safe as possible. For you to keep using Facebook’s products so that they can suck in more and more of your private data to their servers (and not share it with others, because as I discussed “privacy is good now”), you need to trust them to keep your identity safe.

And maybe, the other reason is that accidentally storing plaintext passwords is less of a one-off bug, but rather a rite of passage for any company that stores passwords . It has happened to Twitter and GitHub as Krebs reports, but they are simply the most well known offenders. A common joke schema in Twitter is publicly shaming some organization by sharing screenshots of their customer service representatives asking you questions about your password, which is a tell that they can see your passwords. There is even a Tumblr —which itself got hacked— to just out these companies.

Given an average user has around 100+ accounts and most companies will not even flag this as an issue, and even fewer of those who notice will publicly come out and apologize (why would you?), it’s quite likely that your passwords are in a database or a log file somewhere, waiting to be looked at by some starry-eyed engineer. This is the world we created for ourselves.

Bugs, or just Organizational Chart Artifacts

I speculated on Twitter that this is less of a “bug” but rather a systemic issue based on my previous experiences on how such things happen. The main point I’ve made is that companies like Facebook operate in a way less centralized fashion that it appears outside. There are teams that build these secure systems, such as credential stores (where your email and hashed password go) and associated “adaptors” that help you use them. And there are also teams that need to launch products, who sometimes find those adaptors unfitting to their needs.

If your options are to convince the team that builds the storage system to work with you, which takes time, or just hack up some solution to save the day, generally you pick the latter. In an environment where up and to the right numbers are prized more heavily than practicing good security hygiene, it’s the more rational choice. You can always apologize with a blog post later, with an aspirational title to remind everyone what you did NOT do, but what you SHOULD HAVE done.

Of course, there’s no reason to think that’s what happened. But I would be surprised it was too far off. If you are technically inclined, just replace “credential store” with “logs”. To the discussion on hand, the difference is quite immaterial. Data is data (and data is data is data too), access methods be damned.

Stewart Brand, the publisher of The Whole Earth Catalog, famously quipped “Information wants to be free”, which of course is a more poetic way to describe the zero marginal cost of copying information. It has long become somewhat of a battle cry for certain corners of the internet who used it to criticize digital rights management schemes.

But I always found it more interesting to see how the same saying applied to databases, which all eventually become free as in liberty much to the chagrin of its owners. And when such data becomes free, it does necessarily not make the world a better place, but puts people’s private information at risk. Should we leave information this free?

There’s also second part of the Brend’s soundbite that got a lot less press: “Information also wants to be expensive”. Seemingly our collective inability to decide on the price (and the cost) of data seems to have a long history.

As we collect more and more data, and put it in more and less places at the same time, makes this discomfort more troubling. Trillions of dollars in dollar value is created out of an asset, that we don’t know how to properly value. We are barely recognizing the negative externalities of decades of oil production and consumption now, and it took us almost destroying the planet. We should do a better job for data.


What I’m Reading

I’ve been on an economics and strategy binge lately. Lots of “big things” to think about and keep in mind as global rules are being rewritten.

Schumpeter on Strategy: Columbia Professor and venture capitalist Jerry Neumann has one of the most thoughtful VC blogs, and this piece on Schumpeter is an “Intro to Strategy” course in itself. Jerry is a good writer too; make sure you follow him.

The mainstream of economics, then as now, pretty much tries to describe the economy as if it shouldn’t change. If it is changing, it’s changing towards an equilibrium, where it won’t have to change any more. Schumpeter noticed that this is not how it works. Both the economy as a whole and individual businesses change constantly. His model of the latter, in his Theory of Economic Development, explains how some entrepreneurs make an unusually large amount of money.

Economics After Neoliberalism: Changing gears now. Renowned Harvard Kennedy School economist (and compatriot of yours truly) Dani Rodrik has been arguing the market dogmatism is finally on its way out, and that can be a saving grace. A piece that spawned a great discussion in responses by other economists and policy makers. Not a light reading, but worthwhile.

Economics does have its universals, of course, such as market-based incentives, clear property rights, contract enforcement, macroeconomic stability, and prudential regulation. These higher-order principles are associated with efficiency and are generally presumed to be conducive to superior economic performance. But these principles are compatible with an almost infinite variety of institutional arrangements with each arrangement producing a different distributional outcome and a different contribution to overall prosperity.

Spotify does taxes

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.


Is my profit margin your tax? If you have to pay to me to merely exist in the marketplace, it can feel the same. Yet, taxes are unique, because generally you have to pay them to not do business, but also live freely, not in a jail cell. You just cannot opt out of paying them. If you are a talented (or a crafty) individual, and you can try going to Dubai or Singapore to minimize some of your personal taxes. And if you are a firm, there are many options available on the menu. But taxes as a whole remain as a fact of life for most people, and most companies.

Yet, taxes and profit margins are not the same. If a certain firm has a monopoly on some raw material you need, and you have to pay that firm exorbitant prices, it can feel a bit unfair. But that’s not the same thing as having a single supplier for your business. Lots of firms operate with having a single supplier, buyer, or a single way to reach their customers. It’s generally not a pleasant place to be, but it happens. What else can you do?

I guess you could complain as loudly a possible.

Spotify made waves recently by filing a complaint with the European Commission about Apple’s unfair practices, arguing that the Apple Tax it has to pay is stifling competition and is a big problem for the industry as a whole. The Swedish firm even built a custom website called Time To Play Fair (dot com, what a great domain name!). The ominous phrase “Apple Tax”, appears multiple times throughout the website, the videos and in the CEO Daniel Ek’s remarks, reminding everyone what is at stake.

Timing is certainly convenient. With the backlash against big tech firms in full force both in the EU and US, Spotify is fanning the flames. And Spotify counting the former Microsoft legal counsel, Horacia Gutierrez, in its ranks, you can see this is not just a press bamboozle for the company; it’s an existential issue. If you are going to take on a firm that has the cash reserves of a small European country’s GDP on an international battleground, you probably want someone whose firm was almost divided in half by US Department of Justice on your side.

Apple responded with its own fiery press release a couple days later, and did not pull its punches. It stops barely short of calling Spotify a bad faith actor in the music business but also doesn’t address the streaming giant’s legitimate points about how Apple is seemingly extending its dominant position to other markets. The main thrust of Apple’s argument seems to be that Spotify is in a dire position where it can’t afford to be in the Apple ecosystem, and now wants a free ride. Is that so?

Spotify’s gripes with Apple are numerous, and some of them definitely feel less grounded in reality than others. For me, the most interesting part is Spotify’s argument that Apple is stifling competition by not allowing other payment options other than Apple’s payment system for handling subscriptions. This Apple system doesn’t come cheap. Apple takes a 30% of all sales for the first year, and reduces to 15% the second year. The list of rules don’t stop there. The requirements are so strict, Spotify argues, not only they are not allowed to link people to a simple web page where Spotify can start billing people directly, they aren’t even allowed tell people such an option exists.

Let’s go back to the competition issue. It’s hard to disentangle whether Apple is a monopolist, and it defines on what the relevant market is. Market definition is a notoriously thorny question, and not one I can answer. Legal scholars such as Sally Hubbard argue as such. But we can limit our discussion to a single question: Can Spotify exist as a business without being in the Apple ecosystem?

The short answer seems to be no. We’ll ignore desktop clients for now, and focus on mobile. In the US, Apple has around 50% of smartphone market share. The number is around 25% for EU. It would be tough to make the case that Spotify could drop those users. Even Apple has an app for Apple Music in the Google Play store. Sure, it’s a remnant of its Beats acquisition, but it’s still there, and updated. That has to count for something.

In its response to Spotify, Apple also argues that most of Spotify’s users are ad-supported. That is true. But the main driver of Spotify’s revenues is the premium tier. Revenues from premium subscribers were around €1,320 million in the last quarter, dwarfing the €175 million revenues from the ad-supported tier. The same tier also has lower margins than the premium, especially in less mature markets for Spotify.

Moreover, there’s also the deeper question of why Spotify needs to have as many users as possible. As Spotify sits in between end-users and record companies, its future is dependent on the whim of its suppliers, as Ben Thompson of Stratechery smartly explains (Subscription required). And in order to have leverage over them, it needs to command as much demand as possible. Even the smallest loss of market share hurts Spotify’s negotiation position, which is already quite complicated. In fact, Apple seems to agree when it says “Spotify wouldn’t be the business they are today without the App Store ecosystem[.]”.

This mad quest to earn market share is obviously less of a problem for Apple. While Spotify has to make costly arrangements with mobile operators, and other smartphone manufacturers like Samsung to have its app pre-installed, or run expensive campaigns with Google, Apple’s Music app comes pre-installed on every iPhone. Apple even runs campaigns over push notifications, which is not allowed for other companies.

And there’s the more fundamental question of user experience. In order to comply with Apple’s rules, firms cannot even mention other payment options in their apps. Imagine trying to become a Netflix customer, on your iPad. There’s simply no way. You have to know, or somehow find out, that you can sign up online. Surely, new users having to make a phone call is not the experience Apple wants on its platform.

Apple’s argument is that Apple payment infrastructure is a more user-friendly, secure, and allows for an integrated experience. That might be true. For example, an in-app browser where users can subscribe also comes with its security risks, where a nefarious app might log the credit card information on the side. And Apple might very well argue that users being able to manage their Apple-mediated subscriptions in one place is a better user experience.

Yet, for any non-digital goods, Apple still allows third-party payment options. As bits and atoms (sorry!) merge more and more, the line will be harder to draw. Uber might be a physical service, but what about an Uber gift card? There’ll many more of these lines to draw in the future.

As the owner of its platform, the decision is for Apple to make, but the tingling sense of capriciousness between digital and physical undermines Apple’s “every firm should play by the same rules” rhetoric. People know that makers of big enough apps, like yours truly, have special levers available to them. In its press release, Apple is quick to point out Spotify leaves out Apple’s cut lowers from 30% to 15% after a year. But the same Apple fails to mention how Netflix was able to negotiate a special, instant 15% rate. Even that wasn’t enough to keep Netflix on its platform anyway.

Reading Apple’s response to Spotify, it’s hard at first to not sympathize with the company. I’ve invested, financially and personally, into the Apple ecosystem for more than 15 years and have convinced many people to do the same. I was even a paid customer of Apple’s overly expensive online services like MobileMe, and greatly benefited from them. Services, for better or worse, are Apple’s future.

Therein lies the rub. As Apple leans more heavily into the services revenue, the Cupertino firm will have more of these kinds of decisions. What works for the short term for the health and growth of a platform can be different than what needs to be done for the long-term sustainability, as many platforms are painfully figuring out now. It can also be hard for a company like Apple, where decisions are made in multi-year-long hardware cycles, to adjust itself to changing conditions fast.

The company has already shown signs of maturity, for example, when it changed course about its free trials on Apple Music, following Taylor Swift’s protest. If Apple wants to entice more businesses to its platforms and grow the pie, it should offer more than just access to its billions of users. Building an ecosystem requires long term thinking, and judging the interests of many different stakeholders, including the platform itself.

The best advice for Apple can still be derived from its core principles: do the right thing for the user. If Apple believes that its payment infrastructure is the best in business, it should let it flourish and compete in the marketplace. If makers of physical goods can be trusted with digital payments, the same rights can be extended to digital subscribers too. If Apple finds itself unable charge its double-digit markups, or developers flock to other options, that would only force Apple to either lower its prices or make its services even better to encourage app makers to switch to it. That would be true competition, making things better and cheaper for everyone.

Zuck pens another memo

Posts

This post is cross-posted from my joint newsletter with Ranjan Roy, The Margins. Please check it out, and consider subscribing.

Imagine you are a business analyst for a public company, and one day the CEO of your favorite company says they are shifting the business to an entirely new business model. Think big here; suppose the company shifted to focusing entirely on what it considered the biggest threat to its business.

What would you expect the stock to do? Go up, because it’s now focusing on handling the threats head on? Or should the stock take a dive, because markets generally don’t like such proclamations and favor conservatism?

Mark Zuckerberg, the supreme leader at Facebook recently unveiled a new vision in a 3000 word manifesto. The entire memo is worth your time, especially so if you enjoy reading, like I do for both professional reasons and entertainment, tech CEOs waxing poetic.

Two main takeaways from the memo are “we won’t know what you are saying on Facebook” and “now you can also message your friend on Instagram from WhatsApp but don’t ask how it happens”.

Given Facebook’s business model is built on knowing as much as about you, and then selling your attention —don’t call it data— to the highest bidder, would you expect the stock to go down, since Facebook will seemingly know less? Or would you expect it to go up, because Facebook is clearly doing what it needs to be doing?

The moral seems to be that you should never underestimate the force of inertia, especially when the rock is as big as Facebook.

To any informed observer, there’s very little new in the manifesto. The grand proclamations are less about Facebook actually giving up on its main business, but rather adding some new capabilities and protections. Let’s dive in.

First, the encryption. Zuckerberg might appear to leave data on the table when he decides to encrypt all communications, but that’s hardly the case. Facebook doesn’t use the contents of the messages today for advertising. Yet the company’s targeting is so good and people more predictable than they think, people accuse the company of listening their private conversations. Moreover, even when Facebook encrypts all the messages you send and receive, it will still be collecting tons of other sources of data, such as the metadata about the messages, location information gathered but the apps, your browsing habits via the various trackers on the web, data shared by apps that use Facebook SDKs, and the huge troves of data buys from other data brokers. None of that, seemingly is changing.

In fact, with these changes, Facebook might end up collecting more data, and or least the more valuable kind of data. Personal communications are “interesting”, in almost a voyeuristic sense, but the privacy implications of looking into them surely hasn’t matched the potential economic benefits. Zuckerberg touches on how on top of this “private foundation”, Facebook can build more value added services, probably similar to WeChat. Facebook might be throwing in the towel for expansion in China, but the strategies are free to copy. When you buy the actual pair of sneakers through Instagram, who needs to know you were planning on buying it?

No one expects Facebook to do something bad for Facebook itself, and this memo is no exception. Yet, the Zuckian double-speak is there. Zuckerberg claims that the big reason to merge all the chat applications company owns into one end-to-end encrypted system (which was previously reported) is about interoperability. That sounds unconvincing. By definition if you are talking to someone on a platform, you are already talking to them. This is where Facebook’s notion of privacy, where it’s about keeping your data private from others but not Facebook, clashes with reality.

A WhatsApp user whose profile forcefully gets merged with their Facebook account is at a less private position, not more. Facebook does not and probably never will commit to knowing nothing about you, because the spread between what Facebook needs to know and what you provide is where Facebook earns its margins. And for chat applications, that margin is practically zero.

More cogently, Facebook’s new interoperability and encryption pushes act as a strong defense against government regulation and scrutiny in multiple ways. If Facebook cannot see your messages, it can’t respond to government inquiries on them. Depending on your position on governments’ responsibilities, that can be a good thing. But also, a fully integrated “interoperable” WhatsApp, Instagram and Messenger makes Facebook much harder to “break up”. When there are no seams, where do things break?

Zuckerberg is only human. And Facebook is not just his life’s work, but it’s also the lens through which he experiences reality. That’s understandable. It’s also however hard to shake off the eye-rolls every time he pens a new manifesto. I am old enough to remember when Facebook was like chairs. Then it became building a new community, with once again a liberal and Facebook-specific interpretation of the word. Now, it’s a living room? There’s more to life than blue pixels.

The more commonly experienced reality is that whatever changes Zuckerberg proposes will take years, both organizationally and technically. There are thousands of decisions that are yet to be made. At Facebook’s scale, a subtle nuance might easily affect an entire nation’s worth of people. Contours of the New Facebook are going to be as controversial of sovereign borders, yet will be decided by a lucky few over in Menlo Park. Writing thousands of words, laying out a new vision is fun and exciting, but when you make it a habit, it’s fair for people to wonder if you are more interested in writing memos than executing. With so many years and unknowns, the vision gets blurry and it becomes harder to see what might happen in the future.

This is not to say Facebook cannot make the changes. If anything, company has proven time and time again that it’s not afraid to change course dramatically; be it from desktop to mobile, HTML5 to native, to buying competitors and copying other competitor’s features with reckless abandon.

Some of the current projects, like News Feed will soon show signs of decay. With such fan fare, it’ll be harder to assign the top resources on not “top-of-mind” projects. No one wants to work on the old stuff, especially when your boss tells the entire world there is the new stuff, with a short-story length memo.

Yet, still, there’s less new here than meets the eye. The peaking ad loads on News Feed and the popularity of Stories format has been out there, for years. Merging of all the different chat platforms to a new fully end-to-end encrypted version was also reported, and already “priced in”. What did Zuck actually announce?

Zuckerberg is a shrewd businessman. Just like any company, Facebook’s long term strategy is to entrench its competitive edge. For several years now, this edge has been primarily the scale, and the reach that comes from it. A Facebook product is used by practically everyone in most of the developed world, and it’s all across the web and in all the apps. Wherever it cannot reach organically, it buys its way in, either through buying the data or the companies themselves.

This new privacy focused Facebook changes very little to Facebook’s own business than what we knew from before. Less of a shift, more of an addition of new capabilities and some protective measures. Markets shrugged. Let’s see what happens in a few years.

The End of Internet History and the Last Ad

Posts

People make bad predictions all the time. Getting them wrong is not the problem, some are bound to be wrong. A source of bad predictions is often inherent to the enterprise; things are unpredictable. In almost any field, chaos reigns over order. In some sense, our economic system in the Western world, capitalism brought on by free enterprise, is designed to maximize this chaos and to reap its benefits. You might even call this innovation. Read More

An internet with an elephant memory

Posts

Turns out I always had a penchant for run-on sentences. I have counted over 3 of them in a college application essay I wrote in 2004. It is sitting there, on my Dropbox account, where I moved my “important” documents to from an old Yahoo! email. It’s been there, untouched, seemingly for eternity. Barring a catastrophic event, like Dropbox going out of business or me getting hacked, I suspect it’ll be there for at least 15 more years. Read More

Expect more Facebook Drama

Personal

Ooff. Another Facebook drama on the wires today. This time, a 2016 memo written by Andrew Bosworth made it way to Buzzfeed. It’s a horrible memo. Boz, as he likes to be called, argues that Facebook’s growth at all costs mentality justifies everything. And by everything, he means everything. Everything Facebook does, the scummy growth tactics, such as the contact importers. But more salaciously, the growth, as defined by connecting more people in more ways, justifies what happens due to the growth. Sorry if you’ve been exposed to bullies, Boz says, or if accidentally facilitated some terrorist plot. Read More