how David defeated Goliath
this story started in June of this year. one of Apple AppStore employees made an “error” while reviewing revision of Basecamp authors new app - Hey. David Heinemeier Hansson, one of the Basecamp founders, took it to Twitter David is not only Basecamp CEO, he’s author of series of great books on running teams and companies. he started to relay live on Twitter emails he and his employees exchanged with Apple Application Store team. as Apple takes anything between 15 to 30% of sales cut out of every in-app transaction, they suddenly asked him to include payment option in Hey. David argued, that the app never had and doesn’t plan to have option to buy anything.
David tried to explain clearly that the application never had any payment options. and that’s because it is simply used to log into the portal - to which you already own access to. well, such setup is no longer possible and current reading of AppStore rules is that every application has to have payment option. on top of that, there’s no option to build applications that are “just” frontends to some other services. David again calmly pointed out, that there are a lot of such applications in AppStore, including high profile ones like Spotify. that escalated quickly into all-out war, but at the end Apple succumbed and not only accepted Hey back, they also opened to challenge AppStore verdicts.
in the meantime however, amid stream of scandals with giants like Facebook (and its hypocritical CEO), Google or Apple were accused of abusing users privacy (including obeying to authoritarian jurisdictions like Russia or China), not paying proper taxes (by funneling them through different clever mechanisms to tax havens) and other behaviors, atmosphere around IT giants became a bit smelly.
and here we come to the reason why I decided to write this post. it’s not that I admire Apple in any particular way or intend to defend that company. I wrote this post because I believe the way Epic Games used this opportunity to attack besieged Apple is an example of the worst kind of hypocrisy.
first of all, Epic Games signed the contract by entering both the Apple and Google stores. they saw what kind of sales fees must be reckoned with. and then suddenly, after so many years, they woke up and decided to fight for “their users’ money”? and where has this noble motive been so far?
then, instead of being like Basecamps CEO - transparent from the very start, they chose to play in the privacy of “friendly emails” until it became obvious that they might as well be trying to threaten the Andromeda galaxy. moreover, it seems that the entire action was planned from the beginning with the assumption that Apple would not agree. if going to court would be only option left, Apple would be anyway under pressure to behave less “monopolistic” - just like other IT giants.
last but not least, we already know that Epic was trying to play two sided game from the very beginning. it seems they shipped hidden code in Fortnite app while discussing with Apple option to do this officially. because this option was highly obfuscated and hidden in the code, AppStore review process didn’t detect it.
but let’s cover this step by step.
a little digression.
I have very bad opinion about the distributors of most of the new games, and that’s not because I have something specific against Fortnite. since the launch of Fortnite (which my younger son played) and the newest Modern Warfare (which I played and sometimes I play), I had the opportunity to see how incredibly amateurish platforms supporting them are. companies like Epic Games, Blizzard and Activision seem not to care about how the user gets onboard - and mind it, Blizzard built Diablo once. Activision produced and distributed one of the best 8-bit computer games ever created. in 2020 those companies try to package games into “platforms” that are simply indigestible.
those network platforms, which are designed to connect not only hundreds of thousands of players around the world, but also provide distribution of updates, are built by the worst losers known to the world of software. while it is worth to appreciate quality of the code used by game engines themselves, including the parts used to synchronize the actions of characters in the games, I suspect that they are created and chiseled by the game engine and game authors themselves (such like Unreal engine). and platforms maintained by distributors and publishers, which then operate external ecosystem for such games, like “lobbies” or allow (actually force, because today more and more games cannot be played without a) Internet connection and b) before downloading and installing latest patch) to download updates, are worst, and most unreliable type of software I’ve come across so far. and I witnessed a lot in my life.
why do I say that? because by necessity I became the Internet Service Dad and witnessed not only how (badly) those platforms communicate with players, but also how they work and try (let me emphasize - try) to ensure redundancy. I have a few wired internet uplinks and professional network equipment at my disposal. I carefully control how packets leave my network, how they are QoS-marked and by which routes they travel to their destinations on the Internet. I can see how network game stacks handle IPv4 and IPv6, what filtering they can handle and what not, and so on and so on. in short - I saw a lot of it up to 0s and 1s in Wireshark. and I say with full responsibility that these platforms only work because the internet is so well designed. the people that are responsible for that crappy code should be ashamed of everything they did and I hope that now, instead of in Hawaii, they spend their last days somewhere under the bridge, without money and they do deeply regret what crap they decided to write and then release to the world.
it doesn’t help that as part of “updates”, these platforms need to push literally gigabytes of updates your way every few days. and those updates don’t really change the whole game, on contrary - on the surface they don’t change anything really visible, so apparently have to fix thousands of bugs. they come from a number of distribution centers (why they don’t use existing CDNs for example instead of reinventing the wheel?), which no matter where you are or which game you play, are the most buggy, unreliable and just plain crap pieces of software ever written. it doesn’t help them probably, that crappy software of the “platform” itself, has to work with crappy code that needs those bulky “updates”.
so Epic Games vs Apple
so, as I already wrote, Epic decided to use the negative turmoil around Apple. despite the fact they signed Apple Developer Agreement before entering the AppStore, they suddenly decided to force Apple into submision. while we can always discuss Apple policy with regards to sales tax they take, Epic decided now to get and keep full price directly. funnily enough, Epic Games decided to communicate all of that by writing an email to Apple board - apparently in a rush, because at the end they confused Apple iOS with Google Android. on top of that, in something of a fake digital Robin Hood, they mistook the attempt to have a discussion with the something more akin to extortion tactics that mafia bosses use in movies.
of course, “genuinely” & “deeply” concerned for the rest of the developers, Sweeney, didn’t mention the email in public. well, if they’d agree you end up as hero, right?
here’s the first email from June 30th in its entirety (date is important, as it means Epic is starting its attack after observing for most of the June how well Basecamp was doing):
Dear Tim, Phil, Craig, Matt, Because of restrictions imposed by Apple, Epic is unable to provide consumers with certain features in our iOS apps.
suprised? only now?
We would like to offer consumers the following features:
- Competing payment processing options other than Apple payments, without Apple’s fees, in Fortnite and other Epic Games software distributed through the iOS App Store;
- A competing Epic Games Store app available through the iOS App Store and through direct installation that has equal access to underlying operating system features for software installation and update as the iOS App Store itself has, including the ability to install and update software as seamlessly as the iOS App Store experience.
so, basically: “give us your piece of the cake, because we have grown so much that we think we deserve it and you no longer can threaten us”.
If Epic were allowed to provide these options to iOS device users, consumers would have an opportunity to pay less for digital products and developers would earn more from their sales. Epic is requesting that Apple agree in principle to permit Epic to roll out these options for the benefit of all iOS customers. We hope that Apple will also make these options equally available to all iOS developers in order to make software sales and distribution on the iOS platform as open and competitive as it is on personal computers.
I wonder who was bright enough to add the last sentence, because otherwise the word requesting could be considered as direct blackmail attempt :)
As you know, Epic was required to accept your standard, non-negotiable contracts, like the Apple Developer Program License Agreement, in order to offer products on iOS devices through the iOS App Store. Epic is also required to comply with Apple’s unilateral standards documents to obtain app approval, like Apple’s App Store Review Guidelines. Apple’s contracts and standards documents contain restrictive provisions that prohibit Epic from offering a competing app store and competing payment processing options to consumers.
once again - fake amazement that when I enter someone’s home I have to accept his/her rules.
and the best part of the email, with a failed “tip” on how to get out of the impasse that Epic itself has created:
Apple would need to provide a side letter or alter its contracts and standards documents to remove such restrictions to allow Epic to provide a competing app store and competing payment processing option to iOS customers.
“side letter”, for people unfamiliar with American corporations, is the worst sin you can ever commit.
the ending also leaves no doubt as to on what Epic is focusing in the first place, and what is just a good excuse for attack:
Please confirm within two weeks if Apple agrees in principle to allow Epic to provide a competing app store and competing payment processing, in which case we will meet with your team to work out the details including Epic’s firm commitment to utilize any such features diligently to protect device security, customer privacy, and a high-quality user experience. If we do not receive your confirmation, we will understand that Apple is not willing to make the changes necessary to allow us to provide Android customers with the option of choosing their app store and payment processing system.
Best Regards, Tim Sweeney Founder and CEO Epic Games
pay attention to ‘Android’ in the last sentence. either Sweeney wrote the e-mail in a rush and sent it without consultation, which in itself says a lot about him, or despite the e-mail being consulted nobody caught it. which is even worse thing for all his associates.
because Apple response was not what Tim had expected, a month and a half later, Sweeney decided to go all the way, seeing that he would not win anything anyway. one of the other potential reasons was that in the real world outside, nobody picked Epic side en masse (unlike in the Basecamp case).
however before Tim wrote and sent the email, he made sure that Fortnite app was submitted for AppStore review and sneaked through code that supports direct payment options. that’s about it with regards to being honest and transparent, but what else would you expect from our digital Robin Hood at this point?
Dear Tim, Phil, Craig, Matt, Douglas,
I’m writing to tell you that Epic will no longer adhere to Apple’s payment processing restrictions.
Today, Epic is launching Epic direct payments in Fortnite on iOS, offering customers the choice of paying in-app through Epic direct payments or through Apple payments, and passing on the savings of Epic direct payments to customers in the form of lower prices.
We choose to follow this path in the firm belief that history and law are on our side. Smartphones are essential computing devices that people use to live their lives and conduct their business. Apple’s position that its manufacture of a device gives it free rein to control, restrict, and tax commerce by consumers and creative expression by developers is repugnant to the principles of a free society.
Ending these restrictions will benefit consumers in the form of lower prices, increased product selection, and business model innovation.
Henceforth, all versions of Fortnite that Epic submits to the App Store will contain these two payment options, side by side, for customers to choose among. We hope that Apple will reflect on its platform restrictions and begin to make historic changes that bring to the world’s billion iOS consumers the rights and freedoms enjoyed on the world’s leading open computing platforms including Windows and macOS. In support of this path, Epic’s public explanation of our payment service will be neutral and factual to provide Apple with a chance to consider taking a supportive route and communicating it in a way of Apple’s choosing.
If Apple chooses instead to take punitive action by blocking consumer access to Fortnite or forthcoming updates, then Epic will, regrettably, be in conflict with Apple on a multitude of fronts - creative, technical, business, and legal - for so long as it takes to bring about change, if necessary for many years.
Tim Sweeney
Epic Games
in the meantime Apple warned Epic that because it violates the agreement and the rules, it will close their developer account. and that’s what happened in the end - Google did the same thing in Play store, as Epic violated that agreement as well.
of course, who are we to discuss wars of the giants. but in my opinion, when you enter the river, you know what you are up to. it is brazen to demand that the whole world changes its rules for you while you justify the whole act under false guise of “customers good”. Apple is not without its faults of course, but if we consider moral side of this, in my opinion Tim Sweeney was just promoted to group of cheap crooks.
you can of course be a rich philanthropist or a poor cad.
but why you can’t just be and act like a decent person?