Do you ever scroll through Bring A Trailer, shaking your head at the impossibly low-mileage cars that seem to be the site’s bread and butter? All those 44,000-mile BMWs,28,000-mile Porsches, or 10,000-mile Honda Elements have never really been given the chance to be cars — they’re garage furniture, a decoration to…
Porsche loves to set blistering Nürburgring lap times. The 911 GT3 and GT2 RS each hold a spot among the top ten fastest production cars around the circuit, and the 918 Spyder and previous-generation 911 GT3 RS barely miss the cutoff.
There can only be one of two responses: 1) It's just a harmless name, it doesn't mean anything. THEN CHANGE THE NAME. 2) Names are very powerful, and can be used to build or maintain reputation. THEN CHANGE THE NAME.
Award for longest blog post title ever? Andrey Muzychenko has a great github repository where they decompiled the 25 year old Space Cadet Pinball application from Windows 95/XP and then recompiled it for Linux (and really any platform now that it's portable code!).
NOTE: Because this is a decompilation/recompilation, it doesn't include the original data files. You'll need those from a Windows XP disk or ISO that you'll need to find yourself.
Then I cloned the repo under WSL and built. It builds into bin and creates a Linux executable.
NOTE: Place compiled executable into a folder containing original game resources (not included).
I am a digital hoarder so I have digital copies of basically everything I've worked on for the last 30 years. I happened to have a Windows XP virtual disk drive from a VM from years ago that was saved on my Synology.
I was able to open it and get all the original resources and wav files.
Then I copy all the original resources minus the .exe and then run the newly built Linux version...and it magically pops out and runs on Windows...as a graphical Linux app.
Amazing! Have fun!
Sponsor: Make login Auth0’s problem. Not yours. Provide the convenient login features your customers want, like social login, multi-factor authentication, single sign-on, passwordless, and more. Get started for free.
This is the story of the mastermind behind one of the largest "fake news" operations in the US.
For two years, he ran websites and Facebook groups that spread bogus stories, conspiracy theories, and propaganda. Under him was a dedicated team of writers and editors paid to produce deceptive content—from outright hoaxes to political propaganda—with the supreme goal of tipping the 2016 election to Donald Trump.
Through extensive efforts, he built a secret network of self-reinforcing sites from the ground up. He devised a strategy that got prominent personalities—including Trump—to retweet misleading claims to their followers. And he fooled unwary American citizens, including the hacker's own father, into regarding fake news sources more highly than the mainstream media.
Pundits and governments just might have given Russia too much credit, he says, when a whole system of manipulating people's perception and psychology was engineered and operated from within the US.
"Russia played such a minor role that they weren't even a blip on the radar," the hacker told me recently. "This was normal for politicians, though… if you say a lie enough times, everyone will believe it."
Previously dubbed "Hacker X," he's now ready to reveal who he is—and how he did it.
The fake news impresario who has now decided to break his silence is "ethical hacker" Robert Willis.
Some in the information security community might know "Rob" today as an active member who speaks at conferences and works with the Sakura Samurai ethical hacking group. (The Sakura Samurai have, on many occasions, responsibly disclosed vulnerabilities in the computer systems of government and private entities. I have previously interacted with Rob on about two occasions, minimally, when I had questions regarding Sakura Samurai's vulnerability writeups.)
But back in 2015, Willis was just another hacker looking for an IT job. He had already received one job offer—but still had an interview scheduled at one final company.
"I was thinking of not showing up to the interview," he told me. "I had, after all, just committed to another company."
That final company was opaque—it would not reveal either its name or the actual job duties until Willis showed up in person. But the opacity was itself intriguing. Willis decided to do the interview.
Willis in 2017 when writing for the now-shuttered Conservative Country website. (credit: Robert Willis)
"I showed up at the location, which was a large corporate building. I was given directions to wait downstairs until I was collected. The secretiveness was intriguing. It may have turned some people off, but I love an adventure. I had not been given any information on the job other than that they were very excited, because to find someone like me was very rare—I had tons of random, overlapping, highly technical skills from years of wearing multiple hats at smaller private companies."
Even before his ethical hacking days at Sakura Samurai, Willis had gained an extensive technical skill set in networking, web applications, hacking, security, search engine optimization (SEO), graphic design, entrepreneurship, and management. He knew how to take advantage of search engine algorithms, once, he said, getting a random phrase to the No. 1 spot on one engine within 24 hours. "Many will say this is/was impossible, but I have the receipts," he said, "and so do other credible people."
At the interview site, a man came down to get him, and they rode the elevator to a floor with a nearly empty office. Inside waited a woman beside three chairs. They all sat. His hosts finally revealed the name of their company: Koala Media. The moment felt like an orchestrated Big Reveal.
"I wasn't scared but excited at how crazy this was already turning out [to be]," Willis told me. "I listened. I was told that there were big plans for the office I was sitting in and that they had already hired the initial writers and editor for the new operation."
The interviewers at the company told Willis that "everything was to be built with security in mind—at extreme levels."
Should he get the job, his primary role would be to rapidly expand a single, popular website already owned by Koala Media. For this, they needed someone with Willis' diverse skill set.
Then the interview took a political turn. "They told me that they were against big companies and big government because they are basically the same thing," Willis said. They said they had readers on the right and the left. They said they were about "freedom." That sounded OK to Willis, who describes himself as a social liberal and fiscal conservative—"very punk rock, borderline anarchist."
Then the interviewers told him, "If you work for us, you can help stop Hillary Clinton."
"I hated the establishment, Republicans, and Democrats, and Hillary was the target because she was as establishment as it got and was the only candidate that was all but guaranteed to be running on the main ticket in the future 2016 cycle," said Willis. "If I were to choose a lesser evil at the time, it would have, without a doubt, been the Republican Party, since I had moved to the new city due to the Democrats literally destroying my previous home state. It felt like good revenge."
Willis says he had no indication that the company that was about to recruit him was extreme or would become so in the future. In his perception, the company was just "investigative" with regard to its journalism.
The owners of Koala Media reeled in good money at the time. Koala's main site covered "health" topics and hawked supplements and alternative cures. A tiny front-page ad would bring in $30,000 a month, Willis tells me, with mailing lists enriching the Koala Media empire further.
"Getting highly targeted individuals to sign up was huge for financial gain," he said. "[Koala] would advertise products directly to individuals and sell thousands of them at a time."
Emails were sent out twice a week, one promoting a sale and the other some new product. Additionally, affiliate links and virtual event promotions garnered further income in the "hundreds of thousands of dollars" range for a single opportunity.
But as Willis came on board, Koala's stories got more controversial.
A former Koala Media writer who has worked with Willis told Ars, "In the beginning, the job was fine, writing regular AP-style news articles. Then, it went toward goofy stuff, like 'lemon curing cancer.' And eventually, it went to super-inaccurate stuff." That is when the writer knew it was time to call it quits. But Willis stayed on, even as one of the site owners personally contributed content that made him uncomfortable.
"That was the problem," Willis told me. "We were trying to build a more legitimate network and were reaching more and more millions weekly, but then the owner—who contributed a story once a day, during the best time for reach—would write crazy stuff.
"What saved me was a couple [of Koala Media] employees," he added. "One came into my office and closed the door and looked at me and said, 'You don't actually believe this stuff, do you?' and I let out a sigh of relief when I said, 'God, no'—and laughed. It became an ongoing joke."
From that moment onward, the hacker and office staff would joke about the stuff they were being assigned to write—like a conspiracy-laden writeup on "chemtrails" or a piece on "lemons curing cancer"—thinking that only a small "ultracrazy" percentage of readers actually believed what was being written.
Toward the end of 2015, more and more pro-Trump stories started emerging on Koala. But after Trump won the Republican primary in 2016, the focus shifted heavily toward anti-Clinton stories. During this time, Koala's already-loose editorial standards relaxed even further. Stories became increasingly bizarre or opinionated. Citations that did exist were often placed in a misleading manner, misconstruing the linked stories or pointing to existing stories in the Koala webring, making it hard for readers to fact-check the material. The "search bar" on these news sites even took users to a search engine created by Koala; it showed stories from "independent media," i.e., sites from the webring. Pieces that ran during this crucial period claimed, among other things, that Clinton had plans to "criminalize" gun owners, to kill the free press, to forcefully "drug" conservatives, to vaccinate people against their wills, to euthanize some adults, and to ban the US flag.
Yet Facebook, which directed plenty of traffic to Koala, never cut the site off. In the two years of the operation that Willis oversaw, Facebook banned only one of Koala's posts, Willis said.
Through it all, Willis did what he was hired to do: he put his technical skills in the service of boosting Koala's reach—by any means possible.
The basic approach involved the creation of a massive syndication network of hundreds of specialty "news" websites, where articles from the main Koala website could be linked to or syndicated. But these additional websites were engineered so that they looked independent of each other. They were "a web ring where the websites didn't look like they had any real associations with each other from a technical standpoint and couldn't be traced," said Willis.
Each fake news website was on a separate server and had a unique IP address. Each day's stories were syndicated out to the fake news sites through a multistep sync operation involving "multiple VPNs" with "multiple layers of security." Eventually, each public-facing fake news site received its daily content payload, and the stories would go live at scheduled times. In addition to Americans, Willis' team also comprised outsourced web developers working from Mexico, Eastern Europe, South Africa, and Taiwan.
"I oversaw everything and even had stacks of SIM cards purchased with cash to activate different sites on Facebook since it was needed at that point in time," admitted Willis. "Every website had a fake identity I made up. I had them in a sheet where I put the name, address, and the SIM card phone number. When I accessed their account I created on Facebook, I would VPN into the city I put them in as living in. Everything attached to a website followed these procedures because you needed to have a 'real' person to create a Facebook page for the websites. We wanted no attachment, no trace of the original source. If anyone were to investigate who owned a page, they would be investigating a fake person."
Eventually, carriers started asking for Social Security numbers (SSNs) prior to issuing and activating SIM cards. But "they took anything resembling an SSN, even ones generated from dead people," Willis said. As a test, Willis once provided Elvis Presley's SSN, which he had found on Google Images. The number worked.
Independent studies, seen by Ars, have confirmed that in 2015, shortly after Willis had started at Koala, hundreds of fake news domains sprang up. A British think tank has also linked this network of hundreds of domains to Koala Media.
After carefully studying the Facebook pages maintained by Koala staff, which were reaching about 3 million people weekly, Willis began using information-warfare tactics, some inspired by young Macedonians. Willis studied the connection between Koala headlines and the emotions they triggered among readers. The next time Koala Media's owners came into the office, Willis showed them a carefully outlined posting schedule.
"I surprised them by holding up a large poster board with what became the schedule and deep explanations from a psychological standpoint on what articles to put at what times," he said. "Early morning was positive articles—people will interact with positive things when they first wake up, they had the big story of the day at 11 am already, which they previously noticed was the most powerful slot of the day, afternoon prior to 2 pm was articles to really push hard, late night (11 pm to the early morning) was fringe content."
These claims have been corroborated to Ars by former Koala Media staff who prefer to remain anonymous.
The new publishing strategy, along with the additional fake news sites, caused a rapid spike in traffic. As Willis puts it, this all felt "like playing a video game and getting new high scores to me. I did not think of the readers as people but more like background characters in a video game. I am neurodiverse and have major issues with understanding empathy due to my condition. Crunching numbers is something I love to do; these were numbers I wanted to go up, and I would do it with no emotional attachment to the material or people."
Soon enough, Koala's published "news" pieces reached over 30 million people a week.
"I was completely caught off guard while pushing nonstop Trump news through the election cycle," said Willis. "One of our websites was the No. 1 Google search result for the term 'Trump News'!"
The results of the 2016 election left Willis and his team in shock.
“The shock was around the power of the machine”
Willis and his army of fake news editors knew that millions of Americans targeted by their pro-Trump and anti-Clinton propaganda were real people who actually showed up to vote. "There were other pro-Trump news organizations," Willis told me, "but nothing was built [as] extreme as ours. We had without a doubt contributed to Donald Trump winning the presidency."
Countless studies, including one from Stanford, attempted to pin the election outcome on fake news. Ars has seen the news articles produced by Willis' operation but cannot disclose these, as doing so would divulge the Koala website.
By the end, Willis was hoping that he and his team would be caught, that someone would be able to connect the dots. But it didn't quite happen.
By 2017, after being with the fake news farm for nearly two years, Willis couldn't take it anymore. "I had a soul-searching moment and money in the bank and decided what I liked doing most was hacking, and I wanted to get back to it. So I decided to get a job in the security industry as a hacker," he said.
He knew he should talk about the system he had helped to build. "I helped contribute to the monster of fake news," he said. "I knew I had a responsibility to be a whistleblower on what exactly went down—even though the network I helped build looks like a shell of its former self, especially after they've been banned from basically every platform, along with other 'alternative' news outlets."
In a Zoom call, Stephenson told me that, as a podcast producer, he is constantly looking for ways to expand his network and frequently travels to attend social events like conferences. It was at the interactive media and film festival South by Southwest (SXSW) in March 2018 that Stephenson first met Willis. Over a period of time, the two became well acquainted. When Willis confessed to Stephenson the details of the operation and stressed that he needed the truth to come out, Stephenson agreed to act as his "handler."
Around that time, former White House chief information officer and renowned cybersecurity expert Theresa Payton was working on her book Manipulated. She was seeking sources who were either the victims of misinformation campaigns or "manipulators" involved in fake news production. Payton was already one of Stephenson's connections, and Stephenson put Payton and Willis in touch.
But Payton did not know Willis' real name—referring to him only as "Hacker X" in her book. In the beginning, Payton and "Hacker X" would communicate over Zoom calls with the latter's video turned off. In June 2019, though, the trio met in Austin, Texas. Even then, on meeting "Hacker X" and Stephenson in person, Payton did not know Willis' real name—although Willis did reveal his face. The detailed account of the rendezvous is shared in Payton's book.
One interesting detail that caught my eye in the book was that Payton had no trouble guessing how "Hacker X" might look months before they had even met:
Without ever laying eyes on him—as his video was turned off during our first interview—I ask if he would let me guess what he looks like. He laughs heartily, thinking I'll never guess correctly. Based on our two hours speaking and my profiling skills, I hazard a guess. "You're a five-ten to six-foot twentysomething male. Earnest-looking face, perhaps—someone who could be in a J. Crew or Brooks Brothers ad or lacrosse-team picture of an Ivy League school."
"Wow!" his handler says. "That description was stunningly accurate."
In the same book, "Benefactors" is a catch-all phrase used by Payton to describe Koala Media owners. The chapter titled "Anatomy of a Manipulation Campaign" goes over the Benefactors' intentions:
Hacker X tells me that the Benefactors wanted to initiate a massive manipulation campaign with three goals: "run an online news campaign that would net them a lot of money, make sure they did not get caught, and not let the deep state get Hillary elected." The Benefactors had admired the Obama campaign and its ability to mobilize and energize voters through social media to gather, donate, and get the word out. They told him they had heard Hacker X was the best at building algorithms that could target the right message to the right people—and at hiding his tracks; they wanted to keep this operation and his existence covert. They wanted him to build this operation from the ground up, with security and privacy in mind. "I'm sitting at a table with them, and they say to me, 'Don't you want to be part of something big? You can help us make sure the country picks the right person for the next president. Are you up for the challenge?' And I said, 'Heck, yeah—I'm ready!" Hacker X quietly adds, "I gave up almost two years of my life serving the cause." Not once does he mention anything about "moral values." I have no idea if Hacker X is religious; he never says anything racial, bigoted, anti-immigrant, or antigay.
He does demonstrate a deep dislike for the "elites" and for Hillary Clinton, repeatedly using the phrase "destroy Hillary"; otherwise, he seems to generally love his fellow Americans and all walks of life.
Payton's book was published in the spring of 2020, a time when humanity's focus was on fighting the coronavirus pandemic. With so much devastation all around, unmasking "Hacker X" wasn't on anyone's radar.
Despite all the evidence presented by Willis, and despite the multiple sources who helped me corroborate the story, the skeptic in me couldn't help but wonder: what were Willis' true motives while all this was happening? What were they now? And what would "whistleblowing" achieve years after the 2016 election—and without naming the news empire he ran?
To ditch a job offer Willis had already accepted, diving instead into the murky waters of an elusive antiestablishment "media company," raised questions. Was it just the thrill of exploring the unknown and having found a place where his hacker mindset could be applied to its fullest potential? Or did Willis see some higher purpose in denouncing the political "establishment"?
Willis admits the decision to join Koala Media was at least partially motivated by political revenge, but based on my understanding from people who have known Willis, he isn't actually "right wing." Willis isn't a Democrat or even a proper Republican, his handler Stephenson tells me. He's just antiestablishment. Willis' self-proclaimed title—"original punk rock right-wing millennial"—aptly describes his ideology.
Sources also told Ars that Koala Media owners realized the massive potential for financial gain in pushing out the pro-Trump and anti-Clinton rhetoric after analyzing Trump's voter base and their emotional reactions to the fake news articles all adding to traffic. Had Clinton's voter base earned them more money, the pro-Clinton narrative might have been their focus, claim the sources.
Was Willis the same way? Did he do it for the money?
Born in Stamford, Connecticut, Willis was raised by his mother and her family, who had immigrated to the United States from Italy. "My father was around, but I didn't know his family," Willis told Ars.
The hacker says that he comes from a place of poverty, though as a child he was never aware of it. "I had a unique upbringing; I was poor but didn't know it when I was young," he said. "I remember that I wasn't allowed to cross the street since my neighbors were crackheads. I grew up around the gay community from a young age with gay relatives. My schoolmates and friends were very diverse since I was just outside New York City. I was always very open-minded and accepting of everyone. I was the only white kid on my bus. I grew up as diverse as one could imagine."
In a conversation with Ars, the hacker recounted an episode from his childhood where his younger brother got attacked over food stamps on their school bus, and he described a nomadic life deprived of the basics.
"I built my first computer using parts from computers found at the city dump," he said. But it was getting involved with technology that helped Willis out of a life of poverty.
"By the time I was in my 20s I had moved over 30 times—very much a nomad, always used to moving but within the same general areas. I put myself through school with the help of government aid, taking many years, struggled very hard, because of no support system. Once government aid ran out, I paid upfront for classes before taking loans for my bachelor's degree."
But poverty isn't what fueled his journey into the secretive Koala empire. Rather, what seemed like a "fun-sounding job" came his way years after he had already left Connecticut and a life of poverty behind.
As for naming himself, if he didn't do it in Payton's book, why come out now? After having interviewed Willis several times, Payton believes the hacker just wants to do the right thing. According to her, Willis is an "idealist" and wants people to know the truth. Recounting her meeting with Willis and Stephenson, she tells me, "I saw that in front of my own eyes... He is superimpressed with himself that he got away with it. But he felt disgusted that he got away with it."
"He has remorse for what he did. I feel protective about him," stressed Payton. "Not that he needs protecting… but he wants to right the wrongs that he believes he was a part of."
Willis' decision to reveal his identity now, he told me, is fueled by the continuing damage that he sees from fake news stories about COVID, especially those spreading anti-vaccination propaganda.
"The new war is to wake up those who have been manipulated, while actively taking out the fake news campaigns," writes Willis in a blog post. "COVID has shown me the deadly side of fake news and anti-vaccination people. After multiple conversations with my father, who refuses to wear a mask or get vaccinated, I was getting very concerned. I asked him what sites he would read the conspiracy-based things on, and he mentioned the website that ran the network I had built the machine on."
Prior to approaching me, Willis had disclosed his history with fake news farms to his family, hoping to undo the brainwashing done by these websites. Unfortunately, it was too late. To this day, Willis' father does not believe the hacker's story, Willis said, adding, "He has been too manipulated."
A common topic of discussion among my close friends is where the bottlenecks are in our productivity and how we can execute more quickly. This is very different from what I see in my extended social circles, where people commonly say that velocity doesn't matter. In online discussions about this, I frequently see people go a step further and assign moral valence to this, saying that it is actually bad to try to increase velocity or be more productive or work hard (see appendix for more examples).
The top reasons I see people say that productivity doesn't matter (or is actually bad) fall into one of three buckets:
Working on the right thing is more important than working quickly
Speed at X doesn't matter because you don't spend much time doing X
Thinking about productivity is bad and you should "live life"
I certainly agree that working on the right thing is important, but increasing velocity doesn't stop you from working on the right thing. If anything, each of these is a force multiplier for the other. Having strong execution skills becomes more impactful if you're good at picking the right problem and vice versa.
It's true that the gains from picking the right problem can be greater than the gains from having better tactical execution because the gains from picking the right problem can be unbounded, but it's also much easier to improve tactical execution and doing so also helps with picking the right problem because having faster execution lets you experiment more quickly, which helps you find the right problem.
A concrete example of this is a project I worked on to quantify the machine health of the fleet. The project discovered a number of serious issues (a decent fraction of hosts were actively corrupting data or had a performance problem that would increase tail latency by > 2 orders of magnitude, or both). This was considered serious enough that a new team was created to deal with the problem.
In retrospect, my first attempts at quantifying the problem were doomed and couldn't have really worked (or not in a reasonable amount of time, anyway). I spent a few weeks cranking through ideas that couldn't work and a critical part of getting to the idea that did work after "only" a few weeks was being able to quickly try out and discard ideas that didn't work. In part of a previous post, I described how long a tiny part of that process took and multiple people objected to that being impossibly fast in internet comments.
I find this a bit funny since I'm not a naturally quick programmer. Learning to program was a real struggle for me and I was pretty slow at it for a long time (and I still am in aspects that I haven't practiced). My "one weird trick" is that I've explicitly worked on speeding up things that I do frequently and most people have not. I view the situation as somewhat analogous to sports before people really trained. For a long time, many athletes didn't seriously train, and then once people started trying to train, the training was often misguided by modern standards. For example, if you read commentary on baseball from the 70s, you'll see people saying that baseball players shouldn't weight train because it will make them "muscle bound" (many people thought that weight lifting would lead to "too much" bulk, causing people to be slower, have less explosive power, and be less agile). But today, players get a huge advantage from using performance-enhancing drugs that increase their muscle-bound-ness, which implies that players could not get too "muscle bound" from weight training alone. An analogous comment to one discussed above would be saying that athletes shouldn't worry about power/strength and should increase their skill, but power increases returns to skill and vice versa.
Coming back to programming, if you explicitly practice and train and almost no one else does, you'll be able to do things relatively quickly compared to most people even if, like me, you don't have much talent for programming and getting started at all was a real struggle. Of course, ther's always going to be somewhere more talented out there who's executing faster after having spent less time improving. But, luckily for me, relatively few people seriously attempt to improve, so I'm able to do ok.
Anyway, despite operating at a rate that some internet commenters thought was impossible, it took me weeks of dead ends to find something that worked. If I was doing things at a speed that people thought was normal, I suspect it would've taken long enough to find a feasible solution that I would've dropped the problem after spending maybe one or two quarters on it. The number of plausible-ish seeming dead ends was probably not unrelated to why the problem was still an open problem despite being a critical issue for years. Of course, someone who's better at having ideas than me could've solved the problem without the dead ends, but as we discussed earlier, it's fairly easy to find low hanging fruit on "execution speed" and not so easy to find low hanging fruit on "having better ideas". However, it's possible to, to a limited extent, simulate someone who has better ideas than me by being able to quickly try out and discard ideas (I also work on having better ideas, but I think it makes sense to go after the easier high ROI wins that are available as well). Being able to try out ideas quickly also improves the rate at which I can improve at having better ideas since a key part of that is building intuition by getting feedback on what works.
The next major objection is that speed at a particular task doesn't matter because time spent on that task is limited. At a high level, I don't agree with this objection because, while this may hold true for any particular kind of task, the solution to that is to try to improve each kind of task and not to reject the idea of improvement outright. A sub-objection people have is something like "but I spend 20 hours in unproductive meetings every week, so it doesn't matter what I do with my other time". I think this is doubly wrong, in that if you then only have 20 hours of potentially productive time, whatever productivity multiplier you have on that time still holds for your general productivity. Also, it's generally possible to drop out of meetings that are a lost cause and increase the productivity of meetings that aren't a lost cause1.
More generally, when people say that optimizing X doesn't help because they don't spend time on X and are not bottlenecked on X, that doesn't match my experience as I find I spend plenty of time bottlenecked on X for commonly dismissed Xs. I think that part of this is because getting faster at X can actually increase time spent on X due to a sort of virtuous cycle feedback loop of where it makes sense to spend time. Another part of this is illustrated in this comment by Fabian Giesen:
It is commonly accepted, verging on a cliche, that you have no idea where your program spends time until you actually profile it, but the corollary that you also don't know where you spend your time until you've measured it is not nearly as accepted.
When I've looked how people spend time vs. how people think they spend time, it's wildly inaccurate and I think there's a fundamental reason that, unless they measure, people's estimates of how they spend their time tends to be way off, which is nicely summed in by another Fabian Giesen quote, which happens to be about solving rubik's cubes but applies to other cognitive tasks:
Paraphrasing a well-known cuber, "your own pauses never seem bad while you're solving, because your brain is busy and you know what you're thinking about, but once you have a video it tends to become blindingly obvious what you need to improve". Which is pretty much the usual "don't assume, profile" advice for programs, but applied to a situation where you're concentrated and busy for the entire time, whereas the default assumption in programming circles seems to be that as long as you're actually doing work and not distracted or slacking off, you can't possibly be losing a lot of time
Unlike most people who discuss this topic online, I've actually looked at where my time goes and a lot of it goes to things that are canonical examples of things that you shouldn't waste time improving because people don't spend much time doing them.
An example of one of these, the most commonly cited bad-thing-to-optmize example that I've seen, is typing speed (when discussing this, people usually say that typing speed doesn't matter because more time is spent thinking than typing). But, when I look at where my time goes, a lot of it is spent typing.
A specific example is that I've written a number of influential docs at my current job and when people ask how long some doc took to write, they're generally surprised that the doc only took a day to write. As with the machine health example, a thing that velocity helps with is figuring out which docs will be influential. If I look at the docs I've written, I'd say that maybe 15% were really high impact (caused a new team to be created, changed the direction of existing teams, resulted in significant changes to the company's bottom line, etc.). Part of it is that I don't always know which ideas will resonate with other people, but part of it is also that I often propose ideas that are long shots because the ideas sound too stupid to be taken seriously (e.g., one of my proposed solutions to a capacity crunch was to, for each rack, turn off 10% of it, thereby increasing effective provisioned capacity, which is about as stupid sounding an idea as one could come up with). If I was much slower at writing docs, it wouldn't make sense to propose real long shot ideas. As things are today, if I think an idea has a 5% chance of success, in expectation, I need to spend ~20 days writing docs to have one of those land.
I spend roughly half my writing time typing. If I typed at what some people say median typing speed is (40 WPM) instead of the rate some random typing test clocked me at (110 WPM), this would be a 0.5 + 0.5 * 110/40 = 1.875x slowdown, putting me at nearly 40 days of writing before a longshot doc lands, which would make that a sketchier proposition. If I hadn't optimized the non-typing part of my writing workflow as well, I think I would be, on net, maybe 10x slower2, which would put me at more like ~200 days per high impact longshot doc, which is enough that I think that I probably wouldn't write longshot docs3.
More generally, Fabian Giesen has noted that this kind of non-linear impact of velocity is common:
There are "phase changes" as you cross certain thresholds (details depend on the problem to some extent) where your entire way of working changes.
There's a lot of things I could in theory do at any speed but in practice cannot, because as iteration time increases it first becomes so frustrating that I can't do it for long and eventually it takes so long that it literally drops out of my short-term memory, so I need to keep notes or otherwise organize it or I can't do it at all.
Certainly if I can do an experiment in an interactive UI by dragging on a slider and see the result in a fraction of a second, at that point it's very "no filter", if you want to try something you just do it.
Once you're at iteration times in the low seconds (say a compile-link cycle with a statically compiled lang) you don't just try stuff anymore, you also spend time thinking about whether it's gonna tell you anything because it takes long enough that you'd rather not waste a run.
Once you get into several-minute or multi-hour iteration times there's a lot of planning to not waste runs, and context switching because you do other stuff while you wait, and note-taking/bookkeeping; also at this level mistakes are both more expensive (because a wasted run wastes more time) and more common (because your attention is so divided).
As you scale that up even more you might now take significant resources for a noticeable amount of time and need to get that approved and budgeted, which takes its own meetings etc.
A specific example of something moving from one class of item to another in my work was this project on metrics analytics. There were a number of proposals on how to solve this problem. There was broad agreement that the problem was important with no dissenters, but the proposals were all the kinds of things you'd allocate a team to work on through multiple roadmap cycles. Getting a project that expensive off the ground requires a large amount of organizational buy-in, enough that many important problems don't get solved, including this one. But it turned out, if scoped properly and executed reasonably, the project was actually something a programmer could create an MVP of in a day, which takes no organizational buy-in to get off the ground. Instead of needing to get multiple directors and a VP to agree that the problem is among the org's most important problems, you just need a person who thinks the problem is worth solving.
Going back to Xs where people say velocity doesn't matter because they don't spend a lot time on X, another one I see frequently is coding, and it is also not my personal experience that coding speed doesn't matter. For the machine health example discussed above, after I figured out something that would work, I spent one month working on basically nothing but that, coding, testing, and debugging. I think I had about 6 hours of meetings during that month, but other than that plus time spent eating, etc., I would go in to work, code all day, and then go home. I think it's much more difficult to compare coding speed across people because it's rare to see people do the same or very similar non-trivial tasks, so I won't try to compare to anyone else, but if I look at my productivity before I worked on improving it as compared to where I'm at now, the project probably would have been infeasible without the speedups I've found by looking at my velocity.
Ahmdal's law based arguments can make sense when looking for speedups in a fixed benchmark, like a sub-task of SPECint, but when you have a system where getting better at a task increases returns to doing that task and can increase time spent on the task, it doesn't make sense to say that you shouldn't work on something because you spend a lot of time doing it. I spend time on things that are high ROI, but those things are generally only high ROI because I've spent time improving my velocity, which reduces the "I" in ROI.
The last major argument I see against working on velocity assigns negative moral weight to the idea of thinking about productivity and working on velocity at all. This kind of comment often assigns positive moral weight to various kinds of leisure, such as spending time with friends and family. I find this argument to be backwards. If someone thinks it's important to spend time with friends and family, an easy way to do that is to be more productive at work and spend less time working.
Personally, I deliberately avoid working long hours and I suspect I don't work more than the median person at my company, which is a company where I think work-life balance is pretty good overall. A lot of my productivity gains have gone to leisure and not work. Furthermore, deliberately working on velocity has allowed me to get promoted relatively quickly4, which means that I make more money than I would've made if I didn't get promoted, which gives me more freedom to spend time on things that I value.
For people that aren't arguing that you shouldn't think about productivity because it's better to focus on leisure and instead argue that you simply shouldn't think about productivity at all because it's unnatural and one should live a natural life, that ultimately comes down to personal preference, but for me, I value the things I do outside of work too much to not explicitly work on productivity at work.
As with this post on reasons to measure, while this post is about practical reasons to improve productivity, the main reason I'm personally motivated to work on my own productivity isn't practical. The main reason is that I enjoy the process of getting better at things, whether that's some nerdy board game, a sport I have zero talent at that will never have any practical value to me, or work. For me, a secondary reason is that, given that my lifespan is finite, I want to allocate my time to things that I value, and increasing productivity allows me to do more of that, but that's not a thought i had until I was about 20, at which point I'd already been trying to improve at most things I spent significant time on for many years.
Another common reason for working on productivity is that mastery and/or generally being good at something seems satisfying for a lot of people. That's not one that resonates with me personally, but when I've asked other people about why they work on improving their skills, that seems to be a common motivation.
A related idea, one that Holden Karnofsky has been talking about for a while, is that if you ever want to make a difference in the world in some way, it's useful to work on your skills even in jobs where it's not obvious that being better at the job is useful, because the developed skills will give you more leverage on the world when you switch to something that's more aligned with you want to achieve.
Appendix: one way to think about what to improve
Here's a framing I like from Gary Bernhardt (not set off in a quote block since this entire section, another than this sentence, is his).
People tend to fixate on a single granularity of analysis when talking about efficiency. E.g., "thinking is the most important part so don't worry about typing speed". If we step back, the response to that is "efficiency exists at every point on the continuum from year-by-year strategy all the way down to millisecond-by-millisecond keystrokes". I think it's safe to assume that gains at the larger scale will have the biggest impact. But as we go to finer granularity, it's not obvious where the ROI drops off. Some examples, moving from coarse to fine:
The macro point that you started with is: programming isn't just thinking; it's thinking plus tactical activities like editing code. Editing faster means more time for thinking.
But editing code costs more than just the time spent typing! Programming is highly dependent on short-term memory. Every pause to edit is a distraction where you can forget the details that you're juggling. Slower editing effectively weakens your short-term memory, which reduces effectiveness.
But editing code isn't just hitting keys! It's hitting keys plus the editor commands that those keys invoke. A more efficient editor can dramatically increase effective code editing speed, even if you type at the same WPM as before.
But each editor command doesn't exist in a vacuum! There are often many ways to make the same edit. A Vim beginner might type "hhhhxxxxxxxx" when "bdw" is more efficient. An advanced Vim user might use "bdw", not realizing that it's slower than "diw" despite having the same number of keystrokes. (In QWERTY keyboard layout, the former is all on the left hand, whereas the latter alternates left-right-left hands. At 140 WPM, you're typing around 14 keystrokes per second, so each finger only has 70 ms to get into position and press the key. Alternating hands leaves more time for the next finger to get into position while the previous finger is mid-keypress.)
We have to choose how deep to go when thinking about this. I think that there's clear ROI in thinking about 1-3, and in letting those inform both tool choice and practice. I don't think that (4) is worth a lot of thought. It seems like we naturally find "good enough" points there. But that also makes it a nice fence post to frame the others.
#4 comment: "What if I don't want to improve my productivity ? Just take time."
After the initial indignation, this comment goes on and proves that the commenter missed the point entirely, as the rest of the comment explains how the commenter works productively, which the commenter apparently is ok with as long as it's not phrased as a way to work productively, because one is supposed to be morally outraged by someone wanting to be productive and sharing techniques about how to be productive with other people who might be interested in being productive
In the responses, someone points out that someone who's more productive would be able to spend more time on leisure; that comment is uniformly panned because "work expands so as to fill the time available for its completion", as if how one spends time is some sort of immutable law of nature and not something under anyone's control
Another comment: "Alright. What are we optimizing for? Productivity? Or the end-goals of any of: achieving more, climbing the corporate ladder, making more money, etc..?"
Some positive examples of people who have used their productivity to "fund" things that they value include Andy Kelley (Zig), Jamie Brandon (various), Andy Matuschak (mnemonic medium, various), Saul Pwanson (VisiData), Andy Chu (Oil Shell). I'm drawing from programming examples, but you can find plenty of others, e.g., Nick Adnitt (Darkside Canoes) and, of course, numerous people who've retired to pursue interests that aren't work-like at all.
Appendix: another reason to avoid being productive
An idea that's become increasingly popular in my extended social circles at major tech companies is that one should avoid doing work and waste as much time as possible, often called "antiwork", which seems like a natural extension of "tryhard" becoming an insult. The reason given is often something like, work mainly enriches upper management at your employer and/or shareholders, who are generally richer than you.
I'm sympathetic to the argument and agree that upper management and shareholders capture most of the value from work. But as much as I sympathize with the idea of deliberately being unproductive to "stick it to the man", I value spending my time on things that I want enough that I'd rather get my work done quickly so I can do things I enjoy more than work. Additionally, having been productive in the past has given me good options for jobs, so I have work that I enjoy a lot more than my acquaintances in tech who have embraced the "antiwork" movement.
The less control you have over your environment, the more it makes sense to embrace "antiwork". Programmers at major tech companies have, relatively speaking, a lot of control over their environment, which is why I'm not "antiwork" even though I'm sympathetic to the cause.
Thanks to Fabian Giesen, Gary Bernhardt, Ben Kuhn, David Turner, Marek Majkowski, Anja Boskovic, Aaron Levin, Lifan Zeng, Justin Blank, and Jamie Brandon for comments/corrections/discussion
When I look at the productiveness of meetings, there are some people who are very good at keeping meetings on track and useful. For example, one person who I've been in meetings with who is extraordinarily good at ensuring meetings are productive is Bonnie Eisenman. Early on in my current job, I asked her how she was so effective at keeping meetings productive and have been using that advice since then (I'm not nearly as good at it as she is, but even so, improving at this was a significant win for me).
10x might sound like an implausibly large speedup on writing, but in a discussion on writing speed on a private slack, a well-known newsletter author mentioned that their net writing speed for a 5k word newsletter was a little under 2 words per minute (WPM). My net rate (including time spent editing, etc.) is over 20 WPM per doc.
With a measured typing speed of 110 WPM, that might sound like I spend a small fraction of my time typing, but it turns out it's roughly half the time. If I look at my writing speed, it's much slower than my typing test speed and it seems that it's perhaps half the rate. If I look at where the actual time goes, roughly half of it goes to typing and have goes to thinking, semi-serially, which creates long pauses in my typing.
If I look at where the biggest win here could come, it would be from thinking and typing in parallel, which is something I'd try to achieve by practicing typing more, not less. But even without being able to do that, and with above average typing speed, I still spend half of my time typing!
The reason my net speed is well under the speed that I write is that I do multiple passes and re-write. Some time is spent reading as I re-write, but I read much more quickly than I write, so that's a pretty small fraction of time. In principle, I could adopt an approach that involves less re-writing, but I've tried a number of things that one might expect would lead to that goal and haven't found one that works for me (yet?).
Although the example here is about work, this also holds for my personal blog, where my velocity is similar. If I wrote ten times slower than I do, I don't think I'd have much of a blog. My guess is that I would've written a few posts or maybe even a few drafts and not gotten to the point where I'd post and then stop.
I enjoy writing and get a lot of value out of it in a variety of ways, but I value the other things in my life enough that I don't think writing would have a place in my life if my net writing speed were 2 WPM.
another strategy would be to write shorter docs. There's a style of doc where that works well, but I frequently write docs where I leverage my writing speed to discuss a problem that would be difficult to convincingly discuss without a long document.
One example of a reason that my docs is that I frequently work on problems that span multiple levels of the stack, which means that I end up presenting data from multiple levels of the stack as well as providing enough context about why the problem at some level drives a problem up or down the stack for people who aren't deeply familiar with that level of the stack, which is necessary since few readers will have strong familiarity with every level needed to understand the problem.
In most cases, there have been previous attempts to motivate/fund work on the problem that didn't get traction because there wasn't a case linking an issue at one level of the stack to important issues at other levels of the stack. I could avoid problems that span many levels of the stack, but there's a lot of low hanging fruit among those sorts of problems for technical and organizational reasons, so I don't think it makes sense to ignore them just because it takes a day to write a document explaining the problem (although it might make sense if it took ten days, at least in cases where people might be skeptical of the solution).