From an inside look at Twitter’s problems with management, technology, and trolls:
At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.”
Calls to rethink that approach were ignored by senior executives, according to people familiar with the situation. “There was no real sense of urgency,” the former executive explained, pointing the finger at Harvey’s superiors, including current C.E.O. Jack Dorsey. “It’s a technology company with crappy technologists, a revolving door of product heads and C.E.O.s, and no real core of technological innovation. You had Del saying, ‘Trolls are going to be a problem. We will need a technological solution for this.’” But Twitter never developed a product sophisticated enough to automatically deal with with bots, spam, or abuse. “You had this unsophisticated human army with no real scalable platform to plug into. You fast forward, and it was like, ‘Hey, shouldn’t we just have basic rules in place where if the suggestion is to suspend an account of a verified person, there should be a process in place to have a flag for additional review, or something?’ You’d think it would take, like, one line of code to fix that problem. And the classic response is, ‘That’s on our product road map two quarters from now.’”
None of this means that Twitter is going to vanish soon — after all, COBOL is still around, and software legacies just hang around, decaying slowly, like an assortment of pseudogenes. But still, maybe you should consider jumping ship, since the one way to kill it is to erode its user base. Mastodon is out there, waiting for you with open arms.
I wonder if that’s why I keep getting hang-ups when sending a tweet, or secure connection failures sometimes. Twitter is garbage by every definition!
My only problem with Mastodon has been when new toots are made there’s nothing in the tab name letting me know. So if it actually were more active 20 o r 30 could go by and I wouldn’t even notice until I remember and actually go to that tab page.
It’s a fairly minor issue compared with, um, all of Twitter.
Are people actually leaving twitter though? Granted, I don’t spend much time there at all, outside of getting my posts uploaded, but it doesn’t seem to be desolate of users at this point in time.
I don’t understand why they can’t be building a proper platform, then switch over when its ready, but what do I know about such shit?
The technological issue is just finger pointing. There’s nothing about a particular webdev stack that prevents you from implementing anti-harassment measures. Mastodon is written in Ruby, using Ruby on Rails, for what it’s worth. The real problem is management.
It’s laughable to blame Ruby on Rails for Twitter’s problems with harassment. That really sounds like something an executive would say to avoid responsibility.
Caine@#2:
Are people actually leaving twitter though?
There are people who just ignore twitter and don’t leave. That’s the problem that’s underlying twitter’s membership – even they don’t know what percentage of their audience are bots. It’s bots talking to bots, and there’s still lots of talking going on; you know how bots like to talk…
What’s funny (in an “eeeuh, gross”) kind of way is that their market value is based on eyeballs and ads, which is being propped up by bots reading and clicking about, too. It’s a stock scam; the bots just have to be good enough to look like humans and they can keep charging their advertisers for clicks.
The whole house of cards will collapse when marketers realize that the bots aren’t actually converting into sales. I heard a statistic the other day about Pepsi, I believe, posting ads in 15,000 different online channels and measuring the click-through and impact of the campaign. Zero. None. Zip. Basically, it turned out that huge ad campaigns had absolutely no effect because the only place that anyone sees them is google and there’s a single thread holding that up, which is that people actually search for what they are interested in using google. So, if I google “should I have a nice yummy pepsi sugary product?” google will: give me pepsi ads. Holy shit! Really!
The online ad market is/will collapse soon but it’s unfortunately mostly going to happen when giant aggregators like amazon essentially become adfarms. They already are, but that’s their value: e-tailers are paying for a place at the table. The only parts of the online economy that are worth anything are google, amazon, ebay, baidu and alibaba because they are both the storefront and the search engine. Google is unique in that set in that it’s not a storefront, exactly. But it’s going to survive only because people accidentally “see” the ads as they are searching for stuff; it’s a scam.
I’ve been meaning to do a posting about this for some time, but the whole online marketing bogo-economy is very murky and there are people making huge amounts of money at it – so it looks like it’s lively and popping, but it may actually be moribund; it’s like a zombie animated on meth and nitromethane.
Siggy@#4:
It’s laughable to blame Ruby on Rails for Twitter’s problems with harassment.
Damn right.
It’s actually C’s fault. Because that’s what Ruby on Rails was written in (it’s compilers all the way down!)
No, wait, it’s Linux’ fault. Except that was just a rewrite of UNIX.
Twitter’s problems are all due to the guys in the UNIX Room at Bell Labs.
I am still waiting for “antisocial media” – when that shit comes around I’m gonna jump on it.
Ruby On Rails was incredible for its time. It made rapid prototyping of websites easy and it scaled “well enough” for what it was intended to. Calling it “fisher-price” is an unwarranted slur. It was the first of its breed. Lessons were learned and new frameworks were developed. Some would say too many; we get new ones every week, it seems like. We _can_ fault twitter for not paying back technical debt, but I don’t think it’s reasonable to fault them for using one of the only available web frameworks when they started out.
Rob Bos@#8:
Calling it “fisher-price” is an unwarranted slur
It’s generous. Calling RoR “fisher-price” is grounds for a C&D from Fisher Price’s lawyers.
Oi. I know some really good RoR programmers. While I don’t use it myself, I understand it’s improved quite a bit since those early days. They’re not the only big tech company to use it, either: GitHub was built on RoR in the early days as well, they’re not having these types of problems (or, if they are, I haven’t heard about it yet). Really, it sounds like Twitter’s management screwed the pooch by not letting them work on their technical debt. Of course, I am a programmer, so I’m more sympathetic to the dev team =).
It’s a poor carpenter who blames their tools. If, for some reason, it was impossible or impractical to code something up in Ruby, they could have coded that piece in another language and called into it from the Ruby code. Blaming the technology is just cowardly scapegoating.
Marcus Ranum @#9 – I’m willing to agree that it’s true NOW, but 15 years ago, it was the only game in town.
Twitter only _used_ to be written in Ruby-on-Rails. A number of years, they rewrote it in Scala to make it more scalable, and such. (Mastadon, is still written in Ruby on Rails, BTW). As has been said, Twitter’s issues are managerial, not because of what language the code is written in.
Realistically, if you were going to complain about a “Fisher-Price” framework, that would be WordPress and PHP. Ruby is much better than that. Yet these frame works work.
Regardless of the backend framework used, building an automated system to detect “abuse” without involvement by “non-scalable” human moderators is an extremely hard problem that no one has a solution for.
Detecting spam and bots is easier, but blaming that on Rails is just absurd.
@Marcus Ranum #6:
Pfft. I blame the designer of the PDP-7’s flipchips.
”
What’s funny (in an “eeeuh, gross”) kind of way is that their market value is based on eyeballs and ads, which is being propped up by bots reading and clicking about, too. It’s a stock scam; the bots just have to be good enough to look like humans and they can keep charging their advertisers for clicks.
”
Whenever I get a “Promoted tweet”, I block the promoter. It doesn’t make any impact, I’m sure, but at least I have the small satisfaction of knowing their money had the directly opposite effect of their intention.
@16, Sili
I do this too. Allergy to marketing sure has some weird symptoms, doesn’t it?
One other thing that is seldom mentioned is…
In order to comply with the FBI’s gentle requests for data about users’ activities, any social media site that gets large enough is going to have to have capabilities for pulling up queries like “every comment Marcus ever made” or even “every comment made by anyone Marcus talks to that references thagomizers”
That stuff is all the inputs into a pretty good stalker detector, sockpuppet correlator and bot detector – it’s just that the FBI doesn’t care about bots (or didn’t until recently…) and it was probably a development afterthought and may have been done quick and dirty.
You know that the FBI and NSA have way more data than they need to correlate all the bots to their owners as well as all the robo-callers and most of the spammers; they just don’t because they want to keep the capability secret don’t’cha know? Which gives us the funny situation that anyone who is worried about being watched is practicing operational security and the police state has all the information they could use to actually improve things, but won’t because they don’t want to spook the people they are watching, who are already thoroughly spooked. I would describe it as a self-licking ice cream cone, except it can’t even lick itself.
drransom, @ #14:
Well, to do it perfectly would be very hard, but you can certainly do a great deal better than Twitter currently does without too much difficulty. Combine Bayesian spam filtering techniques with some simple network analysis (e.g. “has this user received a large number messages containing racial slurs in the last hour from people they have no prior connection with?”) and you could probably knock out better than 90% of it. Get some machine learning involved, and you can probably get that up to 99%+. They just don’t because (a) they don’t really care beyond the PR aspects, and (b) abuse is actually good for their business model. As long as your key metrics are number of active users and number of tweets sent, having people and bots using burner accounts to spew huge volumes of abuse at people is a positive benefit.
Marcus, @ #18:
Also, having lots of other people doing all that shady stuff provides useful cover for all the shady stuff they’re doing. (See also: Tor.)
I guess they figured out some of the programming. They apparently upset the MAGA crowd by purging tens of thousands of bots this evening which killed great swaths of their follower lists. Now if they could figure out the programming to get rid of the MAGA crowd.
There are plenty of machine learning tools to be used on Ruby. That is exactly what would be used to do the filtering they’re saying can’t be done. Also, as d3zd3z mentioned, it’s now written in Scala, so yes they can always make whole platform changes without much impact. All they would need to do is write the requisite functions and such, and it’s done. It wouldn’t be easy, but it’s absolutely do-able on Ruby on Rails or any other framework. If needed they could have even had outside function calls to R or Python or another language that could handle the statistical computation more cleanly.
PHP and the things built on top of it have improved a lot over the years, too, you know. Ruby on Rails may beat PHP and its friends from 15 years ago, but at this point RoR is the new PHP. It’s kept all its warts and is pretty much a “grind out thousands of lines of code” world for most developers, right up there with PHP and Java. In particular, Rails has always rather overused abstraction and heavyweight “design” concepts, and just generally prized cleverness over clarity, making it look cool but in the end leaving you with code that’s harder to understand and maintain. Python, which is practically the same language when you look at it from the kind of distance a Haskell programmer would, at least didn’t fall into that trap.
Oh, and RoR also seems to be the main home of cargo-cult “agile” developers, by which I mean those who are more concerned with going through the motions of being agile than actually getting rid of code they don’t need. It’s this combination of relatively poor code and bad culture that really kills your productivity in that world. Though I’ve spent plenty of time working with people who develop in Rails, I’ve never done coding within that platform myself not so much because it looked bad in 2004 and doesn’t look any better now, but more because RoR developers reject the opinions of anybody who hasn’t drunk the kool-aid. (I did like Ruby enough to write my own web framework in it.)
So in the end I think you can blame RoR for IT failures, but it’s not strictly a technology fail, it’s more a culture fail enhanced by technology that’s far from great. (“Better than Perl” was not really a high aspiration in the early 2000s, much less now.)