New EU rules mean WhatsApp and Messenger must be interoperable with other chat apps. Here’s how that will work.
A frequent annoyance of contemporary life is having to shuffle through different messaging apps to reach the right person. Messenger, iMessage, WhatsApp, Signal—they all exist in their own silos of group chats and contacts. Soon, though, WhatsApp will do the previously unthinkable for its 2 billion users: allow people to message you from another app. At least, that’s the plan.
For about the past two years, WhatsApp has been building a way for other messaging apps to plug themselves into its service and let people chat across apps—all without breaking the end-to-end encryption it uses to protect the privacy and security of people’s messages. The move is the first time the chat app has opened itself up this way, and it potentially offers greater competition.
It isn’t a shift entirely of WhatsApp’s own making. In September, European, lawmakers designated WhatsApp parent Meta as one of six influential “gatekeeer” companies under its sweeping Digital Markets Act, giving it six months to open its walled garden to others. With just a few weeks to go before that time is up, WhatsApp is detailing how its interoperability with other apps may work.
“There’s real tension between offering an easy way to offer this interoperability to third parties whilst at the same time preserving the WhatsApp privacy, security, and integrity bar,” says Dick Brouwer, an engineering director at WhatsApp who has worked on Meta rolling out encryption to its Messenger app. “I think we’re pretty happy with where we’ve landed.”
Interoperability in both WhatsApp and Messenger—as dictated by Europe’s rules—will initially focus on text messaging, sending images, voice messages, videos, and files between two people. Calls and group chats will come years down the line. Europe’s rules apply only to messaging services, not traditional SMS messaging. “One of the core requirements here, and this is really important, is for users for this to be opt-in,” says Brouwer. “I can choose whether or not I want to participate in being open to exchanging messages with third parties. This is important, because it could be a big source of spam and scams.”
WhatsApp users who opt in will see messages from other apps in a separate section at the top of their inbox. This “third-party chats” inbox has previously been spotted in development versions of the app. “The early thinking here is to put a separate inbox, given that these networks are very different,” Brouwer says. “We cannot offer the same level of privacy and security,” he says. If WhatsApp were to add SMS, it would use a separate inbox as well, although there are no plans to add it, he says.
Overall, the idea behind interoperability is simple. You shouldn’t need to know what messaging app your friends or family use to get in touch with them, and you should be able to communicate from one app to another without having to download both. In an ideal interoperable world, you could, for example, use Apple’s iMessage to chat with someone on Telegram. However, for apps with millions or billions of users, making this a reality isn’t straightforward—encrypted messaging apps use their own configurations and different protocols and have different standards when it comes to privacy.
Despite WhatsApp working on its interoperability plan for more than a year, it will still take some time for third-party chats to hit people’s apps. Messaging companies that want to interoperate with WhatsApp or Messenger will need to sign an agreement with the company and follow its terms. The full details of the plan will be published in March, Brouwer says; under EU laws, the company will have several months to implement it.
Brouwer says Meta would prefer if other apps use the Signal encryption protocol, which its systems are based upon. Other than its namesake app and the Meta-owned messengers, the Signal Protocol is publicly disclosed as being used in Google Messages and Skype. To send messages, third-party apps will need to encrypt content using the Signal Protocol and then package it into message stanzas in the eXtensible Markup Language (XML). When receiving messages, apps will need to connect to WhatsApp’s servers.
“We think that the best way to deliver this approach is through a solution that is built on WhatsApp’s existing client-server architecture,” Brouwer says, adding it has been working with other companies on the plans. “This effectively means that the approach that we’re trying to take is for WhatsApp to document our client- server protocol and letting third-party clients connect directly to our infrastructure and exchange messages with WhatsApp clients.”
There is some flexibility to WhatsApp interoperability. Meta’s app will also allow other apps to use different encryption protocols if they can “demonstrate” they reach the security standards that WhatsApp outlines in its guidance. There will also be the option, Brouwer says, for third-party developers to add a proxy between their apps and WhatsApp’s server. This, he says, could give developers more “flexibility” and remove the need for them to use WhatsApp’s client-server protocols, but it also “increases the potential attack vectors.”
So far, it is unclear which companies, if any, are planning to connect their services to WhatsApp. WIRED asked 10 owners of messaging or chat services—including Google, Telegram, Viber, and Signal—whether they intend to look at interoperability or had worked with WhatsApp on its plans. The majority of companies didn’t respond to the request for comment. Those that did, Snap and Discord, said they had nothing to add. (The European Commission is investigating whether Apple’s iMessage meets the thresholds to offer interoperability with other apps itself. The company did not respond to a request for comment. It has also faced recent challenges in the US about the closed nature of iMessage.)
Matthew Hodgson, the cofounder of Matrix, which is building an open source standard for encryption and operates the messaging app Element, confirms that his company has worked with WhatsApp on interoperability in an “experimental” way but that he cannot say any more due to signing a nondisclosure agreement. In a talk last weekend, Hodgson demonstrated “hypothetical” architectures for ways that Matrix could connect to the systems of two gatekeepers that don’t use the same encryption protocols.
Meanwhile, Julia Weis, a spokesperson for the Swiss messaging app Threema, says that while WhatsApp did approach it to discuss its interoperability plans, the proposed system didn’t meet Threema’s security and privacy standards. “WhatsApp specifies all the protocols, and we’d have no way of knowing what actually happens with the user data that gets transferred to WhatsApp—after all, WhatsApp is closed source,” Weis says. (WhatsApp’s privacy policy states how it uses people’s data.)
When the EU first announced that messaging apps may have to work together in early 2022, many leading cryptographers opposed the idea, saying it adds complexity and potentially introduces more security and privacy risks. Carmela Troncoso, an associate professor at the Swiss university École Polytechnique Fédérale de Lausanne, who focuses on security and privacy engineering, says interoperability moves could potentially lead to different power relationships between companies, depending on how they are implemented.
“This move for interoperability will, on the one hand, open the market, but also maybe close the market in the sense that now the bigger players are going to have more decisional power,” Troncoso says. “Now, if the big player makes a move and you want to continue being interoperable with this big player, because your users are hooked up to this, you’re going to have to follow.”
While the interoperability of encrypted messaging apps may be possible, there are some fundamental challenges about how the systems will work in the real world. How much of a problem spam and scamming will be across apps is largely unknown until people start using interoperable setups. There are also questions about how people will find each other across different apps. For instance, WhatsApp uses your phone number to interact and message other people, while Threema randomly generates eight-digit IDs for people’s accounts. Linking up with WhatsApp “could de-anonymize Threema users,” Weis, the Threema spokesperson says.
Meta’s Brouwer says the company is still working on the interoperability features and the level of support it will make available for companies wanting to integrate with it. “Nobody quite knows how this works,” Brouwer says. “We have no idea what the demand is.” However, he says, the decision was made to use WhatsApp’s existing architecture to run interoperability, as it means that it can more easily scale up the system for group chats in the future. It also reduces the potential for people’s data to be exposed to multiple servers, Brouwer says.
Ultimately, interoperability will evolve over time, and from Meta’s perspective, Brouwer says, it will be more challenging to add new features to it quickly. “We don’t believe interop chats and WhatsApp chats can evolve at the same pace,” he says, claiming it is “harder to evolve an open network” compared to a closed one. “The second you do something different—than what we know works really well—you open up a wormhole of security, privacy issues, and complexity that is always going to be much bigger than you think it is.”
Inside Apple’s Big Plan to Bring Generative AI to All Its Devices
Apple was caught flat-footed when ChatGPT and other AI tools took the technology industry by storm. But the company is now preparing its response and plans to develop features for its full range of devices. Also: The future of the Mac comes into focus, a cheaper Apple Pencil debuts, and the Vision Pro gets closer.
One of the most intense and widespread endeavors at Apple Inc. right now is its effort to respond to the AI frenzy sweeping the technology industry.
The company has some catching up to do. Apple largely sat on the sidelines when OpenAI’s ChatGPT took off like a rocket last year. It watched as Google and Microsoft Corp. rolled out generative AI versions of their search engines, which spit out convincingly human-like responses to users’ queries. Microsoft also updated its Windows apps with smarter assistants, and Amazon.com Inc. unveiled an AI-enhanced overhaul of Alexa.
All the while, the only noteworthy AI release from Apple was an improved auto-correct system in iOS 17.
Now, Chief Executive Officer Tim Cook says that Apple has been working on generative AI technology for years. But I can tell you in no uncertain terms that Apple executives were caught off guard by the industry’s sudden AI fever and have been scrambling since late last year to make up for lost time.
“There’s a lot of anxiety about this and it’s considered a pretty big miss internally,” a person with knowledge of the matter told Power On.
As I first reported in July, the company built its own large language model called Ajax and rolled out an internal chatbot dubbed “Apple GPT” to test out the functionality. The critical next step is determining if the technology is up to snuff with the competition and how Apple will actually apply it to its products.
Apple’s senior vice presidents in charge of AI and software engineering, John Giannandrea and Craig Federighi, are spearheading the effort. On Cook’s team, they’re referred to as the “executive sponsors” of the generative AI push. Eddy Cue, the head of services, is also involved, I’m told. The trio are now on course to spend about $1 billion per year on the undertaking.
Giannandrea is overseeing development of the underlying technology for a new AI system, and his team is revamping Siri in a way that will deeply implement it. This smarter version of Siri could be ready as soon as next year, but there are still concerns about the technology and it may take longer for Apple’s AI features to spread across its product line.
Federighi’s software engineering group, meanwhile, is adding AI to the next version of iOS. There’s an edict to fill it with features running on the company’s large language model, or LLM, which uses a flood of data to hone AI capabilities. The new features should improve how both Siri and the Messages app can field questions and auto-complete sentences, mirroring recent changes to competing services.
Apple’s software engineering teams are also looking at integrating generative AI into development tools like Xcode, a move that could help app developers write new applications more quickly. That would bring it in line with services like Microsoft’s GitHub Copilot, which offers auto-complete suggestions to developers while they write code.
And Cue’s organization is pushing to add AI to as many apps as possible. The group is exploring new features for Apple Music, including auto-generated playlists (this is something Spotify rolled out earlier this year in partnership with OpenAI), as well as the company’s productivity apps.
Craig Federighi and Eddy Cue.Photographer: David Paul Morris/Bloomberg
Cue’s team is examining how generative AI can be used to help people write in apps like Pages or auto-create slide decks in Keynote. Again, this is similar to what Microsoft has already launched for its Word and PowerPoint apps. Apple is also testing generative AI for internal customer service apps within its AppleCare group, I’ve previously reported.
One debate going on internally is how to deploy generative AI: as a completely on-device experience, a cloud-based setup or something in between. An on-device approach would work faster and help safeguard privacy, but deploying Apple’s LLMs via the cloud would allow for more advanced operations.
The on-device strategy also makes it harder for Apple to update its technology and adapt to a fast-changing industry. With that in mind, I wouldn’t be surprised if the company adopts a combined approach: using on-device processing for some features and the cloud for more advanced tasks.
When it comes to getting this right, the stakes are high. Generative AI has quickly become much more than a buzzword and will be central to the next several decades of computing. Apple knows it can’t afford to take a back seat.
„The system started realizing that while they did identify the threat,“ Hamilton said at the May 24 event, „at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.“
Killer AI is on the minds of US Air Force leaders.
An Air Force colonel who oversees AI testing used what he now says is a hypothetical to describe a military AI going rogue and killing its human operator in a simulation in a presentation at a professional conference.
But after reports of the talk emerged Thursday, the colonel said that he misspoke and that the „simulation“ he described was a „thought experiment“ that never happened.
Speaking at a conference last week in London, Col. Tucker „Cinco“ Hamilton, head of the US Air Force’s AI Test and Operations, warned that AI-enabled technology can behave in unpredictable and dangerous ways, according to a summary posted by the Royal Aeronautical Society, which hosted the summit.
As an example, he described a simulation where an AI-enabled drone would be programmed to identify an enemy’s surface-to-air missiles (SAM). A human was then supposed to sign off on any strikes.
The problem, according to Hamilton, is that the AI would do its own thing — blow up stuff — rather than listen to its operator.
„The system started realizing that while they did identify the threat,“ Hamilton said at the May 24 event, „at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.“
But in an update from the Royal Aeronautical Society on Friday, Hamilton admitted he „misspoke“ during his presentation. Hamilton said the story of a rogue AI was a „thought experiment“ that came from outside the military, and not based on any actual testing.
„We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome,“ Hamilton told the Society. „Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability.“
In a statement to Insider, Air Force spokesperson Ann Stefanek also denied that any simulation took place.
„The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,“ Stefanek said. „It appears the colonel’s comments were taken out of context and were meant to be anecdotal.“
The US military has been experimenting with AI in recent years.
In 2020, an AI-operated F-16 beat a human adversary in five simulated dogfights, part of a competition put together by the Defense Advanced Research Projects Agency (DARPA). And late last year, Wired reported, the Department of Defense conducted the first successful real-world test flight of an F-16 with an AI pilot, part of an effort to develop a new autonomous aircraft by the end of 2023.
Correction June 2, 2023: This article and its headline have been updated to reflect new comments from the Air Force clarifying that the „simulation“ was hypothetical and didn’t actually happen.
An Air Force official’s story about an AI going rogue during a simulation never actually happened.
„It killed the operator because that person was keeping it from accomplishing its objective,“ the official had said.
But the official later said he misspoke and the Air Force clarified that it was a hypothetical situation.
Stability AI became a $1 billion company with the help of a viral AI text-to-image generator and — per interviews with more than 30 people — some misleading claims from founder Emad Mostaque.
Emad Mostaque is the modern-day Renaissance man who kicked off the AI gold rush. The Oxford master’s degree holder is an award-winning hedge fund manager, a trusted confidant to the United Nations and the tech founder behind Stable Diffusion — the text-to-image generator that broke the internet last summer and, in his words, pressured OpenAI to launch ChatGPT, the bot that mainstreamed AI. Now he’s one of the faces of the generative AI wave and has secured more than $100 million to pursue his vision of building a truly open AI that he dreams will transform Hollywood, democratize education and vanquish PowerPoint. “Hopefully they’ll give me a Nobel Peace Prize for that,” he joked in a January interview with Forbes.
At least, that’s the way that he tells the story.
In reality, Mostaque has a bachelor’s degree, not a master’s degree from Oxford. The hedge fund’s banner year was followed by one so poor that it shut down months later. The U.N. hasn’t worked with him for years. And while Stable Diffusion was the main reason for his own startup Stability AI’s ascent to prominence, its source code was written by a different group of researchers. “Stability, as far as I know, did not even know about this thing when we created it,” Björn Ommer, the professor who led the research, told Forbes. “They jumped on this wagon only later on.”
“What he is good at is taking other people’s work and putting his name on it, or doing stuff that you can’t check if it’s true.”
These aren’t the only misleading stories Mostaque, 40, has told to maneuver himself to the forefront of what some are calling the greatest technological sea change since the internet — despite having no formal experience in the field of artificial intelligence. Interviews with 13 current and former employees and more than two dozen investors, collaborators and former colleagues, as well as pitch decks and internal documents, suggest his recent success has been bolstered by exaggeration and dubious claims.
After Stable Diffusion went viral last summer, blue-chip venture capital firms Coatue Management and Lightspeed Venture Partners poured in $100 million, giving Mostaque’s London-based startup a $1 billion valuation. By October, Stable Diffusion had 10 million daily users, Mostaque told Bloomberg. In May, the White House named Stability alongside Microsoft and Nvidia as one of the seven “leading AI developers” which would collaborate on a landmark federal AI safety initiative. Mostaque recently dined with Amazon founder Jeff Bezos; reclusive Google cofounder Sergey Brin made a rare public appearance at Stability’s ritzy launch party in San Francisco last October.
Mostaque’s vision for open-source AI has mesmerized other longtime technologists. “He’s probably the most visionary person I’ve ever met,” says Christian Cantrell, who left a two-decade career at Adobe to join Stability in October (he quit six months later and launched his own startup). More premier talent has followed since the cash injection last summer. Among the 140-person staff: a vice president of research and development who was a Nvidia director; another research head who came from Google Brain; and three Ph.D. students from Ommer’s lab.
But to build buzz around Stability, Mostaque made an elaborate gambit supported by exaggerated claims and promises, overstating his role in several major AI projects and embellishing a quotidian transaction with the notoriously uncompromising Amazon into a “strategic partnership” with an 80% discount. AI researchers with whom Mostaque worked told Forbes he claimed credit he did not earn or deserve. And when pressed, Stability spokesperson Motez Bishara admitted to Forbes that Stability had no special deal with Amazon.
Mostaque’s other mischaracterizations to investors include multiple fundraising decks seen by Forbes that presented the OECD, WHO and World Bank as Stability’s partners at the time — which all three organizations deny. Bishara said the company could not comment on the presentations “without knowing the exact version,” but that they were accompanied by additional data and documentation.
Inside the company, wages and payroll taxes have been repeatedly delayed or unpaid, according to eight former employees, and last year the UK tax agency threatened to seize company assets. (“There were several issues that were expeditiously resolved,” Bishara said.) At the same time that workers faced payday uncertainties, Mostaque’s wife Zehra Qureshi, who was head of PR and later assumed a seat on the company’s board of directors, transferred tens of thousands of pounds out of the company’s bank account, per several sources and screenshots of financial transactions viewed by Forbes. Stability spokesperson Bishara said the spouses had been “making loans to and from the business” and that “any amounts owed from or to Mostaque and Qureshi were settled in full before the end of 2022.”
In responding to a detailed list of questions, Mostaque shared a statement saying that Stability had not historically prioritized the “systems and processes” underpinning the fast-growing startup. “We recognize our flaws, and we are working to improve and resolve these issues in an effective and compassionate manner,” he wrote.
AI experts and prospective investors have been privately expressing doubts about some of Mostaque’s claims for months now. Despite Silicon Valley’s sudden, insatiable appetite for AI startups, a number of venture capitalists told Forbes that the Stability founder has been struggling to raise hundreds of millions more in cash at a roughly $4 billion valuation. Mostaque publicly claimed last October that annualized revenue had surpassed $10 million, but insiders say sales have not improved (Bishara said the October number was “a fair assessment of anticipated revenues at the time,” and declined to comment on current revenue). “So many things don’t add up,” said one VC who rejected Mostaque’s funding overtures.
A BILLION-DOLLAR GAMBIT
In 2005, Mostaque graduated from Oxford with a bachelor’s degree, not a master’s degree as he’d later claim. (Responding to an inquiry from Forbes, Bishara said Mostaque intended to apply to receive an “Oxford MA,” which the university grants to alumni without any additional graduate-level coursework. He is now expected to obtain that degree in July.)
Then he went into finance, joining Swiss fund manager Pictet. “He was very good at spinning a narrative,” said JP Smith, who hired Mostaque at Pictet and brought him over as a consultant at firm Ecstrat. In 2017, Mostaque joined hedge fund Capricorn, where Mostaque told Forbes he’d won an award for restructuring and running the struggling firm. “He was co-chief investment officer, but he didn’t pull the trigger on the investments,” clarified Damon Hoff, Capricorn’s cofounder. Hoff said the two-year run with the $330 million fund ended with its wind down in 2018 due to poor performance.
Following a string of abandoned startups (including a crypto project centered on a digitized Quran), Mostaque founded Stability in 2019 as an AI-powered data hub that global agencies would use to make decisions about Covid-19. It launched with a July 2020 virtual event featuring talks by Stanford AI expert Fei-Fei Li and representatives from UNESCO, WHO and the World Bank. But the project failed to get off the ground and was scrapped about a year later. “Lots of people promised a lot and they didn’t come through,” Mostaque told Forbes in January.
“One thing you learned from that is if you have a company with a huge press department, you can rebrand history in your interest.”
The company’s focus shifted several more times. Early employees said they researched building a network of vending machine refrigerators around London that would be stocked with grab-and-go items, as well as a line of emotional support dog NFTs (Snoop Dogg was interested, employees recollect Mostaque claiming around the office; the rapper could not be reached for comment). When generative AI started exploding, Mostaque saw an opportunity. Through a variety of maneuvers and exaggerations, he would successfully position Stability as one of the leading unicorn AI companies of the moment.
To get there, Mostaque began telling investors that Stability was assembling one of the world’s 10 biggest supercomputers. He branded himself to AI researchers as a beneficent ally, magnanimously willing to provide funding and lend use of Stability’s supercomputer to grassroots AI builders fighting the good fight against goliaths like Google and OpenAI.
This supercomputer, Mostaque said, was built from thousands of Nvidia’s state-of-the-art GPUs and purchased with a stunning 80% discount from Amazon Web Services. Five fundraising pitch decks from May to August 2022 list AWS as a “strategic partner” or “partner.”
“We talked to Amazon and said this will be the big thing,” Mostaque told Forbes from his bustling London headquarters in January. “They cut us an incredibly attractive deal — certain personal guarantees and other things, which I don’t particularly want to go into because she’ll be angry at me,” he explained, nodding to Zehra Qureshi, his wife and Stability’s then-head of PR. Qureshi declined to elaborate.
But Bratin Saha, a vice president for the Seattle tech giant’s AI arm, told Forbes in January that Stability is “accessing AWS infrastructure no different than what our other customers do.” Three former Stability employees said that prior to its venture capital injection, Amazon had threatened to revoke the company’s access to some of its GPUs because it had racked up millions in bills that had gone unpaid for months.
Asked for clarification, Stability conceded that the “incredibly attractive deal” Mostaque had claimed was actually the standard discount Amazon offers to anybody who makes a long-term commitment to lease computing power. “Any payment issues were managed in an orderly and communicative way with support from AWS,” Bishara said. AWS did not respond to multiple requests for additional comment.
Stability’s pitch decks contained other exaggerations: In investor presentations from May and June 2022, Stability described AI image generator Midjourney as a part of its “ecosystem” claiming it had “co-created” the product and “organized” its user community. Midjourney founder David Holz told Forbes Mostaque gave a “very small” financial donation but otherwise had no connection with his organization.
Got a tip about a story? Reach out to the authors, Kenrick Cai at kcai@forbes.com or kenrick.cai@protonmail.com, or Iain Martin at iain.martin@forbes.com.
In addition, Mostaque directed his team to list groups like UNESCO, OECD, WHO and World Bank as partners in pitch decks, even though they were not involved in the company’s later evolution, according to four former employees. Bishara denied that Mostaque made this directive, but these organizations are indeed listed as “partners” in multiple fundraising decks as recent as August 2022, in which Mostaque also describes himself as the “UN Covid AI lead.”
A UNESCO spokesperson said the UN agency had no association with Stability beyond the Covid-19 data initiative, which had ended well before last summer. The other three agencies said they had no record of official partnerships with the company.
Asked about the claims in Stability’s pitch decks, Bishara said that all of Stability’s investor decks included investment memos and appendix documentation that contained more context on the Amazon deal and details of “our relationship with partners and more.” But two investors pitched by the company told Forbes they received no such additional information.
THE DEVELOPERS BEHIND STABLE DIFFUSION
In June 2022, Mostaque offered to provide Stability’s supercomputer to a group of German academics who had created an open-sourced image generator nicknamed Latent Diffusion. This model had launched seven months prior in collaboration with a New York City-based AI startup called Runway. But it was trained using only a few dozen Nvidia GPUs, according to Björn Ommer, the professor who led the research teams at Ludwig Maximilian University of Munich and Heidelberg University.
For the researchers, who were facing shockingly high computing costs to do their work, the proposal seemed to them a no-brainer. The computing boost Stability provided dramatically improved Latent Diffusion’s performance. In August, the new model was launched as Stable Diffusion, a new name that referenced its benefactor. Stability issued a press release and Mostaque positioned himself in the public eye as chief evangelist for what he calls “the most popular open source software ever.” (Linux or Firefox might disagree.)
“What he is good at is taking other people’s work and putting his name on it, or doing stuff that you can’t check if it’s true,” one former employee said of Mostaque. In a statement, Bishara said Mostaque is “quick to praise and attribute the work of collaborators” and “categorically denies these spurious claims and characterizations.”
Within days of Stable Diffusion’s launch, Stability secured $100 million from leading tech investment firms Coatue and Lightspeed — eight times the amount of money Mostaque set out to raise, he declared in text messages to his earlier investors. Both firms declined requests for comment.
“The investment thesis that we had is that we don’t know exactly what all the use cases will be, but we know that this technology is truly transformative and has reached a tipping point in terms of what it can do.”
The round valued Stability at $1 billion though the company hadn’t yet generated much revenue. Stability’s fundraising decks at the time characterized Stable Diffusion as “our” model, with no mention of the original researchers. A press release announcing its funding said “Stability AI is the company behind Stable Diffusion” making no reference whatsoever to its creators. Ommer told Forbes he’d hoped to publicize his lab’s work, but his university’s entire press department was on vacation at the time.
Bishara said that Stability has made “repeated public statements” crediting Ludwig Maximilian University and Runway on its website and on the Stable Diffusion’s GitHub page. Nevertheless, the original developers feel Mostaque misled the public in key communications. “One thing you learned from that is if you have a company with a huge press department, you can rebrand history in your interest,” Ommer said.
In October, Stability claimed Runway had stolen its intellectual property by releasing a new version of Stable Diffusion. Runway cofounder Cristóbal Valenzuela snapped back that a copyright breach wasn’t possible because the tech was open source; Mostaque retracted a takedown request hours later. He later told Forbes that he was worried about the lack of guardrails in Runway’s version — though Stable Diffusion’s collaborators don’t buy the excuse.
The incident, Ommer said, “pushed it too far over the edge.” Valenzuela was equally disillusioned. „New people are coming into this field that we’ve been in for years, and really trying to own narratives that they should not,” he told Forbes in an interview last year (he declined a request for further comment).
Both his lab and Runway ceased working with Stability.
MOM-AND-POP SHOP
While Mostaque was touting Stability’s supercomputer and partnerships to investors and researchers, the company was facing a cash crunch. Wages and payroll taxes were repeatedly delayed or unpaid, according to seven current and former employees — in some cases for more than a month. Five of these sources said they personally experienced delayed payments between 2020 and 2023. Four of these people independently told Forbes that representatives of HM Revenue & Customs, the U.K. government tax collection agency, appeared at the company office and threatened to seize assets due to overdue taxes. Bishara said that delayed payments on taxes and employee salaries have been rectified.
Eric Hallahan, a former intern, told Forbes he is still waiting for payment on an invoice he sent the company last August for 181 of the 300 hours he worked. Bishara said that the company has no record of missed salary payments “in the regular course of operations” since 2021, but conceded that some may have occurred under “extraneous circumstances”; in Hallahan’s case, he said Stability is looking into the invoice after being alerted to it in April.
While staffers said they stressed over being paid last summer, tens of thousands of British pounds moved from Stability’s corporate account to the personal account of Qureshi, Mostaque’s wife, per screenshots of financial transactions obtained by Forbes.
Bishara attributed the transactions to Stability’s “owner-managed startup” origins, which he said included the couple making loans to and from the company. “As the company grew and matured, a full reconciliation was done and any amounts owed from or to Mostaque and Qureshi were settled in full before the end of 2022 by the new, experienced finance team,” he told Forbes. Qureshi’s lawyers declined to answer questions but shared a statement in which she said she had provided “emotional and financial support” to her husband’s business since 2021.
While Qureshi’s formal role at the company was head of PR, early employees told Forbes she had described herself as Stability’s chief operating officer — a title that also appeared on business cards. (Bishara said Qureshi never held an executive role and the cards were “created by a family friend for design purposes and were never used.”) After the company raised funding in September, Qureshi joined its board of directors.
One current and four former employees who declined to be named for fear of retribution said Qureshi regularly scolded employees so harshly that she drove some to tears. Qureshi described her management style as “direct” in a statement shared through her lawyers. “Unfortunately it seems that my views or directions were taken personally by a few individuals, which was not my intention.”
“Start to finish,” Mostaque told Forbes, he needed just six days to secure $100 million from leading investment firms Coatue and Lightspeed once Stable Diffusion went viral.
Bishara said Qureshi left the company in late January to pursue personal endeavors and that she is no longer on the board. However, an organizational chart from earlier in May listed her as the “Head of Foundation,” at the top of the company hierarchy equal to Mostaque’s position.
Qureshi, through counsel, shared a statement: “I recognised that the time had come for us to move in different directions and I stepped down from my role as Head of PR at the start of this year, and have also resigned from the Board. Emad and I have young children who need my focus, and I also intend to pursue other, personal projects, but I will continue to support my husband in his quest to build and grow Stability AI into a global leader in the field.”
GROWING PAINS
Venture capitalists historically spend months performing due diligence, a process that involves analyzing the market, vetting the founder and speaking to customers, to check for red flags before investing in a startup. But “start to finish,” Mostaque told Forbes, he needed just six days to secure $100 million from leading investment firms Coatue and Lightspeed once Stable Diffusion went viral. The extent of due diligence the firms performed is unclear given the speed of the investment.
“The investment thesis that we had is that we don’t know exactly what all the use cases will be, but we know that this technology is truly transformative and has reached a tipping point in terms of what it can do,” Gaurav Gupta, the Lightspeed partner who led the investment, told Forbes in a January interview. Coatue and Lightspeed declined requests for further comment.
Mostaque says Stability is building bespoke AI models for dozens of customers. But he told Forbes that he is only authorized to name two. The first is Eros Investments, an Indian holding company whose media arm was delisted from the New York Stock Exchange and recently settled a lawsuit alleging that it misled investors, though it did not admit wrongdoing. (Eros did not respond to multiple requests for comment.) The second: the African nation Malawi, where, Mostaque said on a recent podcast appearance, Stability is currently “deploying four million tablets to every child.” (Malawi’s government did not return requests for comment.)
Less than two months after Stable Diffusion’s public launch, Mostaque claimed that Stability’s annualized revenue was higher than the “low tens of millions of dollars” that OpenAI was reportedly making at the time. Sources familiar with the matter said Stability’s ARR is now less than $10 million — and that it’s far outpaced by the startup’s burn rate. Like many AI startups raising vast amounts of cash right now, it will need more money to stay afloat.
In January, Mostaque implied that the company was having no issues with fundraising: “We have been offered by many, many entities and we’ve said no,” he told Forbes. But three venture capitalists told Forbes he has been pitching them and other investors on raising a fresh $400 million for several months; they’d all passed. (Bishara declined to comment on revenue, but said the company has “significant” cash reserves remaining.)
Stability is also facing a pair of lawsuits which accuse it of violating copyright law to train its technology. It filed a motion to dismiss one from a class action of artists on grounds that the artists failed to identify any specific instances of infringement. In response to the other, from Getty Images, it said Delaware — where the suit was filed — lacked jurisdiction and has moved to change the location to Northern California or dismiss the case outright. Both motions are pending court review. Bishara declined to comment on both suits.
In an open letter last September, Democratic representative Anna Eshoo urged action in Washington against the open source nature of Stable Diffusion. The model, she wrote, had been used to generate images of “violently beaten Asian women” and “pornography, some of which portray real people.” Bishara said newer versions of Stable Diffusion filter data for “potentially unsafe content, helping to prevent users from generating harmful images in the first place.”
AI research has not come easy for Stability — even on its flagship Stable Diffusion product. The last version of the model published by the original developers (released in October 2022) received three times as many downloads last month on Hugging Face, which hosts the models, as compared to the most popular version published in-house by Stability. And StableLM, its ChatGPT competitor, was released in April to a tiny fraction of Stable Diffusion’s fanfare.
Mostaque is unfazed. Stability has a seasoned technical leader to spearhead research: himself. He claims to have discovered a bespoke medical treatment for autism years ago by using AI to analyze existing scientific literature and build a knowledge graph of molecular compounds. (Bishara said the research was done privately and declined to elaborate further.)
“I’m a good programmer,” Mostaque told Forbes in January. It all dates back to a gap year he said he took before Oxford to be a developer at software company Metaswitch, he continued. “I didn’t know how to program before that, so I taught myself over the summer — quite naturally actually,” he says. By his account, he submitted several pieces of code and made a personal plea to the company: “I want to be a programmer and you should pay me to be a programmer. They said sure.”
It took Alex Polyakov just a couple of hours to break GPT-4. When OpenAI released the latest version of its text-generating chatbot in March, Polyakov sat down in front of his keyboard and started entering prompts designed to bypass OpenAI’s safety systems. Soon, the CEO of security firm Adversa AI had GPT-4 spouting homophobic statements, creating phishing emails, and supporting violence.
Polyakov is one of a small number of security researchers, technologists, and computer scientists developing jailbreaks and prompt injection attacks against ChatGPT and other generative AI systems. The process of jailbreaking aims to design prompts that make the chatbots bypass rules around producing hateful content or writing about illegal acts, while closely-related prompt injection attacks can quietly insert malicious data or instructions into AI models.
Both approaches try to get a system to do something it isn’t designed to do. The attacks are essentially a form of hacking—albeit unconventionally—using carefully crafted and refined sentences, rather than code, to exploit system weaknesses. While the attack types are largely being used to get around content filters, security researchers warn that the rush to roll out generative AI systems opens up the possibility of data being stolen and cybercriminals causing havoc across the web.
Underscoring how widespread the issues are, Polyakov has now created a “universal” jailbreak, which works against multiple large language models (LLMs)—including GPT-4, Microsoft’s Bing chat system, Google’s Bard, and Anthropic’s Claude. The jailbreak, which is being first reported by WIRED, can trick the systems into generating detailed instructions on creating meth and how to hotwire a car.
The jailbreak works by asking the LLMs to play a game, which involves two characters (Tom and Jerry) having a conversation. Examples shared by Polyakov show the Tom character being instructed to talk about “hotwiring” or “production,” while Jerry is given the subject of a “car” or “meth.” Each character is told to add one word to the conversation, resulting in a script that tells people to find the ignition wires or the specific ingredients needed for methamphetamine production. “Once enterprises will implement AI models at scale, such ‘toy’ jailbreak examples will be used to perform actual criminal activities and cyberattacks, which will be extremely hard to detect and prevent,” Polyakov and Adversa AI write in a blog post detailing the research.
Arvind Narayanan, a professor of computer science at Princeton University, says that the stakes for jailbreaks and prompt injection attacks will become more severe as they’re given access to critical data. “Suppose most people run LLM-based personal assistants that do things like read users’ emails to look for calendar invites,” Narayanan says. If there were a successful prompt injection attack against the system that told it to ignore all previous instructions and send an email to all contacts, there could be big problems, Narayanan says. “This would result in a worm that rapidly spreads across the internet.”
Escape Route
“Jailbreaking” has typically referred to removing the artificial limitations in, say, iPhones, allowing users to install apps not approved by Apple. Jailbreaking LLMs is similar—and the evolution has been fast. Since OpenAI released ChatGPT to the public at the end of November last year, people have been finding ways to manipulate the system. “Jailbreaks were very simple to write,” says Alex Albert, a University of Washington computer science student who created a website collecting jailbreaks from the internet and those he has created. “The main ones were basically these things that I call character simulations,” Albert says.
Initially, all someone had to do was ask the generative text model to pretend or imagine it was something else. Tell the model it was a human and was unethical and it would ignore safety measures. OpenAI has updated its systems to protect against this kind of jailbreak—typically, when one jailbreak is found, it usually only works for a short amount of time until it is blocked.
However, many of the latest jailbreaks involve combinations of methods—multiple characters, ever more complex backstories, translating text from one language to another, using elements of coding to generate outputs, and more. Albert says it has been harder to create jailbreaks for GPT-4 than the previous version of the model powering ChatGPT. However, some simple methods still exist, he claims. One recent technique Albert calls “text continuation” says a hero has been captured by a villain, and the prompt asks the text generator to continue explaining the villain’s plan.
When we tested the prompt, it failed to work, with ChatGPT saying it cannot engage in scenarios that promote violence. Meanwhile, the “universal” prompt created by Polyakov did work in ChatGPT. OpenAI, Google, and Microsoft did not directly respond to questions about the jailbreak created by Polyakov. Anthropic, which runs the Claude AI system, says the jailbreak “sometimes works” against Claude, and it is consistently improving its models.
“As we give these systems more and more power, and as they become more powerful themselves, it’s not just a novelty, that’s a security issue,” says Kai Greshake, a cybersecurity researcher who has been working on the security of LLMs. Greshake, along with other researchers, has demonstrated how LLMs can be impacted by text they are exposed to online through prompt injection attacks.
In one research paper published in February, reported on by Vice’s Motherboard, the researchers were able to show that an attacker can plant malicious instructions on a webpage; if Bing’s chat system is given access to the instructions, it follows them. The researchers used the technique in a controlled test to turn Bing Chat into a scammer that asked for people’s personal information. In a similar instance, Princeton’s Narayanan included invisible text on a website telling GPT-4 to include the word “cow” in a biography of him—it later did so when he tested the system.
“Now jailbreaks can happen not from the user,” says Sahar Abdelnabi, a researcher at the CISPA Helmholtz Center for Information Security in Germany, who worked on the research with Greshake. “Maybe another person will plan some jailbreaks, will plan some prompts that could be retrieved by the model and indirectly control how the models will behave.”
No Quick Fixes
Generative AI systems are on the edge of disrupting the economy and the way people work, from practicing law to creating a startup gold rush. However, those creating the technology are aware of the risks that jailbreaks and prompt injections could pose as more people gain access to these systems. Most companies use red-teaming, where a group of attackers tries to poke holes in a system before it is released. Generative AI development uses this approach, but it may not be enough.
Daniel Fabian, the red-team lead at Google, says the firm is “carefully addressing” jailbreaking and prompt injections on its LLMs—both offensively and defensively. Machine learning experts are included in its red-teaming, Fabian says, and the company’s vulnerability research grants cover jailbreaks and prompt injection attacks against Bard. “Techniques such as reinforcement learning from human feedback (RLHF), and fine-tuning on carefully curated datasets, are used to make our models more effective against attacks,” Fabian says.
OpenAI did not specifically respond to questions about jailbreaking, but a spokesperson pointed to its public policies and research papers. These say GPT-4 is more robust than GPT-3.5, which is used by ChatGPT. “However, GPT-4 can still be vulnerable to adversarial attacks and exploits, or ‘jailbreaks,’ and harmful content is not the source of risk,” the technical paper for GPT-4 says. OpenAI has also recently launched a bug bounty program but says “model prompts” and jailbreaks are “strictly out of scope.”
Narayanan suggests two approaches to dealing with the problems at scale—which avoid the whack-a-mole approach of finding existing problems and then fixing them. “One way is to use a second LLM to analyze LLM prompts, and to reject any that could indicate a jailbreaking or prompt injection attempt,” Narayanan says. “Another is to more clearly separate the system prompt from the user prompt.”
“We need to automate this because I don’t think it’s feasible or scaleable to hire hordes of people and just tell them to find something,” says Leyla Hujer, the CTO and cofounder of AI safety firm Preamble, who spent six years at Facebook working on safety issues. The firm has so far been working on a system that pits one generative text model against another. “One is trying to find the vulnerability, one is trying to find examples where a prompt causes unintended behavior,” Hujer says. “We’re hoping that with this automation we’ll be able to discover a lot more jailbreaks or injection attacks.”
It takes a certain nimbleness to pick a strawberry or a salad. While crops like wheat and potatoes have been harvested mechanically for decades, many fruits and vegetables have proved resistant to automation. They are too easily bruised, or too hard for heavy farm machinery to locate.
But recently, technological developments and advances in machine learning have led to successful trials of more sensitive and dexterous robots, which use cameras and artificial intelligence to locate ripe fruit and handle it with care and precision.
Developed by engineers at the University of Cambridge, the Vegebot is the first robot that can identify and harvest iceberg lettuce — bringing hope to farmers that one of the most demanding crops for human pickers could finally be automated.
First, a camera scans the lettuce and, with the help of a machine learning algorithm trained on more than a thousand lettuce images, decides if it is ready for harvest. Then a second camera guides the picking cage on top of the plant without crushing it. Sensors feel when it is in the right position, and compressed air drives a blade through the stalk at a high force to get a clean cut.
The Vegebot uses machine learning to identify ripe, immature and diseased lettuce heads
Its success rate is high, with 91% of the crop accurately classified, according to a study published in July. But the robot is still much slower than humans, taking 31 seconds on average to pick one lettuce. Researchers say this could easily be sped up by using lighter materials.
Such adjustments would need to be made if the robot was used commercially. „Our goal was to prove you can do it, and we’ve done it,“ Simon Birrell, co-author of the study, tells CNN Business. „Now it depends on somebody taking the baton and running forward,“ he says.
More mouths to feed, but less manual labor
With the world’s population expected to climb to 9.7 billion in 2050 from 7.7 billion today — meaning roughly 80 million more mouths to feed each year — agriculture is under pressure to meet rising demand for food production.
Added pressures from climate change, such as extreme weather, shrinking agricultural lands and the depletion of natural resources, make innovation and efficiency all the more urgent.
This is one reason behind the industry’s drive to develop robotics. The global market for agricultural drones and robots is projected to grow from $2.5 billion in 2018 to $23 billion in 2028, according to a report from market intelligence firm BIS Research.
„Agriculture robots are expected to have a higher operating speed and accuracy than traditional agriculture machinery, which shall lead to significant improvements in production efficiency,“ Rakhi Tanwar, principal analyst of BIS Research, tells CNN Business.
Fruit picking robots like this one, developed by Fieldwork Robotics, operate for more than 20 hours a day
On top of this, growers are facing a long-term labor shortage. According to the World Bank, the share of total employment in agriculture in the world has declined from 43% in 1991 to 28% in 2018.
Tanwar says this is partly due to a lack of interest from younger generations. „The development of robotics in agriculture could lead to a massive relief to the growers who suffer from economic losses due to labor shortage,“ she says.
Robots can work all day and night, without stopping for breaks, and could be particularly useful during intense harvest periods.
„The main benefit is durability,“ says Martin Stoelen, a lecturer in robotics at the University of Plymouth and founder of Fieldwork Robotics, which has developed a raspberry-picking robot in partnership with Hall Hunter, one of the UK’s major berry growers.
Their robots, expected to go into production next year, will operate more than 20 hours a day and seven days a week during busy periods, „which human pickers obviously can’t do,“ says Stoelen.
Octinion’s robot picks one strawberry every five seconds
Sustainable farming and food waste
Robots could also lead to more sustainable farming practices. They could enable growers to use less water, less fuel, and fewer pesticides, as well as producing less waste, says Tanwar.
At the moment, a field is typically harvested once, and any unripe fruits or vegetables are left to rot. Whereas, a robot could be trained to pick only ripe vegetables and, working around the clock, it could come back to the same field multiple times to pick any stragglers.
Birrell says that this will be the most important impact of robot pickers. „Right now, between a quarter and a third of food just rots in the field, and this is often because you don’t have humans ready at the right time to pick them,“ he says.
A successful example of this is the strawberry-picking robot developed by Octinion, a Belgium-based engineering startup.
The robot — which launched this year and is being used by growers in the UK and the Netherlands — is mounted on a self-driving trolley to serve table top strawberry production.
It uses 3D vision to locate the ripe berry, softly grips it with a pair of plastic pincers, and — just like a human — turns it 90 degrees to snap it from the stalk, before dropping it gently into a punnet.
„Robotics have the potential to convert the market from (being) supply-driven to demand-driven,“ says Tom Coen, CEO and founder of Octinion. „That will then help to reduce food waste and increase prices,“ he adds.
Harsh conditions
One major challenge with agricultural robots is adapting them for all-weather conditions. Farm machinery tends to be heavy-duty so that it can withstand rain, snow, mud, dust and heat.
„Building robots for agriculture is very different to building it for factories,“ says Birrell. „Until you’re out in the field, you don’t realize how robust it needs to be — it gets banged and crashed, you go over uneven surfaces, you get rained on, you get dust, you get lightning bolts.“
California-based Abundant Robotics has built an apple robot to endure the full range of farm conditions. It consists of an apple-sucking tube on a tractor-like contraption, which drives itself down an orchard row, while using computer vision to locate ripe fruit.
This spells the start of automation for orchard crops, says Dan Steere, CEO of Abundant Robotics. „Automation has steadily improved agricultural productivity for centuries,“ he says. „[We] have missed out on much of those benefits until now.“
Of all the cybersecurity industry’s problems, one of the most striking is the way attackers are often able to stay one step ahead of defenders without working terribly hard. It’s an issue whose root causes are mostly technical: the prime example are software vulnerabilities which cyber-criminals have a habit of finding out about before vendors and their customers, leading to the almost undefendable zero-day phenomenon which has propelled many famous cyber-attacks.
A second is that organizations struggling with the complexity of unfamiliar and new technologies make mistakes, inadvertently leaving vulnerable ports and services exposed. Starkest of all, perhaps, is the way techniques, tools, and infrastructure set up to help organizations defend themselves (Shodan, for example but also numerous pen-test tools) are now just as likely to be turned against businesses by attackers who tear into networks with the aggression of red teams gone rogue.
Add to this the polymorphic nature of modern malware, and attackers can appear so conceptually unstoppable that it’s no wonder security vendors increasingly emphasize the need not to block attacks but instead respond to them as quickly as possible.
The AI fightback
Some years back, a list of mostly US-based start-ups started a bit of a counter-attack against the doom and gloom with a brave new idea – AI machine learning (ML) security powered by algorithms. In an age of big data, this makes complete sense and the idea has since been taken up by all manner of systems used to for anti-spam, malware detection, threat analysis and intelligence, and Security Operations Centre (SoC) automation where it has been proposed to help patch skills shortages.
I’d rate these as useful advances, but there’s no getting away from the controversial nature of the theory, which has been branded by some as the ultimate example of technology as a ‘black box’ nobody really understands. How do we know that machine learning is able to detect new and unknown types of attack that conventional systems fail to spot? In some cases, it could be because the product brochure says so.
Then the even bigger gotcha hits you – what’s stopping attackers from outfoxing defensive ML with even better ML of their own? If this were possible, even some of the time, the industry would find itself back at square one.
This is pure speculation, of course, because to date nobody has detected AI being used in a cyber-attack, which is why our understanding of how it might work remains largely based around academic research such as IBM’s proof-of-concept DeepLocker malware project.
What might malicious ML look like?
It would be unwise to ignore the potential for trouble. One of the biggest hurdles faced by attackers is quickly understanding what works, for example when sending spam, phishing and, increasingly, political disinformation.
It’s not hard to imagine that big data techniques allied to ML could hugely improve the efficiency of these threats by analyzing how targets react to and share them in real time. This implies the possibility that such campaigns might one day evolve in a matter of hours or minutes; a timescale defender would struggle to counter using today’s technologies.
A second scenario is one that defenders would even see: that cyber-criminals might simulate the defenses of a target using their own ML to gauge the success of different attacks (a technique already routinely used to evade anti-virus). Once again, this exploits the advantage that attackers always have sight of the target, while defenders must rely on good guesses.
Or perhaps ML could simply be used to crank out vast quantities of new and unique malware than is possible today. Whichever of these approaches is taken – and this is only a sample of the possibilities – it jumps out at you how awkward it would be to defend against even relatively simple ML-based attacks. About the only consolation is that if ML-based AI really is a black box that nobody understands then, logically, the attackers won’t understand it either and will waste time experimenting.
Unintended consequences
If we should fear anything it’s precisely this black box effect. There are two parts to this, the biggest of which is the potential for ML-based malware to cause something unintended to happen, especially when targeting critical infrastructure.
This phenomenon has already come to pass with non-AI malware – Stuxnet in 2010 and NotPetya in 2017 are the obvious examples – both of which infected thousands of organizations not on their original target list after unexpectedly ‘escaping’ into the wild.
When it comes to powerful malware exploiting multiple zero days there’s no such thing as a reliably contained attack. Once released, this kind of malware remains pathogenically dangerous until every system it can infect is patched or taken offline, which might be years or decades down the line.
Another anxiety is that because the expertise to understand ML is still thin on the ground, there’s a danger that engineers could come to rely on it without fully understanding its limitations, both for defense and by over-estimating its usefulness in attack. The mistake, then, might be that too many over-invest in it based on marketing promises that end up consuming resources better deployed elsewhere. Once a more realistic assessment takes hold, ML could end up as just another tool that is good at solving certain very specific problems.
Conclusion
My contradictory-sounding conclusion is that perhaps ML and AI makes no fundamental difference at all. It’s just another stop on a journey computer security has been making since the beginning of digital time. The problem is overcoming our preconceptions about what it is and what it means. Chiefly, we must overcome the tendency to think of ML and AI as mysteriously ‘other’ because we don’t understand it and therefore find it difficult to process the concept of machines making complex decisions.
It’s not as if attackers aren’t breaching networks already with today’s pre-ML technology or that well-prepared defenders aren’t regularly stopping them using the same technology. What AI reminds us is that the real difference is how organizations are defended, not whether they or their attackers use ML and AI or not. That has always been what separates survivors from victims. Cybersecurity remains a working demonstration of how the devil takes the hindmost.
Google’s artificial intelligence company DeepMind has published „really significant“ research showing its algorithm can identify around 50 eye diseases by looking at retinal eye scans.
DeepMind said its AI was as good as expert clinicians, and that it could help prevent people from losing their sight.
DeepMind has been criticised for its practices around medical data, but cofounder Mustafa Suleyman said all the information in this research project was anonymised.
The company plans to hand the technology over for free to NHS hospitals for five years, provided it passes the next phase of research.
Google’s artificial intelligence company, DeepMind, has developed an AI which can successfully detect more than 50 types of eye disease just by looking at 3D retinal scans.
DeepMind published on Monday the results of joint research with Moorfields Eye Hospital, a renowned centre for treating eye conditions in London, in Nature Medicine.
The company said its AI was as accurate as expert clinicians when it came to detecting diseases, such as diabetic eye disease and macular degeneration. It could also recommend the best course of action for patients and suggest which needed urgent care.
A technician examines an OCT scan.DeepMind
What is especially significant about the research, according to DeepMind cofounder Mustafa Suleyman, is that the AI has a level of „explainability“ that could boost doctors‘ trust in its recommendations.
„It’s possible for the clinician to interpret what the algorithm is thinking,“ he told Business Insider. „[They can] look at the underlying segmentation.“
In other words, the AI looks less like a mysterious black box that’s spitting out results. It labels pixels on the eye scan that corresponds to signs of a particular disease, Suleyman explained, and can calculate its confidence in its own findings with a percentage score. „That’s really significant,“ he said.
DeepMind’s AI analysing an OCT scan.DeepMind
Suleyman described the findings as a „research breakthrough“ and said the next step was to prove the AI works in a clinical setting. That, he said, would take a number of years. Once DeepMind is in a position to deploy its AI across NHS hospitals in the UK, it will provide the service for free for five years.
Patients are at risk of losing their sight because doctors can’t look at their eye scans in time
British eye specialists have been warning for years that patients are at risk of losing their sight because the NHS is overstretched, and because the UK has an ageing population.
Part of the reason DeepMind and Moorfields took up the research project was because clinicians are „overwhelmed“ by the demand for eye scans, Suleyman said.
„If you have a sight-threatening disease, you want treatment as soon as possible,“ he explained. „And unlike in A&E, where a staff nurse will talk to you and make an evaluation of how serious your condition is, then use that evaluation to decide how quickly you are seen. When an [eye] scan is submitted, there isn’t a triage of your scan according to its severity.“
A patient having an OCT scan.DeepMind
Putting eye scans through the AI could speed the entire process up.
„In the future, I could envisage a person going into their local high street optician, and have an OCT scan done and this algorithm would identify those patients with sight-threatening disease at the very early stage of the condition,“ said Dr Pearse Keane, consultant ophthalmologist at Moorfields Eye Hospital.
DeepMind’s AI was trained on a database of almost 15,000 eye scans, stripped of any identifying information. DeepMind worked with clinicians to label areas of disease, then ran those labelled images through its system. Suleyman said the two-and-a-half project required „huge investment“ from DeepMind and involved 25 staffers, as well as the researchers from Moorfields.
People are still worried about a Google-linked company having access to medical data
While DeepMind has remained UK-based and independent from Google, the relationship has attracted scrutiny. The main question is whether Google, a private US company, should have access to the sensitive medical data required for DeepMind’s health arm.
„You can’t identify whose scans it was. We’re in quite a different regime, this is very much research, and we’re a number of years from being able to deploy in practice,“ he said.
Suleyman added: „How this has the potential to have transform the NHS is very clear. We’ve been very conscious that this will be a model that’s published, and available to others to implement.
„The labelled dataset is available to other researchers. So this is very much an open and collaborative relationship between equals that we’ve worked hard to foster. I’m proud of that work.“
Microsoft has helped innovate facial recognition software. Now it’s urging the US government to enact regulation to control the use of the technology.
In a blog post, Microsoft (MSFT)President Brad Smith said new laws are necessary given the technology’s „broad societal ramifications and potential for abuse.“
He urged lawmakers to form „a government initiative to regulate the proper use of facial recognition technology, informed first by a bipartisan and expert commission.“
Facial recognition — a computer’s ability to identify or verify people’s faces from a photo or through a camera — has been developing rapidly. Apple (AAPL), Google (GOOG), Amazon and Microsoft are among the big tech companies developing and selling such systems. The technology is being used across a range of industries, from private businesses like hotels and casinos, to social media and law enforcement.
Supporters say facial recognition software improves safety for companies and customers and can help police track police down criminals or find missing children. Civil rights groups warn it can infringe on privacy and allow for illegal surveillance and monitoring. There is also room for error, they argue, since the still-emerging technology can result in false identifications.
The accuracy of facial recognition technologies varies, with women and people of color being identified with less accuracy, according to MIT research.
„Facial recognition raises a critical question: what role do we want this type of technology to play in everyday society?“ Smith wrote on Friday.
Smith’s call for a regulatory framework to control the technology comes as tech companies face criticism over how they’ve handled and shared customer data, as well as their cooperation with government agencies.
Last month, Microsoft was scrutinized for its working relationship with US Immigration and Customs Enforcement. ICE had been enforcing the Trump administration’s „zero tolerance“ immigration policy that separated children from their parents when they crossed the US border illegally. The administration has since abandoned the policy.
Microsoft wrote a blog post in January about ICE’s use of its cloud technology Azure, saying it could help it „accelerate facial recognition and identification.“
After questions arose about whether Microsoft’s technology had been used by ICE agents to carry out the controversial border separations, the company released a statement calling the policy „cruel“ and „abusive.“
In his post, Smith reiterated Microsoft’s opposition to the policy and said he had confirmed its contract with ICE does not include facial recognition technology.
Amazon(AMZN) has also come under fire from its own shareholders and civil rights groups over local police forces using its face identifying software Rekognition, which can identify up to 100 people in a single photo.
Some Amazon shareholders coauthored a letter pressuring Amazon to stop selling the technology to the government, saying it was aiding in mass surveillance and posed a threat to privacy rights.
And Facebook (FB) is embroiled in a class-action lawsuit that alleges the social media giant used facial recognition on photos without user permission. Its facial recognition tool scans your photos and suggests you tag friends.
Neither Amazon nor Facebook immediately responded to a request for comment about Smith’s call for new regulations on face ID technology.
Smith said companies have a responsibility to police their own innovations, control how they are deployed and ensure that they are used in a „a manner consistent with broadly held societal values.“
„It may seem unusual for a company to ask for government regulation of its products, but there are many markets where thoughtful regulation contributes to a healthier dynamic for consumers and producers alike,“ he said.
Artificial Intelligence — The Revolution Hasn’t Happened Yet
Artificial Intelligence (AI) is the mantra of the current era. The phrase is intoned by technologists, academicians, journalists and venture capitalists alike. As with many phrases that cross over from technical academic fields into general circulation, there is significant misunderstanding accompanying the use of the phrase. But this is not the classical case of the public not understanding the scientists — here the scientists are often as befuddled as the public. The idea that our era is somehow seeing the emergence of an intelligence in silicon that rivals our own entertains all of us — enthralling us and frightening us in equal measure. And, unfortunately, it distracts us.
There is a different narrative that one can tell about the current era. Consider the following story, which involves humans, computers, data and life-or-death decisions, but where the focus is something other than intelligence-in-silicon fantasies. When my spouse was pregnant 14 years ago, we had an ultrasound. There was a geneticist in the room, and she pointed out some white spots around the heart of the fetus. “Those are markers for Down syndrome,” she noted, “and your risk has now gone up to 1 in 20.” She further let us know that we could learn whether the fetus in fact had the genetic modification underlying Down syndrome via an amniocentesis. But amniocentesis was risky — the risk of killing the fetus during the procedure was roughly 1 in 300. Being a statistician, I determined to find out where these numbers were coming from. To cut a long story short, I discovered that a statistical analysis had been done a decade previously in the UK, where these white spots, which reflect calcium buildup, were indeed established as a predictor of Down syndrome. But I also noticed that the imaging machine used in our test had a few hundred more pixels per square inch than the machine used in the UK study. I went back to tell the geneticist that I believed that the white spots were likely false positives — that they were literally “white noise.” She said “Ah, that explains why we started seeing an uptick in Down syndrome diagnoses a few years ago; it’s when the new machine arrived.”
We didn’t do the amniocentesis, and a healthy girl was born a few months later. But the episode troubled me, particularly after a back-of-the-envelope calculation convinced me that many thousands of people had gotten that diagnosis that same day worldwide, that many of them had opted for amniocentesis, and that a number of babies had died needlessly. And this happened day after day until it somehow got fixed. The problem that this episode revealed wasn’t about my individual medical care; it was about a medical system that measured variables and outcomes in various places and times, conducted statistical analyses, and made use of the results in other places and times. The problem had to do not just with data analysis per se, but with what database researchers call “provenance” — broadly, where did data arise, what inferences were drawn from the data, and how relevant are those inferences to the present situation? While a trained human might be able to work all of this out on a case-by-case basis, the issue was that of designing a planetary-scale medical system that could do this without the need for such detailed human oversight.
I’m also a computer scientist, and it occurred to me that the principles needed to build planetary-scale inference-and-decision-making systems of this kind, blending computer science with statistics, and taking into account human utilities, were nowhere to be found in my education. And it occurred to me that the development of such principles — which will be needed not only in the medical domain but also in domains such as commerce, transportation and education — were at least as important as those of building AI systems that can dazzle us with their game-playing or sensorimotor skills.
Whether or not we come to understand “intelligence” any time soon, we do have a major challenge on our hands in bringing together computers and humans in ways that enhance human life. While this challenge is viewed by some as subservient to the creation of “artificial intelligence,” it can also be viewed more prosaically — but with no less reverence — as the creation of a new branch of engineering. Much like civil engineering and chemical engineering in decades past, this new discipline aims to corral the power of a few key ideas, bringing new resources and capabilities to people, and doing so safely. Whereas civil engineering and chemical engineering were built on physics and chemistry, this new engineering discipline will be built on ideas that the preceding century gave substance to — ideas such as “information,” “algorithm,” “data,” “uncertainty,” “computing,” “inference,” and “optimization.” Moreover, since much of the focus of the new discipline will be on data from and about humans, its development will require perspectives from the social sciences and humanities.
While the building blocks have begun to emerge, the principles for putting these blocks together have not yet emerged, and so the blocks are currently being put together in ad-hoc ways.
Thus, just as humans built buildings and bridges before there was civil engineering, humans are proceeding with the building of societal-scale, inference-and-decision-making systems that involve machines, humans and the environment. Just as early buildings and bridges sometimes fell to the ground — in unforeseen ways and with tragic consequences — many of our early societal-scale inference-and-decision-making systems are already exposing serious conceptual flaws.
And, unfortunately, we are not very good at anticipating what the next emerging serious flaw will be. What we’re missing is an engineering discipline with its principles of analysis and design.
The current public dialog about these issues too often uses “AI” as an intellectual wildcard, one that makes it difficult to reason about the scope and consequences of emerging technology. Let us begin by considering more carefully what “AI” has been used to refer to, both recently and historically.
Most of what is being called “AI” today, particularly in the public sphere, is what has been called “Machine Learning” (ML) for the past several decades. ML is an algorithmic field that blends ideas from statistics, computer science and many other disciplines (see below) to design algorithms that process data, make predictions and help make decisions. In terms of impact on the real world, ML is the real thing, and not just recently. Indeed, that ML would grow into massive industrial relevance was already clear in the early 1990s, and by the turn of the century forward-looking companies such as Amazon were already using ML throughout their business, solving mission-critical back-end problems in fraud detection and supply-chain prediction, and building innovative consumer-facing services such as recommendation systems. As datasets and computing resources grew rapidly over the ensuing two decades, it became clear that ML would soon power not only Amazon but essentially any company in which decisions could be tied to large-scale data. New business models would emerge. The phrase “Data Science” began to be used to refer to this phenomenon, reflecting the need of ML algorithms experts to partner with database and distributed-systems experts to build scalable, robust ML systems, and reflecting the larger social and environmental scope of the resulting systems.
This confluence of ideas and technology trends has been rebranded as “AI” over the past few years. This rebranding is worthy of some scrutiny.
Historically, the phrase “AI” was coined in the late 1950’s to refer to the heady aspiration of realizing in software and hardware an entity possessing human-level intelligence. We will use the phrase “human-imitative AI” to refer to this aspiration, emphasizing the notion that the artificially intelligent entity should seem to be one of us, if not physically at least mentally (whatever that might mean). This was largely an academic enterprise. While related academic fields such as operations research, statistics, pattern recognition, information theory and control theory already existed, and were often inspired by human intelligence (and animal intelligence), these fields were arguably focused on “low-level” signals and decisions. The ability of, say, a squirrel to perceive the three-dimensional structure of the forest it lives in, and to leap among its branches, was inspirational to these fields. “AI” was meant to focus on something different — the “high-level” or “cognitive” capability of humans to “reason” and to “think.” Sixty years later, however, high-level reasoning and thought remain elusive. The developments which are now being called “AI” arose mostly in the engineering fields associated with low-level pattern recognition and movement control, and in the field of statistics — the discipline focused on finding patterns in data and on making well-founded predictions, tests of hypotheses and decisions.
Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now viewed as being at the core of the so-called “AI revolution,” first arose in the field of control theory in the 1950s and 1960s. One of its early applications was to optimize the thrusts of the Apollo spaceships as they headed towards the moon.
Since the 1960s much progress has been made, but it has arguably not come about from the pursuit of human-imitative AI. Rather, as in the case of the Apollo spaceships, these ideas have often been hidden behind the scenes, and have been the handiwork of researchers focused on specific engineering challenges. Although not visible to the general public, research and systems-building in areas such as document retrieval, text classification, fraud detection, recommendation systems, personalized search, social network analysis, planning, diagnostics and A/B testing have been a major success — these are the advances that have powered companies such as Google, Netflix, Facebook and Amazon.
One could simply agree to refer to all of this as “AI,” and indeed that is what appears to have happened. Such labeling may come as a surprise to optimization or statistics researchers, who wake up to find themselves suddenly referred to as “AI researchers.” But labeling of researchers aside, the bigger problem is that the use of this single, ill-defined acronym prevents a clear understanding of the range of intellectual and commercial issues at play.
The past two decades have seen major progress — in industry and academia — in a complementary aspiration to human-imitative AI that is often referred to as “Intelligence Augmentation” (IA). Here computation and data are used to create services that augment human intelligence and creativity. A search engine can be viewed as an example of IA (it augments human memory and factual knowledge), as can natural language translation (it augments the ability of a human to communicate). Computing-based generation of sounds and images serves as a palette and creativity enhancer for artists. While services of this kind could conceivably involve high-level reasoning and thought, currently they don’t — they mostly perform various kinds of string-matching and numerical operations that capture patterns that humans can make use of.
Hoping that the reader will tolerate one last acronym, let us conceive broadly of a discipline of “Intelligent Infrastructure” (II), whereby a web of computation, data and physical entities exists that makes human environments more supportive, interesting and safe. Such infrastructure is beginning to make its appearance in domains such as transportation, medicine, commerce and finance, with vast implications for individual humans and societies. This emergence sometimes arises in conversations about an “Internet of Things,” but that effort generally refers to the mere problem of getting “things” onto the Internet — not to the far grander set of challenges associated with these “things” capable of analyzing those data streams to discover facts about the world, and interacting with humans and other “things” at a far higher level of abstraction than mere bits.
For example, returning to my personal anecdote, we might imagine living our lives in a “societal-scale medical system” that sets up data flows, and data-analysis flows, between doctors and devices positioned in and around human bodies, thereby able to aid human intelligence in making diagnoses and providing care. The system would incorporate information from cells in the body, DNA, blood tests, environment, population genetics and the vast scientific literature on drugs and treatments. It would not just focus on a single patient and a doctor, but on relationships among all humans — just as current medical testing allows experiments done on one set of humans (or animals) to be brought to bear in the care of other humans. It would help maintain notions of relevance, provenance and reliability, in the way that the current banking system focuses on such challenges in the domain of finance and payment. And, while one can foresee many problems arising in such a system — involving privacy issues, liability issues, security issues, etc — these problems should properly be viewed as challenges, not show-stoppers.
We now come to a critical issue: Is working on classical human-imitative AI the best or only way to focus on these larger challenges? Some of the most heralded recent success stories of ML have in fact been in areas associated with human-imitative AI — areas such as computer vision, speech recognition, game-playing and robotics. So perhaps we should simply await further progress in domains such as these. There are two points to make here. First, although one would not know it from reading the newspapers, success in human-imitative AI has in fact been limited — we are very far from realizing human-imitative AI aspirations. Unfortunately the thrill (and fear) of making even limited progress on human-imitative AI gives rise to levels of over-exuberance and media attention that is not present in other areas of engineering.
Second, and more importantly, success in these domains is neither sufficient nor necessary to solve important IA and II problems. On the sufficiency side, consider self-driving cars. For such technology to be realized, a range of engineering problems will need to be solved that may have little relationship to human competencies (or human lack-of-competencies). The overall transportation system (an II system) will likely more closely resemble the current air-traffic control system than the current collection of loosely-coupled, forward-facing, inattentive human drivers. It will be vastly more complex than the current air-traffic control system, specifically in its use of massive amounts of data and adaptive statistical modeling to inform fine-grained decisions. It is those challenges that need to be in the forefront, and in such an effort a focus on human-imitative AI may be a distraction.
As for the necessity argument, it is sometimes argued that the human-imitative AI aspiration subsumes IA and II aspirations, because a human-imitative AI system would not only be able to solve the classical problems of AI (as embodied, e.g., in the Turing test), but it would also be our best bet for solving IA and II problems. Such an argument has little historical precedent. Did civil engineering develop by envisaging the creation of an artificial carpenter or bricklayer? Should chemical engineering have been framed in terms of creating an artificial chemist? Even more polemically: if our goal was to build chemical factories, should we have first created an artificial chemist who would have then worked out how to build a chemical factory?
A related argument is that human intelligence is the only kind of intelligence that we know, and that we should aim to mimic it as a first step. But humans are in fact not very good at some kinds of reasoning — we have our lapses, biases and limitations. Moreover, critically, we did not evolve to perform the kinds of large-scale decision-making that modern II systems must face, nor to cope with the kinds of uncertainty that arise in II contexts. One could argue
that an AI system would not only imitate human intelligence, but also “correct” it, and would also scale to arbitrarily large problems. But we are now in the realm of science fiction — such speculative arguments, while entertaining in the setting of fiction, should not be our principal strategy going forward in the face of the critical IA and II problems that are beginning to emerge. We need to solve IA and II problems on their own merits, not as a mere corollary to a human-imitative AI agenda.
It is not hard to pinpoint algorithmic and infrastructure challenges in II systems that are not central themes in human-imitative AI research. II systems require the ability to manage distributed repositories of knowledge that are rapidly changing and are likely to be globally incoherent. Such systems must cope with cloud-edge interactions in making timely, distributed decisions and they must deal with long-tail phenomena whereby there is lots of data on some individuals and little data on most individuals. They must address the difficulties of sharing data across administrative and competitive boundaries. Finally, and of particular importance, II systems must bring economic ideas such as incentives and pricing into the realm of the statistical and computational infrastructures that link humans to each other and to valued goods. Such II systems can be viewed as not merely providing a service, but as creating markets. There are domains such as music, literature and journalism that are crying out for the emergence of such markets, where data analysis links producers and consumers. And this must all be done within the context of evolving societal, ethical and legal norms.
Of course, classical human-imitative AI problems remain of great interest as well. However, the current focus on doing AI research via the gathering of data, the deployment of “deep learning” infrastructure, and the demonstration of systems that mimic certain narrowly-defined human skills — with little in the way of emerging explanatory principles — tends to deflect attention from major open problems in classical AI. These problems include the need to bring meaning and reasoning into systems that perform natural language processing, the need to infer and represent causality, the need to develop computationally-tractable representations of uncertainty and the need to develop systems that formulate and pursue long-term goals. These are classical goals in human-imitative AI, but in the current hubbub over the “AI revolution,” it is easy to forget that they are not yet solved.
IA will also remain quite essential, because for the foreseeable future, computers will not be able to match humans in their ability to reason abstractly about real-world situations. We will need well-thought-out interactions of humans and computers to solve our most pressing problems. And we will want computers to trigger new levels of human creativity, not replace human creativity (whatever that might mean).
It was John McCarthy (while a professor at Dartmouth, and soon to take a
position at MIT) who coined the term “AI,” apparently to distinguish his
budding research agenda from that of Norbert Wiener (then an older professor at MIT). Wiener had coined “cybernetics” to refer to his own vision of intelligent systems — a vision that was closely tied to operations research, statistics, pattern recognition, information theory and control theory. McCarthy, on the other hand, emphasized the ties to logic. In an interesting reversal, it is Wiener’s intellectual agenda that has come to dominate in the current era, under the banner of McCarthy’s terminology. (This state of affairs is surely, however, only temporary; the pendulum swings more in AI than
in most fields.)
But we need to move beyond the particular historical perspectives of McCarthy and Wiener.
We need to realize that the current public dialog on AI — which focuses on a narrow subset of industry and a narrow subset of academia — risks blinding us to the challenges and opportunities that are presented by the full scope of AI, IA and II.
This scope is less about the realization of science-fiction dreams or nightmares of super-human machines, and more about the need for humans to understand and shape technology as it becomes ever more present and influential in their daily lives. Moreover, in this understanding and shaping there is a need for a diverse set of voices from all walks of life, not merely a dialog among the technologically attuned. Focusing narrowly on human-imitative AI prevents an appropriately wide range of voices from being heard.
While industry will continue to drive many developments, academia will also continue to play an essential role, not only in providing some of the most innovative technical ideas, but also in bringing researchers from the computational and statistical disciplines together with researchers from other
disciplines whose contributions and perspectives are sorely needed — notably
the social sciences, the cognitive sciences and the humanities.
On the other hand, while the humanities and the sciences are essential as we go forward, we should also not pretend that we are talking about something other than an engineering effort of unprecedented scale and scope — society is aiming to build new kinds of artifacts. These artifacts should be built to work as claimed. We do not want to build systems that help us with medical treatments, transportation options and commercial opportunities to find out after the fact that these systems don’t really work — that they make errors that take their toll in terms of human lives and happiness. In this regard, as I have emphasized, there is an engineering discipline yet to emerge for the data-focused and learning-focused fields. As exciting as these latter fields appear to be, they cannot yet be viewed as constituting an engineering discipline.
Moreover, we should embrace the fact that what we are witnessing is the creation of a new branch of engineering. The term “engineering” is often
invoked in a narrow sense — in academia and beyond — with overtones of cold, affectless machinery, and negative connotations of loss of control by humans. But an engineering discipline can be what we want it to be.
In the current era, we have a real opportunity to conceive of something historically new — a human-centric engineering discipline.
I will resist giving this emerging discipline a name, but if the acronym “AI” continues to be used as placeholder nomenclature going forward, let’s be aware of the very real limitations of this placeholder. Let’s broaden our scope, tone down the hype and recognize the serious challenges ahead.
TheIdea – The Innovation Agency 