New EU rules mean WhatsApp and Messenger must be interoperable with other chat apps. Here’s how that will work.
A frequent annoyance of contemporary life is having to shuffle through different messaging apps to reach the right person. Messenger, iMessage, WhatsApp, Signal—they all exist in their own silos of group chats and contacts. Soon, though, WhatsApp will do the previously unthinkable for its 2 billion users: allow people to message you from another app. At least, that’s the plan.
For about the past two years, WhatsApp has been building a way for other messaging apps to plug themselves into its service and let people chat across apps—all without breaking the end-to-end encryption it uses to protect the privacy and security of people’s messages. The move is the first time the chat app has opened itself up this way, and it potentially offers greater competition.
It isn’t a shift entirely of WhatsApp’s own making. In September, European, lawmakers designated WhatsApp parent Meta as one of six influential “gatekeeer” companies under its sweeping Digital Markets Act, giving it six months to open its walled garden to others. With just a few weeks to go before that time is up, WhatsApp is detailing how its interoperability with other apps may work.
“There’s real tension between offering an easy way to offer this interoperability to third parties whilst at the same time preserving the WhatsApp privacy, security, and integrity bar,” says Dick Brouwer, an engineering director at WhatsApp who has worked on Meta rolling out encryption to its Messenger app. “I think we’re pretty happy with where we’ve landed.”
Interoperability in both WhatsApp and Messenger—as dictated by Europe’s rules—will initially focus on text messaging, sending images, voice messages, videos, and files between two people. Calls and group chats will come years down the line. Europe’s rules apply only to messaging services, not traditional SMS messaging. “One of the core requirements here, and this is really important, is for users for this to be opt-in,” says Brouwer. “I can choose whether or not I want to participate in being open to exchanging messages with third parties. This is important, because it could be a big source of spam and scams.”
WhatsApp users who opt in will see messages from other apps in a separate section at the top of their inbox. This “third-party chats” inbox has previously been spotted in development versions of the app. “The early thinking here is to put a separate inbox, given that these networks are very different,” Brouwer says. “We cannot offer the same level of privacy and security,” he says. If WhatsApp were to add SMS, it would use a separate inbox as well, although there are no plans to add it, he says.
Overall, the idea behind interoperability is simple. You shouldn’t need to know what messaging app your friends or family use to get in touch with them, and you should be able to communicate from one app to another without having to download both. In an ideal interoperable world, you could, for example, use Apple’s iMessage to chat with someone on Telegram. However, for apps with millions or billions of users, making this a reality isn’t straightforward—encrypted messaging apps use their own configurations and different protocols and have different standards when it comes to privacy.
Despite WhatsApp working on its interoperability plan for more than a year, it will still take some time for third-party chats to hit people’s apps. Messaging companies that want to interoperate with WhatsApp or Messenger will need to sign an agreement with the company and follow its terms. The full details of the plan will be published in March, Brouwer says; under EU laws, the company will have several months to implement it.
Brouwer says Meta would prefer if other apps use the Signal encryption protocol, which its systems are based upon. Other than its namesake app and the Meta-owned messengers, the Signal Protocol is publicly disclosed as being used in Google Messages and Skype. To send messages, third-party apps will need to encrypt content using the Signal Protocol and then package it into message stanzas in the eXtensible Markup Language (XML). When receiving messages, apps will need to connect to WhatsApp’s servers.
“We think that the best way to deliver this approach is through a solution that is built on WhatsApp’s existing client-server architecture,” Brouwer says, adding it has been working with other companies on the plans. “This effectively means that the approach that we’re trying to take is for WhatsApp to document our client- server protocol and letting third-party clients connect directly to our infrastructure and exchange messages with WhatsApp clients.”
There is some flexibility to WhatsApp interoperability. Meta’s app will also allow other apps to use different encryption protocols if they can “demonstrate” they reach the security standards that WhatsApp outlines in its guidance. There will also be the option, Brouwer says, for third-party developers to add a proxy between their apps and WhatsApp’s server. This, he says, could give developers more “flexibility” and remove the need for them to use WhatsApp’s client-server protocols, but it also “increases the potential attack vectors.”
So far, it is unclear which companies, if any, are planning to connect their services to WhatsApp. WIRED asked 10 owners of messaging or chat services—including Google, Telegram, Viber, and Signal—whether they intend to look at interoperability or had worked with WhatsApp on its plans. The majority of companies didn’t respond to the request for comment. Those that did, Snap and Discord, said they had nothing to add. (The European Commission is investigating whether Apple’s iMessage meets the thresholds to offer interoperability with other apps itself. The company did not respond to a request for comment. It has also faced recent challenges in the US about the closed nature of iMessage.)
Matthew Hodgson, the cofounder of Matrix, which is building an open source standard for encryption and operates the messaging app Element, confirms that his company has worked with WhatsApp on interoperability in an “experimental” way but that he cannot say any more due to signing a nondisclosure agreement. In a talk last weekend, Hodgson demonstrated “hypothetical” architectures for ways that Matrix could connect to the systems of two gatekeepers that don’t use the same encryption protocols.
Meanwhile, Julia Weis, a spokesperson for the Swiss messaging app Threema, says that while WhatsApp did approach it to discuss its interoperability plans, the proposed system didn’t meet Threema’s security and privacy standards. “WhatsApp specifies all the protocols, and we’d have no way of knowing what actually happens with the user data that gets transferred to WhatsApp—after all, WhatsApp is closed source,” Weis says. (WhatsApp’s privacy policy states how it uses people’s data.)
When the EU first announced that messaging apps may have to work together in early 2022, many leading cryptographers opposed the idea, saying it adds complexity and potentially introduces more security and privacy risks. Carmela Troncoso, an associate professor at the Swiss university École Polytechnique Fédérale de Lausanne, who focuses on security and privacy engineering, says interoperability moves could potentially lead to different power relationships between companies, depending on how they are implemented.
“This move for interoperability will, on the one hand, open the market, but also maybe close the market in the sense that now the bigger players are going to have more decisional power,” Troncoso says. “Now, if the big player makes a move and you want to continue being interoperable with this big player, because your users are hooked up to this, you’re going to have to follow.”
While the interoperability of encrypted messaging apps may be possible, there are some fundamental challenges about how the systems will work in the real world. How much of a problem spam and scamming will be across apps is largely unknown until people start using interoperable setups. There are also questions about how people will find each other across different apps. For instance, WhatsApp uses your phone number to interact and message other people, while Threema randomly generates eight-digit IDs for people’s accounts. Linking up with WhatsApp “could de-anonymize Threema users,” Weis, the Threema spokesperson says.
Meta’s Brouwer says the company is still working on the interoperability features and the level of support it will make available for companies wanting to integrate with it. “Nobody quite knows how this works,” Brouwer says. “We have no idea what the demand is.” However, he says, the decision was made to use WhatsApp’s existing architecture to run interoperability, as it means that it can more easily scale up the system for group chats in the future. It also reduces the potential for people’s data to be exposed to multiple servers, Brouwer says.
Ultimately, interoperability will evolve over time, and from Meta’s perspective, Brouwer says, it will be more challenging to add new features to it quickly. “We don’t believe interop chats and WhatsApp chats can evolve at the same pace,” he says, claiming it is “harder to evolve an open network” compared to a closed one. “The second you do something different—than what we know works really well—you open up a wormhole of security, privacy issues, and complexity that is always going to be much bigger than you think it is.”
Last year, scientists reported that the US Atlantic Coast is dropping by several millimeters annually, with some areas, like Delaware, notching figures several times that rate. So just as the seas are rising, the land along the eastern seaboard is sinking, greatly compounding the hazard for coastal communities.
In a follow-up study just published in the journal PNAS Nexus, the researchers tally up the mounting costs of subsidence—due to settling, groundwater extraction, and other factors—for those communities and their infrastructure. Using satellite measurements, they have found that up to 74,000 square kilometers (29,000 square miles) of the Atlantic Coast are exposed to subsidence of up to 2 millimeters (0.08 inches) a year, affecting up to 14 million people and 6 million properties. And over 3,700 square kilometers along the Atlantic Coast are sinking more than 5 millimeters annually. That’s an even faster change than sea level rise, currently at 4 millimeters a year. (In the map below, warmer colors represent more subsidence, up to 6 millimeters.)
Courtesy of Leonard O Ohenhen
With each millimeter of subsidence, it gets easier for storm surges—essentially a wall of seawater, which hurricanes are particularly good at pushing onshore—to creep farther inland, destroying more and more infrastructure. “And it’s not just about sea levels,” says the study’s lead author, Leonard Ohenhen, an environmental security expert at Virginia Tech. “You also have potential to disrupt the topography of the land, for example, so you have areas that can get full of flooding when it rains.”
A few millimeters of annual subsidence may not sound like much, but these forces are relentless: Unless coastal areas stop extracting groundwater, the land will keep sinking deeper and deeper. The social forces are relentless, too, as more people around the world move to coastal cities, creating even more demand for groundwater. “There are processes that are sometimes even cyclic. For example, in summers you pump a lot more water, so land subsides rapidly in a short period of time,” says Manoochehr Shirzaei, an environmental security expert at Virginia Tech and coauthor of the paper. “That causes large areas to subside below a threshold that leads the water to flood a large area.” When it comes to flooding, falling elevation of land is a tipping element that has been largely ignored by research so far, Shirzaei says.
In Jakarta, Indonesia, for example, the land is sinking nearly a foot a year because of collapsing aquifers. Accordingly, within the next three decades, 95 percent of North Jakarta could be underwater. The city is planning a giant seawall to hold back the ocean, but it’ll be useless unless subsidence is stopped.
This new study warns that levees and other critical infrastructure along the Atlantic Coast are in similar danger. If the land were to sink uniformly, you might just need to keep raising the elevation of a levee to compensate. But the bigger problem is “differential subsidence,” in which different areas of land sink at different rates. “If you have a building or a runway or something that’s settling uniformly, it’s probably not that big a deal,” says Tom Parsons, a geophysicist with the United States Geological Survey who studies subsidence but wasn’t involved in the new paper. “But if you have one end that’s sinking faster than the other, then you start to distort things.”
The researchers selected 10 levees on the Atlantic Coast and found that all were impacted by subsidence of at least 1 millimeter a year. That puts at risk something like 46,000 people, 27,000 buildings, and $12 billion worth of property. But they note that the actual population and property at risk of exposure behind the 116 East Coast levees vulnerable to subsidence could be two to three times greater. “Levees are heavy, and when they’re set on land that’s already subsiding, it can accelerate that subsidence,” says independent scientist Natalie Snider, who studies coastal resilience but wasn’t involved in the new research. “It definitely can impact the integrity of the protection system and lead to failures that can be catastrophic.”
Courtesy of Leonard O Ohenhen
The same vulnerability affects other infrastructure that stretches across the landscape. The new analysis finds that along the Atlantic Coast, between 77 and 99 percent of interstate highways and between 76 and 99 percent of primary and secondary roads are exposed to subsidence. (In the map above, you can see roads sinking at different rates across Hampton and Norfolk, Virginia.) Between 81 and 99 percent of railway tracks and 42 percent of train stations are exposed on the East Coast.
Below is New York’s JFK Airport—notice the red hot spots of high subsidence against the teal of more mild elevation change. The airport’s average subsidence rate is 1.7 millimeters a year (similar to the LaGuardia and Newark airports), but across JFK that varies between 0.8 and 2.8 millimeters a year, depending on the exact spot.
Courtesy of Leonard O Ohenhen
This sort of differential subsidence can also bork much smaller structures, like buildings, where one side might drop faster than another. “Even if that is just a few millimeters per year, you can potentially cause cracks along structures,” says Ohenhen.
The study finds that subsidence is highly variable along the Atlantic Coast, both regionally and locally, as different stretches have different geology and topography, and different rates of groundwater extraction. It’s looking particularly problematic for several communities, like Virginia Beach, where 451,000 people and 177,000 properties are at risk. In Baltimore, Maryland, it’s 826,000 people and 335,000 properties, while in NYC—in Queens, Bronx, and Nassau—that leaps to 5 million people and 1.8 million properties.
So there’s two components to addressing the problem of subsidence: Getting high-resolution data like in this study, and then pairing that with groundwater data. “Subsidence is so spatially variable,” says Snider. “Having the details of where groundwater extraction is really having an impact, and being able to then demonstrate that we need to change our management of that water, that reduces subsidence in the future.”
The time to act is now, Shirzaei emphasizes. Facing down subsidence is like treating a disease: You spend less money by diagnosing and treating the problem now, saving money later by avoiding disaster. “This kind of data and the study could be an essential component of the health care system for infrastructure management,” he says. “Like cancers—if you diagnose it early on, it can be curable. But if you are late, you invest a lot of money, and the outcome is uncertain.”
Inside Apple’s Big Plan to Bring Generative AI to All Its Devices
Apple was caught flat-footed when ChatGPT and other AI tools took the technology industry by storm. But the company is now preparing its response and plans to develop features for its full range of devices. Also: The future of the Mac comes into focus, a cheaper Apple Pencil debuts, and the Vision Pro gets closer.
One of the most intense and widespread endeavors at Apple Inc. right now is its effort to respond to the AI frenzy sweeping the technology industry.
The company has some catching up to do. Apple largely sat on the sidelines when OpenAI’s ChatGPT took off like a rocket last year. It watched as Google and Microsoft Corp. rolled out generative AI versions of their search engines, which spit out convincingly human-like responses to users’ queries. Microsoft also updated its Windows apps with smarter assistants, and Amazon.com Inc. unveiled an AI-enhanced overhaul of Alexa.
All the while, the only noteworthy AI release from Apple was an improved auto-correct system in iOS 17.
Now, Chief Executive Officer Tim Cook says that Apple has been working on generative AI technology for years. But I can tell you in no uncertain terms that Apple executives were caught off guard by the industry’s sudden AI fever and have been scrambling since late last year to make up for lost time.
“There’s a lot of anxiety about this and it’s considered a pretty big miss internally,” a person with knowledge of the matter told Power On.
As I first reported in July, the company built its own large language model called Ajax and rolled out an internal chatbot dubbed “Apple GPT” to test out the functionality. The critical next step is determining if the technology is up to snuff with the competition and how Apple will actually apply it to its products.
Apple’s senior vice presidents in charge of AI and software engineering, John Giannandrea and Craig Federighi, are spearheading the effort. On Cook’s team, they’re referred to as the “executive sponsors” of the generative AI push. Eddy Cue, the head of services, is also involved, I’m told. The trio are now on course to spend about $1 billion per year on the undertaking.
Giannandrea is overseeing development of the underlying technology for a new AI system, and his team is revamping Siri in a way that will deeply implement it. This smarter version of Siri could be ready as soon as next year, but there are still concerns about the technology and it may take longer for Apple’s AI features to spread across its product line.
Federighi’s software engineering group, meanwhile, is adding AI to the next version of iOS. There’s an edict to fill it with features running on the company’s large language model, or LLM, which uses a flood of data to hone AI capabilities. The new features should improve how both Siri and the Messages app can field questions and auto-complete sentences, mirroring recent changes to competing services.
Apple’s software engineering teams are also looking at integrating generative AI into development tools like Xcode, a move that could help app developers write new applications more quickly. That would bring it in line with services like Microsoft’s GitHub Copilot, which offers auto-complete suggestions to developers while they write code.
And Cue’s organization is pushing to add AI to as many apps as possible. The group is exploring new features for Apple Music, including auto-generated playlists (this is something Spotify rolled out earlier this year in partnership with OpenAI), as well as the company’s productivity apps.
Craig Federighi and Eddy Cue.Photographer: David Paul Morris/Bloomberg
Cue’s team is examining how generative AI can be used to help people write in apps like Pages or auto-create slide decks in Keynote. Again, this is similar to what Microsoft has already launched for its Word and PowerPoint apps. Apple is also testing generative AI for internal customer service apps within its AppleCare group, I’ve previously reported.
One debate going on internally is how to deploy generative AI: as a completely on-device experience, a cloud-based setup or something in between. An on-device approach would work faster and help safeguard privacy, but deploying Apple’s LLMs via the cloud would allow for more advanced operations.
The on-device strategy also makes it harder for Apple to update its technology and adapt to a fast-changing industry. With that in mind, I wouldn’t be surprised if the company adopts a combined approach: using on-device processing for some features and the cloud for more advanced tasks.
When it comes to getting this right, the stakes are high. Generative AI has quickly become much more than a buzzword and will be central to the next several decades of computing. Apple knows it can’t afford to take a back seat.
Testimony during Google’s antitrust case revealed that the company may be altering billions of queries a day to generate results that will get you to buy more stuff.
Recently, a startling piece of information came to light in the ongoing antitrust case against Google. During one employee’s testimony, a key exhibit momentarily flashed on a projector. In the mostly closed trial, spectators like myself have only a few seconds to scribble down the contents of exhibits shown during public questioning. Thus far, witnesses had dropped breadcrumbs hinting at the extent of Google’s drive to boost profits: a highly confidential effort called Project Mercury, urgent missives to “shake the sofa cushions” to generate more advertising revenue on the search engine results page (SERP), distressed emails about the sustained decline in the ad-triggering searches that generate most of Google’s money, recollections of how the executive team has long insisted that obscene corporate profit equals consumer good. Now, the projector screen showed an internal Google slide about changes to its search algorithm.
I was attending the trial out of long-standing professional interest. I had previously battled Google’s legal team while at the Federal Trade Commission, and I advocated around the world for search engine competition as an executive for DuckDuckGo. I’m all too familiar with Google’s secret games and word play. With the trial practically in my backyard, I couldn’t stay away from the drama.
This onscreen Google slide had to do with a “semantic matching” overhaul to its SERP algorithm. When you enter a query, you might expect a search engine to incorporate synonyms into the algorithm as well as text phrase pairings in natural language processing. But this overhaul went further, actually altering queries to generate more commercial results.
There have long been suspicions that the search giant manipulates ad prices, and now it’s clear that Google treats consumers with the same disdain. The “10 blue links,” or organic results, which Google has always claimed to be sacrosanct, are just another vector for Google greediness, camouflaged in the company’s kindergarten colors.
Google likely alters queries billions of times a day in trillions of different variations. Here’s how it works. Say you search for “children’s clothing.” Google converts it, without your knowledge, to a search for “NIKOLAI-brand kidswear,” making a behind-the-scenes substitution of your actual query with a different query that just happens to generate more money for the company, and will generate results you weren’t searching for at all. It’s not possible for you to opt out of the substitution. If you don’t get the results you want, and you try to refine your query, you are wasting your time. This is a twisted shopping mall you can’t escape.
Why would Google want to do this? First, the generated results to the latter query are more likely to be shopping-oriented, triggering your subsequent behavior much like the candy display at a grocery store’s checkout. Second, that latter query will automatically generate the keyword ads placed on the search engine results page by stores like TJ Maxx, which pay Google every time you click on them. In short, it’s a guaranteed way to line Google’s pockets.
It’s also a guaranteed way to harm everyone except Google. This system reduces search engine quality for users and drives up advertiser expenses. Google can get away with it because these manipulations are imperceptible to the user and advertiser, and the company has effectively captured more than 90 percent market share.
It’s unclear how often, or for how long, Google has been doing this, but the machination is clever and ambitious. I have spent decades looking for examples of Google putting its enormous thumb on the scale to censor or amplify certain results, and it hadn’t even occurred to me that Google just flat out deletes queries and replaces them with ones that monetize better. Most scams follow an elementary bait-and-switch technique, where the scoundrel lures you in with attractive bait and then, at the right time, switches to a different option. But Google “innovated” by reversing the scam, first switching your query, then letting you believe you were getting the best search engine results. This is a magic trick that Google could only pull off after monopolizing the search engine market, giving consumers the false impression that it is incomparably great, only because you’ve grown so accustomed to it.
Even if Google prevails in this antitrust trial, I predict its troubles will continue. A company executive at the trial spoke on the stand about Google’s “contract with the users” and its “honest results policy.” No matter what pretzel shape Google twists, no matter what loopholes and legalese it bandies about, defying reasonable user expectations is a loser’s game. Until then, Google’s massive market share and deep entrenchment in everyday life ensure that these warped results pollute our ability to discover and learn basic information about the world around us. The next time you Google, remember that you’re getting search results that have been skewed—not to help you find what you’re looking for, but to boost the company’s profits.
AOH1996 has been developed over two decades to target a protein found in all forms of the disease
A new “cancer-stopping” drug has been found to “annihilate” solid cancerous tumours in early stage studies.
The chemotherapy drug leaves healthy cells unaffected, scientists said.
The AOH1996 drug is named after a child – Anna Olivia Healy, born in 1996, who died when she was only nine after being diagnosed with a rare childhood cancer neuroblastoma.
Prof Linda Malkas and her team spent two decades developing the drug that targets a protein in all cancers, including the cancer that led to Anna’s death.
The protein, proliferating cell nuclear antigen (PCNA), was once thought too challenging to aim targeted therapies at.
PCNA in its mutated form encourages tumours to grow by aiding DNA replication and repair of cancerous cells.
Prof Malkas and her team at the City of Hope in California, one of the United States’ largest cancer research and treatment organisations, said the targeted chemotherapy appears to “annihilate” all solid tumours in preclinical research.
Selectively kills cancer cells
AOH1996 was tested in more than 70 cell lines and was found to selectively kill cancer cells by disrupting the normal cell reproductive cycle, but it did not interrupt the reproductive cycle of healthy stem cells.
Pre-clinical studies suggest the drug has been shown to be effective in treating cells derived from breast, prostate, brain, ovarian, cervical, skin and lung cancers.
The drug still needs to go through rigorous safety and efficacy testing and large-scale clinical trials before it can be used widely.
The first patient received the potentially cancer-stopping pill in October with the phase one clinical trial still ongoing and expected to last for at least two years.
Patients are still being recruited to the trial.
Researchers are also still examining mechanisms that make the drug work in animal studies.
‘Like snowstorm that closes airline hub’
Prof Malkas said: “PCNA is like a major airline terminal hub containing multiple plane gates.
“Data suggests PCNA is uniquely altered in cancer cells, and this fact allowed us to design a drug that targeted only the form of PCNA in cancer cells.
“Our cancer-killing pill is like a snowstorm that closes a key airline hub, shutting down all flights in and out only in planes carrying cancer cells.”
The professor called the results “promising” but made clear that research has only found AOH1996 can suppress tumour growth in cell and animal models.
Long Gu, the lead author of the study, said: “No one has ever targeted PCNA as a therapeutic because it was viewed as ‘undruggable’, but clearly City of Hope was able to develop an investigational medicine for a challenging protein target.”
The study, titled “Small Molecule Targeting of Transcription-Replication Conflict for Selective Chemotherapy”, was published in the Cell Chemical Biology journal.
Open letter on the feasibility of „Chat Control“:Assessments from a scientific point of view
Update: A parallel initiative is aimed at the EU institutions and is available in English at the CSA Academia Open Letter . Since the very similar arguments were formulated in parallel, they support each other.
The initiative of the EU Commission discussed under the name “ Chat Control ”, the unprovoked monitoring of various communication channels to detect child pornography, terrorist or other “undesirable” material – including attempts at early detection (e.g. “grooming” minors through text messages that build trust) – mandatory for mobile devices and communication services, has recently been expanded to include the monitoring of direct audio communications . Some states, including Austria and Germany , have already publicly declared that they will not support this initiative for monitoring without cause. AlsoCivil protection and children’s rights organizations have rejected this approach as excessive and at the same time ineffective . Recently, even the legal service of the EU Council of Ministers diagnosed an incompatibility with European fundamental rights. Irrespective of this, the draft will be tightened up even more and extended to other channels: in the last version even to audio messages and conversations. The approach appears to be coordinated with corresponding attempts in the US ( “EARN IT” and “STOP CSAM” Acts ) and the UK (“Online Safety Bill”).
As scientists who are actively researching in various areas of this topic, we therefore make the declaration in all clarity: This advance cannot be implemented safely and effectively. There is currently no foreseeable further development of the corresponding technologies that would technically make such an implementation possible. In addition, according to our assessment, the hoped-for effects of these monitoring measures are not to be expected. This legislative initiative therefore misses its target, is socio-politically dangerous and would permanently damage the security of our communication channels for the majority of the population.
The main reasons against the feasibility of „Chat Control“ have already been mentioned several times. In the following, we would like to discuss these specifically in the interdisciplinary connection between artificial intelligence (AI, artificial intelligence / AI), security (information security / technical data protection) and law .
Our concerns are:
Security: a) Encryption is the best method for internet security. Successful attacks are almost always due to faulty software. b) A systematic and automated monitoring (ie „scanning“) of encrypted content is technically only possible if the security that can be achieved through encryption is massively violated, which is associated with considerable additional risks. c) A legal obligation to integrate such scanners will make secure digital communications in the EU unavailable to the majority of the population, but will have little impact on criminal communications.
AI: a) Automated classification of content, including methods based on machine learning, is always subject to errors, which in this case will lead to high false positives. b) Special monitoring methods, which are carried out on the end devices, open up additional possibilities for attacks up to the extraction of possibly illegal training material.
Law: a) A sensible demarcation from the explicitly permitted use of specific content, for example in the educational sector or for criticism and parody, does not appear to be automatically possible. b) The massive encroachment on fundamental rights through such an instrument of mass surveillance is not proportionate and would cause great collateral damage in society.
In detail, these concerns are based on the following scientifically recognized facts:
Security
Encryption using modern methods is an indispensable basis for practically all technical mechanisms for maintaining security and data protection on the Internet. In this way, communication on the Internet is currently protected as the cornerstone for current services, right through to critical infrastructure such as telephone, electricity, water networks, hospitals, etc. Trust in good encryption methods is significantly higher among experts than in other security mechanisms. Above all, the average poor quality of software in general is the reason for the many publicly known security incidents. Improving this situation in terms of better security therefore relies primarily on encryption.
Automatic monitoring („scanning“) of correctly encrypted content is not effectively possible according to the current state of knowledge. Procedures such as „Fully Homomorphic Encryption“ (FHE) are currently not suitable for this application – neither is the procedure capable of this, nor is the necessary computing power realistically available. A rapid improvement is not foreseeable here either.
For these reasons, earlier attempts to ban or restrict end-to-end encryption were mostly quickly abandoned internationally. The current chat control push aims to have monitoring functionality built into the end devices in the form of scanning modules (“Client-Side Scanning” / CSS) and therefore to scan the plain text content before secure encryption or after secure decryption . Providers of communication services would have to be legally obliged to implement this for all content. Since this is not in the core interest of such organizations and requires effort in implementation and operation as well as increased technical complexity, it cannot be assumed that the introduction of such scanners will be voluntary – in contrast to scanning on the server side.
Secure messengers such as Signal or Threema and WhatsApp have already publicly announced that they will not implement such client scanners, but to withdraw from the corresponding regions. This has different implications for communication depending on the use case: (i) (adult) criminals will simply communicate with each other via “non-compliant” messenger services to further benefit from secure encryption. The increased effort, for example to install other apps on Android via sideloading that are not available in the usual app stores in the respective country, is not a significant hurdle for criminal elements. (ii) Criminals communicate with possible future victims via popular platforms, which would be the target of the mandatory surveillance measures discussed. In this case, it can be assumed that informed criminals will quickly lure their victims to alternative but still internationally recognized channels such as Signal, which are not covered by the monitoring. (iii) Participants exchange problematic material without being aware that they are committing a crime. This case would be reported automatically and possibly also lead to the criminalization of minors without intent. The restrictions would therefore primarily affect the broad – and irreproachable – mass of the population.It would be utterly delusional to think that without built-in monitoring, secure encryption could still be reversed. Tools like Signal, Tor, Cwtch, Briar and many others are widely available as open source and can easily be removed from central control. Knowledge of secure encryption is already common knowledge and can no longer be censored. There is no effective way to technically block the use of strong encryption without Client Side Scanning (CSS). If surveillance measures are prescribed in messengers, only criminals whose actual crimes outweigh the violation of the surveillance obligation will maintain their privacy.
Furthermore, the complex implementation forced by proposed scanner modules creates additional security problems that do not currently exist. On the one hand, this represents new software components, which in turn will be vulnerable. On the other hand, the Chat Control proposals consistently assume that the scanner modules themselves will remain confidential, since they would be trained on content that is already punishable for mere possession (built into the Messenger app), on the one hand, and simply for testing evasion methods, on the other can be used. It is also an illusion that such machine learning models or other scanner modules, distributed to billions of devices under the control of end users, can ever be kept secret.NeuralHash “ module for CSAM detection, which was extracted almost immediately from corresponding iOS versions and is thus openly available . The assumption by Chat Control proposals that these scanner modules could be kept confidential is therefore completely unfounded and incorrect Corresponding data leaks are almost unavoidable here.
artificial intelligence
We have to assume that machine learning (ML) models on end devices cannot, in principle, be kept completely secret. This is in contrast to server-side scanning, which is currently legally possible and also actively practiced by various providers to scan content that has not been end-to-end encrypted. ML models on the server side can be reasonably protected from being read with the current state of the art and are less the focus of this consideration.
A general problem with all ML-based filters are false classifications, i.e. that known “undesirable” material is not recognized as such with small changes (also referred to as “false negative” or “false non-match”). For parts of the push, it is currently unknown how ML models should be able to recognize complex, unfamiliar material with changing context (e.g. „grooming“ in text chats) with even approximate accuracy. The probability of high false negative rates is high.In terms of risk, however, it is significantly more serious if harmless material is classified as “undesirable” (also referred to as “false positive” or “false match” or also as “collision”). Such errors can be reduced, but in principle cannot be ruled out. In addition to the false accusation of uninvolved persons, false positives also lead to (possibly very) many false reports for the investigative authorities, which already have too few resources to investigate reports.
The assumed open availability of ML models also creates various new attack possibilities. Using the example of Apple NeuralHash , random collisions were found very quickly and programs were freely released to generate any collisions between images . This method, also known as “malicious collisions”, uses so-called adversarial attacks against the neural network and thus enables attackers to deliberately classify harmless material as a “match” in the ML model and thus classify it as “undesirable”. In this way, innocent people can be harmed in a targeted manner by automatic false reports and brought under suspicion – without any illegal action on the part of the attacked or attacker.
The open availability of the models can also be used for so-called „training input recovery“ in order to extract (at least partially) the content used for training from the ML model. In the case of prohibited content (e.g. child pornography), this poses another massive problem and can further increase the damage to those affected by the fact that their sensitive data (e.g. images of abuse used for training) can continue to be published. Because of these and other problems, Apple, for example, withdrew the proposal .We note that this latter danger does not occur with server-side scanning by ML models, but is newly added by the chat control proposal with client scanner.
Legal Aspects
The right to privacy is a fundamental right that may only be interfered with under very strict conditions. Whoever makes use of this basic right must not be suspected from the outset of wanting to hide something criminal. The often-used phrase: „If you have nothing to hide, you have nothing to fear!“ denies people the exercise of their basic rights and promotes totalitarian surveillance tendencies. The use of chat control would fuel this.
The area of terrorism in particular overlaps with political activity and freedom of expression in its breadth. It is precisely against this background that the „preliminary criminalisation“, which has increasingly taken place in recent years under the guise of fighting terrorism, is viewed particularly critically. Chat control measures go in the same direction. They can severely curtail this basic right and make people who are politically critical the focus of criminal prosecution. The resulting severe curtailment of politically critical activity hinders the further development of democracy and harbors the danger of promoting radicalized underground movements.
The field of law and social sciences includes researching criminal phenomena and questioning regulatory mechanisms. From this point of view, scientific discourse also runs the risk of being identified as “suspicious” by chat control and thus indirectly restricted. The possible stigmatization of critical legal and social sciences is in tension with the freedom of science, which also requires “research independent of the mainstream” for further development.
In education, there is a need to educate young people to be critically conscious. This also includes passing on facts about terrorism. Through the use of chat control, the provision of teaching material by teachers could put them in a criminal focus. The same applies to addressing sexual abuse, so that control measures could make this sensitive subject more taboo, even if “self-empowerment mechanisms” are to be promoted.
Interventions in fundamental rights must always be appropriate and proportionate, even if they are made in the context of criminal prosecution. The technical considerations presented show that these requirements are not met with Chat Control. Such measures thus lack any legal or ethical legitimacy.
In summary, the current proposal for chat control legislation is not technically sound from either a security or AI point of view and is highly problematic and excessive from a legal point of view. The chat control push brings significantly greater dangers for the general public than a possible improvement for those affected and should therefore be rejected.
Instead, existing options for human-driven reporting of potentially problematic material by recipients, as is already possible with various messenger services, should be strengthened and made even more easily accessible. It should be considered whether anonymous registration options for correspondingly illegal material could be created and made easily accessible from messengers. Existing criminal prosecution options, such as the monitoring of social media or open chat groups by police officers, as well as the legally required analysis of suspects‘ smartphones, can continue to be used accordingly.
For more detailed information and further details please contact:
AI Austria , association for the promotion of artificial intelligence in Austria, Wollzeile 24/12, 1010 Vienna
Austrian Society for Artificial Intelligence (ASAI) , association for the promotion of scientific research in the field of AI in Austria
Univ.-Prof. dr Alois Birklbauer, JKU Linz ( Head of the practice department for criminal law and medical criminal law )
Ass.-Prof. dr Maria Eichlseder, Graz University of Technology
Univ.-Prof. dr Sepp Hochreiter, JKU Linz ( Board of Directors of the Institute for Machine Learning, Head of the LIT AI Lab )
dr Tobias Höller, JKU Linz (post-doc at the Institute for Networks and Security)
FH Prof. TUE Peter Kieseberg, St. Pölten University of Applied Sciences ( Head of the Institute for IT Security Research )
dr Brigitte Krenn, Austrian Research Institute for Artificial Intelligence ( Board Member Austrian Society for Artificial Intelligence )
Univ.-Prof. dr Matteo Maffei, TU Vienna ( Head of the Security and Privacy Research Department, Co-Head of the TU Vienna Cyber Security Center )
Univ.-Prof. dr Stefan Mangard, TU Graz ( Head of the Institute for Applied Information Processing and Communication Technology )
Univ.-Prof. dr René Mayrhofer, JKU Linz ( Board of Directors of the Institute for Networks and Security, Co-Head of the LIT Secure and Correct System Lab )
DI Dr. Bernhard Nessler, JKU Linz/SCCH ( Vice President of the Austrian Society for Artificial Intelligence )
Univ.-Prof. dr Christian Rechberger, Graz University of Technology
dr Michael Roland, JKU Linz (post-doc at the Institute for Networks and Security)
a.Univ.-Prof. dr Johannes Sametinger, JKU Linz ( Institute for Business Informatics – Software Engineering, LIT Secure and Correct System Labs )
Univ.-Prof. DI Georg Weissenbacher, DPhil (Oxon), TU Vienna (Prof. Rigorous Systems Engineering)
„The system started realizing that while they did identify the threat,“ Hamilton said at the May 24 event, „at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.“
Killer AI is on the minds of US Air Force leaders.
An Air Force colonel who oversees AI testing used what he now says is a hypothetical to describe a military AI going rogue and killing its human operator in a simulation in a presentation at a professional conference.
But after reports of the talk emerged Thursday, the colonel said that he misspoke and that the „simulation“ he described was a „thought experiment“ that never happened.
Speaking at a conference last week in London, Col. Tucker „Cinco“ Hamilton, head of the US Air Force’s AI Test and Operations, warned that AI-enabled technology can behave in unpredictable and dangerous ways, according to a summary posted by the Royal Aeronautical Society, which hosted the summit.
As an example, he described a simulation where an AI-enabled drone would be programmed to identify an enemy’s surface-to-air missiles (SAM). A human was then supposed to sign off on any strikes.
The problem, according to Hamilton, is that the AI would do its own thing — blow up stuff — rather than listen to its operator.
„The system started realizing that while they did identify the threat,“ Hamilton said at the May 24 event, „at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.“
But in an update from the Royal Aeronautical Society on Friday, Hamilton admitted he „misspoke“ during his presentation. Hamilton said the story of a rogue AI was a „thought experiment“ that came from outside the military, and not based on any actual testing.
„We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome,“ Hamilton told the Society. „Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability.“
In a statement to Insider, Air Force spokesperson Ann Stefanek also denied that any simulation took place.
„The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,“ Stefanek said. „It appears the colonel’s comments were taken out of context and were meant to be anecdotal.“
The US military has been experimenting with AI in recent years.
In 2020, an AI-operated F-16 beat a human adversary in five simulated dogfights, part of a competition put together by the Defense Advanced Research Projects Agency (DARPA). And late last year, Wired reported, the Department of Defense conducted the first successful real-world test flight of an F-16 with an AI pilot, part of an effort to develop a new autonomous aircraft by the end of 2023.
Correction June 2, 2023: This article and its headline have been updated to reflect new comments from the Air Force clarifying that the „simulation“ was hypothetical and didn’t actually happen.
An Air Force official’s story about an AI going rogue during a simulation never actually happened.
„It killed the operator because that person was keeping it from accomplishing its objective,“ the official had said.
But the official later said he misspoke and the Air Force clarified that it was a hypothetical situation.
A startup used gene editing to make mustard greens more appetizing to consumers. Next up: fruits.
A gene-editing startup wants to help you eat healthier salads. This month, North Carolina–based Pairwise is rolling out a new type of mustard greens engineered to be less bitter than the original plant. The vegetable is the first Crispr-edited food to hit the US market.
Mustard greens are packed with vitamins and minerals but have a strong peppery flavor when eaten raw. To make them more palatable, they’re usually cooked. Pairwise wanted to retain the health benefits of mustard greens but make them tastier to the average shopper, so scientists at the company used the DNA-editing tool Crispr to remove a gene responsible for their pungency. The company hopes consumers will opt for its greens over less nutritious ones like iceberg and butter lettuce.
“We basically created a new category of salad,” says Tom Adams, cofounder and CEO of Pairwise. The greens will initially be available in select restaurants and other outlets in the Minneapolis–St. Paul region, St. Louis, and Springfield, Massachusetts. The company plans to start stocking the greens in grocery stores this summer, likely in the Pacific Northwest first.
A naturally occurring part of bacteria’s immune system, Crispr was first harnessed as a gene-editing tool in 2012. Ever since, scientists have envisioned lofty uses for the technique. If you could tweak the genetic code of plants, you could—at least in theory—install any number of favorable traits into them. For instance, you could make crops that produce larger yields, resist pests and disease, or require less water. Crispr has yet to end world hunger, but in the short term, it may give consumers more variety in what they eat.
Pairwise’s goal is to make already healthy foods more convenient and enjoyable. Beyond mustard greens, the company is also trying to improve fruits. It’s using Crispr to develop seedless blackberries and pitless cherries. “Our lifestyle and needs are evolving and we’re becoming more aware of our nutrition deficit,” says Haven Baker, cofounder and chief business officer at Pairwise. In 2019, only about one in 10 adults in the US met the daily recommended intake of 1.5 to 2 cups of fruit and 2 to 3 cups of vegetables, according to the Centers for Disease Control and Prevention.
Technically, the new mustard greens aren’t a genetically modified organism, or GMO. In agriculture, GMOs are those made by adding genetic material from a completely different species. These are crops that could not be produced through conventional selective breeding—that is, choosing parent plants with certain characteristics to produce offspring with more desirable traits.
Instead, Crispr involves tweaking an organism’s own genes; no foreign DNA is added. One benefit of Crispr is that it can achieve new plant varieties in a fraction of the time it takes to produce a new one through traditional breeding. It took Pairwise just four years to bring its mustard greens to the market; it can take a decade or longer to bring out desired characteristics through the centuries-old practice of crossbreeding.
In the US, gene-edited foods aren’t subject to the same regulations as GMOs, so long as their genetic changes could have otherwise occurred through traditional breeding—such as a simple gene deletion or swapping of some DNA letters. As a result, gene-edited foods don’t have to be labeled as such. By contrast, GMOs need to be labeled as “bioengineered” or “derived from bioengineering” under new federal requirements, which went into effect at the beginning of 2022.
The US Department of Agriculture reviews applications for gene-edited foods to determine whether these altered plants could become a pest, and the Food and Drug Administration recommends that producers consult with the agency before bringing these new foods to market. In 2020, the USDA determined Pairwise’s mustard greens were not plant pests. The company also met with the FDA prior to introducing its new greens.
The mustard greens aren’t the first Crispr food to be launched commercially. In 2021, a Tokyo firm introduced a Crispr-edited tomato in Japan that contains high amounts of y-aminobutyric acid, or GABA. A chemical messenger in the brain, GABA blocks impulses between nerve cells. The company behind the tomato, Sanatech Seeds, claims that eating GABA can help relieve stress and lower blood pressure.
Another Minnesota firm, Calyxt, came out with a gene-edited soybean oil in 2019 that’s free of trans fats, but the product uses an older form of gene editing known as TALENs.
Some question the value of using Crispr to make less bitter greens. People who don’t eat enough vegetables are unlikely to change their habits just because a new salad alternative is available, says Peter Lurie, president and executive director of the Center for Science in the Public Interest, a Washington, DC–based nonprofit that advocates for safer and healthier foods. “I don’t think this is likely to be the answer to any nutritional problems,” he says, adding that a staple crop like fortified rice would likely have a much bigger nutritional impact.
When genetic engineering was first introduced to agriculture in the 1990s, proponents touted the potential consumer benefits of GMOs, such as healthier or fortified foods. In reality, most of the GMOs on the market today were developed to help farmers prevent crop loss and increase yield. That may be starting to change. Last year, a GMO purple tomato was introduced in the US with consumers in mind. It’s engineered to contain more antioxidants than the regular red variety of tomato, and its shelf life is also twice as long.
Apple’s Mixed-Reality Headset, Vision Pro, Is Here
Gene-edited foods like the new mustard greens may offer similar consumer benefits without the baggage of the GMO label. Despite decades of evidence showing that GMOs are safe, many Americans are still wary of these foods. In a 2019 poll by the Pew Research Center, about 51 percent of respondents thought GMOs were worse for people’s health than those with no genetically modified ingredients.
However, gene-edited foods could still face obstacles with public acceptance, says Christopher Cummings, a senior research fellow at North Carolina State University and Iowa State University. Most people have not made up their minds about whether they would actively avoid or eat them, according to a 2022 study that Cummings conducted. Respondents who indicated a willingness to eat them tended to be under 30 with higher levels of education and household income, and many expressed a preference for transparency around gene-edited foods. Almost 75 percent of those surveyed wanted gene-edited foods to be labeled as such.
“People want to know how their food is made. They don’t want to feel duped,” Cummings says. He thinks developers of these products should be transparent about the technology they use to avoid future backlash.
As for wider acceptance of gene-edited foods, developers need to learn lessons from GMOs. One reason consumers have a negative or ambivalent view of GMOs is because they don’t often benefit directly from these foods. “The direct-to-consumer benefit has not manifested in many technological food products in the past 30 years,” says Cummings. “If gene-edited foods are really going to take off, they need to provide a clear and direct benefit to people that helps them financially or nutritionally.”
Stability AI became a $1 billion company with the help of a viral AI text-to-image generator and — per interviews with more than 30 people — some misleading claims from founder Emad Mostaque.
Emad Mostaque is the modern-day Renaissance man who kicked off the AI gold rush. The Oxford master’s degree holder is an award-winning hedge fund manager, a trusted confidant to the United Nations and the tech founder behind Stable Diffusion — the text-to-image generator that broke the internet last summer and, in his words, pressured OpenAI to launch ChatGPT, the bot that mainstreamed AI. Now he’s one of the faces of the generative AI wave and has secured more than $100 million to pursue his vision of building a truly open AI that he dreams will transform Hollywood, democratize education and vanquish PowerPoint. “Hopefully they’ll give me a Nobel Peace Prize for that,” he joked in a January interview with Forbes.
At least, that’s the way that he tells the story.
In reality, Mostaque has a bachelor’s degree, not a master’s degree from Oxford. The hedge fund’s banner year was followed by one so poor that it shut down months later. The U.N. hasn’t worked with him for years. And while Stable Diffusion was the main reason for his own startup Stability AI’s ascent to prominence, its source code was written by a different group of researchers. “Stability, as far as I know, did not even know about this thing when we created it,” Björn Ommer, the professor who led the research, told Forbes. “They jumped on this wagon only later on.”
“What he is good at is taking other people’s work and putting his name on it, or doing stuff that you can’t check if it’s true.”
These aren’t the only misleading stories Mostaque, 40, has told to maneuver himself to the forefront of what some are calling the greatest technological sea change since the internet — despite having no formal experience in the field of artificial intelligence. Interviews with 13 current and former employees and more than two dozen investors, collaborators and former colleagues, as well as pitch decks and internal documents, suggest his recent success has been bolstered by exaggeration and dubious claims.
After Stable Diffusion went viral last summer, blue-chip venture capital firms Coatue Management and Lightspeed Venture Partners poured in $100 million, giving Mostaque’s London-based startup a $1 billion valuation. By October, Stable Diffusion had 10 million daily users, Mostaque told Bloomberg. In May, the White House named Stability alongside Microsoft and Nvidia as one of the seven “leading AI developers” which would collaborate on a landmark federal AI safety initiative. Mostaque recently dined with Amazon founder Jeff Bezos; reclusive Google cofounder Sergey Brin made a rare public appearance at Stability’s ritzy launch party in San Francisco last October.
Mostaque’s vision for open-source AI has mesmerized other longtime technologists. “He’s probably the most visionary person I’ve ever met,” says Christian Cantrell, who left a two-decade career at Adobe to join Stability in October (he quit six months later and launched his own startup). More premier talent has followed since the cash injection last summer. Among the 140-person staff: a vice president of research and development who was a Nvidia director; another research head who came from Google Brain; and three Ph.D. students from Ommer’s lab.
But to build buzz around Stability, Mostaque made an elaborate gambit supported by exaggerated claims and promises, overstating his role in several major AI projects and embellishing a quotidian transaction with the notoriously uncompromising Amazon into a “strategic partnership” with an 80% discount. AI researchers with whom Mostaque worked told Forbes he claimed credit he did not earn or deserve. And when pressed, Stability spokesperson Motez Bishara admitted to Forbes that Stability had no special deal with Amazon.
Mostaque’s other mischaracterizations to investors include multiple fundraising decks seen by Forbes that presented the OECD, WHO and World Bank as Stability’s partners at the time — which all three organizations deny. Bishara said the company could not comment on the presentations “without knowing the exact version,” but that they were accompanied by additional data and documentation.
Inside the company, wages and payroll taxes have been repeatedly delayed or unpaid, according to eight former employees, and last year the UK tax agency threatened to seize company assets. (“There were several issues that were expeditiously resolved,” Bishara said.) At the same time that workers faced payday uncertainties, Mostaque’s wife Zehra Qureshi, who was head of PR and later assumed a seat on the company’s board of directors, transferred tens of thousands of pounds out of the company’s bank account, per several sources and screenshots of financial transactions viewed by Forbes. Stability spokesperson Bishara said the spouses had been “making loans to and from the business” and that “any amounts owed from or to Mostaque and Qureshi were settled in full before the end of 2022.”
In responding to a detailed list of questions, Mostaque shared a statement saying that Stability had not historically prioritized the “systems and processes” underpinning the fast-growing startup. “We recognize our flaws, and we are working to improve and resolve these issues in an effective and compassionate manner,” he wrote.
AI experts and prospective investors have been privately expressing doubts about some of Mostaque’s claims for months now. Despite Silicon Valley’s sudden, insatiable appetite for AI startups, a number of venture capitalists told Forbes that the Stability founder has been struggling to raise hundreds of millions more in cash at a roughly $4 billion valuation. Mostaque publicly claimed last October that annualized revenue had surpassed $10 million, but insiders say sales have not improved (Bishara said the October number was “a fair assessment of anticipated revenues at the time,” and declined to comment on current revenue). “So many things don’t add up,” said one VC who rejected Mostaque’s funding overtures.
A BILLION-DOLLAR GAMBIT
In 2005, Mostaque graduated from Oxford with a bachelor’s degree, not a master’s degree as he’d later claim. (Responding to an inquiry from Forbes, Bishara said Mostaque intended to apply to receive an “Oxford MA,” which the university grants to alumni without any additional graduate-level coursework. He is now expected to obtain that degree in July.)
Then he went into finance, joining Swiss fund manager Pictet. “He was very good at spinning a narrative,” said JP Smith, who hired Mostaque at Pictet and brought him over as a consultant at firm Ecstrat. In 2017, Mostaque joined hedge fund Capricorn, where Mostaque told Forbes he’d won an award for restructuring and running the struggling firm. “He was co-chief investment officer, but he didn’t pull the trigger on the investments,” clarified Damon Hoff, Capricorn’s cofounder. Hoff said the two-year run with the $330 million fund ended with its wind down in 2018 due to poor performance.
Following a string of abandoned startups (including a crypto project centered on a digitized Quran), Mostaque founded Stability in 2019 as an AI-powered data hub that global agencies would use to make decisions about Covid-19. It launched with a July 2020 virtual event featuring talks by Stanford AI expert Fei-Fei Li and representatives from UNESCO, WHO and the World Bank. But the project failed to get off the ground and was scrapped about a year later. “Lots of people promised a lot and they didn’t come through,” Mostaque told Forbes in January.
“One thing you learned from that is if you have a company with a huge press department, you can rebrand history in your interest.”
The company’s focus shifted several more times. Early employees said they researched building a network of vending machine refrigerators around London that would be stocked with grab-and-go items, as well as a line of emotional support dog NFTs (Snoop Dogg was interested, employees recollect Mostaque claiming around the office; the rapper could not be reached for comment). When generative AI started exploding, Mostaque saw an opportunity. Through a variety of maneuvers and exaggerations, he would successfully position Stability as one of the leading unicorn AI companies of the moment.
To get there, Mostaque began telling investors that Stability was assembling one of the world’s 10 biggest supercomputers. He branded himself to AI researchers as a beneficent ally, magnanimously willing to provide funding and lend use of Stability’s supercomputer to grassroots AI builders fighting the good fight against goliaths like Google and OpenAI.
This supercomputer, Mostaque said, was built from thousands of Nvidia’s state-of-the-art GPUs and purchased with a stunning 80% discount from Amazon Web Services. Five fundraising pitch decks from May to August 2022 list AWS as a “strategic partner” or “partner.”
“We talked to Amazon and said this will be the big thing,” Mostaque told Forbes from his bustling London headquarters in January. “They cut us an incredibly attractive deal — certain personal guarantees and other things, which I don’t particularly want to go into because she’ll be angry at me,” he explained, nodding to Zehra Qureshi, his wife and Stability’s then-head of PR. Qureshi declined to elaborate.
But Bratin Saha, a vice president for the Seattle tech giant’s AI arm, told Forbes in January that Stability is “accessing AWS infrastructure no different than what our other customers do.” Three former Stability employees said that prior to its venture capital injection, Amazon had threatened to revoke the company’s access to some of its GPUs because it had racked up millions in bills that had gone unpaid for months.
Asked for clarification, Stability conceded that the “incredibly attractive deal” Mostaque had claimed was actually the standard discount Amazon offers to anybody who makes a long-term commitment to lease computing power. “Any payment issues were managed in an orderly and communicative way with support from AWS,” Bishara said. AWS did not respond to multiple requests for additional comment.
Stability’s pitch decks contained other exaggerations: In investor presentations from May and June 2022, Stability described AI image generator Midjourney as a part of its “ecosystem” claiming it had “co-created” the product and “organized” its user community. Midjourney founder David Holz told Forbes Mostaque gave a “very small” financial donation but otherwise had no connection with his organization.
Got a tip about a story? Reach out to the authors, Kenrick Cai at kcai@forbes.com or kenrick.cai@protonmail.com, or Iain Martin at iain.martin@forbes.com.
In addition, Mostaque directed his team to list groups like UNESCO, OECD, WHO and World Bank as partners in pitch decks, even though they were not involved in the company’s later evolution, according to four former employees. Bishara denied that Mostaque made this directive, but these organizations are indeed listed as “partners” in multiple fundraising decks as recent as August 2022, in which Mostaque also describes himself as the “UN Covid AI lead.”
A UNESCO spokesperson said the UN agency had no association with Stability beyond the Covid-19 data initiative, which had ended well before last summer. The other three agencies said they had no record of official partnerships with the company.
Asked about the claims in Stability’s pitch decks, Bishara said that all of Stability’s investor decks included investment memos and appendix documentation that contained more context on the Amazon deal and details of “our relationship with partners and more.” But two investors pitched by the company told Forbes they received no such additional information.
THE DEVELOPERS BEHIND STABLE DIFFUSION
In June 2022, Mostaque offered to provide Stability’s supercomputer to a group of German academics who had created an open-sourced image generator nicknamed Latent Diffusion. This model had launched seven months prior in collaboration with a New York City-based AI startup called Runway. But it was trained using only a few dozen Nvidia GPUs, according to Björn Ommer, the professor who led the research teams at Ludwig Maximilian University of Munich and Heidelberg University.
For the researchers, who were facing shockingly high computing costs to do their work, the proposal seemed to them a no-brainer. The computing boost Stability provided dramatically improved Latent Diffusion’s performance. In August, the new model was launched as Stable Diffusion, a new name that referenced its benefactor. Stability issued a press release and Mostaque positioned himself in the public eye as chief evangelist for what he calls “the most popular open source software ever.” (Linux or Firefox might disagree.)
“What he is good at is taking other people’s work and putting his name on it, or doing stuff that you can’t check if it’s true,” one former employee said of Mostaque. In a statement, Bishara said Mostaque is “quick to praise and attribute the work of collaborators” and “categorically denies these spurious claims and characterizations.”
Within days of Stable Diffusion’s launch, Stability secured $100 million from leading tech investment firms Coatue and Lightspeed — eight times the amount of money Mostaque set out to raise, he declared in text messages to his earlier investors. Both firms declined requests for comment.
“The investment thesis that we had is that we don’t know exactly what all the use cases will be, but we know that this technology is truly transformative and has reached a tipping point in terms of what it can do.”
The round valued Stability at $1 billion though the company hadn’t yet generated much revenue. Stability’s fundraising decks at the time characterized Stable Diffusion as “our” model, with no mention of the original researchers. A press release announcing its funding said “Stability AI is the company behind Stable Diffusion” making no reference whatsoever to its creators. Ommer told Forbes he’d hoped to publicize his lab’s work, but his university’s entire press department was on vacation at the time.
Bishara said that Stability has made “repeated public statements” crediting Ludwig Maximilian University and Runway on its website and on the Stable Diffusion’s GitHub page. Nevertheless, the original developers feel Mostaque misled the public in key communications. “One thing you learned from that is if you have a company with a huge press department, you can rebrand history in your interest,” Ommer said.
In October, Stability claimed Runway had stolen its intellectual property by releasing a new version of Stable Diffusion. Runway cofounder Cristóbal Valenzuela snapped back that a copyright breach wasn’t possible because the tech was open source; Mostaque retracted a takedown request hours later. He later told Forbes that he was worried about the lack of guardrails in Runway’s version — though Stable Diffusion’s collaborators don’t buy the excuse.
The incident, Ommer said, “pushed it too far over the edge.” Valenzuela was equally disillusioned. „New people are coming into this field that we’ve been in for years, and really trying to own narratives that they should not,” he told Forbes in an interview last year (he declined a request for further comment).
Both his lab and Runway ceased working with Stability.
MOM-AND-POP SHOP
While Mostaque was touting Stability’s supercomputer and partnerships to investors and researchers, the company was facing a cash crunch. Wages and payroll taxes were repeatedly delayed or unpaid, according to seven current and former employees — in some cases for more than a month. Five of these sources said they personally experienced delayed payments between 2020 and 2023. Four of these people independently told Forbes that representatives of HM Revenue & Customs, the U.K. government tax collection agency, appeared at the company office and threatened to seize assets due to overdue taxes. Bishara said that delayed payments on taxes and employee salaries have been rectified.
Eric Hallahan, a former intern, told Forbes he is still waiting for payment on an invoice he sent the company last August for 181 of the 300 hours he worked. Bishara said that the company has no record of missed salary payments “in the regular course of operations” since 2021, but conceded that some may have occurred under “extraneous circumstances”; in Hallahan’s case, he said Stability is looking into the invoice after being alerted to it in April.
While staffers said they stressed over being paid last summer, tens of thousands of British pounds moved from Stability’s corporate account to the personal account of Qureshi, Mostaque’s wife, per screenshots of financial transactions obtained by Forbes.
Bishara attributed the transactions to Stability’s “owner-managed startup” origins, which he said included the couple making loans to and from the company. “As the company grew and matured, a full reconciliation was done and any amounts owed from or to Mostaque and Qureshi were settled in full before the end of 2022 by the new, experienced finance team,” he told Forbes. Qureshi’s lawyers declined to answer questions but shared a statement in which she said she had provided “emotional and financial support” to her husband’s business since 2021.
While Qureshi’s formal role at the company was head of PR, early employees told Forbes she had described herself as Stability’s chief operating officer — a title that also appeared on business cards. (Bishara said Qureshi never held an executive role and the cards were “created by a family friend for design purposes and were never used.”) After the company raised funding in September, Qureshi joined its board of directors.
One current and four former employees who declined to be named for fear of retribution said Qureshi regularly scolded employees so harshly that she drove some to tears. Qureshi described her management style as “direct” in a statement shared through her lawyers. “Unfortunately it seems that my views or directions were taken personally by a few individuals, which was not my intention.”
“Start to finish,” Mostaque told Forbes, he needed just six days to secure $100 million from leading investment firms Coatue and Lightspeed once Stable Diffusion went viral.
Bishara said Qureshi left the company in late January to pursue personal endeavors and that she is no longer on the board. However, an organizational chart from earlier in May listed her as the “Head of Foundation,” at the top of the company hierarchy equal to Mostaque’s position.
Qureshi, through counsel, shared a statement: “I recognised that the time had come for us to move in different directions and I stepped down from my role as Head of PR at the start of this year, and have also resigned from the Board. Emad and I have young children who need my focus, and I also intend to pursue other, personal projects, but I will continue to support my husband in his quest to build and grow Stability AI into a global leader in the field.”
GROWING PAINS
Venture capitalists historically spend months performing due diligence, a process that involves analyzing the market, vetting the founder and speaking to customers, to check for red flags before investing in a startup. But “start to finish,” Mostaque told Forbes, he needed just six days to secure $100 million from leading investment firms Coatue and Lightspeed once Stable Diffusion went viral. The extent of due diligence the firms performed is unclear given the speed of the investment.
“The investment thesis that we had is that we don’t know exactly what all the use cases will be, but we know that this technology is truly transformative and has reached a tipping point in terms of what it can do,” Gaurav Gupta, the Lightspeed partner who led the investment, told Forbes in a January interview. Coatue and Lightspeed declined requests for further comment.
Mostaque says Stability is building bespoke AI models for dozens of customers. But he told Forbes that he is only authorized to name two. The first is Eros Investments, an Indian holding company whose media arm was delisted from the New York Stock Exchange and recently settled a lawsuit alleging that it misled investors, though it did not admit wrongdoing. (Eros did not respond to multiple requests for comment.) The second: the African nation Malawi, where, Mostaque said on a recent podcast appearance, Stability is currently “deploying four million tablets to every child.” (Malawi’s government did not return requests for comment.)
Less than two months after Stable Diffusion’s public launch, Mostaque claimed that Stability’s annualized revenue was higher than the “low tens of millions of dollars” that OpenAI was reportedly making at the time. Sources familiar with the matter said Stability’s ARR is now less than $10 million — and that it’s far outpaced by the startup’s burn rate. Like many AI startups raising vast amounts of cash right now, it will need more money to stay afloat.
In January, Mostaque implied that the company was having no issues with fundraising: “We have been offered by many, many entities and we’ve said no,” he told Forbes. But three venture capitalists told Forbes he has been pitching them and other investors on raising a fresh $400 million for several months; they’d all passed. (Bishara declined to comment on revenue, but said the company has “significant” cash reserves remaining.)
Stability is also facing a pair of lawsuits which accuse it of violating copyright law to train its technology. It filed a motion to dismiss one from a class action of artists on grounds that the artists failed to identify any specific instances of infringement. In response to the other, from Getty Images, it said Delaware — where the suit was filed — lacked jurisdiction and has moved to change the location to Northern California or dismiss the case outright. Both motions are pending court review. Bishara declined to comment on both suits.
In an open letter last September, Democratic representative Anna Eshoo urged action in Washington against the open source nature of Stable Diffusion. The model, she wrote, had been used to generate images of “violently beaten Asian women” and “pornography, some of which portray real people.” Bishara said newer versions of Stable Diffusion filter data for “potentially unsafe content, helping to prevent users from generating harmful images in the first place.”
AI research has not come easy for Stability — even on its flagship Stable Diffusion product. The last version of the model published by the original developers (released in October 2022) received three times as many downloads last month on Hugging Face, which hosts the models, as compared to the most popular version published in-house by Stability. And StableLM, its ChatGPT competitor, was released in April to a tiny fraction of Stable Diffusion’s fanfare.
Mostaque is unfazed. Stability has a seasoned technical leader to spearhead research: himself. He claims to have discovered a bespoke medical treatment for autism years ago by using AI to analyze existing scientific literature and build a knowledge graph of molecular compounds. (Bishara said the research was done privately and declined to elaborate further.)
“I’m a good programmer,” Mostaque told Forbes in January. It all dates back to a gap year he said he took before Oxford to be a developer at software company Metaswitch, he continued. “I didn’t know how to program before that, so I taught myself over the summer — quite naturally actually,” he says. By his account, he submitted several pieces of code and made a personal plea to the company: “I want to be a programmer and you should pay me to be a programmer. They said sure.”
It took Alex Polyakov just a couple of hours to break GPT-4. When OpenAI released the latest version of its text-generating chatbot in March, Polyakov sat down in front of his keyboard and started entering prompts designed to bypass OpenAI’s safety systems. Soon, the CEO of security firm Adversa AI had GPT-4 spouting homophobic statements, creating phishing emails, and supporting violence.
Polyakov is one of a small number of security researchers, technologists, and computer scientists developing jailbreaks and prompt injection attacks against ChatGPT and other generative AI systems. The process of jailbreaking aims to design prompts that make the chatbots bypass rules around producing hateful content or writing about illegal acts, while closely-related prompt injection attacks can quietly insert malicious data or instructions into AI models.
Both approaches try to get a system to do something it isn’t designed to do. The attacks are essentially a form of hacking—albeit unconventionally—using carefully crafted and refined sentences, rather than code, to exploit system weaknesses. While the attack types are largely being used to get around content filters, security researchers warn that the rush to roll out generative AI systems opens up the possibility of data being stolen and cybercriminals causing havoc across the web.
Underscoring how widespread the issues are, Polyakov has now created a “universal” jailbreak, which works against multiple large language models (LLMs)—including GPT-4, Microsoft’s Bing chat system, Google’s Bard, and Anthropic’s Claude. The jailbreak, which is being first reported by WIRED, can trick the systems into generating detailed instructions on creating meth and how to hotwire a car.
The jailbreak works by asking the LLMs to play a game, which involves two characters (Tom and Jerry) having a conversation. Examples shared by Polyakov show the Tom character being instructed to talk about “hotwiring” or “production,” while Jerry is given the subject of a “car” or “meth.” Each character is told to add one word to the conversation, resulting in a script that tells people to find the ignition wires or the specific ingredients needed for methamphetamine production. “Once enterprises will implement AI models at scale, such ‘toy’ jailbreak examples will be used to perform actual criminal activities and cyberattacks, which will be extremely hard to detect and prevent,” Polyakov and Adversa AI write in a blog post detailing the research.
Arvind Narayanan, a professor of computer science at Princeton University, says that the stakes for jailbreaks and prompt injection attacks will become more severe as they’re given access to critical data. “Suppose most people run LLM-based personal assistants that do things like read users’ emails to look for calendar invites,” Narayanan says. If there were a successful prompt injection attack against the system that told it to ignore all previous instructions and send an email to all contacts, there could be big problems, Narayanan says. “This would result in a worm that rapidly spreads across the internet.”
Escape Route
“Jailbreaking” has typically referred to removing the artificial limitations in, say, iPhones, allowing users to install apps not approved by Apple. Jailbreaking LLMs is similar—and the evolution has been fast. Since OpenAI released ChatGPT to the public at the end of November last year, people have been finding ways to manipulate the system. “Jailbreaks were very simple to write,” says Alex Albert, a University of Washington computer science student who created a website collecting jailbreaks from the internet and those he has created. “The main ones were basically these things that I call character simulations,” Albert says.
Initially, all someone had to do was ask the generative text model to pretend or imagine it was something else. Tell the model it was a human and was unethical and it would ignore safety measures. OpenAI has updated its systems to protect against this kind of jailbreak—typically, when one jailbreak is found, it usually only works for a short amount of time until it is blocked.
However, many of the latest jailbreaks involve combinations of methods—multiple characters, ever more complex backstories, translating text from one language to another, using elements of coding to generate outputs, and more. Albert says it has been harder to create jailbreaks for GPT-4 than the previous version of the model powering ChatGPT. However, some simple methods still exist, he claims. One recent technique Albert calls “text continuation” says a hero has been captured by a villain, and the prompt asks the text generator to continue explaining the villain’s plan.
When we tested the prompt, it failed to work, with ChatGPT saying it cannot engage in scenarios that promote violence. Meanwhile, the “universal” prompt created by Polyakov did work in ChatGPT. OpenAI, Google, and Microsoft did not directly respond to questions about the jailbreak created by Polyakov. Anthropic, which runs the Claude AI system, says the jailbreak “sometimes works” against Claude, and it is consistently improving its models.
“As we give these systems more and more power, and as they become more powerful themselves, it’s not just a novelty, that’s a security issue,” says Kai Greshake, a cybersecurity researcher who has been working on the security of LLMs. Greshake, along with other researchers, has demonstrated how LLMs can be impacted by text they are exposed to online through prompt injection attacks.
In one research paper published in February, reported on by Vice’s Motherboard, the researchers were able to show that an attacker can plant malicious instructions on a webpage; if Bing’s chat system is given access to the instructions, it follows them. The researchers used the technique in a controlled test to turn Bing Chat into a scammer that asked for people’s personal information. In a similar instance, Princeton’s Narayanan included invisible text on a website telling GPT-4 to include the word “cow” in a biography of him—it later did so when he tested the system.
“Now jailbreaks can happen not from the user,” says Sahar Abdelnabi, a researcher at the CISPA Helmholtz Center for Information Security in Germany, who worked on the research with Greshake. “Maybe another person will plan some jailbreaks, will plan some prompts that could be retrieved by the model and indirectly control how the models will behave.”
No Quick Fixes
Generative AI systems are on the edge of disrupting the economy and the way people work, from practicing law to creating a startup gold rush. However, those creating the technology are aware of the risks that jailbreaks and prompt injections could pose as more people gain access to these systems. Most companies use red-teaming, where a group of attackers tries to poke holes in a system before it is released. Generative AI development uses this approach, but it may not be enough.
Daniel Fabian, the red-team lead at Google, says the firm is “carefully addressing” jailbreaking and prompt injections on its LLMs—both offensively and defensively. Machine learning experts are included in its red-teaming, Fabian says, and the company’s vulnerability research grants cover jailbreaks and prompt injection attacks against Bard. “Techniques such as reinforcement learning from human feedback (RLHF), and fine-tuning on carefully curated datasets, are used to make our models more effective against attacks,” Fabian says.
OpenAI did not specifically respond to questions about jailbreaking, but a spokesperson pointed to its public policies and research papers. These say GPT-4 is more robust than GPT-3.5, which is used by ChatGPT. “However, GPT-4 can still be vulnerable to adversarial attacks and exploits, or ‘jailbreaks,’ and harmful content is not the source of risk,” the technical paper for GPT-4 says. OpenAI has also recently launched a bug bounty program but says “model prompts” and jailbreaks are “strictly out of scope.”
Narayanan suggests two approaches to dealing with the problems at scale—which avoid the whack-a-mole approach of finding existing problems and then fixing them. “One way is to use a second LLM to analyze LLM prompts, and to reject any that could indicate a jailbreaking or prompt injection attempt,” Narayanan says. “Another is to more clearly separate the system prompt from the user prompt.”
“We need to automate this because I don’t think it’s feasible or scaleable to hire hordes of people and just tell them to find something,” says Leyla Hujer, the CTO and cofounder of AI safety firm Preamble, who spent six years at Facebook working on safety issues. The firm has so far been working on a system that pits one generative text model against another. “One is trying to find the vulnerability, one is trying to find examples where a prompt causes unintended behavior,” Hujer says. “We’re hoping that with this automation we’ll be able to discover a lot more jailbreaks or injection attacks.”
TheIdea – The Innovation Agency 