5,000 AI-Controlled Fake X Accounts Linked to China Disinformation Campaign, Researchers Reveal

cross-posted from: https://feddit.org/post/1940634

Archived link

5,000 AI-Controlled Fake X Accounts Linked to China Disinformation Campaign

Researchers have uncovered a network of at least 5,000 fake X (formerly Twitter) accounts that appear to be controlled by AI in a disinformation campaign linked to China – and the activity appears to be heating up as the U.S. election approaches.

The X disinformation network, dubbed “Green Cicada” by researchers, “primarily engages with divisive U.S. political issues and may plausibly be staged to interfere in the upcoming presidential election.”

The network has also amplified divisive political issues in other democracies, including Australia, western Europe, India, Japan and other democratic countries.

The finding is the latest example of attempted interference in the U.S. presidential election, which just this month has seen reports of increasing activity by Iran.

[...]

The researchers, from CyberCX. [...] said the network is unlikely to be very effective in its current state, but they added that it “is plausible that the network operators are preparing to increase activities in the lead up to the U.S. presidential election.”

Most accounts on the network are currently dormant, but activity increased sharply in July. The network has been rectifying operational errors over time – including reducing malformed outputs – which could make its activities more effective and harder to detect over time.

The network uses a Chinese-language LLM system and links to an AI researcher affiliated with Tsinghua University and Zhipu AI, a prominent Chinese AI company. So far the actors haven’t had specific political leanings, but instead have focused on amplification of divisive content, “consistent with China’s information operation playbook,” the researchers said.

[...]

The researchers said [that] "our findings also indicate* key gaps in X’s willingness and ability to detect inauthentic content. *While we have observed X taking sporadic action against Green Cicada Network accounts during our period of monitoring, we have observed a failure to take systemic action against overtly linked accounts."

“We note that X has reversed initiatives put in place by Twitter to combat inauthentic activity, including efforts to detect, label and/or ban inauthentic accounts.”

The researchers said the network is a sign of things to come, with generative AI able to produce “a significant scale of malicious output with limited human oversight, at low cost and with low barriers to entry. It is possible that the system underpinning the network is operated by high-end consumer-grade hardware and is developed by just one individual.

“We assess that a more mature, future version of the system underlying the Green Cicada Network would be extremely difficult for parties other than X to detect.”