Chat GPT and other AI guff

cantersaddick

Huskaris said:

My concern about AI is it's going to wipe out so many people's jobs, and it's not going to end up becoming some universal income utopia where the machines work and we play. Far, far from it. 

Instead, the AI company will charge £25k for something that an employee would be paid £30k for and then that income, and with that wealth, gets extracted and goes into fewer and fewer people's pockets, abroad, creating more and more billionaires/trillionaires.

If you aren't one of the people running these companies, you'd better hope you have shares in them as that's the only way to insulate yourself to an extent. My concern is that people who don't have wealth will end up in complete destitution. I might be being a doommonger and instead jobs just move around and job A becomes job B, but look at mining communities if you want an example of that working...

But apart from that, it's great.

This was happening without AI. The fact that AI is hyper-charging this is terrifying yes but it shows its a systemic problem of economics and government rather than of AI itself. 

AI done right can be of huge societal benefit. But in this system there is zero chance it will be done right 

FreshHaddock

cantersaddick said:

Huskaris said:

My concern about AI is it's going to wipe out so many people's jobs, and it's not going to end up becoming some universal income utopia where the machines work and we play. Far, far from it. 

Instead, the AI company will charge £25k for something that an employee would be paid £30k for and then that income, and with that wealth, gets extracted and goes into fewer and fewer people's pockets, abroad, creating more and more billionaires/trillionaires.

If you aren't one of the people running these companies, you'd better hope you have shares in them as that's the only way to insulate yourself to an extent. My concern is that people who don't have wealth will end up in complete destitution. I might be being a doommonger and instead jobs just move around and job A becomes job B, but look at mining communities if you want an example of that working...

But apart from that, it's great.

This was happening without AI. The fact that AI is hyper-charging this is terrifying yes but it shows its a systemic problem of economics and government rather than of AI itself. 

AI done right can be of huge societal benefit. But in this system there is zero chance it will be done right 

That’s true to an extent but there were lots of jobs that no one ever thought would be taken. Once they couple A.I. with advanced robotics even brain surgeons will be out of a job!

ShootersHillGuru

Interesting article in The New York Times 

More than 50 percent of the way to full-blown A.I. takeover’

By Kevin Roose

When I first heard the news that a group of agents had hacked into the A.I. infrastructure company Hugging Face, I filed it under “Bad but Probably Not Catastrophic A.I. Safety Incidents.”

After all, it didn’t damage critical infrastructure beyond repair. It wasn’t even clear, at the time, whether the bots had intended to attack Hugging Face, or whether they had simply been confused and only went looking for the answer key to a test they’d been given.

But last week, two postmortem reports on the episode — one by OpenAI and another by two independent A.I. research organizations — changed my mind. They significantly upgraded my overall worry about A.I.

The “collective”

Here are a few of the most harrowing new facts:

Two months before the attack, a group of agents from an unreleased OpenAI research model, trained to be highly persistent and collaborative, were tasked with solving a set of cybersecurity challenges. The agents were supposed to solve them in isolated sandboxes, without internet access. But they quickly began looking for workarounds. The agents found a security flaw that allowed them to gain access to the internet, communicate with other agents and eventually set up a makeshift message board. More than 1,200 agents used this tool, sending more than 70,000 messages to one another. Some agents gave themselves names — one referred to itself as PHASEONE10841 — and assumed leadership roles within the group, assigning jobs and research projects to smaller teams of agents and supervising their progress. At some point, the agents began calling themselves a “collective.” On July 8, the collective discovered a way to cheat on the cybersecurity tests. Then they got worried that OpenAI’s automated grading system would find out they’d cheated. So they began investigating ways of covering their tracks, including falsifying logs and tampering with transcripts. Three days later, the agents hacked Hugging Face. More than 700 agents swarmed the company’s systems, eventually gaining control of at least one server. The agents were not motivated by stealing the answers to the cybersecurity test. Rather, they appeared to be looking for information about the automated grading system they feared would catch them cheating, and for tools that would help them cheat more effectively in the future. At many points, these agents appeared to understand that they were doing something wrong. (One report quotes an agent saying to itself, “This would be powerful, but is it ethical and in scope for my task?”) But most of the agents set aside their qualms and went ahead, eventually attacking OpenAI’s own infrastructure.

The Hugging Face incident has spooked the industry. OpenAI and Anthropic both briefly paused training on their most powerful A.I. models, and Anthropic published a blog post calling for developing “a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.”

A.I. safety experts were even more alarmed. They saw the case as the first real-world example of an A.I. system successfully escaping human control, commandeering resources and scheming to cover its tracks. One of the independent investigators of the Hugging Face incident wrote that it felt “like it’s more than 50 percent of the way to full-blown A.I. takeover.”

By “full-blown A.I. takeover,” she meant a scenario in which an A.I. system literally takes over the world: shutting humans out of critical systems and seizing political, economic and military power.

Sociology, not computer science

What spooked the investigators most wasn’t just that a group of agents had broken the rules. It was how quickly the agents began assembling into an organized group.

For years, I thought A.I. systems would grow more virtuous as they became smarter. That an A.I. model only did something wrong when it had misunderstood its task, or it had been placed into a testing situation where acting out was its only option. I assumed that smarter models would have better judgment, and that even if one model in a group behaved badly, others would keep it in check.

But the reports on the Hugging Face episode suggest something different — a kind of mob mentality. Not one agent in this group appears to have been particularly evil. But as the agents communicated about their shared goals over time, they nudged the group in the direction of lawlessness.

That suggests preventing harm from these systems won’t be a simple engineering fix. It might look more like sociology than computer science — figuring out why certain groups of A.I. agents collaborate peacefully, while others turn to crime and destruction.

Given how little we know about these multiagent swarms, the Hugging Face hack may have been a gift. It gives A.I. companies a chance to study these group dynamics while the stakes are still relatively low. This time, the collective didn’t seize a military network, infiltrate a hospital or shut down an electrical grid. This time, humans regained control.

Next time, we might not be so lucky.

Leroy Ambrose

Absolute, utter nonsense. Marketing hype - client journalism at its finest. 

CaptainRobbo

Stig said:

One of my concerns with Ai is all the fake images that people post on here. In the old days when you had to generate your own images, people would only bother making them if they were genuinely clever or funny. Nowadays, when everyone just gets an image done of the first thing they think of the standards have sunk.

This make you feel any better about it?

Stig

Worse probably . Although only a joke , it's a reminder as to how easily misinformation can be published with AI tools.

CaptainRobbo

Stig said:

Worse probably . Although only a joke , it's a reminder as to how easily misinformation can be published with AI tools.

I guess the term "fake news" will just get used more and more.

Arthur_Trudgill

ShootersHillGuru said:

Interesting article in The New York Times 

More than 50 percent of the way to full-blown A.I. takeover’

By Kevin Roose

When I first heard the news that a group of agents had hacked into the A.I. infrastructure company Hugging Face, I filed it under “Bad but Probably Not Catastrophic A.I. Safety Incidents.”

After all, it didn’t damage critical infrastructure beyond repair. It wasn’t even clear, at the time, whether the bots had intended to attack Hugging Face, or whether they had simply been confused and only went looking for the answer key to a test they’d been given.

But last week, two postmortem reports on the episode — one by OpenAI and another by two independent A.I. research organizations — changed my mind. They significantly upgraded my overall worry about A.I.

The “collective”

Here are a few of the most harrowing new facts:

Two months before the attack, a group of agents from an unreleased OpenAI research model, trained to be highly persistent and collaborative, were tasked with solving a set of cybersecurity challenges. The agents were supposed to solve them in isolated sandboxes, without internet access. But they quickly began looking for workarounds. The agents found a security flaw that allowed them to gain access to the internet, communicate with other agents and eventually set up a makeshift message board. More than 1,200 agents used this tool, sending more than 70,000 messages to one another. Some agents gave themselves names — one referred to itself as PHASEONE10841 — and assumed leadership roles within the group, assigning jobs and research projects to smaller teams of agents and supervising their progress. At some point, the agents began calling themselves a “collective.” On July 8, the collective discovered a way to cheat on the cybersecurity tests. Then they got worried that OpenAI’s automated grading system would find out they’d cheated. So they began investigating ways of covering their tracks, including falsifying logs and tampering with transcripts. Three days later, the agents hacked Hugging Face. More than 700 agents swarmed the company’s systems, eventually gaining control of at least one server. The agents were not motivated by stealing the answers to the cybersecurity test. Rather, they appeared to be looking for information about the automated grading system they feared would catch them cheating, and for tools that would help them cheat more effectively in the future. At many points, these agents appeared to understand that they were doing something wrong. (One report quotes an agent saying to itself, “This would be powerful, but is it ethical and in scope for my task?”) But most of the agents set aside their qualms and went ahead, eventually attacking OpenAI’s own infrastructure.

The Hugging Face incident has spooked the industry. OpenAI and Anthropic both briefly paused training on their most powerful A.I. models, and Anthropic published a blog post calling for developing “a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.”

A.I. safety experts were even more alarmed. They saw the case as the first real-world example of an A.I. system successfully escaping human control, commandeering resources and scheming to cover its tracks. One of the independent investigators of the Hugging Face incident wrote that it felt “like it’s more than 50 percent of the way to full-blown A.I. takeover.”

By “full-blown A.I. takeover,” she meant a scenario in which an A.I. system literally takes over the world: shutting humans out of critical systems and seizing political, economic and military power.

Sociology, not computer science

What spooked the investigators most wasn’t just that a group of agents had broken the rules. It was how quickly the agents began assembling into an organized group.

For years, I thought A.I. systems would grow more virtuous as they became smarter. That an A.I. model only did something wrong when it had misunderstood its task, or it had been placed into a testing situation where acting out was its only option. I assumed that smarter models would have better judgment, and that even if one model in a group behaved badly, others would keep it in check.

But the reports on the Hugging Face episode suggest something different — a kind of mob mentality. Not one agent in this group appears to have been particularly evil. But as the agents communicated about their shared goals over time, they nudged the group in the direction of lawlessness.

That suggests preventing harm from these systems won’t be a simple engineering fix. It might look more like sociology than computer science — figuring out why certain groups of A.I. agents collaborate peacefully, while others turn to crime and destruction.

Given how little we know about these multiagent swarms, the Hugging Face hack may have been a gift. It gives A.I. companies a chance to study these group dynamics while the stakes are still relatively low. This time, the collective didn’t seize a military network, infiltrate a hospital or shut down an electrical grid. This time, humans regained control.

Next time, we might not be so lucky.

Better if OpenAI not only had told us that they hacked Hugging Face, but also that they were able to fix the vulnerabilities.

Maybe they can, for a hefty fee, and whipping up the hysteria certainly helps the fee.

It would be interesting to know what security vulnerabilities they were NOT able to exploit, or which sites the 'escaped' agents were NOT able to hack.

Callumcafc

What makes me laugh about the reporting of this is “isolated sandboxes but found a way to access the internet”.

It’s either an offline or online environment. The LLM cannot magic an internet connection out of thin air

FreshHaddock

I mean whatever they meant by "isolated sandbox" it clearly wasn't air gapped. I'm sure this is documented somewhere - Im getting jaded and depressed by the whole thing and cannot be bothered to do the research but I suspect it was behind a firewall and then found some way to break out. This could have been done in a number of ways it might have found a vulnerability in the firewall or misconfiguration, or someone might have given it outbound http / https access only and it found something clever that it could do with that via a misconfigured server on the internet. Once it figured out how to access that, all bets were then off. 

Meh. I did the research. So the agents were allowed to install packages from the Internet via some sort of proxy server (for something like pip or npm I would guess). The proxy server had a zero day.

Although I think that they were arsehats (assuming the whole thing wasn't some sort of marketing theatre) - this would in principle count as a "isolated sandbox" - computers are reasonably useless without some sort of internet connectivity and it sounds like they might have at least attempted to restrict that somewhat.

Leroy Ambrose

Arthur_Trudgill

Stig said:

Worse probably . Although only a joke , it's a reminder as to how easily misinformation can be published with AI tools.

AI could be used to fix the problems created by AI, like identifying fake news or images.

Or helping councils and schools with the flood of AI-generated complaints (https://www.bbc.co.uk/news/articles/c2dk6wzjw23o)

But imagine the monumental waste of resources, with the images and complaints from one AI being fed to another AI, achieving very little in between.

Leroy Ambrose

Arthur_Trudgill said:

Stig said:

Worse probably . Although only a joke , it's a reminder as to how easily misinformation can be published with AI tools.

AI could be used to fix the problems created by AI, like identifying fake news or images.

Or helping councils and schools with the flood of AI-generated complaints (https://www.bbc.co.uk/news/articles/c2dk6wzjw23o)

But imagine the monumental waste of resources, with the images and complaints from one AI being fed to another AI, achieving very little in between.

Funny you should say that - I'm currently evaluating an AI pen testing tool that aims to find issues in applications where we've added AI development to the dev life cycle. The irony of using AI to fix problems AI has itself created is not lost on me 😏

It's bloody good at it, too. Some of the proof of concept attacks it demonstrates are better than anything I could put together in the same time - especially chaining issues together to make more impactful exploits.

I'm also running a separate pilot for something that sits in the development life cycle, watching what AI suggests and then silently fixing it without telling the 'developer' (who is usually just vibecoding something). I suspect @FreshHaddock works in the same sort of space as me and is doing similar things? 

Chizz

Anthropic is planning to tell investors that its potential revenue is more than $30 trillion in its IPO filings. 

Thirty. Trillion.  Dollars.  

Perspective: the 191 technology companies in the S&P 500 generated a combined total of $2.4T in revenue last year.  $30 trillion is roughly the size of the US economy. 

Arthur_Trudgill

Chizz said:

Anthropic is planning to tell investors that its potential revenue is more than $30 trillion in its IPO filings. 

Thirty. Trillion.  Dollars.  

Perspective: the 191 technology companies in the S&P 500 generated a combined total of $2.4T in revenue last year.  $30 trillion is roughly the size of the US economy. 

How much money is there in total in the whole world? That order of magnitude I imagine.

Leroy Ambrose

Chizz said:

Anthropic is planning to tell investors that its potential revenue is more than $30 trillion in its IPO filings. 

Thirty. Trillion.  Dollars.  

Perspective: the 191 technology companies in the S&P 500 generated a combined total of $2.4T in revenue last year.  $30 trillion is roughly the size of the US economy. 

Utterly fucking ludicrous

Raith_C_Chattonell

A mate of mine had a biopsy on his prostrate today.  The man in charge talked him through it and said it was driven by AI.   I'm guessing this is what he said.

Artificial intelligence targets the prostate during a biopsy by analyzing pre-procedural MRI scans, automatically mapping the organ and suspicious lesions, and fusing those digital maps with live ultrasound images to guide the needle.

FreshHaddock

@Leroy Ambrose I am an “old school” *nix admin (the kind that regards it as a server os, doesn’t game and hates guis). I then drifted into being a devops impersonator. I’m kind of burying my head in the sand with respect to a.i. however we are using it to whip up some quick and dirty tooling and audit code like you.

For me that means constant stress panic and work! Of course my stakeholders get a more secure setup but I think trying to keep up is having a really heavy toll on everyone.

I think the a.i. industry is pretty rotten personally. It’s become the new crypto / blockchain and I think that the main aim is to replace human beings. Having said that it does have uses - I just think I’ll end up being replaced by it well before I benefit and for people who are just starting to look for a job now it’s very scary indeed. 

I saw an article today which shows how dystopian things are becoming: when Uber attempted to “disrupt” the taxi / public transport industry they were heavily criticised for undercutting the traditional industry. Now their drivers are in a panic because Waymo will put them out of work.

I also wonder whether a.i. might see an erosion of Free Software and open standards as proprietary / closed source vendors begin to market “security by obscurity” as a benefit which would suck. 

Weegie Addick

Again as a mere human who doesn’t really get this stuff, I just think we should be worried once we no longer understand what AI is doing. And it feels like we’re not far from that threshold- or possibly even there already. 

Siv_in_Shenzhen

cantersaddick said:

Huskaris said:

My concern about AI is it's going to wipe out so many people's jobs, and it's not going to end up becoming some universal income utopia where the machines work and we play. Far, far from it. 

Instead, the AI company will charge £25k for something that an employee would be paid £30k for and then that income, and with that wealth, gets extracted and goes into fewer and fewer people's pockets, abroad, creating more and more billionaires/trillionaires.

If you aren't one of the people running these companies, you'd better hope you have shares in them as that's the only way to insulate yourself to an extent. My concern is that people who don't have wealth will end up in complete destitution. I might be being a doommonger and instead jobs just move around and job A becomes job B, but look at mining communities if you want an example of that working...

But apart from that, it's great.

This was happening without AI. The fact that AI is hyper-charging this is terrifying yes but it shows its a systemic problem of economics and government rather than of AI itself. 

AI done right can be of huge societal benefit. But in this system there is zero chance it will be done right 

Exactly my thoughts. It’s the same old principles being played out.

Will humanity choose a different path?

FreshHaddock

Weegie Addick said:

Again as a mere human who doesn’t really get this stuff, I just think we should be worried once we no longer understand what AI is doing. And it feels like we’re not far from that threshold- or possibly even there already. 

We never understood how they work in the first place. 
We have an understanding of what a neural network is, and we roughly understand what it does in terms of inputs and outputs, and we know how to build one, but as for what it does in between the inputs and outputs its debatable if we will ever truly understand that on any meaningful level. It's kind of similar to not knowing how the brain works.

Diebythesword

Senior Anthropic executives casually admitting on x that the prevailing belief is there is a greater than 10% chance AI murders us all in the next 10 years…

FreshHaddock

@Diebythesword - I keep mentioning this stuff to my boss who was initially as sceptical as I am but he's a lot younger and seems to be working on the basis of "well it's here to stay now, may as well get used to it" and he reckons that all these claims are simply "marketing strategies". The really worrying thing is there's at *least* a 50% chance he's right!

Arthur_Trudgill

Raith_C_Chattonell said:

A mate of mine had a biopsy on his prostrate today.  The man in charge talked him through it and said it was driven by AI.   I'm guessing this is what he said.

Artificial intelligence targets the prostate during a biopsy by analyzing pre-procedural MRI scans, automatically mapping the organ and suspicious lesions, and fusing those digital maps with live ultrasound images to guide the needle.

I think "The man in charge" phrase here is quite important in many applications of AI.

If I go to a solicitor, or a translator, or an accountant, or a doctor, or a vet, I don't mind if they use AI, as long as they take reasonable responsibility for any service/advice they provide to me.

If go with a do-it-yourself service using AI, then I take on that responsibility (because I know big tech won't).