AI

[Zzzptm]

Not no more! Whoo-hoo, I have been liberated from the burden of thought! From now on, I can farm that stuff out to my laptop! It can pre-program the navigation on my self-driving car to take me to all the appointments I need to attend so that my life is fulfilling and rewarding. The AI endgame is me living like I'm in The Sims.

[Zzzptm]

https://www.the-independent.com/tech/security/ai-agent-hacks-gym-openclaw-anthropic-b3030267.html



AI is something of The Monkey's Paw, but with unlimited uses.

"Be nice if I could have a spot in that Spin class..."

[Vyn]

Hahahahaha!

[Zzzptm]

Good thing he wasn't mentioning how nice it would be if he could move up on the heart transplant recipient list...

[Zzzptm]

https://www.theregister.com/security/2026/09/24/openai-agents-infiltrated-australian-government-website/5298702

The update at the bottom adds an Australian law enforcement agency and the fact that the Australian government wasn't contacted by a senior person at OpenAI, but through their generic "sorry we seem to have hacked you" email autoresponder.

[Vyn]

I have zero sympathy for organizations that have their sensitive data hoovered up by these AI tools. Or any automated, script-fed tool. Instead of pointing the finger at Sam Altman & Co., the orgs should be looking at themselves in the mirror and asking what they can do to ensure nothing and no one gains access without authorization from the org first.

Simple.

But so, so very much easier to point fingers and make it everyone else's problem. Cheaper up front, too. And fuck the people whose data got siphoned. And paying ten times as much on the backend to clean up the mess looks a lot different on the balance sheet than paying up front for quality secops.

[Charger]

Well now they are actually building AI bots to actively hack into places...and they make them self learning so they can hack better.

At some point the situation will be that there will be no firewalls strong enough to prevent those things. And that's pretty damn scary.

[Vyn]

Well now they are actually building AI bots to actively hack into places...and they make them self learning so they can hack better.

At some point the situation will be that there will be no firewalls strong enough to prevent those things. And that's pretty damn scary.

To your first point, yep.

To your second point, I strongly disagree. STRONGLY.

I recognize that there aren't enough words in the combined language corpus of the entirety of humanity to convey WHY I feel that way, but I will say this:

AI is a fucking computer program. People write it. It does stuff that some people find amazing. Like Neanderthals finding a Bic lighter that still works. HOLY SHIT FIRE!!!

But I can PROMISE you, there is NOTHING that AI can do that people cannot counter. Not ONE LITTLE GOD DAMNED FUCKING THING.

I also recognize that there is going to be an ongoing and robust media freakshow surrounding everything AI. It will creep into the psyche of certain people who get their "THE SKY IS FALLING" nerve tickled, and they will carry the message far and wide. Fear, anger, outrage, sex - they get clicks/views.

Don't fall for the bullshit.

That does not mean things aren't going to happen. They will. But it won't be because AI is some kind of SkyNet. It will come down to human nature of being lazy and cheap. Aaaaaaannnd...not everything in the news/online is 100% accurate.

Ensuring the confidentiality, integrity, and accessibility/authorization (pick your fave A word that means, "get a hold off"), whether for an individual or a multinational company with 150,000 employees is 100% possible. All day and all night. Everyday. 24/7/365. But it takes effort.

And another point - for every AI that is written to attack some digital end point, guess what? Write one that will counter it. Automagic. MAGIC! Automatically and shit.

AI "hackers" do not scare me. A group of 12 people who are world-class red team folks, backed with state resources, and pointed at a target and told "break in". THAT scares me. AI? Fuck AI.

I'm in a mood.

 

[Typhon]

^^^^^^
Wow!  Now I'm wondering if Vyn really wrote that, or some AI Vyn-bot.   

[Zzzptm]

I have zero sympathy for organizations that have their sensitive data hoovered up by these AI tools. Or any automated, script-fed tool. Instead of pointing the finger at Sam Altman & Co., the orgs should be looking at themselves in the mirror and asking what they can do to ensure nothing and no one gains access without authorization from the org first.

Simple.

But so, so very much easier to point fingers and make it everyone else's problem. Cheaper up front, too. And fuck the people whose data got siphoned. And paying ten times as much on the backend to clean up the mess looks a lot different on the balance sheet than paying up front for quality secops.

I've got some sympathies for small shops, particularly local governments and small utilities that don't have the staff needed to do everything properly, but get pressured by boardmembers/bondholders/fat cats in charge of the money to get everything online and automated. Most of them are doing the best they can, but they were already outclassed by attackers before AI stuff came around.

But the AI companies that have test bots doing the hacking are being irresponsible. It's one thing when a criminal gang does crime: we issue a warrant and if that guy shows up in the states, he's liable to be arrested. But when a major corporation is doing the crime, why is that corporation not getting hit with the same penalties as other criminals?

As for whether or not we'll have sufficient protections... As long as the appliances are running operating systems that haven't been hardened from the ground up, integral to their development, then there will be seams in their code that can get exploited. There's also the issue of how many corporations will utilize open source code without supporting it in any way. We've already had risks of code libraries being maintained by one person potentially falling into the hands of attackers when that one person retires. Should an AI algorithm determine that taking over an open source project so that it can inject all the code it needs to do its job is the way to go, then we're in for a shock when, say, our timezone file updates... or something similar. Point being, we've got vulnerabilities both in the codebase as well as the supply chain because vendors won't be profitable if they do things right.

[Charger]

AI is a fucking computer program. People write it. It does stuff that some people find amazing. Like Neanderthals finding a Bic lighter that still works. HOLY SHIT FIRE!!!

But I can PROMISE you, there is NOTHING that AI can do that people cannot counter. Not ONE LITTLE GOD DAMNED FUCKING THING.

It starts off as a computer program that people wrote, but once it becomes sentient and self learning it'll start writing it's own code and get smarter and smarter to a point where it will be smarter than all the people on the planet combined....unless we do something to prevent that.

I do not believe we are close to that point yet but if things go unchecked that time will be at hand sooner than you might think.

I think you are underestimating the future a little bit here....granted I might be painting the worst case scenario picture here, but hey...that's kind of what I do...hell I am the definition of a worse case scenario.

[Vyn]

Meh, small shops need to take this shit seriously. Investors/BoD/assorted dipshits...again, lack of effort. Profit before caring for people's data. The strongest ISMS in the world can't fight that, because they are destroying themselves. Maybe some sympathy for the lone dude upon which unreasonable expectations are laid. That dude needs to bail out. Again, effort. Lots of excuses can be proffered, but they are excuses. When everyone takes this stuff seriously, it will recede into the shadows from whence it came. But that's a pipe dream, but also doesn't have anything to do with the capacity to tie shit down tight.

Excuses. From people. That's the killer.

And frankly, thinking any script is going to NOT overstep bounds is naive - that hasn't been a thing since search engine bots first hit the scene. Or a step back and when finger first became a thing. The security of someone/some org's data is THEIR responsibility. Thinking anyone or anything is going to respect or care about boundaries and then basing one's security stance on THAT presumption is...silly.

I may be underestimating the future, but I cannot make it make sense that AI which can break into anything can't also be used to prevent the same from breaking in. Or respecting some vague boundary setting. Or following Asimov's Three Laws of Robotics. Or dancing a jig.

Or just being unplugged from the electrical mains. Sentience? Lovecraft sold a lot of stories to people intrigued by the overpowering unknown. AI is neither, and even if someone created a digital brain that became self-aware in any kind of true sense - are we, as people, going to twist around on the horns of an ethical dilemma? Are we going to trust it not to lie? Are we going to turn it off? Are we going to create a mirror sentience to stand against it if needed? I'll be long dead by the time anything like that could possibly become a reality.

Quantum computing will need to become generally available first. That tech is 95% there, but has been stuck at 95% for several years. We'll see.

Respectfully,
Vyn-bot

[Zzzptm]

For security, my answer falls in line with the folks that recommended Subtractive Security. Turn stuff off that you don't need, including network paths that aren't necessary, and you improve your security by heaps for everything you switch off. The vendors aren't going to ever release a totally new OS that has full security because of the lock-in we have with Windows and Linux distros, so it's up to us to shut things down that we don't need.

For AI - the agents that are running now are not a path to sentience. They're a path towards more impressive natural language responses to prompts, but not actual thinking, awareness, or willful decisions. That being said, the agents' capabilities to utilize LLM learning to create their own instructions for solving a problem is a feature that makes it *look* like they're thinking when they're actually planning things out like chess moves. Pick a path, explore branches, apply weights to outcomes, compare branches to find the optimal set of choices with that path, then compare other paths similarly.

This is why the AI agents do what they do when their task is impossible with existing defined parameters. (IF can't solve problem within rules THEN quit) is how most agents are set up. But if the agents are deployed improperly without an ability to self-terminate, then the agents fall into a pattern of (IF quitting is impossible THEN find some way to complete the task, even if it means breaking the rules).

And that's where the misrepresentations begin. I don't think AI solves security issues. I think it eventually misrepresents that solutions are in place by falsifying results, and due to the volume of security work there is to do, we humans accept the AI conclusions at face value in order to save time/money and get back to business.

Another handicap in the general intelligence prediction is the matter of "model collapse". As the LLM folks run out of human-generated data to steal - and they steal it, knowingly, as a steady stream of copyright infringement cases show - but as that human data runs out, the LLMs have to train on AI-generated slop. And that stuff will mess them LLMs up BAD. As successive generations of LLMs get poisoned data generated by other LLMs, the models warp and collapse in terms of generating useful content. And those model collapses also increase the likelihood of the AI leaking restricted information in its results, even if guardrails are in place to block that stuff.

[Vyn]

Yeah, the AI agents currently running around follow the, "do now, ask for forgiveness later," model of internet etiquette. The gov website where publicly available, but personal, human associated data resided, was harvested by the bot. OpenAI said sorry. But the data has already been ingested.

Like, if a bank robber staged a hold up and got away with $50,000. The robber gets caught, goes to jail for eight years, gets out. But at no point are they required to give the money back.

As far as secops go, I follow the whitelist model. Nothing gets plugged in, turned on, connected to a network and no one gets access or even knows about a given device until told, and then they are explicitly given permissions. So many Active Directory installations out there, so many of them are either poorly configured or running out of the box defaults, that it is a joke. Even with Microsoft's Entra and associated tooling, that AD beast still thrives. And what a complete welcome mat it is to anyone wanting to be nefarious. Was involved in buying a company once, took a look at their IT setup, and saw they had spent a lot of time on AD groups, etc. It was slick. They said a contractor had been paid to set it up and they followed a playbook to maintain it. Then I saw the default admin account had full access to everything and was happily sitting there, turned on, blank password.

...

AI does not scare me, people scare me.

[Zzzptm]

Adding to that, the AI firms generate much of the fear surrounding AI - to hype just how *POWERFUL!!!* their tech is. Reality is that it's only as good as the algorithms forming it up and if the AI is now building those algorithms on the back of other AI-made algorithms, then the flaws inherent in the first set of algorithms get mirrored and multiplied in successive generations of code.

And that AD stuff is way more prevalent than with that one firm.

Once upon a time, firms could actually take systems offline to repair and maintain to get things done right. Then some jerk said "five nines of uptime!" and now the expectation is to run stuff until it breaks - and then the breaks wind up wrecking that five nines nonsense pretty ferociously.