A year ago, I did some reverse engineering. I figured a pattern matching engine has to be good matching patterns, so I threw the messed-up output of Ghidra into Claude to clean it up and to give me an overview of what the fuck happens in a certain part.
I was absolutely surprised. Sure, it got some things wrong, but it knew way more than it should have been able to guess from that short bit of fucked up, optimised not-quite-C. Including guessing context that it should not be able to guess. This was a firmware with not-yet-released functions that, to my knowledge, shouldn’t exist outside of this bit of binary firmware. At least not for us non-NDA mortals.
At the time, Mistral had a new-ish model as well, so I figured I’d give it a try. It worked as expected, cleaned up things, gave variables reasonable names, made educated guesses and so on, but didn’t act as if it had background info. And I was using Sonnet 4.x, so the small Claude which shouldn’t be that much larger.
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too. I have no idea how, maybe aggressively training on user inputs? Anyway, this does not help to allalleviate my suspicions.
I specifically asked our CEO about this just to cover my ass. It seems insane to trust these guys at all. Here in the USA, small businesses have de facto no legal protection from large businesses.
Absolutely. Especially if you have company info / IP that someone else would pay a lot for. This is a question for lawyers and managers to negotiate with the lawyers at OpenAI, Anthropic etc, not for us little people to decide. I’m not throwing company data in an online service that is not cleared for us to use, and I don’t even work with spicy stuff. Back in my IT days we would have called that a data breach, which would have lead to something from an unpleasant hour in the boss’ office to a promotion to customer.
They’re using gig workers by the hundreds of thousands, if not millions to do “data annotation”. You can actually make quite a bit if you’ve got higher credentials like Phd, masters, or a very good resume.
My theory is they’ve been using professionals to inadvertently leak proprietary methodologies. They could be doing so by creating problems for them to solve that lead people to inadvertently expose solutions based on their actual jobs.
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too.
An enormous amount of people use this technology for privat company data without batting an eye; not just software developers, but business people for reports, etc.
They all throw their confidential data in there without even considering the implications, just “trusting” Anthropic. claude code bots are running on tens of thousands of proprietary code bases that used to be private, and of course they are using all this data to train their model.
It surely gets better at coding in general, after all there are smart people at anthropic with near limitless budgets, but my conspiracy thesis is that it gets a lot better at replicating stuff it has seen before, but really shouldn’t have.
Makes me wonder how much of the frontier models hacking is just “our devs uploaded the codebase to Claude and suddenly it knows where all the backdoors are”?
I threw about 10 short phrased meeting notes (4-5 words each) into an LLM to get some document formatting started with titles and sections etc. It built a full scope document and inferred a ton of information about the project and product.
It was surprising how much it could infer, and suggested very sensible testing criteria and alternate solutions should the main product path not pan out.
I can see why people start believing it “thinks”, because it was remarkable what it “understood” about a vague new product. Had I sent the info out to stake holders, they would have assumed this person was a key team member that was involved from day 1.
A coworker did something similar recently. Besides pasting internal stuff into a foreign model like a moron, it had some nice ideas, but lacked a lot of information it simply could not have known. Things like, we already did this, but in an unrelated repository, and we didn’t mention it in this part of the docs. Or, since the person did not know / input the overall purpose and constrains correctly, the LLM proposal was cool, but missed the point completely. Or, the LLM forgot to mention that its plan would require buying hardware we simply don’t have yet, and which would be really expensive and hard to obtain right now, because, idk, hallucinations or it just assumed it was all already there.
Yeah, not impressed. Neither by that idiotic coworker nor by the proposal.
I figured you meant the coworker, but its something I think any AI user should be aware of, especially just using a public web browser version where they definetly will store you data and could leak private info.
I don’t use it often enough to have stats, but I did notice large differences between public AI models.
I had an odd Secure Linux permission issue after an update so I figure WTH let’s try AI debugging.
I pasted same info into Chatgpt and Googles AI. One suggested a bunch of debug steps which made sense, but it talked ahead of itself. It would say check this, then do this, then this (assuming you had answered a Y/N scenario for step 1 already. So it was a complicated interaction, because it gets your response, formulates new plans, but assumes you also completed step 2 and 3 from before.
It led itself around in circles with pages and pages of suggestions. Even though none of the steps were poor choices, they made logical sense, it just couldn’t sort it out.
The other one took my paste, asked one question, and gave me the solution. The update had stripped the SELinux rule off the file and become an undefined default.
“Intelligence”, as a word has always been ill defined. What’s it mean? It’s mostly been used to create prejudices via standardized tests, the bell curve, and IQ tests.
I don’t believe it’s “intelligent” as we don’t really know how to define that. But I do think it’s really really really good at writing reasonable seeming statements in various languages using probability.
Coding and programming, are just languages. So it will likely be able to get pretty damn good at it, just as it has with the English language.
I think with LLMs accurate regurgitation of facts, is the colary to good code.
But I don’t believe that means it “thinks” (which requires introspection, and an internal experience). It does multi-agent computing, and this might even simulate some aspects of thinking… But it’s not thinking. It’s computing.
It won’t be able to do any more than that, until it consistently gets data from having a body (and something like glial thought-interactions/organs/flaws/survival).
By the way, this also means if you could download a mind to a computer - it would slowly lose it’s humanity as it lived longer and longer interfacing with computer systems rather than human systems/organs/experiencs.
Eventually that “uploaded human” would become a Clankr, and stop understanding human life, the will to survive/think. Eventually it would just compute (not think/care or have flaws).
Many sci-fi stories are based on the man-to-technology-to-cold-rational-evil pipeline… Less have gone in the other direction, even though it often produces very popular characters: Spock, Data, the Terminator, Pinnoccio.
Nah, I don’t think so. We’re getting into philosophical / religious territory here, but from a technical standpoint I believe there is no difference between a brain and an accurate simulation of a brain, and an accurate simulation of a brain is “just” an engineering challenge at this point. We might now have the computing power or architecture yet, but there is no conceptual issue prohibiting us from building it.
But an LLM is not a simulation of a brain. It is an autocomplete on steroids. Very different architecture. It is not a brain, but it can sometimes produce the same outputs a brain would. But it is nowhere close to a human brain.
Oh it learned enough from code that it has developed some basic, and hear me out of this, artificial verbal intelligence. I know I know, not just pattern matching, but actual artificial intelligence at a low level.
Please don’t downvote me (sorry don’t remember if this is fuckai or not) but I’m just saying it’s one possible explanation to this phenomenon you describe.
Don’t get me wrong, the models are really impressive, and something I would have called impossible two years ago. From an information theory point of view, it is super cool how they manage to compress so much knowledge and connections in so little size (relatively speaking).
Itelligence is very ill defined, as someone else pointed out. It still has a very weird model of the world (if at all).
My point is however, it sometimes seems to be too good at guessing facts that should not be guessable from the inputs and publically available information alone.
I mean even if it did train on the proprietary code it can’t memorize it. It has to “learn what code like this looks like”. It has to figure out what the function is. That can I think only be called verbal intelligence. The ability to understand what is meant behind words. Not very good yet but having to pretend it’s not there is just denialism. Denialism exactly like climate change denial. And of course some AI fans do have AI psychosis, different side of the same coin.
The whole dancing around LLMs clearly being able to reason (which can be shown in scientific experiments) and therefor clearly having some sort intelligence is so stupid. Because meanwhile 90% of lemmy users are too brainwashed to accept basic reality any more.
Because you know, this isn’t just funny. There are real danger as the technology progresses. But it’s politically incorrect and gets you massive downvotes in the wrong community if you talk about those dangers. You can’t talk about facts any more.
It’s not just about the historical facts about e.g. the coup in Ukraine that you can’t talk objectively about any more. Facts in general become anathema. Only the feeling of correctness and manufacturing social consent matters. Collectively we are becoming more and more stupid animals. We are trained to become less sentient or sapient. Less able to perceive and react to reality and more and more act on trained and instinctual impulses. But I see little difference between the MAGA fascists and like 90% of lemmy users now. And if you don’t want to offend (or get fired) you have to mask and that seemed funny in your post.
There are studies and tools like LiCoEval and others that investigate LLM memorization. It does happen but it’s not all the time, and it’s very infrequent for real world (complex) examples. Mostly this concerns textbook examples or standard boilerplate code - from what I read. And I obviously don’t know the specific example mentioned. This is also a problem that can be fixed.
But I’m really just half mocking the very careful language used by the comment I replied to.
I’m sure it’s all but officially confirmed, that all the people upload to it goes to training data unsanitised. And Microslop for sure uploaded entirety of their codebase to all the bots. This is at least part of the reason why claude is so surprisingly OK at reverse-engineering all the big stuff.
Tinfoil hat time:
A year ago, I did some reverse engineering. I figured a pattern matching engine has to be good matching patterns, so I threw the messed-up output of Ghidra into Claude to clean it up and to give me an overview of what the fuck happens in a certain part.
I was absolutely surprised. Sure, it got some things wrong, but it knew way more than it should have been able to guess from that short bit of fucked up, optimised not-quite-C. Including guessing context that it should not be able to guess. This was a firmware with not-yet-released functions that, to my knowledge, shouldn’t exist outside of this bit of binary firmware. At least not for us non-NDA mortals.
At the time, Mistral had a new-ish model as well, so I figured I’d give it a try. It worked as expected, cleaned up things, gave variables reasonable names, made educated guesses and so on, but didn’t act as if it had background info. And I was using Sonnet 4.x, so the small Claude which shouldn’t be that much larger.
Since then, I have the conspiracy theory that the Claude people threw a bunch of proprietary sources in the training, too. I have no idea how, maybe aggressively training on user inputs? Anyway, this does not help to allalleviate my suspicions.
I specifically asked our CEO about this just to cover my ass. It seems insane to trust these guys at all. Here in the USA, small businesses have de facto no legal protection from large businesses.
Absolutely. Especially if you have company info / IP that someone else would pay a lot for. This is a question for lawyers and managers to negotiate with the lawyers at OpenAI, Anthropic etc, not for us little people to decide. I’m not throwing company data in an online service that is not cleared for us to use, and I don’t even work with spicy stuff. Back in my IT days we would have called that a data breach, which would have lead to something from an unpleasant hour in the boss’ office to a promotion to customer.
They’re using gig workers by the hundreds of thousands, if not millions to do “data annotation”. You can actually make quite a bit if you’ve got higher credentials like Phd, masters, or a very good resume.
My theory is they’ve been using professionals to inadvertently leak proprietary methodologies. They could be doing so by creating problems for them to solve that lead people to inadvertently expose solutions based on their actual jobs.
It’s lowkey corporate espionage.
An enormous amount of people use this technology for privat company data without batting an eye; not just software developers, but business people for reports, etc.
They all throw their confidential data in there without even considering the implications, just “trusting” Anthropic. claude code bots are running on tens of thousands of proprietary code bases that used to be private, and of course they are using all this data to train their model.
It’s standard operating practice to steal everything. The world’s full of dumb fucks.
Then people are surprised that claude code gets better at coding
Better at replicating closed source software.
It surely gets better at coding in general, after all there are smart people at anthropic with near limitless budgets, but my conspiracy thesis is that it gets a lot better at replicating stuff it has seen before, but really shouldn’t have.
Meta had an explicit higher price for using their model without retention of prompts for training. This all but confirms your theory.
Wait… wait… you believe Facebook?
Oh, facebook will steal the data even if they say they won’t. What’s interesting is they put a price on blatant prompt copying.
We steal your data: $5 We steal your data and tell you we don’t: $10
Pretty much.
Makes me wonder how much of the frontier models hacking is just “our devs uploaded the codebase to Claude and suddenly it knows where all the backdoors are”?
I guess most, honestly
I threw about 10 short phrased meeting notes (4-5 words each) into an LLM to get some document formatting started with titles and sections etc. It built a full scope document and inferred a ton of information about the project and product. It was surprising how much it could infer, and suggested very sensible testing criteria and alternate solutions should the main product path not pan out.
I can see why people start believing it “thinks”, because it was remarkable what it “understood” about a vague new product. Had I sent the info out to stake holders, they would have assumed this person was a key team member that was involved from day 1.
A coworker did something similar recently. Besides pasting internal stuff into a foreign model like a moron, it had some nice ideas, but lacked a lot of information it simply could not have known. Things like, we already did this, but in an unrelated repository, and we didn’t mention it in this part of the docs. Or, since the person did not know / input the overall purpose and constrains correctly, the LLM proposal was cool, but missed the point completely. Or, the LLM forgot to mention that its plan would require buying hardware we simply don’t have yet, and which would be really expensive and hard to obtain right now, because, idk, hallucinations or it just assumed it was all already there.
Yeah, not impressed. Neither by that idiotic coworker nor by the proposal.
Well i was careful and used a private LLM, and didn’t include any customer specific info just in case.
That “moron” was not directed at you, but only at my coworker. I just wanted to point out that it seems to be a bit hit-or-miss sometimes :)
Apparently you write good meeting notes.
I figured you meant the coworker, but its something I think any AI user should be aware of, especially just using a public web browser version where they definetly will store you data and could leak private info.
I don’t use it often enough to have stats, but I did notice large differences between public AI models.
I had an odd Secure Linux permission issue after an update so I figure WTH let’s try AI debugging.
I pasted same info into Chatgpt and Googles AI. One suggested a bunch of debug steps which made sense, but it talked ahead of itself. It would say check this, then do this, then this (assuming you had answered a Y/N scenario for step 1 already. So it was a complicated interaction, because it gets your response, formulates new plans, but assumes you also completed step 2 and 3 from before. It led itself around in circles with pages and pages of suggestions. Even though none of the steps were poor choices, they made logical sense, it just couldn’t sort it out.
The other one took my paste, asked one question, and gave me the solution. The update had stripped the SELinux rule off the file and become an undefined default.
“Intelligence”, as a word has always been ill defined. What’s it mean? It’s mostly been used to create prejudices via standardized tests, the bell curve, and IQ tests.
I don’t believe it’s “intelligent” as we don’t really know how to define that. But I do think it’s really really really good at writing reasonable seeming statements in various languages using probability.
Coding and programming, are just languages. So it will likely be able to get pretty damn good at it, just as it has with the English language.
I think with LLMs accurate regurgitation of facts, is the colary to good code.
But I don’t believe that means it “thinks” (which requires introspection, and an internal experience). It does multi-agent computing, and this might even simulate some aspects of thinking… But it’s not thinking. It’s computing.
It won’t be able to do any more than that, until it consistently gets data from having a body (and something like glial thought-interactions/organs/flaws/survival).
By the way, this also means if you could download a mind to a computer - it would slowly lose it’s humanity as it lived longer and longer interfacing with computer systems rather than human systems/organs/experiencs.
Eventually that “uploaded human” would become a Clankr, and stop understanding human life, the will to survive/think. Eventually it would just compute (not think/care or have flaws).
Many sci-fi stories are based on the man-to-technology-to-cold-rational-evil pipeline… Less have gone in the other direction, even though it often produces very popular characters: Spock, Data, the Terminator, Pinnoccio.
Nah, I don’t think so. We’re getting into philosophical / religious territory here, but from a technical standpoint I believe there is no difference between a brain and an accurate simulation of a brain, and an accurate simulation of a brain is “just” an engineering challenge at this point. We might now have the computing power or architecture yet, but there is no conceptual issue prohibiting us from building it.
But an LLM is not a simulation of a brain. It is an autocomplete on steroids. Very different architecture. It is not a brain, but it can sometimes produce the same outputs a brain would. But it is nowhere close to a human brain.
Oh it learned enough from code that it has developed some basic, and hear me out of this, artificial verbal intelligence. I know I know, not just pattern matching, but actual artificial intelligence at a low level.
Please don’t downvote me (sorry don’t remember if this is fuckai or not) but I’m just saying it’s one possible explanation to this phenomenon you describe.
Don’t get me wrong, the models are really impressive, and something I would have called impossible two years ago. From an information theory point of view, it is super cool how they manage to compress so much knowledge and connections in so little size (relatively speaking).
Itelligence is very ill defined, as someone else pointed out. It still has a very weird model of the world (if at all).
My point is however, it sometimes seems to be too good at guessing facts that should not be guessable from the inputs and publically available information alone.
I mean even if it did train on the proprietary code it can’t memorize it. It has to “learn what code like this looks like”. It has to figure out what the function is. That can I think only be called verbal intelligence. The ability to understand what is meant behind words. Not very good yet but having to pretend it’s not there is just denialism. Denialism exactly like climate change denial. And of course some AI fans do have AI psychosis, different side of the same coin.
The whole dancing around LLMs clearly being able to reason (which can be shown in scientific experiments) and therefor clearly having some sort intelligence is so stupid. Because meanwhile 90% of lemmy users are too brainwashed to accept basic reality any more.
Because you know, this isn’t just funny. There are real danger as the technology progresses. But it’s politically incorrect and gets you massive downvotes in the wrong community if you talk about those dangers. You can’t talk about facts any more.
It’s not just about the historical facts about e.g. the coup in Ukraine that you can’t talk objectively about any more. Facts in general become anathema. Only the feeling of correctness and manufacturing social consent matters. Collectively we are becoming more and more stupid animals. We are trained to become less sentient or sapient. Less able to perceive and react to reality and more and more act on trained and instinctual impulses. But I see little difference between the MAGA fascists and like 90% of lemmy users now. And if you don’t want to offend (or get fired) you have to mask and that seemed funny in your post.
Anyway, sorry just rambling lol
i would err towards it ingesting proprietary code before this, it’s not wrong but AI spits out copyrighted and should-be private stuff all the time.
There are studies and tools like LiCoEval and others that investigate LLM memorization. It does happen but it’s not all the time, and it’s very infrequent for real world (complex) examples. Mostly this concerns textbook examples or standard boilerplate code - from what I read. And I obviously don’t know the specific example mentioned. This is also a problem that can be fixed.
But I’m really just half mocking the very careful language used by the comment I replied to.
I’m sure it’s all but officially confirmed, that all the people upload to it goes to training data unsanitised. And Microslop for sure uploaded entirety of their codebase to all the bots. This is at least part of the reason why claude is so surprisingly OK at reverse-engineering all the big stuff.