FaceDeer

That’s not how synthetic data generation generally works. It uses AI to process data sources, generating well-formed training data based on existing data that’s not so useful directly. Not to generate it entirely from its own imagination.

The comments assuming otherwise are ironic because it’s misinformation that people keep telling each other.

The “how will we know if it’s real” question has the same answer as it always has. Check if the source is reputable and find multiple reputable sources to see if they agree.

“Is there a photo of the thing” has never been a particularly great way of judging whether something is accurately described in the news. This is just people finding out something they should have already known.

If the concern is over the verifiability of the photos themselves, there are technical solutions that can be used for that problem.

You’re welcome to try other methods but LLMs seem to be working best so far.

With a decompiler it should be pretty straightforward to automatically check for “hallucinations,” the compiled code is still right there and you can compare the decompiled logic to the original.

The time at which the source code was lost is irrelevant for decompilation, decompilation uses the binary files. Those are the files that are out there being played right now.

Until recently decompilers tended to produce rough and useless code for the most part, but I’m looking forward to seeing what modern LLMs will bring to decompilation. They could be trained specifically for the task.

Workarounds for those sorts of limitations have been developed, though. Chain-of-thought prompting has been around for a while now, and I recall recently seeing an article about a model that had that built right into it; it had been trained to use <thought></thought> tags to enclose invisible chunks of its output that would be hidden from the end user but would be used by the AI to work its way through a problem. So if you asked it whether cats had feathers it might respond “<thought>Feathers only grow on birds and dinosaurs. Cats are mammals.</thought> No, cats don’t have feathers.” And you’d only see the latter bit. It was a pretty neat approach to improving LLM reasoning.

And they’re overlooking that radionuclide contamination of steel actually isn’t much of a problem any more, since the surge in background radionuclides caused by nuclear testing peaked in 1963 and has since gone down almost back to the original background level again.

I guess it’s still a good analogy, though. People bring up Low Background Steel because they think radionuclide contamination is an unsolved problem (despite it having been basically solved), and they bring up “model collapse” because they think it’s an unsolved problem (despite it having been basically solved). It’s like newspaper stories, everyone sees the big scary front page headline but nobody pays attention to the little block of text retracting it on page 8.

Which is actually a pretty good thing.

I wouldn’t call it a “dud” on that basis. Lots of models come out with lagging support on the various inference engines, it’s a fast-movibg field.

Why does the rule need to be specific to data centers? Why not just try to encourage renewable energy in general?

The intersection of “Luddite hooligan” and “stops to think about technological capabilities and future consequences before vandalizing stuff” is not large.

Makes it all the more amusing how OpenAI staff were fretting about how GPT-2 was “too dangerous to release” back in the day. Nowadays that class of LLM is a mere toy.

Shush, this is an opportunity for people to dump on Microsoft, if you take it from them they’ll turn on you.

Though bear in mind that parameter count alone is not the only measure of a model’s quality. There’s been a lot of work done over the past year or two on getting better results from the same or smaller parameter counts, lots of discoveries have been made on how to train better and run inferencing better. The old ChatGPT3 from back at the dawn of all this was really big and was trained on a huge number of tokens but nowadays the small downloadable models fine-tuned by hobbyists would compete with it handily.

You can get decent results with much less these days, actually. I don’t have personal experience (I do have a 24GB GPU) but the open source community has put a lot of work into getting models to run on lower-spec machines. Aim for smaller models (8B parameters is common) and low quantization (the values of the parameters get squished into smaller numbers of bits). It’s slower and the results can be of noticeably lower quality but I’ve seen people talk about usable LLMs running CPU-only.

particularly for companies entrusted with vast amounts of sensitive personal information.

I nodded along to most of your comment but this cast a discordant and jarring tone over it. Why particularly those companies? The CrowdStrike failure didn’t actually result in sensitive information being deleted or revealed, it just caused computers to shut down entirely. Throwing that in there as an area of particular concern seems clickbaity.

One of the background details I liked in Ghost in the Shell was how the high-end data analysts and programmers employed by the government did their work using cybernetic hands whose fingers could separate into dozens of smaller fingers to let them operate keyboards extremely quickly. They didn’t use direct cybernetic links because that was a security vulnerability for their brains.

Different countries have a variety of very different approaches to appointing judges, and some of those methods are not nearly as easy to corrupt as the American system.

Americans are subject to a lot of cultural indoctrination about how their system is the “greatest democracy in the world,” “leader of the free world,” and other such platitudes. It’s really not the case, though. America’s system is one of the earliest that’s still around, and unfortunately that means it’s got a lot of problems that have been corrected in democracies that were founded later on but have remained embedded in America’s.

Doesn’t help that America has a somewhat problematic electorate as well.

Not necessarily. Curation can also be done by AIs, at least in part.

As a concrete example, NVIDIA’s Nemotron-4 is a system specifically intended for generating “synthetic” training data for other LLMs. It consists of two separate LLMs; Nemotron-4 Instruct, which generates text, and Nemotron-4 Reward, which evaluates the outputs of Instruct to determine whether they’re good to train on.

Humans can still be in that loop, but they don’t necessarily have to be. And the AI can help them in that role so that it’s not necessarily a huge task.

It means that even if AI is having more environmental impact right now, there’s no reason to say “you can’t improve it that much.” Maybe you can improve it. As I said previously, a lot of research is being done on exactly that - methods to train and run AIs much more cheaply than it has so far. I see developments along those lines being discussed all the time in AI forums such as /r/localllama.

Much like with blockchains, though, it’s really popular to hate AI and “they waste enormous amounts of electricity” is an easy way to justify that. So news of such developments doesn’t spread easily.

Funny you should mention blockchains. Ethereum, the second-largest blockchain after Bitcoin, switched from proof-of-work to a proof-of-stake validation system two and a half years ago. That cut its energy use by 99.95%. The “blockchains are inherently a huge waste of energy” narrative is just firmly lodged in the popular view of them now, though, despite it being long proven false.