Just tried out Bard with what I thought was a not to exotic question.

It made up a feature for a library, then lied about when it was added, and finally conocted some imaginary source code for it.

It’s seriously a mystery for me how anyone gets a net benefit from using these LLMs when you have to second-guess literally everything they output.

I was thinking of something like that too, but probably never would’ve gotten around to implement it myself, so thanks! Could you put your script on Github or something maybe?

Also it would be great if it could copy at least top-level comments by OP, because that’s often used for linking to a source.