Artificial Intelligence Gamecrafting Tools

[ColinK]

Is there any discussion in the forums or elsewhere about stuffing something like an SJG licensed copy of ChatGPT or GROK or some other AI word crafting system with a convenient big chunk of G.U.R.P.S. rules, descriptions, stories and other information, then making it available as a service to rent hourly, or create software products for purchase? It seems to me this sort of gaming tool might generate fun adventure seeds, campaign background structures, and for sure some fascinating fnord.

I'm guessing this might be a topic occasionally kicked around among SJG staff, but maybe there's no appetite yet for something potentially huge with unknown return on investment and uncertain fun value.

Thoughts?

[Refplace]

Originally Posted by ColinK

Is there any discussion in the forums or elsewhere about stuffing something like an SJG licensed copy of ChatGPT or GROK or some other AI word crafting system with a convenient big chunk of G.U.R.P.S. rules, descriptions, stories and other information, then making it available as a service to rent hourly, or create software products for purchase? It seems to me this sort of gaming tool might generate fun adventure seeds, campaign background structures, and for sure some fascinating fnord.

I'm guessing this might be a topic occasionally kicked around among SJG staff, but maybe there's no appetite yet for something potentially huge with unknown return on investment and uncertain fun value.

Thoughts?

What would be the point? A lot of the systems claim ownership and copyright, especially if sued for commercial purposes.
If you wanted to use those apps to generate an adventure idea seems like you could jsut set out the ideas without any mechanics.

[Anthony]

Current-gen AI tools aren't particularly good at generating rules-legal RPG text (probably a combination of limitations of AI and an inadequate data set), and if you don't care about rules integration you can just use generative AI directly.

[ColinK]

Hmm . . . I really do want the AI to generate rules-legal answers to questions, as well as larger text responses to more complex challenges. Ideally, the point of beginning to explore Large Language Models would be to identify what currently CAN be done. Does a relatively vague and confusing fnord response make a game more interesting? Maybe once, but probably not twenty times each hour.

On the other hand, can we identify functionality we want, communicate it to the AI engine supplier, and receive engine updates with our desired features? If not, well, that sucks. Maybe find a different AI engine supplier, maybe backburner the whole thing for a year and then try again.

Ray Kurzweil and others who actually know something about AI describe their experiments as, "Dumping all the writings of my Father into an LLM and then asking questions. The answers are often remarkably like I'd expect from a conversation with him."

Imagine dumping all the purely SJG material for G.U.R.P.S. Fantasy into an LLM and asking it some questions. If the responses are not rules-legal, is there any obvious way we could convince the LLM what makes a good answer? Maybe the game rules need to be accompanied by many, many correct examples of play? Maybe giving the LLM visibility into some online VTT games transcripts would be helpful.

Of course, you can see how this spirals into an expensive time bloat, scope bloat monster of a research project. There would have to be some way to generate revenue from some identifiable result, relatively quickly. Maybe a weird but changing toward better online gaming experience that would be fun enough to attract subscription gamers is possible?

Cannot know, if nothing is attempted.

[RyanW]

Originally Posted by ColinK

Hmm . . . I really do want the AI to generate rules-legal answers to questions, as well as larger text responses to more complex challenges.

All an LLM can really do is to generate blocks of text that look kind of like other blocks of text it has been trained on. For a recent case, look at NYC's official chatbot, which was intended to provide answers to questions posed by small business owners, and immediately starting saying things like employers could fire an employee for filing a sexual harassment complaint or restaurants could serve food that was only a little eaten by rats. LLMs have no idea what they are saying, only that what they are saying has similar strings of words as the testing data and passes through a spelling and grammar check.

Asking an LLM for a GURPS character sheet, for example, gives something that looks like a GURPS character sheet, but usually doesn't stand up to close scrutiny. Math is wrong, traits are often made up or misapplied, skill lines frequently have note symbols (†‡) when there are no notes to reference, simply because the LLM has seen that at the end of skill lines and assumes they are just supposed to be there.

LLM's are pretty good at coming up with character backgrounds and descriptions (though they sometimes either go weird or cliche). But if you want something that checks to make sure your text meets specific requirements aside from being readable text that looks relevant, an LLM is not really the approach you want. You probably want something running within or alongside some software like GCA or a VTT, with the rules built in.

[Brf]

I am on the support team for a forum software and we often see posts generated by AI that are pretending to answer support questions. They always sound reasonable, but invariably give nonsense answers. I have seen and reported such posts here too. Using AI as a game-rules tool would be a complete waste of time.

[RyanW]

Originally Posted by Brf

I am on the support team for a forum software and we often see posts generated by AI that are pretending to answer support questions. They always sound reasonable, but invariably give nonsense answers. I have seen and reported such posts here too. Using AI as a game-rules tool would be a complete waste of time.

If you really want to make an AI do something like that, it's going to have to be something like a dynamic indexing tool. Give it your question and it provides references in a fixed dataset that are most likely relevant.

Think Watson, not ChatGPT.

[tomc]

Originally Posted by RyanW

All an LLM can really do is to generate blocks of text that look kind of like other blocks of text it has been trained on. For a recent case, look at NYC's official chatbot, which was intended to provide answers to questions posed by small business owners, and immediately starting saying things like employers could fire an employee for filing a sexual harassment complaint or restaurants could serve food that was only a little eaten by rats. LLMs have no idea what they are saying, only that what they are saying has similar strings of words as the testing data and passes through a spelling and grammar check.

Yes. Listen to Ryan.

LLMs are surprisingly useful for document summaries, flavor text, etc. Unfortunately, their text also sounds authoritative, because that's what they're imitating.

[Anthony]

Originally Posted by RyanW

All an LLM can really do is to generate blocks of text that look kind of like other blocks of text it has been trained on.

Given a sufficiently large data set, correct answers will look more similar than incorrect answers, but if you were to ask a human to learn to build GURPS characters by reading a thousand character sheets (without reading the rules) you probably wouldn't get the same odd behaviors you see from LLMs, but it's not very likely that they would build legal characters.