The truth? ChatGPT can’t handle the truth! 

But can you make it tell the truth? Can you make it give you an answer you can rely on? (Spoiler: Yes. And it’s not a prompt.  And we built it!)

This is one of the most important discussions we have in our workshops around the effective use of Large Language Models like ChatGPT. Large Language Models ‘merely’ predict the next word, trying to satisfy us with their answers.  The truth is incidental (instrumental) to that goal.  If you ask ChatGPT a question, you may get a correct answer. But how will you know it is right? 

This is what we often want. An answer we can rely on.  And Large Language Models are bad at this.

ChatGPT is magical. Its answers flow so fluidly. It understands our questions so well!  But. BUT. You cannot tell if the answers are correct or if they are just correct looking.

A (mysterious secret) tech partner and I have been playing with this concept of using 1st party data to make ChatGPT not only true, but reliably true. We have created this great proof of concept that allows you to ask any question about the South African Constitution and get a validated answer. (Thanks to Scott Gray for the original idea) It not only answers you but it also tells you exactly which section it got the answer from, and gives you the relevant section.

Try it out and tell us what you think! http://demo.baobabai.com/constitution

This isn’t like those creepy hallucinated sources that ChatGPT is rapidly becoming famous for.  This is the real McCoy, Chapter and Verse.

Here’s how it works:
1) The data, in this case the original text of the Constitution, is vectorised so that it can be easily processed.  
2) The user question is translated into a search over the 1st party data. (Data scientists seem to want to say things like ‘we look for closeness between the vectorised user prompt and our source data in high dimensional space’)
3) The ‘found’ text is inserted into the prompt and the LLM is instructed to answer the question using only the data found in the prompt
4) This answer is returned to the user, as well as the reference used to feed the prompt.

It’s just a simple first attempt but it shows how powerful these models can be if you combine their natural language ability with the reliability of first party data.

So – Does anyone have a large lake of text that they would like to be able to get reliable data from? Let’s talk!

(Because seeing the power of this is addictive, we also built one for the rules of golf here: http://demo.baobabai.com/golf . Now you have no excuse when you make a noise during your partners swing)