Love it or hate it, AI is here to stay. Like crypto/blockchain, we will eventually see a reduction in the devastating environmental costs of AI, and we need to push for that as quickly as possible, but AI isn’t going anywhere. We’ll find new techniques beyond LLMs, or return to older techniques.

I’ve recently taken a job at an AI start-up, and have had to dive head first into learning the edges of LLMs, and “agentic development”. It’s been eye opening with exactly how good these tools can be, but it also has been thinking about how bad these tools can be. The old adage of garbage-in, garbage-out is in play here.

How a novice prompts AI

When I first discovered ChatGPT, it was a private beta, and you had to apply for access. I went to an art show, and there was a reading of a generated and curated, LLM generated novel in the world of Harry Potter. If I remember, the AI shipped Harry-Draco pretty hard. It was ChatGPT 2 or maybe even 1. It all seemed so magical.

And it was. Fast forward through the Covid pandemic, and the release of ChatGPT, and I started playing with it in earnest. And my prompts where as unsophisticated as the models themselves.

Create me a diet. I’m 45, I like protein. I have a shellfish allergy.

And it dutifully went off and created a diet that was bad.

What is wrong with simple prompting.

LLMs are a perfect example of Dunning-Kreuger. They will confidently make up stuff and present it as facts. The best analogy I have for this is that they are basically a white, privileged cis-het male. And it makes sense give the data they were trained on (hi, reddit). They are over-confident and cock-sure, so they’ll happily make up whatever is statistically plausible, like a startup founder bullshitting his way through an investor pitch.

This novice style prompt doesn’t provide enough context for the math to work out. It’s all about probability and next word prediction, so we need context in order to be able to get a good answer from the LLMs.

Utilizing your own knowledge with the LLM.

Lately, I’ve been having alot of joint pain, and is probably diet related. My diet has changed significantly in the last two years as I settle into new routines with my partner. Now, I am a nutrition coach. I know a thing or two about diet and exercise, and I should use that knowledge to provide context to the LLM.

I decided this morning that I wanted to put together an anti-inflammatory diet to help with my joint pain. I suspect it’s autoimmune related so my diet would play a large part in regulating the inflammation.

My initial prompt was this:

I have alot of joint pain. Can you help me research and develop an anti-inflammatory diet. It should be a balanced diet with 40% protein, 30% fat, and 30% carbs. I don’t eat shellfish. Ask clarifying questions and only pull from reputable sources like JAMA, and other industry sources. Use the most recent research, nothing older than 10 years

It asked me a few questions and then came up with a comprehensive plan. One thing that it recommended was certain supplements which I knew I should be taking anyway. I live in Seattle, we are all chronically vitamin D deficient. I wanted to see what supplements it would recommend, so I asked a follow up question:

Find product recommendations for the supplements that you suggested. Research them to find high quality supplements. Use independent lab-verified and NSF certified supplements.

It’s not my first-rodeo on supplementation, so I know what brands are quality. And I know it’s important to be verified by an independent lab since supplements aren’t regulated by the FDA and are prone to scams. I also know that if they are NSF-certified, they are regulated and considered safe for athletes. For me, that’s the minimum.

And it recommended the brands I would expect: Thorne (my favorite), Momentous, and NatureWise.

Because I knew what “good” looked like, the LLM was able to provide what I needed and produced a relative high-quality result. It’s not perfect, but overall, it’s good.

The key take away from this is: know the field you are working in, and provide the LLM with the right amount of context to allow it to produce a high quality output

---

• Image from freepik