ChatGPT & Salmon Data - the Trouble with Tribbles

A defeated salmon farmer.

For those of you who had social lives in high school and university, the title of this blog post makes reference to a classic Star Trek episode where the star ship Enterprise is invaded by adorable furry creatures that confound the crews attempts to manage them, ultimately putting James T. Kirk and his band of heroes in mortal peril. Perhaps a laboured metaphor but I request you bear with me.

I had grand visions this week of creating a custom ChatGPT program to compare, analyze and report on the quarterly statements of publicly listed salmon farming companies. I wanted to compare EBIT values, Fair Value adjustments, biomass values etc. and to lord it over my fish farming compatriots on how cutting edge and youthful I am.

After hours of effort, I am giving up (for now), I have not felt this much frustration and defeat for a while. I will survive and live to consult/advise another day, but it has been a real rollercoaster.

I am a fan of ChatGPT. I used it to great effect earlier this year to develop a production planning tool which compared farming scenarios using 10 regional temperature profiles, regionally specific costs and inputs, flexible harvesting scenarios etc., to arrive at an overall production cost comparison. Using ChatGPT over several weeks allowed me to achieve things with Excel formulas that I never, in my wildest dreams, would have been able to figure out on my own.

ChatGPT and Excel

If you haven’t used ChatGPT to develop excel formulas, you should give it a try. Here’s an example of a detailed interaction I used to develop a complicated formula.

ChatGPT Screenshot

ChatGPT analyzes the detailed prompt, explains the approach and then produces an excel formula you can copy and paste into your spreadsheet. If the formula doesn’t work, you can ask it to check the function and create an alternative formula. In some cases, it took a lot of back and forth and clarification of details but, in pretty much every case, I was able to get extremely complicated formulas to work. Here’s an example of the harvest biomass formula I was able to develop:

=IF(AND(AND(TEXT(LR2, "dddd")<>"Friday", TEXT(LR2, "dddd")<>"Saturday"), LR$4 > 'Harvest Parameters'!$B$3, 'Scenario 3'!LR$8 > 0), IF('Scenario 3'!LR$8 < 'Harvest Parameters'!$B$7, 'Scenario 3'!LR$8, 'Harvest Parameters'!$B$7), 0)

The formula checks the average weight of the fish each day, once average weight hits 5 kgs, the it begins harvesting fish based on the plant capacity listed on the “Harvest Parameters” tab. It checks the day of the week to make sure the plant is operating that day and once the biomass on the farm falls below the daily harvest amount, it empties the farm by taking whatever is left.

It’s an incredible tool really. It will suggest layouts for data tables, guide you through troubleshooting when things aren’t working properly etc. If you do any work with spreadsheets, tables or charts, it can be an invaluable resource.

Creating a Custom GPT

Flush with success in using the system for Excel, I was full of ambition and ego when I decided to design my own data analysis tool for analyzing quarterly statements and annual reports – a tool that would put me squarely atop the world of salmon farming analysts so I could dole out sage advice and wisdom to all comers.

My experience over the past few days have left me sad, defeated and deflated as I’ve realized how little I know about how to program and manage the actions of a large language model like ChatGPT.

How to take your own journey of frustration

While I’m not giving up entirely, this foray into prompt engineering has been like trying to manage an extremely bright but utterly lazy teenager. The system continues to make obvious errors, fabricates information if it thinks it can get away with it, and almost never takes the same approach to solving a problem twice.

When confronted with its failings, the system adopts an obsequious tone, apologizes profusely and promises to do better. Then fails again in some new and bizarre way.

How to get started

You will need a paid subscription to ChatGPT to create a custom GPT. On the left side of the home page, you’ll see an option to “Explore GPTs”

ChatGPT Screenshot

Click on “Explore GPTs”.

On the next page, on the top right of the screen, you will see an option for “+create”

ChatGPT Screenshot

Once you hit “+create” a new window opens:

ChatGPT Screenshot

I won’t go into depth on how the system works – here is a link to a Youtube video on how to program a custom gpt.

Essentially, the configuration section has 4 main sections.

Name – name your GPT – I’ve named them as what I want to achieve but I lack imagination

Description – in this section, you need to frame the objective of the GPT – for example, careful, error-free analysis of financial statements by international farming companies.

Instructions – this is the meat of the programing and will include careful step-by-step instructions on where to look (or not look) for key data, which data to extract, calculations to perform and how to present the results.

Conversation starters – these guide users in how to interact with the program. For example, in my program, one of the conversation starters just confirms the reporting period to be analyzed.

Knowledge – ChatGPT only has data up to 2023 (depending on the version you are using) so, if you want the system to analyze reports from Q3 2024, you’ll need to download the reports from the companies you want to examine and upload them into the GPT knowledge section.

Capabilities – if you want to do data analysis, you need to check the box for “Code Interpreter & Data Analysis”

Actions – no idea, I was soundly defeated by this point.

How it went

Developing the instructions was an iterative learning experience. I watched YouTube videos, consulted young people, and asked ChatGPT for support in creating effective instructions. I provided the system with detailed context, specified where in the reports I wanted it to source information, how the results were to be laid out and worked through a painful step-by-step troubleshooting process to identify problems. I spent hours and went through many, many iterations.

Here's an example:

ChatGPT Screenshot

On the third step of my instructions, I had instructed the system to review the uploaded reports and identify the page numbers where the profit and loss, balance sheet and cash flow statements could be found. I assure you the search terms were not “broad” as the system indicates, I had gone through many iterations, each more specific than the last. I wanted the system to focus on the full financial statements for data extraction rather than the narrative summaries and key figures sections. In the example above, the system omitted the pages from the Mowi report despite explicit instructions on how to find them and having successfully found them on several previous attempts. I went through a troubleshooting process more than 30 times yesterday and the system would not return correct values consistently.

I guess that’s really the core of my frustration at this point. When working with excel formulas, I frequently had to go through many iterations to arrive at a formula that worked. I would occasionally have to restate objectives, confirm cell references, troubleshoot elements of the formula etc., but the result was a formula that worked consistently and correctly. When working on an excel formula, the system would reliably work. This was not the case at all with more complicated processes.

Data Integrity

Another surprising and frustrating aspect of the system was its tendency to just throw numbers at the page and hope they stick. Here’s another example:

ChatGPT Screenshot

This was yesterday. The system had produced the correct harvest volume at least 25 times and then, on the next iteration, delivered a number that was obviously incorrect. When I accused the system of “making shit up” it responded with a hurt tone and profuse apologies. I asked it to tell me exactly where it found the number it used and it would not do so – it just went ahead and corrected the stupid entry.

Enough complaining

I could drone on for hours about my frustrations with the system, but I do recognize that the problem is probably me. My failures are likely a reflection of a steep learning curve and limited experience rather than a failing of the system. I’ll update all of you on my future progress if any is made. I will make two closing points:

1)        Do not trust the numbers you receive from ChatGPT for any important purpose unless they are backed by specific references. If it can’t find the specifics, it will provide an answer it thinks you want to hear.

2)        De-bugging a system that is incredibly inconsistent with the processes it uses to find information, lacks clear standards of data integrity, and can’t follow instructions consistently (Even if my instructions are crap, shouldn’t they return the same crap results each time?) seems utterly impossible to me.

You should give it a try. Either I’ll have some “misery loves company” companionship from you, or you may have more success than me.

For those still reading, I salute you. Feedback/comments can be may via the new comment section on the blog, LinkedIn or via email at Info@AlanWCook.com.