March 16, 2023

This is an experiment in using ChatGPT 3.5 to debug a student question.

The responses shown were the first responses. Nothing was cherry-picked.

Debugging a student sketch

ℹ️

Note:

My prompts to ChatGPT are in the left column, in black text with a white background.
ChatGPT responses are in the left column, on a light blue background; except for code, which is white (and other colors) on black.
Commentary about the conversation is in the right column, on gray.

Transcript

Notes

ℹ️

The conversation begins with a student question that was posted to Slack.

This is the original student code (minus a final “Thank you”, that I omitted for no good reason), copied from Slack and posted into ChatGPT.

ℹ️

Notes:

The issue that val is uninitialized is correct. The consequence of this is that the code behaves nondeterministically if the button is pressed when it starts: it might detect this as a button press; it might not.
The comment “you should check for a rising edge” is not correct. The submitted code already does this.
ChatGPT has made a couple of non-functional improvements: it has added comments, and it has changed digitalRead(7) to digitalRead(button).

Experimenting With Stylistic Choices

ℹ️

Note: The code only supports one round (it does not contain logic to reset count back to 0), so this prompt is misleading.

However, it is the answer that was supplied by a (human) instructor in the channel, so I decided to see what ChatGPT would do with it…

…and, in addition to adding code to turn off the LEDs that should not be lit, ChatGPT adds code (the new “overflow” condition) to reset the count back to zero, so that there is another round.

ℹ️

I was fishing for ChatGPT to remove some lines from the count == 2 and count == 3 cases that turned on pins that were already HIGH.

ChatGPT went in a different direction, though, and eliminated all calls to digitalWrite() by a write to PORTB. (I didn’t know about PORTB.)

ℹ️

Notes:

ChatGPT ends its response in the middle of a sentence output. Either it ran out of tokens (sometimes this happens), or there was a service interruption (I tried this the day after ChatGPT 4.0 was announced, and there was a lot of traffic, and a large number of outages, that day).
This could have been even more concise (I’ll stop short of better) if I’d continued to let it use PORTB.

State Machine

ℹ️

Note that ChatGPT came up with descriptive names (OFF, LED1_ON, etc.) for the states of the state machine.

ChatGPT is a Compiler?

ℹ️

Not shown: a series of questions where I asked ChatGPT to use Object-Oriented Programming. It defined a C++ class to represent states, and a subclass to represent each specific state. It was an explosion of overly verbose (although, on a quick read, correct) code, that didn’t demonstrate much. Rather than try to coach it back on track, I restarted the conversation (in the colloquial sense, not by resetting the chat) by asking it to refer back to the code before these changes.

I haven’t scrutinized or tested this assembly code.

ℹ️

This doesn’t look right, but I tried it anyway. (Who knows — ChatGPT had known about PORTB, and I hadn’t…)

ℹ️

…and, as (mostly) expected, it failed, so I pasted in the error message.

ℹ️

The following instructions (untested) are closer to what I had been expecting the first time.

ℹ️

I pressed “Regenerate” to request another answer for the previous prompt, and it generated a completely different (untested) solution.

Finding an Arduino Simulator

ℹ️

ChatGPT is hallucinating. There are Arduino simulators (or emulators), but there is nothing called the “Arduino Simulator”, and there is no site https://arduinosimulator.org.

ℹ️

I hadn’t expected actually ChatGPT to produce a genuine link to an Arduino simulator. My intuition — which is not based on calculation, but is slightly calibrated by exploring ChatGPT’s capabilities — is that GPT-3~3.5 have too few parameters to memorize information that its training corpus contains as little discussion of Arduino emulators as I would expect.

Let’s direct this question to a couple of Chatbots that are actually connected to the web. Bing Chat uses ChatGPT 4.0; I’m guessing Perplexity uses GPT 3.5, plus some secret sauce. Both of them are “retrieval-augmented” models, that combine web search (and possibly other corpora and other augmentations) with language models.

Bing Chat

Perpexity.ai

ℹ️

Note: Perplexity suggests followup questions. One of them is “which simulator is recommended for learning and debugging”. I clicked on this one, in order to generate the following answer:

Debugging a Student Arduino Sketch