Some time ago, I tested AI with some simple questions concerning writing a Fortran subroutine doing something I asked. What I asked was deliberately somewhat obscure, to see how it will react. It failed miserably (I actually posted an example about that before.) It didn’t ask for more details (as it should.) It just gave me a ridiculously wrong answer.
Two weeks or so ago, I tested it with Numerical Methods. This time it failed even more. I will give an example here, but trust me, it wasn’t the only one.
Specifically, I asked if the Cash-Karp method has the FSAL (First Same As Last) property. It happily answered that yes, Cash-Karp method has the FSAL property. This is wrong, it doesn’t. Not only that, but it also added the property was first introduced in the Fehlberg (RKF) method which, again, is wrong (but I guess I can somehow forgive that, since a modified version of RKF with FSAL was introduced way later than the original, although it definitely wasn’t the first method introducing the FSAL property.)
I asked “Are you sure the RKF method has the FSAL property?” This time it replied with apologies that no, it doesn’t, but there is a modified version which does (although I can safely say that nobody uses it.)
Anyway, back to Cash-Karp: As proof of its claim, it gave me two links. The first one was about the Dormand-Prince method (which does indeed has the FSAL property but that’s not the method I asked about,) and the second one didn’t mention Cash-Karp at all.
I insisted asking again: “Are you sure the Cash-Karp method has the FSAL property?” It replied “Yes, I’m sure!” (sic). Then it added some text, basically explaining what that property means. I kept “pushing” by asking it to give me more proof about its claim. This time it gave me three links, two of them dead, one with a text that mentioned Cash-Karp. The syntax was not very good, and a careless reader could interpret it as if Cash-Karp method has the FSAL property (although if you read the text carefully, it clearly says it doesn’t.)
I finally replied with “I think you are wrong. The Cash-Karp method does not have the FSAL property, at least not without heavy custom modifications”. This time it replied that "Yes, you are right, I apologize for my misunderstood answer (sic), the Cash-Karp method does not have the FSAL property. I misunderstood nothing, it clearly gave wrong answers.
Now, suppose I was a student who didn’t know much about those methods and asked AI about it. I would get a wrong answer which, unaware as I would probably be, I could trust it is correct, and further ask for a Fortran subroutine to implement the method, which I doubt it will be correct or optimal, because like I said it failed many times in rather simple Fortran programs.
Oh, and the cherry on top: Two weeks later, I asked again, the same questions as above, assuming the AI learns from its mistakes. It doesn’t seem it does. I got the same answers as before.
I don’t know about you, but my conclusion was, whoever trusts AI for serious work is shooting his/her own foot with a rocket launcher: Even if he misses a bit, the damage will still be devastating. Now, I didn’t use ChatGPT (because it asks to create an account which I refuse to do,) but as far I know, the AI I used has the same algorithms behind the scenes. My overall impression was it acted like a careless fast reader: It collected information from the Internet and combined it to make an answer, but the process was careless and the result may or may not be a correct answer.
Last but not least, maybe one day the AI will be “smart” enough to give better answers but even then, I wouldn’t trust it blindly. And frankly, I hope this will not happen anytime soon. I certainly hope if the time the AI will be good enough to be trusted will ever come, I won’t be around anymore to see that.