-
On March 28, 2022
Using GPT-3 to Automate TrackVia App Script Development
TrackVia App Scripts are a great tool to handle complex operations and math in TrackVia tables. However, writing App Scripts requires a decent understanding of programming fundamentals and the Groovy programming language.
Or does it? What if we could train GPT-3, the most potent publicly-accessible artificial intelligence, to write TrackVia App Scripts from natural language?
What is Artificial Intelligence?
Artificial intelligence is the intelligence possessed, created, or developed by machines. It can also be defined as showcasing a capacity for intelligent behavior.
Modern AI is an extremely complex topic, one far beyond the scope of this blog post. However, the most popular general-use model available is called GPT-3 (Generative Pre-trained Transformer 3), released in 2020 by OpenAI and later acquired by Microsoft. In simple terms, GPT-3 was initially trained on over 430 billion unique language and code data points and can do everything from writing somewhat original text, to maintaining a conversation well enough to pass the Turing Test, to even writing code.
Training GPT-3 on TrackVia Functions
The key to GPT-3’s artificial intelligence is learning through experience. To train GPT-3, you need to give examples of existing code and tell it what to write.
As a company that thrives on automation and artificial intelligence, we knew we had to try to automate TrackVia development. One section of implementation that immediately seemed trivial to automate was the development of App Scripts.
We went through several iterations of training data for GPT-3. The original data set does not include many examples of TrackVia App Scripts, though it is aware of several older blog posts with code examples.
To start, we first tried to directly copy the function list from the TrackVia developer site into a GPT-3 friendly prompt. This strategy worked to an extent. GPT-3 was aware of TrackVia’s unique functions, but since it still did not understand TrackVia’s underlying architecture, the code was full of errors and garbage strings. This strategy is also expensive in the AI world, as it accounts for approximately 1,020 tokens of the overall request. Practically speaking, this is still less than 10 US cents per generation.
The next step was to write a separate training document that would describe how low-code/no-code platforms work on a fundamental level, then explain how TrackVia implements well-known technologies (such as SQL) into these LCNC concepts. The result is a dense training file written in natural language that accounts for approximately 200 tokens. As we wanted to keep our original training data set under 1,000 tokens, we simplified everything as much as possible and barely squeezed in under that goal.
GPT-3 now understands how data inside TrackVia is connected and what layman’s terms such as “child records” mean. It is self-aware of all possible TrackVia data types, most best practices (e.g., exception handling), and limitations (such as the inability to import third-party libraries).
The Generated Code
We approached GPT-3 with three final prompt types:
- As written by a programmer or TrackVia Expert Admin
- As written by a manager
- As written by an end-user
For example, a prompt written by a programmer may look like this:
A Company has many Contacts. We need to find the average age of all contacts in a company. Write a TrackVia app script that loads the children records of the "Companies" table using the relationship "Associated Company". Loop through each child record, also known as contacts, and get the "age" field from its current values. In each loop, validate the age field. If null, show an exception. If not null, add the value of the age field to top-level variable "sumOfAges". Outside of the loop, divide "sumOfAges" by the number of contact records. This will be the average age of all contacts in the company. Save the average age to the "Average Age" field on the company record.
Whereas a manager might write:
I have the following table structure: "Companies" has many "Contacts" Write an app script that takes the average age of each contact, stored in contact field "Age" (if this field is empty, show the user an error message), and save the final average result to company field "Average Age"
And the end-user writes:
Average the age of each contact in the company.
Surprisingly, the more detailed the description is, the worse the quality code GPT-3 generates. We believe that this is due to how GPT-3 interprets and connects natural language phrases. The mid-tier (manager) prompt returns the best results.
The programmer’s prompt returns a possibly functional, but bloated and comment-filled solution. This is the output as GPT-3 generated it, we have not changed anything beyond basic formatting. We believe the comments are based on previous TrackVia blogs, thus the uncanny valley. Additionally, the comments can be easily explained by GPT-3 incorrectly misinterpreting the more detailed prompt as rambling. Our training set specifically instructs GPT-3 on when and how to use comments, as well as providing examples of good comments.
// Get the children records of the "Companies" table using the relationship "Associated Company".
List < Map > contacts = getChildren("Contacts", "Associated Company");
// Initialize a top-level variable to hold the sum of all ages. This will be used to calculate average age.
Int sumOfAges = 0;
// Loop through each child record, also known as contacts, and get the "age" field from its current values.
contacts.each {
contact ->
// In each loop, validate the age field. If null, show an exception. If not null, add the value of the age field to top-level variable "sumOfAges".
if (contact["Age"] == null) {
throw new RuntimeException("Contact must have an Age.");
} else {
// Add this contact's age to our running total for all ages in this company (sumOfAges). We'll use this later to calculate average age for all contacts in this company. Note: we are adding it as a number (Int), not a string (String). That's why we use .toInteger() instead of .toString(). Also note that we are adding it directly into our top-level variable called sumOfAges which is outside of our loop and outside of any function or method definition block! This is because Groovy allows us to do that! It's one reason why Groovy is so cool! :)
sumOfAges += contact["Age"].toInteger();
}
/* Outside of our loop now... divide "sumOfAges" by number of contact records in order to find average age for all contacts in company */
Int avgAge = sumOfAges / contacts.size();
/* Save average age back onto parent record ("Company") */
currentValues["Average Age"] = avgAge;
The manager’s prompt returns a much more elegant and readable solution, though functionally the same as the programmer’s prompt:
// Get the average age of all contacts for a company List
Finally, the end-user’s prompt returns what appears to be another possibly functional script, and is the only one to include some form of error-handling without being explicitly told:
// Get the contacts for the company
List < Map > contacts = getChildren("Contacts", "Link to Company");
Int totalAge = 0;
contacts.each {
contact ->
totalAge += contact["Age"];
}
// Calculate average age of all contacts in company and set on record. If no contacts, set to 0. If only one contact, set to that age. Otherwise calculate average of all ages.
if (contacts == null || contacts.size() == 0) {
currentValues["Average Age"] = 0
} else if (contacts != null && contacts.size() == 1) {
currentValues["Average Age"] = totalAge
} else {
currentValues["Average Age"] = totalAge / (contacts != null ? contacts : [0]).size();
}
Next Steps
Hartmann Industries Labs has determined that it is indeed feasible to rely on GPT-3 to generate baseline TrackVia App Scripts. However, we conclude that the generated code should not be directly used in production. GPT-3 generates TrackVia App Script code that is seemingly functional and impressively elegant considering it’s machine-generated.
There is still a lot for GPT-3, and GPT-4, to learn before they can be considered directly usable by a manager or citizen developer. In the meantime, Hartmann Industries Labs continues to invest in TrackVia development automation technologies and fine-tuning GPT-3 for our back-end implementation use in App Scripts and Microservices.