How AI Systems That Understand and Generate Code Can Improve Software Development.

AI and machine learning (ML) systems are gaining more traction among large enterprises by the day. And it’s not hard to see why: ML models help organizations scale and improve various tasks, from medical device manufacturing to predicting Leukemia relapses. 

Other AI models, such as diffusion and generative models, can generate synthetic voices, videos, or images.

The latest emerging AI trend? Deep learning (DP) techniques for natural language processing (NLP) models that understand and even generate text for tasks such as language translation, blog writing, and even creating computer code for software and other applications. 

AI making inroads in the enterprise, but with room to grow

Recent research shows most companies are relative novices in the AI game, with a 2022 Accenture survey indicating that nearly 65 percent of companies are “AI experimenters” that “lack mature AI strategies” and capabilities.

Only 12 percent of companies were considered “AI achievers” – organizations with differentiated and operationalized AI strategies.

That means there’s still plenty of opportunity for enterprises dabbling in AI to scale and become more efficient in several ways. 

Indeed, additional research from McKinsey shows that nearly 30 percent of respondents can attribute at least five percent of their earnings to AI – an increase of five percent from the year prior. 

Most companies use AI models for service or operations optimization, product enhancements, contact center automation, and product feature optimization. Hidden within these broad categories, however, are an increasing number of enterprises that now use AI-based language models to help developers write code more efficiently.

Types of AI models used for code development

The application of classic NLP models to help developers generate source code and improve productivity isn’t all that new. Domain-specific language-guided, probabilistic grammars, n-gram language models, and simple neural program models have been in use in this regard for years, but are limited in terms of flexibility and time required to configure to specific languages or tasks.

But there’s a new crop of NLP models on the block, based on DL techniques, making massive inroads in code development. These include large language models, fine-tuned language models, and edge language models. 

Each has strengths and weaknesses, along with specific use cases that make the most sense based on model type.

Large language models (LLM)

Large language models are pretty much what they sound like: Language-based models with very large numbers of parameters (or values that can change as the model learns) and that require massive amounts of text-based training data.

These models include: 

• EleutherAI’s GPT-J (six billion parameters)
• OpenAI’s GPT-3 (175 billion parameters) 
• Megatron-Turing Natural Language Generation (MT-NLG) model by Microsoft and Nvidia (530 billion parameters)
• Pathways Language Model (PaLM) by Google (540 billion parameters)

LLMs are very flexible and can be configured to perform several operations, including answering questions, summarizing documents, generating original text, and translation. 

Their downside, however, is that they’re expensive to develop: One study by AI21 Labs indicated that configuring and training a relatively small LLM of 1.5 billion parameters could cost as much as $1.6 million.

Fine-tuned language models

As the name suggests, fine-tuned language models are smaller than LLMs and take far less time, money, and computing power to develop and train for specific use cases. For example: While GPT-3 initially required 45 terabytes of training data, OpenAI’s Codex (a descendant of GPT-3) required just 159 gigabytes.

Fine-tuned models are especially useful for well-established tasks with large amounts of available training data that can benefit from such fine-tuning to generate added value. But despite their smaller size, these models are often still too large to run on local machines.

Edge language models

Edge language models are similar to fine-tuned models in terms of their diminutive stature and data/cost requirements and are deployed primarily to edge devices (such as telecom equipment). 

Code-writing AI models in the real world

The world of code-writing AI models is expanding quickly – almost too quickly, if you ask teachers within some education systems. Here are some of the most popular.

Generative Pre-Trained Transformer (ChatGPT)

OpenAI’s ChatGPT took the world by storm following its Nov. 30 release – people just couldn’t get enough of its uncannily conversational and well-written answers to specific questions. 

Originally designed as an intelligent chatbot, ChatGPT can also develop web pages and applications in various languages, find bugs in code, and even create new programming languages. It can’t yet write complex code on its own, but it’s likely only a matter of time.

Codex

OpenAI’s Codex is based on GPT-3 and is already used within several coding applications and editors, including:

• GitHub Copilot
• Pygma
• Replit
• Warp
• Machinet

Codex can translate English-language text into more than a dozen programming languages. OpenAI says it can help supplement the work of developers and increase their productivity. GitHub Copilot, for example, can suggest entire lines of code or functions within the code editor itself, turn comments into code, and autofill repetitive code. 

Alphacode

There are other examples of code-writing AI that weren’t developed by OpenAI, including DeepMind’s Alphacode. Alphacode was created to write code at a competitive level and ranks within the top 54 percent of programming competitors – placing it as a median competitor among human developers. DeepMind says it’s the “first time an AI code generation system has reached a competitive level of performance in programming competitions.”

Operationalize AI and improve productivity with CapeStart

The Accenture survey mentioned earlier confirms there’s still a huge opportunity for most enterprises – even those already using AI in various capacities – to improve efficiency even further through innovative applications and use cases. 

But developing and operationalizing AI models on your own isn’t easy. Contact us today to learn more about how CapeStart’s teams of AI and machine learning experts can help scale your business and productivity.