Generative AI for embedded applications

The recent periodic buzz around AI has been almost exclusively around large language models (LLM) and generative AI models, a trend that reflects the growing influence and popularity of these topics in recent years. Applications related to large language models and generative AI models cover a wide range of fields, from open chatbots to task-based assistants. While LLM is primarily focused on cloud-based and server-side applications, there is also growing interest in deploying these models in embedded systems and edge devices.

Embedded systems (such as microprocessors in appliances, industrial equipment, automobiles, etc.) need to accommodate limited computing power and memory availability at limited cost and power consumption. This makes deploying high-precision and high-performance language models on edge devices extremely challenging.

Deploy LLM on edge devices

In embedded solutions, a key area to leverage LLM is the natural conversational interaction between the operator and the machine, known as the human-machine interface (HMI). Embedded systems can simplify a variety of input options, such as microphones, cameras, or other sensors, but most systems won't have a full keyboard to interact with the LLM model, as PCS, laptops, and mobile phones do. Therefore, embedded systems must be practical when using audio and vision as LLM inputs.

This requires a pre-processing module for automatic speech recognition (ASR) or image recognition and classification. Similarly, the output options for interactions are limited. The embedded solution may not have a screen, or it may not be easy for users to read the screen information. Therefore, after the generative AI model, a post-processing step is required to convert the model output to audio using a text-to-speech (TTS) algorithm. NXP is building eIQ® GenAI Flow to make edge-generative AI more practical by adding the necessary pre-processing and post-processing modules to make it a modular process.

Revolutionize applications with LLM

By integrating LLM-based speech recognition, natural language understanding, and text generation, embedded devices are able to provide a more intuitive and conversational user experience. This includes smart home devices that respond to voice commands, industrial machinery that is controlled by natural language, and automotive entertainment central control systems that are capable of hands-free conversation to direct users or operate in-car functions.

LLM also plays a role in embedded predictive analytics and decision support systems in health applications. Devices can embed language models trained with domain-specific data and then use natural language processing to analyze sensor data, identify patterns, and generate insights, all while operating in real time at the edge and protecting patient privacy without sending data to the cloud.

Addressing generative AI challenges

There are many challenges to deploying accurate and powerful generative AI models in embedded environments. The size and memory usage of the model need to be optimized so that the LLM can accommodate the resource limitations of the target hardware. Models with billions of parameters require thousands of megabytes of storage, which can be costly and difficult to implement in edge systems. Model optimization techniques such as quantization and pruning are applicable not only to convolutional neural networks, but also to converter models - an important way for generative AI to overcome the model size problem.

Generative AI models like LLM also have knowledge limitations. For example, their understanding is limited, often providing inconsistent answers, also known as "hallucination," and their knowledge is limited by the timeliness of the training data. Training a model or fine-tuning a model through retraining can improve accuracy and context awareness, but this can be costly in terms of data collection and required training calculations. Fortunately, where there is demand, there is innovation; This problem can be solved through retrieval Enhanced Generation (RAG). The RAG approach uses context-specific data to create a knowledge database that the LLM can refer to at run time to accurately answer queries.

eIQ GenAI Flow applies the benefits of generative AI and LLM to edge scenarios practically. By integrating RAG into this process, we provide domain-specific knowledge for embedded devices without exposing user data to the training data of the original AI model. This ensures that any changes to the LLM are private and only used locally at the edge.