Business data analytics is becoming increasingly reliant on AI, with many teams using Large Language Models (LLMs) to distill insights from proprietary datasets, but this practice requires careful handling to preserve data security and privacy.
According to the Immuta AI Security and Governance report, 80% of data experts agree that AI is making data security more challenging. Additionally, 88% of data professionals say employees at their organizations are using AI, but only 50% say their organization’s data security strategy is keeping up with AI’s rate of evolution.
The main problem is that people often send data to LLMs as part of their ongoing workflows without considering security. Many organizations even lack basic monitoring or visibility into what AI is being used, a phenomenon known as “shadow AI.” The top concern regarding AI for 56% of data professionals is the risk of sensitive data exposure via an AI prompt.
Two elements need to be in place for a prompt leak to happen: user input (sensitive data uploaded by employees) and model output (when the LLM generates or reveals confidential information to someone else based on prior interactions or training data). These leaks are more common than assumed, as shown by the DeepSeek incident from January 2025, where millions of lines of chat logs, API keys, and other sensitive information were exposed, affecting many organizations.
There are five ways organizations can prevent data leaks when using LLMs. Firstly, organizations should avoid giving AI direct access to data. LLMs should never connect directly to production databases or sensitive systems, protecting against situations like the DeepSeek leak. A layer can be built to obscure the data before queries are passed to the LLM.
Pyramid Analytics is a decision intelligence solution that separates the AI layer from actual data without compromising output quality. When a user asks Pyramid’s AI chatbot a question, the engine sends a high-level version of the question to the external AI model, along with a description of the data. Pyramid then runs the query within the organization’s environment and returns results in interactive dashboards, charts, or reports, ensuring the LLM never directly interacts with the organization’s data.
Secondly, implementing strong access controls is crucial. Access to LLMs should be restricted based on employee roles, with appropriate restrictions for both data and model access. Role-based access controls (RBAC) support enforcing the principle of least privilege, ensuring users, models, and connected tools have minimum access and capabilities needed to perform tasks. Mature AI organizations may use an MCP server to control how LLMs interact with external resources.
Thirdly, rethinking prompt engineering is necessary. Prompts are how users interact with LLMs, and if not designed with security in mind, they become a serious vulnerability. AI systems must differentiate legitimate prompts from harmful ones by implementing validation rules for incoming prompts, checking for suspicious patterns, and deploying tools like LLM Guard to analyze prompts in real-time.
Fourthly, logging and monitoring AI output and usage is essential. LLMs should be treated like any other business technology, with oversight on what’s being asked, responses generated, and who interacts with the model. This helps identify inappropriate use, policy violations, or issues leading to data leakage, ensuring the LLM operates securely and ethically.
Fifthly, training employees on LLM risks is vital. Even with restrictive security controls, employees can pose a risk by sharing sensitive data or relying on unapproved AI tools. Awareness training platforms like Ninjio offer AI-specific modules that educate employees on best practices and risks associated with LLM use, teaching them to avoid sharing sensitive info and evaluating AI-generated outputs critically.
LLMs are a powerful but relatively immature technology, and despite efforts to standardize security practices, there is still significant risk for organizations, potentially resulting in data leaks. Security should be built into every layer of an LLM implementation, and the measures discussed provide a solid foundation for integrating LLMs safely and responsibly.
