Enterprise Generative AI Architecture

Generative artificial intelligence introduces a new way for enterprises to interact with information, applications, and business processes.

Traditional enterprise applications generally operate through predefined interfaces, business rules, and workflows. Users select options, enter information, and receive outputs based on application logic. Generative AI introduces a different interaction model. Users can express requests in natural language, and applications can interpret those requests, retrieve information, generate content, and—in some architectures—invoke tools or initiate controlled actions.

This capability creates opportunities across the enterprise. Employees may use AI assistants to search policies, summarize documents, prepare reports, analyze information, or interact with business applications. Customer-facing applications may provide conversational experiences. Engineering teams may use generative AI to support software development, documentation, and operational processes.

However, enterprise generative AI is not simply a model connected to a chat interface. A production architecture must address information access, model integration, context, application design, security, privacy, evaluation, observability, cost, and operational responsibility. It must also distinguish between generating information and taking action.

The Enterprise Data Architect has an important role in this environment because generative AI depends on enterprise information and context. The architect must work with AI, application, security, and business teams to ensure that information is made available through appropriate interfaces and that the resulting architecture remains governed and maintainable.

This chapter examines the major components and design decisions involved in enterprise generative AI architecture.

12.1 From Traditional Applications to Generative AI Applications

Traditional enterprise applications are usually designed around defined workflows. A user may open a customer record, select a transaction, complete a form, or run a report. The application follows predefined logic and presents information through a structured interface. Generative AI applications introduce a more flexible interaction pattern. A user might ask:

“Summarize the recent activity on this customer account and identify any unresolved service issues.”

The application may need to interpret the request, identify relevant information, retrieve data from multiple sources, organize the results, and generate a response. The user does not necessarily need to know which application contains each piece of information. This can simplify interaction, but it does not eliminate the underlying complexity. The application still needs to determine:

• What the user is asking
• What information is relevant
• Whether the user is authorized to access it
• Which systems or data products should be queried
• How the retrieved information should be interpreted
• How the response should be generated
• Whether the response requires review before use

The conversational interface is only one part of the system. The enterprise architecture must support everything behind it.

12.2 What Is Enterprise Generative AI Architecture?

Enterprise generative AI architecture is the set of capabilities, components, interfaces, controls, and operating practices required to build and operate generative AI applications within an organization. It typically includes:

• User interaction
• Application logic
• Model access
• Prompt and context management
• Retrieval
• Enterprise data access
• Tool integration
• Identity and authorization
• Safety and policy controls
• Evaluation
• Monitoring
• Operational management

The exact design depends on the use case. A document summarization application may require document ingestion, model access, and output validation. An enterprise knowledge assistant may require retrieval, metadata, permissions, citations, and source freshness. An AI assistant that interacts with business systems may also require tool orchestration, transaction controls, and human approval. There is therefore no single generative AI architecture suitable for every enterprise application. The architect should define reusable enterprise capabilities while allowing application-specific designs where justified.

12.3 The Core Components of a Generative AI Application

A useful conceptual model separates a generative AI application into several components.

User Experience

The interface through which users interact with the application. It may be conversational, embedded within an existing application, or integrated into a business workflow.

Application Orchestration

The logic that manages the interaction, determines which capabilities are needed, and coordinates the application flow.

Model Access Layer

The mechanism through which the application accesses one or more language models.

Prompt and Context Management

The process of constructing instructions and providing the information required for a particular task.

Retrieval and Enterprise Information Access

The capabilities used to locate and obtain relevant enterprise information.

Tool Integration

The interfaces through which the application can access approved services, APIs, or business functions.

Security and Policy Controls

The controls that govern identity, permissions, information use, and application behavior.

Evaluation and Observability

The mechanisms used to assess output quality, monitor performance, and investigate issues.

These components may be implemented within one application framework or distributed across enterprise services. The architectural requirement is that their responsibilities and interfaces are understood.

12.4 The Model Access Layer

The model access layer provides a controlled way for enterprise applications to interact with generative AI models. An enterprise may use:

• Models hosted by an external provider
• Models available through a cloud platform
• Internally hosted models
• Multiple models for different tasks

The model access layer can help standardize how applications connect to these models. It may provide:

• Authentication
• Request routing
• Usage controls
• Model selection
• Rate limiting
• Logging
• Cost tracking
• Policy enforcement

This is particularly useful when multiple applications use different models. Without a common approach, each application may implement its own authentication, logging, configuration, and usage controls. That can create duplication and inconsistent oversight. However, a shared model access layer should not conceal important model differences. Different models may vary in capability, latency, cost, context limits, supported modalities, and deployment constraints. The architecture should make model selection explicit and allow applications to use models appropriate to their requirements.

12.5 Model Selection as an Architectural Decision

Model selection is often treated as a technical comparison. In enterprise architecture, it is also a question of business fit, risk, and operating requirements. The architect should consider:

• Task complexity
• Response quality
• Latency
• Cost
• Data sensitivity
• Deployment options
• Availability
• Model capabilities
• Integration requirements
• Evaluation results

A larger model is not automatically required for every task. A simple classification or extraction task may be handled by a more specialized or smaller model, depending on the requirements and evaluation results. A complex reasoning or synthesis task may require a different capability profile. The architecture should avoid unnecessary dependence on a single model when the enterprise has legitimate requirements for flexibility or resilience. At the same time, supporting many models introduces operational complexity. The organization must balance choice with the cost of maintaining, evaluating, securing, and monitoring multiple model integrations.

12.6 Prompt and Context Architecture

Prompts are often discussed as text instructions supplied to a model. In an enterprise application, prompt management is broader. A prompt may include:

• System-level instructions
• Application instructions
• User requests
• Retrieved enterprise information
• Conversation history
• Structured context
• Tool descriptions
• Output constraints

These elements influence how the model responds. The architect should treat prompt construction as part of application design rather than as an informal string assembled inside code. Prompts should have ownership, versioning, testing, and change management appropriate to the application. This is especially important when prompts contain business rules or influence consequential workflows.

Context Is Not Just More Text

Adding more information to a prompt does not necessarily improve the result. Context should be relevant, current, authorized, and appropriate for the task. A customer-service assistant may need the current service policy and the customer’s authorized service record. It may not need unrelated historical documents or information from other customers. The architecture should therefore provide mechanisms to select context rather than simply pass large collections of information to the model.

12.7 Retrieval-Augmented Generation

Retrieval-augmented generation, commonly called RAG, combines information retrieval with generative AI. Instead of relying only on information learned during model training, a RAG application retrieves relevant information from an external knowledge source and provides it as context to the model. A simplified process is:

1. The user submits a question.
2. The application interprets the request.
3. A retrieval process searches relevant information.
4. The application selects suitable content.
5. The content is provided to the model as context.
6. The model generates a response.
7. The application presents the response and, where supported, its sources.

RAG can help an application use enterprise information that is not part of a model’s training data or that changes more frequently than model training cycles. It does not guarantee correctness. The quality of the result depends on source quality, retrieval relevance, permissions, context construction, model behavior, and application design. RAG architecture is examined in greater depth in Chapter 13. For this chapter, the important point is that retrieval is one component of a larger generative AI application architecture.

12.8 Enterprise Knowledge Sources

Generative AI applications may need information from many enterprise sources. These can include:

• Document management systems
• Policy repositories
• Knowledge bases
• Data products
• Operational applications
• Analytical platforms
• APIs
• Approved external information

The architecture should identify which sources are authoritative for each use case. A document repository may contain several versions of a policy. A knowledge base may include both approved guidance and informal discussion. A data product may provide a governed business definition that differs from an application’s local label. The application needs a way to distinguish these sources. Source authority, effective date, ownership, classification, and lineage can all affect whether retrieved information is suitable. This reinforces the need for enterprise metadata and semantic architecture.

A RAG system that indexes documents without preserving their business context may retrieve content that is textually relevant but operationally inappropriate.

12.9 Tool Integration and Enterprise APIs

Some generative AI applications need to do more than retrieve information. They may need to invoke tools. A tool can be an API, service, function, or controlled business capability exposed to the AI application. Examples include:

• Searching a product catalog
• Checking an order status
• Calculating a value
• Retrieving an account balance
• Creating a service request
• Preparing a transaction for approval

Tool integration introduces a major architectural distinction. Retrieving information and changing enterprise state are different activities. A read-only query may have relatively limited consequences. A tool that changes a customer record, initiates a payment, or modifies an operational process requires stronger controls. The architecture should consider:

• User authorization
• Application authorization
• Allowed operations
• Input validation
• Transaction boundaries
• Confirmation requirements
• Audit records
• Error handling
• Rollback or compensation where applicable

The model should not be treated as the authority for deciding whether a business action is permitted. The application and enterprise control layers must enforce those rules.

12.10 Human Approval and Controlled Action

Generative AI applications can support different levels of autonomy. At one end, the system generates information for a human to review. At another, it may execute a predefined action after satisfying application rules. Between these points are designs in which the AI prepares an action, but a person must approve it. The appropriate design depends on:

• Business impact
• Reversibility
• Risk
• Policy
• User expectations
• Reliability of the process

For example, an assistant might summarize a customer complaint and draft a response. A human service representative reviews the response before it is sent. A different application might automatically categorize a low-risk internal request, provided that the action is reversible and subject to monitoring. The architect should make the autonomy boundary explicit.

The central principle is that authority to act should be established by enterprise policy and application design, not inferred from the model’s ability to generate a proposed action.

12.11 Security Architecture for Generative AI

Generative AI introduces familiar security concerns in new interaction patterns. The application may process confidential information, retrieve restricted documents, or interact with enterprise services. Security must therefore apply across the full request lifecycle. Important concerns include:

• User authentication
• Application identity
• Authorization
• Data classification
• Encryption
• Secrets management
• Network controls
• Logging
• Retention
• Access to model providers
• Output handling

A particularly important requirement is preserving permissions during retrieval. If a user is not authorized to access a document, the application should not expose its contents through a generated response. This means access control must be enforced before or during retrieval, not merely after the model has generated an answer. The architecture should also consider whether sensitive information is sent to external model services and what contractual, technical, and policy controls apply. These decisions depend on the organization’s risk requirements and the specific deployment model.

12.12 Prompt Injection and Untrusted Content

Generative AI applications may process instructions contained in user input or retrieved content. Some content may be misleading, malicious, or simply unrelated to the task. A document could contain text that attempts to influence the application’s behavior. A user could submit a request designed to override application instructions. This is commonly discussed under the broader topic of prompt injection. The architectural response should not depend on assuming that prompts alone can eliminate the risk. Controls may include:

• Treating retrieved content as untrusted data
• Separating instructions from source content
• Limiting available tools
• Enforcing authorization outside the model
• Validating tool inputs
• Restricting sensitive operations
• Monitoring unusual behavior
• Requiring human approval for consequential actions

The most important design principle is to avoid giving the model unrestricted authority over enterprise systems. A model may help determine what action is relevant, but application controls should determine whether the action is allowed.

12.13 Privacy and Sensitive Information

Generative AI applications may process personal, confidential, financial, or otherwise sensitive information. The architecture should identify:

• What information is collected
• Why it is required
• Where it is processed
• Which components receive it
• How long it is retained
• Who can access it
• Whether it is used for model training or other purposes

Data minimization remains important. An application should provide only the information required for its intended task. For example, a policy assistant may need the policy text but not customer records. A customer-service assistant may need selected customer information but not unrelated employee or financial records. Privacy controls should be applied to source data, derived data, prompts, logs, retrieved content, and generated outputs. The architecture should also consider whether conversational history is retained and whether it contains sensitive information.

12.14 Evaluation and Testing

Traditional software testing often focuses on whether the application produces expected outputs for defined inputs. Generative AI systems introduce variability. The same request may produce different wording or structure. A response may be fluent but unsupported. A retrieval process may return relevant information while missing a critical source. Evaluation must therefore examine more than whether the application runs. A generative AI evaluation approach may include:

• Task completion
• Factual consistency
• Relevance
• Completeness
• Source support
• Policy compliance
• Refusal behavior
• Latency
• Cost
• Robustness

The appropriate measures depend on the use case. For a document summarization application, reviewers may assess whether important facts are preserved and unsupported claims are avoided. For a RAG application, evaluation may examine whether relevant sources were retrieved and whether the response is supported by those sources. For a tool-using application, tests may examine whether the correct tool was selected and whether unauthorized actions were prevented. Evaluation should be performed before release and continue during operation.

12.15 Observability and Operational Monitoring

Generative AI applications require observability across several layers. Traditional infrastructure monitoring can identify service availability, resource consumption, and latency. Application monitoring can identify request failures and user experience issues. AI-specific monitoring may need to examine:

• Model response characteristics
• Retrieval performance
• Tool invocation
• Token consumption
• Response latency
• Evaluation results
• Policy violations
• User feedback

The architecture should also support investigation. When an application produces an unexpected result, the team may need to understand:

• The user request
• The application configuration
• The prompt version
• The retrieved sources
• The model used
• The tools invoked
• The resulting output

Logging must be designed carefully because prompts and outputs may contain sensitive information. The organization should define what is captured, how it is protected, and how long it is retained.

12.16 Cost and Performance Architecture

Generative AI introduces cost factors that differ from many traditional applications. Depending on the deployment, costs may be affected by:

• Model usage
• Input and output size
• Number of requests
• Retrieval operations
• Embedding generation
• Storage
• Compute
• Network transfer
• Monitoring and evaluation

Performance may be influenced by:

• Model selection
• Prompt size
• Retrieval latency
• Number of tool calls
• Application orchestration
• Concurrency
• Model service availability

The architect should consider these factors as part of the solution design. For example, a system that repeatedly retrieves and includes large amounts of irrelevant information may increase both latency and cost. A system that uses a high-capability model for every task may incur unnecessary expense. However, reducing cost should not compromise the application’s required quality or controls. A practical architecture may use different model or processing paths for different task types, provided the selection is governed and evaluated.

12.17 Reusable Enterprise Generative AI Capabilities

As AI adoption expands, enterprises should avoid building every application independently. Common capabilities may include:

• Model access
• Identity integration
• Prompt management
• Retrieval services
• Document processing
• Tool integration
• Evaluation
• Monitoring
• Policy enforcement

These capabilities can form an enterprise AI enablement layer. The objective is not to centralize every application. Business teams may still own their use cases, workflows, and user experiences. The shared layer provides reusable services and controls.

This approach can reduce duplicated engineering and improve consistency. It also creates a clearer place for enterprise architecture to establish standards. However, shared services should be designed around real application needs. A centralized AI platform that is difficult to use or too restrictive may encourage teams to build outside it. As with data platform engineering, the internal developer experience matters.

12.18 Generative AI Architecture Patterns

Different use cases call for different patterns.

Pattern 1: Model-Only Generation

The application sends a request to a model and returns the generated result. This may be suitable for tasks that do not require enterprise-specific information, such as rewriting user-provided text.

Pattern 2: Retrieval-Augmented Generation

The application retrieves relevant enterprise information and supplies it as context to the model. This is useful when responses must be grounded in enterprise knowledge.

Pattern 3: Tool-Enabled Assistant

The application allows the model to request approved tools through a controlled orchestration layer. This is useful when the task requires interaction with enterprise services.

Pattern 4: Workflow-Embedded AI

Generative AI is embedded within an existing business application or process. This can make AI available within established workflows and controls.

Pattern 5: Human-in-the-Loop Generation

The application produces a draft, recommendation, or proposed action for human review. This is useful where human judgment or approval is required.

These patterns are not mutually exclusive. A single application may combine retrieval, tool access, workflow integration, and human approval.

12.19 Choosing an Architecture for the Use Case

The architect should begin with the business requirement rather than selecting a model or framework first. A structured assessment can examine:

The purpose is not to create a universal decision tree. It is to ensure that important architectural requirements are identified before implementation choices become difficult to change.

Case study: Meridian Financial Group

12.20 Enterprise Generative AI Architecture at Meridian Financial Group

Meridian Financial Group had several independent generative AI experiments. One team had developed a document assistant. Another had created a customer-service prototype. A third was exploring an assistant for financial analysts. Each initiative used different approaches to model access, prompt management, information retrieval, and monitoring. The experiments demonstrated potential value, but Meridian identified architectural concerns. There was no consistent way to understand which enterprise information an application used. Some applications maintained separate copies of documents. Tool access was not standardized. Evaluation practices varied. Cost and usage information was difficult to compare.

Meridian established a reusable enterprise generative AI architecture.

Shared Enablement Capabilities

The architecture introduced common capabilities for:

• Model access
• Identity integration
• Retrieval services
• Prompt and configuration management
• Approved tool integration
• Evaluation
• Monitoring
• Usage and cost visibility

Business units continued to own their applications and use cases.

Controlled Information Access

Meridian connected applications to selected governed data products and knowledge services. This reduced the need for each application to build independent connections to operational systems. The architecture also required each use case to document its information sources, access rules, and traceability requirements.

Controlled Tool Use

For applications that could interact with enterprise systems, Meridian established a distinction between read-only tools and tools capable of changing business state. Actions with significant business impact required additional application controls and, where appropriate, human approval.

Operational Oversight

Meridian introduced a common approach to evaluating and monitoring applications. The approach considered not only availability and latency but also retrieval behavior, response quality, source support, and cost.

The result was a reusable architecture that supported different applications without requiring every business unit to design its own foundation. The key was not forcing all use cases into one implementation. It was establishing consistent enterprise capabilities and controls around them.

Practical Artifact: Enterprise Generative AI Architecture Assessment

This assessment can be used when evaluating a proposed generative AI application or reviewing an existing implementation.

Step 1: Define the Business Use Case

Document:

• Business objective
• Intended users
• Tasks supported
• Expected outcomes
• Systems involved
• Consequences of incorrect output
• Human responsibilities

Be specific about what the application is intended to do and what it is not authorized to do.

Step 2: Identify the Application Pattern

Determine whether the application is:

• Model-only generation
• Retrieval-augmented generation
• Tool-enabled
• Workflow-embedded
• Human-in-the-loop
• A combination of these patterns

Record why the selected pattern fits the use case.

Step 3: Map the Information Flow

Identify:

• Source systems
• Data products
• Documents
• Retrieval services
• Metadata
• User context
• Model inputs
• Generated outputs

Determine where information is copied, transformed, indexed, or retained.

Step 4: Define Model Access

Document:

• Model provider or hosting approach
• Model selection criteria
• Authentication
• Routing
• Usage limits
• Fallback behavior
• Model version management

Identify dependencies on specific model capabilities.

Step 5: Define Context and Prompt Management

Document:

• Instruction ownership
• Prompt versioning
• Context sources
• Conversation history
• Retrieval behavior
• Output requirements
• Testing approach

Determine how changes to prompts and context will be reviewed and deployed.

Step 6: Define Security and Privacy Controls

Assess:

• User identity
• Application identity
• Authorization
• Data classification
• Sensitive information handling
• External processing
• Logging
• Retention
• Tool permissions

Verify that access controls are preserved through retrieval and generation.

Step 7: Define Tool and Action Boundaries

For each tool, document:

• Purpose
• Permitted operations
• Authorized users
• Application permissions
• Input validation
• Approval requirements
• Audit requirements
• Error handling

Separate read-only access from operations that change business state.

Step 8: Define Evaluation

Establish how the application will be tested for:

• Relevance
• Factual support
• Completeness
• Policy compliance
• Retrieval quality
• Tool correctness
• Robustness
• Latency
• Cost

Use representative examples from the intended business environment.

Step 9: Define Observability and Operations

Determine what will be monitored, who owns incidents, and how the application will be supported. Include:

• Service availability
• Response latency
• Model usage
• Retrieval behavior
• Tool execution
• Quality indicators
• Security events
• Cost

Define logging and retention controls for sensitive content.

Step 10: Review Enterprise Alignment

Confirm alignment with:

• Enterprise data architecture
• Data products
• Information and semantic architecture
• Security standards
• AI governance
• Platform strategy
• Business operating model

Identify capabilities that should be reused rather than independently implemented.

Enterprise Generative AI Architecture Checklist

Remember these

Key Takeaways

1. Enterprise generative AI architecture is much more than connecting a model to a chat interface.
2. A production application requires coordinated capabilities for model access, context, retrieval, tools, security, evaluation, and operations.
3. The model access layer can provide reusable controls while preserving the ability to select models appropriate to different tasks.
4. Prompt and context management should be treated as application architecture, with appropriate ownership, testing, and change control.
5. Retrieval can connect generative AI applications to enterprise knowledge, but it does not guarantee correctness.
6. Tool integration creates a distinction between retrieving information and changing enterprise state.
7. Authorization and business policy must be enforced by application and enterprise controls, not delegated to the model.
8. Security and privacy must apply throughout the information path, including retrieval, prompts, outputs, logs, and tools.
9. Evaluation should examine relevance, source support, policy compliance, tool behavior, and operational characteristics.
10. Observability must cover infrastructure, applications, models, retrieval, tools, and usage.
11. Reusable enterprise AI capabilities can reduce duplication, provided they meet the needs of application teams.
12. The Enterprise Data Architect helps ensure that generative AI applications use trusted, governed enterprise information through explicit interfaces.

Think about your own organization

Architect’s Reflection

Generative AI changes the way people interact with enterprise systems, but it does not remove the need for enterprise architecture. In some respects, it makes architecture more important. A conversational interface can make a complex enterprise environment appear simple. A user asks a question and receives an answer. Yet behind that interaction may be multiple data products, document repositories, APIs, models, retrieval services, and security controls. If these components are not designed together, the simplicity of the user experience can conceal significant complexity. The Enterprise Data Architect must therefore look beyond the interface. The architect should ask where the information originates, what it means, how it is retrieved, who is allowed to access it, and how the application uses it. The same discipline applies to action. A model may propose an action, but the enterprise must define whether that action is permitted, who authorizes it, and how it is recorded. This distinction between generated output and enterprise authority is fundamental.

It is also important not to over-centralize. A common enterprise AI foundation can provide valuable shared capabilities, but different business problems may require different application patterns. A document assistant, a customer-service application, and an operational agent do not necessarily need identical architectures. The goal is to standardize the capabilities that should be shared while preserving appropriate flexibility at the application level. For the Enterprise Data Architect, this creates a broader responsibility. The architect must help design not only where enterprise information is stored, but how it becomes usable context for intelligent applications.

The data platform, data products, semantic services, model access, retrieval, and application controls must operate as a coherent architecture. When these capabilities are connected, generative AI can become part of the enterprise’s information and application landscape rather than remaining a collection of disconnected experiments. That is the architectural transition: from isolated model adoption to governed enterprise generative AI capabilities.