Welcome to COGNITO
AI Readiness Framework Assessment Tool
COGNITO is a structured AI Readiness Self-Guide Framework developed by Idaho National Laboratory, specifically tailored for grid operators of all types and sizes—including investor-owned utilities, municipal utilities, electric cooperatives, public power authorities, independent system operators, regional transmission organizations, and balancing authorities. It helps organizations evaluate their readiness to adopt AI, pinpoint optimal areas that would benefit from AI adoption, identify appropriate use cases based on consequence profiles and technology maturity, assess potential operational and safety risks, and implement appropriate engineering controls.
The Five-Step Process
A structured journey from assessment to implementation
Identify Business Context
Where are we today? What capabilities and gaps define our readiness for AI?
Map current capabilities to identify where AI can fix critical gaps, optimize processes, or enhance high-performing operations. Document specific opportunities with quantified impacts.
Align AI Use Cases
What should we tackle first and what are the stakes? Which use cases align with our mission and risk tolerance?
Match opportunities to appropriate AI applications using a catalog organized by use case risk level (Low to Highest) and technology readiness.
Analyze AI Principles
What do we actually have, what do we need, and what gaps require attention before proceeding?
Explore readiness through structured questions across seven critical principles including risk, data, infrastructure, and compliance.
Implementation Planning & Readiness Validation
What exactly are we building? How will we measure success, and should we build, buy, or partner?
Develop detailed scenarios connecting current operations to AI-enabled futures. Determine build/buy/partner approach and validate resources are in place to execute.
Evaluate Engineering Controls & Mitigations
How will we deploy and sustain it safely? What controls are required for dependable AI operation?
Identify safeguards across engineering, operational, and governance domains to support responsible deployment.
How This Tool Works
- Auto-Save: Your progress is automatically saved in your browser as you work. You may save your progress at any point by clicking the Save button at the top of the screen.
- Navigate & Track Progress: Navigate and monitor your completion status for each step using the navigation pane on the left-hand side of the screen. The navigation pane is collapsible.
- Backup & Collaborate: Use the "Export JSON" button at the top of the screen to export your work for later, share with team members, and/or transfer between browsers. The exported file contains code-like text (this is normal)—simply use "Import JSON" to load your saved progress back into the tool.
- Export Reports: Generate professional PDF or TXT assessment reports for documentation and stakeholder review. Reports can be exported upon completion of Phase 1 and Phase 2. Additional details can be found below.
What You'll Get From COGNITO
1 Phase 1 Deliverable: Strategic AI Readiness Report
Complete Steps 1-2 and Step 3 Risk Analysis
Your Phase 1 Assessment provides everything needed for initial decision-making and stakeholder discussions:
- ✓ Business Maturity Analysis - Documented assessment of your current capabilities across key operational areas with maturity scores (1-5 scale)
- ✓ Identified AI Opportunities - Prioritized list of specific opportunities mapped to AI domains (Detection, Prediction, Control & Optimization, Business & Customer Applications)
- ✓ Selected Use Cases - Recommended AI applications matched to your consequence tolerance and technology readiness
- ✓ Quantified Risk Assessment - Calculated risk scores (1-45 scale) with detailed consequence analysis across 9 categories including safety, service continuity, and asset integrity
- ✓ Security Threat Analysis - Identified AI-specific vulnerabilities (data poisoning, model theft, adversarial attacks) relevant to your selected use cases
- ✓ Initial Control Recommendations - Engineering, operational, and governance controls proportional to your risk level
Use this report to: Justify AI investment, align stakeholders on priorities, determine if you're ready to proceed, and establish baseline expectations.
2 Phase 2 Deliverable: Full AI Readiness and Implementation Analysis
Complete all 5 Steps
Your Full Assessment provides comprehensive project documentation ready for execution:
- ✓ Everything from Phase 1, plus:
- ✓ Complete Readiness Validation - Assessment across all AI principles (Risk Management, Data Availability & Infrastructure, Investment Capacity, Skilled Personnel, Governance and Compliance, Clear Objectives) with documented gaps and remediation paths
- ✓ Detailed Implementation Scenarios - Step-by-step current-to-future-state mapping with specific technical requirements, data flows, and system integration points
- ✓ Build/Buy/Partner Decision - Systematic evaluation and recommendation with documented rationale based on your capabilities, timeline, and resources
- ✓ Success Metrics & KPIs - Defined baselines, targets, and measurement approaches with clear pilot-to-production advancement criteria
- ✓ Comprehensive Control Framework - Full engineering controls, operational procedures, and governance mechanisms tailored to your consequence profile
Use this report as: Project charter for your AI initiative, executive brief for funding approval, vendor RFP foundation, or internal project justification document.
Preparation Tip
Phase 2 assessments require specific organizational information and input from subject matter experts (SMEs). You can complete Phase 1 first, then gather the necessary stakeholders and details before proceeding to Phase 2.
Both reports export as formatted PDF or text files suitable for distribution, editing, and integration into your project planning and executive/board briefings.
Step 1: Identify Business Context
Purpose
Identifying the business context helps you understand your organization's current operational capabilities and identify where AI can deliver meaningful value. AI initiatives routinely fail not due to technology limitations, but because organizations pursue use cases that don't align with their actual business capabilities, data readiness, or strategic priorities.
Key Objective: This step assesses your organization's current operational capabilities, identifies specific pain points, and outlines opportunities where AI can deliver meaningful value.
This assessment serves two distinct organizational needs:
Exploring Where AI Can Add Value?
The business capability assessment provides a comprehensive view across multiple functions, revealing opportunities you may not have considered.
Specific AI Application in Mind?
This framework enables precision readiness evaluation for targeted use cases like predictive maintenance or wildfire monitoring.
What You'll Do:
Select Capabilities to Assess
Choose one or more capabilities to evaluate (select at least one):
Need to change your capability selections?
You can go back and modify which capabilities to assess.
Ready to continue?
Complete Step 1 and move to Step 2: Align AI Use Cases
Step 2: Align AI Use Cases
Match your identified opportunities to appropriate AI applications using use case risk level and technology readiness criteria.
Align Opportunities to Use Cases
For each opportunity identified in Step 1, assign a use case from the catalog and confirm the use case risk level and technology readiness.
|
Capability
|
Opportunity
|
Use Case |
Use Case Risk Level
|
Technology Readiness
|
Status |
|---|
Step 3: Analyze AI Principles
Instructions:
Step 3 evaluates YOUR organization's specific risk profile for implementing the selected AI use case. While Step 2 identified the inherent risk of the AI application itself, Step 3 determines whether your organization possesses the foundational capabilities, data readiness, governance structures, and technical expertise required to implement the use case safely and effectively. Structured questions help you explore critical dimensions of AI readiness, evaluate your current state, and identify gaps requiring attention. This approach ensures you ask the right questions about organizational capabilities and produces documentation that reveals whether you're ready to proceed, need to address specific gaps, or require deeper investigation.
COGNITO uses a two-phase approach that allows organizations to choose their depth of analysis based on their current needs: Phase 1 offers an abridged assessment for entry-level readiness, while Phase 2 completes the full COGNITO framework with comprehensive analysis.
Understanding Two-Phase Approach: Phase 1 and Phase 2
Risk Management is the essential foundation that accompanies Steps 1 and 2. Completing Phase 1 provides sufficient insight for initial decision-making, stakeholder discussions, and go/no-go decisions. You can export a Phase 1 report at this point.
The remaining five principles provide comprehensive depth of analysis for organizations actively approaching AI implementation. Phase 2 validates resources, capabilities, and governance structures needed for successful execution.
The Six AI Readiness Principles
Ability to identify, assess, and mitigate AI-specific risks including model failures and unexpected outputs
Access to quality data and technical systems needed to deploy and scale AI solutions
Financial resources for development, implementation, and ongoing maintenance
Staff with technical skills and domain expertise to develop, deploy, and maintain AI
Ability to meet industry regulations, data privacy requirements, and AI governance standards
Well-defined business goals, success metrics, and expected outcomes for the AI initiative
Phase 1 - Critical Assessment
Risk Management is the essential foundation that accompanies Steps 1 and 2. Complete this for initial decision-making and stakeholder discussions.
Why Phase 1 Matters: Completing Phase 1 provides sufficient insight for go/no-go decisions. You can export a Phase 1 report at this point, or continue to Phase 2 for comprehensive analysis.
Phase 1 Complete! 🎉
You've completed the initial assessment (Business Context, Use Case Alignment, and Risk Analysis). This provides enough insight for initial decision-making and stakeholder discussions.
✓ Export your Phase 1 Report for stakeholder review
→ Continue to Phase 2 to complete the remaining 5 principles for full implementation readiness
Phase 2 - Deep Readiness & Implementation Planning
Complete the remaining 5 principles for comprehensive readiness assessment and implementation guidance.
Why Phase 2 Matters: The remaining five principles provide comprehensive depth of analysis for organizations actively approaching AI implementation. Phase 2 validates resources, capabilities, and governance structures needed for successful execution.
Preparation & SME Involvement Required
Phase 2 assessments require detailed organizational information and input from subject matter experts across the organization who have knowledge of the AI Principle areas. Consider gathering stakeholders familiar with data infrastructure, investment capacity, personnel capabilities, governance and compliance, and strategic objectives.
Tip: Completing these assessments in a working session with appropriate SMEs will ensure accuracy and efficiency.
Data & Infrastructure Assessment
Validate data readiness and technical infrastructure for AI implementation.
Why This Assessment Matters
AI systems are fundamentally dependent on data quality, availability, and accessibility. This structured assessment helps you validate whether you have the data foundation needed for successful implementation before investing significant resources.
What You'll Evaluate
- Ideal Data Profile: What data your AI model needs
- Source Mapping: Where that data exists today
- Gap Analysis: What's missing or incomplete
- Integration Complexity: How hard is it to connect systems
- Governance & Security: Privacy, compliance, access controls
Assessment Outcome
At the end of this assessment, you'll make one of three decisions:
Completed Assessments
Investment Capacity
Financial resources, budget approvals, and multi-year funding commitments necessary for both initial implementation and ongoing operations.
Investment Intensity for Your Selected Use Cases
The following summary provides indicative investment intensity ratings for your selected use cases based on implementation complexity, infrastructure needs, integration requirements, and technology maturity.
⚠️ Disclaimer: These ratings are estimates derived from general assessment of implementation complexity factors including technology maturity, infrastructure dependencies, integration scope, and organizational change requirements. Actual investment requirements will vary based on utility-specific conditions, existing capabilities, vendor selection, and scope decisions. This resource is intended to be illustrative only, in support of Step 3 AI Principle "Investment Capacity" discussions and should not be used as a basis for budgeting or procurement decisions without further analysis.
📊 Investment Intensity Scale
Your selected use cases from Step 2 will appear here...
Consider This Information as You Answer the Questions Below
Review the investment intensity ratings for your selected use cases before answering the Investment Capacity questions. If your available budget, resource commitments, or funding timeline do not align with the complexity levels shown above, flag this as a potential gap in your assessment. Misalignment between investment intensity and allocated resources is a common cause of project delays, scope reductions, or failed implementations.
Assessment Questions
1. Cost Understanding (Budget Risk)
Has a preliminary cost estimate been developed for this AI initiative?
Context: Understanding cost magnitude early helps secure appropriate funding and avoid scope surprises during implementation
2. ROI/Value Case (Value Risk)
Is there a documented business case showing expected ROI or value creation from this AI initiative?
Context: A clear value proposition is essential for sustained executive support, budget approval, and measuring success
3. Budget Approval Status (Budget Risk)
Has funding been secured for this AI initiative?
Context: Uncertain funding can delay timelines, limit vendor negotiations, and slow project momentum
4. Total Cost of Ownership Awareness (Budget Risk)
Do decision-makers understand the total cost of ownership including infrastructure, personnel, vendors, data preparation, and ongoing operations?
Context: Incomplete TCO understanding often leads to budget shortfalls mid-implementation when hidden costs emerge
🚩 Red Flag Indicators
Check any that apply to your organization:
Overall Assessment
Skilled Personnel
Availability of staff with necessary technical skills, domain expertise, and operational experience to develop, deploy, and maintain AI systems.
Assessment Questions
1. AI/ML Experience Level (Resource Risk)
What level of AI/ML development experience exists within your organization or team?
Context: Current and relevant technical AI expertise is critical for successful implementation, vendor management, and long-term system ownership
2. Operational Domain Expertise Involvement (Resource Risk)
Are operational domain experts from the relevant business area actively involved in or supporting this AI initiative?
Context: Domain expertise ensures AI solutions address real operational needs and constraints—projects without active domain expert involvement often miss critical requirements
3. Training & Upskilling Strategy (Resource Risk)
Is there a plan to train or upskill team members working on this specific AI initiative to reduce knowledge gaps and dependencies?
Context: Single points of failure in knowledge create project risk—building broader team capabilities ensure resilience
4. Resource Availability (Timeline Risk)
Can qualified personnel dedicate sufficient time to this AI initiative?
Context: Part-time attention to AI projects often leads to delays, quality issues, and missed requirements—dedicated resources accelerate success
🚩 Red Flag Indicators
Check any that apply to your organization:
Overall Assessment
Governance and Compliance
Ability to meet industry regulations, data privacy requirements, cybersecurity standards, and emerging AI governance frameworks.
Assessment Questions
1. Applicable Regulations Identified (Compliance Risk)
Have you identified which regulations apply to this AI use case (NERC CIP, PUC requirements, data privacy laws, etc.)?
Context: Unknown regulatory requirements discovered late in implementation can halt projects or require costly redesigns
2. AI Governance Framework (Compliance Risk)
Are AI governance policies, procedures, and documentation practices established?
Context: Governance frameworks demonstrate due diligence to regulators and provide audit trails—absence creates compliance exposure
3. Cybersecurity Controls (Security Risk)
Have required cybersecurity controls for AI systems, data protection, and model integrity been identified and implemented?
Context: AI systems introduce new attack surfaces and data vulnerabilities—proactive security prevents breaches and regulatory penalties
4. IT/OT Requirements Alignment (Technical Risk)
Has it been confirmed that this AI initiative can operate within existing internal IT/OT policies, security requirements, and technical constraints?
Context: Conflicts with IT/OT requirements (like cloud restrictions, network segmentation, or data handling policies) discovered late can require costly redesigns or halt projects entirely
🚩 Red Flag Indicators
Check any that apply to your organization:
Overall Assessment
Clear Objectives
Specific, measurable success criteria with stakeholder alignment on what the AI system must accomplish.
Note: Detailed KPI Development in Step 4
Step 4 provides detailed worksheets for defining KPIs, success metrics, and baseline measurements. The questions below help assess whether you're ready to complete that detailed planning.
Assessment Questions
1. Defined Success Metrics (Value Risk)
Have you defined specific, measurable success metrics and the thresholds that will determine if this AI initiative should move forward?
Context: Clear metrics and decision criteria enable objective evaluation and prevent endless pilots—vague goals lead to scope creep and unclear results
2. Baseline Measurements (Value Risk)
Have baseline measurements been captured to compare AI system performance against current state?
Context: Without baselines, you can't prove improvement or ROI—capturing current performance is essential for demonstrating value
3. Stakeholder Alignment (Resource Risk)
Do all key stakeholders (operations, IT, leadership, affected departments) agree on what success looks like?
Context: Misaligned expectations create friction during implementation and dissatisfaction with results—early alignment prevents conflicts
4. Technical-Business Alignment (Value Risk)
Do the technical capabilities of the proposed AI solution align with the business outcomes you need to achieve?
Context: Technology-business misalignment wastes resources on impressive capabilities that don't solve the actual problem
🚩 Red Flag Indicators
Check any that apply to your organization:
Overall Assessment
Complete all Phase 2 assessments before proceeding to Step 4
Step 4: Implementation Planning & Readiness Validation
Step 4 transforms the readiness insights from Steps 1 through 3 into a concrete implementation blueprint. By this point, you have identified clear business opportunities, selected feasible AI use cases, and validated your organizational readiness. Step 4 now requires you to define exactly what you intend to build, how the AI system will operate within your environment, and whether it should be developed internally, procured, or implemented with support from external partners.
Many AI initiatives fail not because the technology is insufficient, but because organizations never clearly define what success looks like, how the AI system fits into existing workflows, or who will maintain it once deployed. Step 4 prevents these issues by forcing precision before development begins. The AI Scenario Definition creates a shared understanding across operations, IT/OT, leadership, and external partners, reducing ambiguity and surfacing integration challenges early. The Build vs. Buy evaluation ensures your implementation choices match your capabilities, constraints, and strategic priorities, rather than defaulting to a technology-first approach.
Step 4 Core Outputs
1. AI Scenario Definition
A detailed description of your proposed AI system, including purpose, functionality, data flows, integration requirements, and performance expectations. This scenario becomes the foundation for technical design, stakeholder alignment, and pilot scoping.
2. Build vs. Buy Decision
A structured evaluation of whether you should develop a custom solution, purchase a commercial platform, or pursue a hybrid partnership model. This decision informs your timelines, cost expectations, required skills, and long-term sustainability.
Select Use Cases for Implementation Planning
By default, all use cases from Step 2 are selected below. Review each use case and:
Keep selected: Use cases where you're ready to develop detailed implementation plans, define success criteria, and assess build/buy/partner decisions
Unselect: Use cases that are not ready based on your Step 3 assessment (data gaps, insufficient resources, high risk without adequate controls, etc.)
You must select at least one use case to proceed.
Only selected use cases will have implementation plans developed in Step 4 and controls evaluated in Step 5.
No use cases selected in Step 2
Please return to Step 2 and select at least one use case before proceeding with implementation planning.
Step 5: Evaluate Engineering Controls & Mitigations
Step 5 translates the risk insights and implementation plans you developed in earlier steps into informed decisions about the safeguards your AI deployment requires. This step focuses on determining which engineering controls, monitoring mechanisms, and governance practices are necessary to manage the risks associated with your selected use case.
This step uses the consequence levels and failure modes you identified in Step 3's AI Risk Analysis Framework, along with the implementation details from Step 4, to guide a structured evaluation of control needs. The principle underlying this evaluation is proportionality: higher-consequence applications require more rigorous safeguards, while lower-consequence applications allow for controls scaled to their risk profile.
Step 5 does not prescribe a mandatory checklist. Instead, it provides question-based prompts that help you determine the right level of protection based on your specific context, risk tolerance, and regulatory obligations. You'll work through questions organized across three control categories: Engineering Controls, Operational Controls, and Governance Controls. Each category addresses a distinct aspect of safe AI deployment, and the depth of consideration you give to each should reflect the consequence profile you established in earlier steps.
Key Outputs from Step 5
Control Strategy Documentation
Documented decisions about engineering controls (fail-safes, redundancy, manual overrides), operational controls (monitoring, incident response, model maintenance), and governance controls (oversight, vendor management, compliance integration) tailored to your specific risk profile.
Framework & Resource References
Identification of relevant industry frameworks, standards, and best practices (NIST AI RMF, NERC CIP, IEC 62443, Cyber-Informed Engineering) that support your control implementation based on your consequence level and regulatory environment.
Important Reminder
The controls and frameworks referenced in this step are guidance resources, not mandatory requirements unless specified by your regulatory environment. Use them to inform your decisions based on your organization's unique needs, risk tolerance, and operational context.
This section displays the use cases you selected and developed implementation plans for in Step 4. These are the AI applications you're actively preparing to deploy, not all the use cases you evaluated in Step 2.
Note: If you don't see any use cases below, return to Step 4 and complete at least one implementation worksheet. Step 5 controls are specific to each use case you're implementing, so you need Step 4 context first.
Control Assessment Questions
Click each tab below to document your approach for Engineering, Operational, and Governance controls.
Engineering Controls
Engineering controls are safeguards embedded in system architecture and design that ensure safe operation when AI models fail, degrade, or produce unexpected outputs. These controls define how AI systems interact with grid operations and what happens when things go wrong. For higher-consequence applications, engineering controls represent the primary line of defense against operational disruption or safety incidents.
Manual Override Capability
Does your AI system include manual override capability that enables operators to assume control seamlessly? High-consequence applications may require immediate operator control without system restart or reconfiguration. Consider how operators will transition from AI-assisted to manual operation.
Fail-Safe Mechanisms
What fail-safe mechanisms revert the system to a known safe state when AI outputs exceed expected bounds or confidence thresholds fall below acceptable levels? Systems might default to conservative, rule-based operation rather than halting entirely. Consider what "safe state" means for your specific application and operational context.
Redundancy Design
Have you designed redundancy to avoid single points of failure in critical functions? Options to consider include multiple models with diverse architectures or maintaining traditional control systems as active backups. Evaluate whether your use case risk profile justifies the additional complexity and cost of redundant systems.
Deployment Architecture
What is your deployment architecture, and how does it align with your consequence profile? On-premises, edge, hybrid, and cloud deployments each present different risk tradeoffs around latency, data residency, availability, and control. Consider how communication failures, processing delays, or cloud service interruptions would impact your operations.
Operator Trust Indicators
How will operators know when to trust AI outputs versus when to intervene? Consider whether confidence indicators, explainability features, and defined thresholds for escalation are appropriate for your application. Evaluate what information operators need to make informed decisions about AI recommendations.
Operational Controls
Operational controls address ongoing monitoring, performance management, and incident response throughout the AI system's operational lifecycle. While engineering controls establish how the system is built, operational controls ensure it continues to perform as expected under real-world conditions. These controls detect problems during operation and enable timely intervention before issues escalate.
Monitoring Mechanisms
What watchdog systems or monitoring mechanisms track AI model health, including input data quality, processing latency, output distributions, and model drift? Consider what constitutes abnormal behavior for your specific model and what response timeframes are appropriate. Evaluate how monitoring alerts will integrate with existing operational procedures.
AI-Specific Incident Response
How will your organization detect and respond to AI-specific incidents such as model poisoning, adversarial inputs, data exfiltration, or unexpected model behavior? AI incident response may require specialized playbooks beyond traditional IT security. Consider scenarios specific to your deployment and how they map to existing incident response procedures.
Performance Baselines and Thresholds
What baseline metrics establish normal AI system performance, and what thresholds trigger investigation or intervention? Consider defining acceptable ranges for accuracy, latency, and availability based on your operational requirements. Balance sensitivity (catching real issues) against specificity (avoiding alert fatigue).
Model Maintenance and Updates
How frequently will models be retrained or recalibrated, and what testing validates model updates before production deployment? Consider what triggers retraining (time-based, performance degradation, data drift) and what validation is sufficient to confirm model updates are safe to deploy. Evaluate testing requirements against your consequence profile.
Governance Controls
Governance controls establish organizational oversight, accountability structures, and alignment with regulatory and compliance requirements. These controls ensure AI deployments remain auditable, transparent, and consistent with existing risk management and cybersecurity programs. For utilities operating within regulated environments, governance controls also address how AI systems fit within established compliance frameworks.
Vendor Requirements
What security and performance requirements apply to AI vendors, and how are these documented in contracts and service level agreements? Consider vendor responsibilities for security, updates, vulnerability disclosure, and incident notification. Evaluate what contractual protections are appropriate for your use case and consequence profile.
Integration with Existing Governance
How does AI deployment integrate with your existing governance programs for cybersecurity, data privacy, regulatory compliance, and operational risk management? Consider how AI systems fit within existing frameworks rather than creating parallel governance structures. Evaluate which existing policies and procedures need to be extended to cover AI-specific considerations.
System Component Visibility
What visibility do you have into AI system components, including model architectures, training data provenance, third-party libraries, and software dependencies? Consider what level of transparency is necessary for your risk profile and compliance obligations. Evaluate vendor willingness to provide documentation and how you will track component changes over time.
Approval Authority and Oversight
Who in your organization has authority to approve AI deployments, modifications, or decommissioning, and what review processes ensure appropriate oversight? Consider whether existing approval authorities are sufficient or whether AI applications require specialized review. Evaluate what cross-functional input (IT, OT, legal, compliance, business units) is appropriate for your governance model.
The following frameworks and standards are provided as reference material to support your AI implementation planning. No inputs or responses are needed for this section.
| Framework | Description & When to Use | Link |
|---|