The discipline that's revealing why most AI agent platforms will disappear by 2027

TL;DR - The Bottom Line

Agent platforms are building on quicksand. While the AI agents market explodes from $5.4 billion in 2024 to a projected $50.31 billion by 2030, foundational LLM providers (OpenAI, Anthropic, Google) are systematically absorbing the functionality that makes specialized platforms valuable. The platforms that survive won't be those with the best AI—they'll be those that own irreplaceable business data and processes.

Key takeaway: The difference between a temporary demo platform and a defensible business is the switching cost created by deep data integration, not AI capability.

The Evolution: Why Agent Platform Consolidation is Inevitable

From Wrappers to Acquisitions

The numbers tell a stark story of ongoing consolidation:

• ServiceNow acquired Moveworks for close to $3 billion (March 2025)
• Google orchestrated a $2.7 billion reverse acquihire with Character.AI (August 2024)
• Salesforce acquired Convergence.ai to strengthen Agentforce (May 2025)
• AI agent companies led M&A activity in Q1 2025, securing the 3 largest deals among 85 acquisitions

The AI landscape has fundamentally shifted:

• 2023: Specialized agent platforms carve out niches (Platform Building Era)
• 2024: Foundational providers begin vertical integration
• 2025: Most platform failures are not AI failures anymore—they are moat failures

The Critical Business Reality

Market Transformation: Organizations building agent platforms are seeing survival rates drop from promising early adoption to brutal consolidation as foundational providers expand their capabilities.

Resource Efficiency: Studies show that foundational providers can:

• Replicate specialized platform features in 6-12 months
• Offer direct model access without API overhead
• Leverage existing user bases for instant distribution
• Undercut pricing through vertical integration

Strategic Positioning: The question shifts from "How do we build better agents?" to "How do we become irreplaceable to our users' business operations?"

What Makes Agent Platforms Vulnerable?

The Core Dependency Problem

Agent platforms ≠ Defensible businesses

Traditional View: Platform = AI capabilities + user interface Reality: Platform = Everything users depend on for critical business functions

The Complete Vulnerability Assessment

Most current agent platforms are built on five foundational weaknesses:

1. Model Dependency

• Platform Reality: "Choose the brain behind your agent. Use OpenAI, Claude, Gemini, and more"
• Strategic Problem: Core value proposition controlled by others
• Example: Khoj explicitly states it can "chat with any local or online LLM (e.g llama3, qwen, gemma, mistral, gpt, claude, gemini, deepseek)"—highlighting their fundamental dependency

2. Feature Replication Risk

• Current Capabilities: Workflow automation, knowledge retrieval, conversational interfaces
• Provider Response: OpenAI's Codex for developers, Anthropic's computer-use agents, Google's Workspace integration
• Time to Replication: 6-18 months for most specialized features

3. Distribution Disadvantage

• Platform Reach: Thousands to millions of users
• Provider Reach: Hundreds of millions (ChatGPT: 200M+ weekly active users)
• Integration Advantage: Microsoft 365 Copilot already used by 70% of Fortune 500 companies

4. Cost Structure Vulnerability

• Platform Economics: Pay API costs + infrastructure + development
• Provider Economics: Direct model access + existing infrastructure + amortized R&D
• Inevitable Outcome: Foundational providers can always undercut pricing

5. Innovation Lag

• Platform Development: Months to integrate new model capabilities
• Provider Development: Instant access to cutting-edge features
• Market Impact: Users migrate to latest capabilities regardless of platform loyalty

Agent Platforms vs. Foundational Providers: The Decisive Comparison

Beyond Features: The Next Competitive Battlefield

The Consolidation Pattern:

• Traditional Platforms: Compete on AI model performance and features
• Foundational Providers: Own the models and can replicate any feature
• Survivor Platforms: Own irreplaceable business processes and data

Example: The Relevance AI Reality Check

• Funding: $37 million Series B with impressive metrics
• Architecture: "Model-agnostic—use OpenAI, Claude, Gemini, and more"
• Vulnerability: 40,000 agents created in January 2025, but all dependent on external AI providers
• Strategic Problem: Building sophisticated wrappers around capabilities they don't control

What Creates Defensible Agent Platforms?

The Data Moat Framework

The Four Pillars of Platform Survival:

1. Proprietary Business Data

• Customer transaction histories and behavioral patterns
• Industry-specific knowledge bases and best practices
• Compliance documentation and audit trails
• Institutional memory and decision-making frameworks

2. Deep Process Integration

• Multi-system workflow orchestration
• Role-based approval hierarchies
• Custom business logic and rules
• Legacy system interconnections

3. Regulatory Lock-in

• Industry-specific compliance requirements
• Data sovereignty and governance policies
• Security certifications and audit frameworks
• Legal and contractual obligations

4. Network Effects

• User-generated content and configurations
• Team collaboration and shared knowledge
• Vendor ecosystem integrations
• Platform-specific training and expertise

Implementation Architecture: The Survival Blueprint

Foundation Layer: Business-Critical Functions

# Integrated Business Platform Structure
class DefensiblePlatform:
    core_business_functions = {
        'customer_management': 'CRM with historical data',
        'project_orchestration': 'Team workflows and approvals',
        'financial_systems': 'Billing, reporting, compliance',
        'knowledge_management': 'Institutional documentation',
        'communication_workflows': 'Internal processes and approvals'
    }
    
    ai_enhancement_layer = {
        'intelligent_insights': 'Data analysis using proprietary datasets',
        'workflow_optimization': 'Process improvement based on usage patterns',
        'predictive_analytics': 'Forecasting using business-specific models',
        'automation_suggestions': 'AI recommendations based on institutional knowledge'
    }

Strategic Layer: Switching Cost Creation

The magic isn't in a smarter AI model or a more clever interface. It's about becoming integral to daily business operations that cannot be easily replaced.

# Switching Cost Calculator
def calculate_replacement_cost(platform_integration):
    return {
        'data_migration': estimate_data_transfer_complexity(),
        'process_reconfiguration': map_workflow_dependencies(),
        'compliance_recertification': assess_regulatory_requirements(),
        'team_retraining': calculate_productivity_loss(),
        'integration_rebuilding': evaluate_system_connections(),
        'institutional_knowledge_loss': quantify_embedded_expertise()
    }

Real-World Implementation Examples

Case Study 1: Professional Services Transformation

The "Demo Platform" Approach:

• AI agent that answers questions about project management
• Generic workflow templates and basic automation
• Dependent on external AI APIs for all intelligence

The "Defensible Platform" Approach:

• Integrated project management system with years of historical data
• AI agents that predict project risks based on past performance patterns
• Custom billing workflows integrated with financial systems
• Client communication history informing AI response personalization

Result: 60% reduction in project delivery time, but more importantly, 18 months of accumulated client data and process customization that cannot be easily replicated.

Case Study 2: Healthcare Operations Integration

Traditional Agent Platform:

• Chatbot for answering medical questions
• Basic appointment scheduling automation
• General medical knowledge retrieval

Integrated Healthcare Platform:

• Electronic health record system with patient history
• AI diagnostic assistance based on institutional treatment outcomes
• Compliance management integrated with regulatory requirements
• Billing and insurance processing with historical optimization

Strategic Advantage: HIPAA compliance, patient data sovereignty, and institutional clinical knowledge create 24+ month switching costs.

Case Study 3: Manufacturing Operations Intelligence

Commoditized Approach:

• AI agent for supply chain inquiries
• Basic inventory management automation
• Generic manufacturing optimization recommendations

Defensible Integration:

• Equipment maintenance history and performance data
• Quality control documentation with AI-powered pattern recognition
• Supplier relationship management with predictive analytics
• Regulatory compliance tracking with automated reporting

Competitive Moat: Years of operational data, compliance documentation, and process optimization that foundational providers cannot access or replicate.

Implementation Guide: From Vulnerable to Defensible

Phase 1: Assessment & Foundation (Weeks 1-2)

Step 1: Vulnerability Audit

# Platform Defensibility Assessment
def audit_platform_defensibility():
    return {
        'ai_dependency_risk': measure_external_ai_reliance(),
        'data_ownership': assess_proprietary_information(),
        'switching_costs': calculate_user_migration_difficulty(),
        'process_integration': evaluate_business_criticality(),
        'competitive_moats': identify_unique_advantages()
    }

Step 2: Business Integration Mapping

Identify Integration Opportunities:

• Map existing business processes that could benefit from AI enhancement
• Assess data assets that could create competitive advantages
• Evaluate regulatory and compliance requirements that create lock-in
• Analyze workflow dependencies that increase switching costs

Phase 2: Strategic Pivot (Weeks 3-6)

Business-First Development Framework:

Agent platforms that survive will answer this question: "If our AI capabilities disappeared tomorrow, would users still depend on our platform for critical business functions?"

Implementation Strategy:

1. Start with business functions that users cannot easily replace
2. Add AI enhancement to improve these core functions
3. Create data flywheel where more usage improves AI performance
4. Build switching costs through customization and integration

Integration Pattern Example:

# Business-Critical Platform Architecture
integrated_platform = {
    'core_value': 'Customer relationship management system',
    'proprietary_data': 'Customer interaction history and preferences',
    'ai_enhancement': 'Predictive lead scoring based on historical success',
    'switching_cost': 'Years of sales data and team workflow customization',
    'network_effect': 'Better data improves AI, driving more usage'
}

Phase 3: Competitive Moat Construction (Weeks 7-12)

Advanced Defensibility Patterns:

Pattern 1: Regulatory Fortress

• Build compliance workflows that are industry-specific
• Create audit trails and documentation that satisfy regulatory requirements
• Develop certification processes that lock in usage patterns

Pattern 2: Data Network Effects

• Design systems where more users create better AI performance
• Build shared knowledge bases that improve with organization usage
• Create collaborative workflows that become more valuable over time

Pattern 3: Process Automation Lock-in

• Integrate with critical business systems (ERP, CRM, financial)
• Automate approval workflows and business logic
• Become integral to daily operations rather than nice-to-have enhancement

Phase 4: Optimization & Scale (Ongoing)

Performance Monitoring Framework:

# Defensibility Metrics
class PlatformSurvival:
    def __init__(self):
        self.switching_cost_score = 0.0  # Months to replace platform
        self.data_moat_strength = 0.0    # Uniqueness of proprietary data
        self.process_integration_depth = 0.0  # Business criticality
        self.network_effect_momentum = 0.0    # Value increase with usage
        self.competitive_replication_time = 0.0  # Time for competitors to match

Common Pitfalls & How to Avoid Them

Pitfall 1: AI-First Thinking

The Problem: Building around AI capabilities rather than business needs The Solution: Start with irreplaceable business functions, then add AI enhancement

Example:

• Wrong: "We build the best customer service AI agent"
• Right: "We build customer service workflow systems that get smarter with AI"

Pitfall 2: Feature Competition

The Problem: Trying to out-feature foundational providers The Solution: Focus on integration depth and switching costs

Strategic Framework: Instead of asking: "How can we make our AI better?" Ask: "How can we make our platform irreplaceable?"

Pitfall 3: Model Dependency

The Problem: Building core value proposition around AI model performance The Solution: Make AI model selection invisible to users while focusing on business outcomes

# Strategic Positioning
value_proposition = {
    'wrong': 'We provide access to the best AI models',
    'right': 'We solve your business problems, and AI makes us better at it'
}

The Future of Agent Platform Survival

Emerging Survival Patterns for 2025 and Beyond

Industry Vertical Dominance

Platforms that survive will own specific industry verticals with deep regulatory and process integration rather than trying to be horizontal AI solutions.

Winning Examples:

• Healthcare: EHR systems with AI diagnostic assistance
• Legal: Case management platforms with AI research capabilities
• Manufacturing: Operations management systems with AI optimization
• Financial Services: Compliance platforms with AI risk assessment

Platform Ecosystem Strategy

Future platforms will become ecosystems where removing the platform breaks multiple business-critical workflows, not just AI functionality.

Strategic Elements:

• API Ecosystem: Third-party integrations that depend on the platform
• Data Partnerships: Industry data sharing that creates network effects
• Compliance Frameworks: Regulatory adherence that locks in usage
• Workflow Orchestration: Business process automation that becomes institutional knowledge

The Consolidation Timeline

2025: Continued aggressive acquisition by foundational providers 2026: Survival platforms emerge with strong business integration moats 2027: Market settles into integrated business platforms vs. foundational AI providers

Investment Priorities:

• Infrastructure: Business process integration systems
• Data Architecture: Proprietary data collection and analysis
• Compliance Engineering: Industry-specific regulatory frameworks
• Integration Platforms: Deep connections with existing business systems

Getting Started: Your Platform Survival Strategy

Week 1: Critical Assessment

• Audit current AI dependencies and identify vulnerability points
• Map business functions that users depend on beyond AI capabilities
• Assess proprietary data assets that could create competitive advantages
• Evaluate switching costs users would face to replace your platform

Week 2: Strategic Planning

• Identify integration opportunities with critical business systems
• Design data collection strategies that improve with usage
• Plan compliance and regulatory positioning for your industry
• Develop switching cost creation roadmap

Week 3-4: Foundation Building

• Begin business function integration that creates user dependency
• Implement data collection systems that improve AI performance over time
• Design workflow automation that becomes integral to daily operations
• Launch pilot programs with deep business integration

Beyond: Competitive Moat Construction

The agent platform future will be built one business integration at a time. Engineer them to be irreplaceable.

Essential Resources & Tools

Survival Frameworks & Platforms

• Salesforce Platform: Study Agentforce integration with core CRM functions
• ServiceNow: Analyze Moveworks acquisition and integration strategy
• Microsoft Power Platform: Observe AI integration within business workflows
• Industry-Specific Platforms: Research vertical-specific survival strategies

Strategic Learning Resources

• CB Insights AI M&A Report 2025
• McKinsey AI in the Workplace Report
• Deloitte Autonomous AI Agents Study

Community & Analysis

• Industry consolidation tracking and M&A analysis
• Platform survival case studies and failure analysis
• Business integration patterns and switching cost creation strategies

Conclusion: The Platform Survival Imperative

Agent platform survival represents more than just an evolution in AI development—it's a fundamental paradigm shift that separates businesses that will exist in 2027 from those that won't. Building defensible AI platforms is becoming less about AI model performance and more about business integration depth and switching cost creation.

The organizations that master business-first platform design today will have insurmountable advantages tomorrow. They'll build irreplaceable systems, create sustainable competitive moats, and solve business problems that foundational providers cannot easily replicate.

Key Strategic Insight: The future belongs to platforms that make AI invisible by embedding it in irreplaceable business functions, not those that make AI the primary value proposition.

Remember: Foundational providers can replicate features, interfaces, and even user experiences. They cannot replicate years of accumulated business data, institutional processes, and regulatory compliance that becomes embedded in well-integrated platforms.

The Bottom Line

The age of "AI-first platforms" is ending before it truly began. Welcome to the era of business-first platforms enhanced by AI.

Ready to transform your vulnerable agent platform into a defensible business? Start by asking: "If we removed all AI functionality tomorrow, would our users still depend on us for critical business operations?" If the answer is no, you know exactly where to begin.

The time for building on others' foundations is over. The time for building foundations that others depend on has begun.

The discipline that's redefining how we build intelligent AI applications in 2025

TL;DR - The Bottom Line

Context Engineering is the discipline of designing and building dynamic systems that provides the right information and tools, in the right format, at the right time, to give a LLM everything it needs to accomplish a task. While prompt engineering focused on crafting clever single instructions, context engineering is about architecting complete information environments that enable AI systems to handle complex, multi-step tasks reliably.

Key takeaway: The difference between a cheap demo and a "magical" agent is about the quality of the context you provide.

The Evolution: Why Context Engineering Matters Now

From Prompts to Systems

Early on, developers focused on phrasing prompts cleverly to coax better answers. But as applications grow more complex, it's becoming clear that providing complete and structured context to the AI is far more important than any magic wording.

The AI landscape has fundamentally shifted:

• 2023: "You are an expert. Do X like Y" (Prompt Engineering Era)
• 2024: Building systems that dynamically gather, structure, and provide context
• 2025: Most agent failures are not model failures anymore, they are context failures

The Critical Business Impact

Performance Transformation: Organizations implementing proper context engineering principles see success rates jump from about 30% to over 90%

Cost Efficiency: Studies show that well-engineered context can:

• Reduce token usage by 20-50% in commercial AI operations
• Improve task completion rates by up to 40%
• Minimize computational resources needed for desired outcomes

Production Readiness: Context engineering shifts the mindset from just writing prompts to designing a system. The prompt you write becomes only a subset of that system.

What Is Context Engineering?

The Core Definition

Context engineering is the delicate art and science of filling the context window with just the right information for the next step — a definition that's gained traction from AI luminaries like Andrej Karpathy.

But let's break this down further:

Context ≠ Just Your Prompt

Traditional View: Context = Single prompt/instruction
Reality: Context = Everything the model sees before generating

The Complete Context Architecture

Context engineering involves assembling a variety of components, including a basic prompt, memory, output from RAG pipelines, output from tool invocation, well-defined and structured output format, and guardrails.

The Five Pillars of Context:

1. Instructions & System Prompts
   • Role definitions and behavioral guidelines
   • Task-specific instructions and constraints
   • Examples and few-shot learning patterns
2. Memory & Historical Context
   • Conversation history and session state
   • User preferences and behavioral patterns
   • Long-term interaction memory
3. Retrieved Knowledge (RAG)
   • External documents and knowledge bases
   • Real-time data feeds and API responses
   • Domain-specific information injection
4. Tool & Environment Context
   • Available functions and their descriptions
   • System capabilities and limitations
   • External service integrations
5. Output Formatting & Constraints
   • Response structure and style guidelines
   • Safety guardrails and content policies
   • Quality control mechanisms

Context Engineering vs. Traditional Approaches

The Fundamental Differences

Beyond RAG: The Next Evolution

GraphRAG and knowledge graphs are to context engineering, what RAG and vector databases are to prompt engineering.

Traditional RAG Limitations:

• Documents lose context when chunked, which affects the retrieval quality and subsequent response quality
• The vector embedding approach to storing and retrieving information is inherently lossy and may miss out on retrieving chunks with exact lexical matches

Context Engineering Solutions:

• Multi-modal context integration
• Hierarchical information structures
• Dynamic context adaptation
• Cross-domain knowledge synthesis

Implementation Architecture & Best Practices

The Four-Layer Context Stack

1. Foundation Layer: Information Architecture

# Context Template Structure
class ContextTemplate:
    role: str              # Define AI's expertise and behavior
    task: str              # Specific objective or goal  
    background: str        # Relevant domain context
    constraints: str       # Limitations and requirements
    examples: List[str]    # Few-shot learning samples
    format: str           # Output structure specification

2. Dynamic Layer: Real-Time Context Assembly The magic isn't in a smarter model or a more clever algorithm. It's about providing the right context for the right task.

# Dynamic Context Pipeline
def assemble_context(user_query, session_state):
    context = {
        'instructions': load_task_instructions(user_query),
        'memory': retrieve_relevant_history(session_state),
        'knowledge': rag_retrieve(user_query),
        'tools': select_relevant_tools(user_query),
        'constraints': apply_safety_guardrails()
    }
    return optimize_context_window(context)

3. Orchestration Layer: Multi-Agent Coordination Instead of cramming everything into a single LLM call and hoping for the best, you can break complex tasks into focused steps, each with its own optimized context window.

4. Optimization Layer: Context Management

Advanced Techniques for Production Systems

Context Compression & Optimization Context pruning means removing outdated or conflicting information as new details arrive. Context offloading, like Anthropic's "think" tool, gives models a separate workspace to process information without cluttering the main context.

Memory Selection Strategies Embeddings and / or knowledge graphs for memory indexing are commonly used to assist with selection. Still, memory selection is challenging.

Tool Context Management Agents use tools, but can become overloaded if they are provided with too many. One approach is to apply RAG to tool descriptions in order to fetch the most relevant tools for a task based upon semantic similarity.

Real-World Applications & Case Studies

Enterprise Knowledge Integration

Enterprises often struggle with knowledge fragmented across countless silos: Confluence, Jira, SharePoint, Slack, CRMs, and various databases. Context engineering provides the architecture to unify these disparate sources.

Case Study: Financial Services AI Assistant

• Challenge: Integrate customer data, regulatory documents, and market data
• Solution: Multi-source RAG with hierarchical context prioritization
• Result: 60% reduction in resolution time, 35% improvement in accuracy

Advanced Code Generation Systems

The next evolution of coding assistants is moving beyond simple autocomplete. Systems are being built that have full context of an entire codebase, integrating with Language Server Protocols (LSP) to understand type errors, parsing production logs to identify bugs, and reading recent commits to maintain coding style.

Implementation Pattern:

# Agentic Code Assistant Context
code_context = {
    'codebase_analysis': analyze_project_structure(),
    'recent_commits': get_git_history(days=7),
    'error_logs': parse_production_errors(),
    'coding_standards': load_style_guide(),
    'dependencies': analyze_package_dependencies()
}

Customer Service Automation

The "Magical" vs "Demo" Agent Example:

The "Cheap Demo" Agent has poor context. It sees only the user's request and nothing else. The "Magical" Agent is powered by rich context.

Demo Agent Response:

User: "Hey, just checking if you're around for a quick sync tomorrow."
Demo: "Thank you for your message. Tomorrow works for me. 
May I ask what time you had in mind?"

Magical Agent Context:

• Calendar integration showing availability
• Previous meeting patterns with this contact
• Project context and urgency levels
• Communication style preferences

Magical Agent Response:

"Hi! I see you're working on the Q4 launch project. I have a 
30-minute slot open tomorrow at 2 PM EST, which aligns with 
our usual check-in time. Should I send a calendar invite?"

Implementation Guide: From Theory to Production

Phase 1: Foundation Setup (Weeks 1-2)

Step 1: Context Audit

# Assess current context usage
def audit_existing_prompts():
    return {
        'static_elements': extract_reusable_components(),
        'dynamic_needs': identify_variable_content(),
        'failure_points': analyze_poor_responses(),
        'optimization_opportunities': find_efficiency_gaps()
    }

Step 2: Template Framework Context Engineering is 10x better than prompt engineering and 100x better than vibe coding.

Phase 2: Dynamic Context Systems (Weeks 3-4)

RAG Integration Best Practices: Introduce context back into the chunks. This can be as simple as prepending chunks with the document and section titles, a method sometimes known as contextual chunk headers.

# Enhanced RAG Context
def enhance_chunks_with_context(chunks, document_metadata):
    enhanced_chunks = []
    for chunk in chunks:
        enhanced_chunk = {
            'content': chunk['text'],
            'context_header': f"Document: {document_metadata['title']}\n"
                             f"Section: {chunk['section']}\n"
                             f"Context: {chunk['summary']}",
            'metadata': document_metadata
        }
        enhanced_chunks.append(enhanced_chunk)
    return enhanced_chunks

Phase 3: Advanced Optimization (Weeks 5-6)

Context Engineering Patterns: Patterns for agent context engineering are still evolving, but we can group common approaches into 4 buckets — write, select, compress, and isolate

1. Write: Store context externally for future retrieval
2. Select: Intelligently choose relevant context for current task
3. Compress: Distill information to essential elements
4. Isolate: Separate contexts to prevent interference

Phase 4: Production Deployment

Performance Monitoring Framework:

# Context Engineering Metrics
class ContextMetrics:
    def __init__(self):
        self.task_completion_rate = 0.0
        self.context_efficiency_ratio = 0.0
        self.user_satisfaction_score = 0.0
        self.cost_per_successful_interaction = 0.0
        self.context_window_utilization = 0.0

Advanced Techniques & Emerging Patterns

Hierarchical Context Architecture

Enterprise experts in 2025 often highlight RAG as a key to prevent hallucinations and enforce truthfulness in AI outputs. They also emphasize long-term session memory so that AI assistants can be truly helpful.

Multi-Layer Context Design:

├── Global Context (System-wide)
│   ├── Organization policies and guidelines
│   ├── Security and compliance requirements
│   └── Brand voice and communication standards
├── Session Context (User-specific)
│   ├── User profile and preferences
│   ├── Conversation history and patterns
│   └── Current task and project context
└── Task Context (Request-specific)
    ├── Immediate query requirements
    ├── Retrieved knowledge and tools
    └── Real-time environmental data

Context Versioning & A/B Testing

Context engineering turned out to be anything but straightforward. It's an experimental science—and we've rebuilt our agent framework four times, each time after discovering a better way to shape context.

Implementation Strategy:

# Context Version Control
class ContextVersion:
    def __init__(self, version_id, context_config):
        self.version_id = version_id
        self.config = context_config
        self.performance_metrics = {}
        self.last_updated = datetime.now()
    
    def compare_performance(self, other_version):
        return self.performance_metrics.compare(other_version.performance_metrics)

Structured Context Encoding

A 2025 article suggests using an "XML-like structure to pack various types of information" (messages, tool outputs, errors) into the context. This is an example of low-level context engineering.

Example Implementation:

<context>
    <role>Senior Software Architect</role>
    <task>Design microservices architecture</task>
    <knowledge>
        <document source="company_guidelines.pdf">
            Architecture must follow 12-factor app principles...
        </document>
        <api_docs>Current service endpoints and schemas</api_docs>
    </knowledge>
    <constraints>
        <security>SOC2 compliance required</security>
        <performance>Sub-100ms response time</performance>
    </constraints>
</context>

Common Pitfalls & How to Avoid Them

Context Poisoning & Mitigation

Context Poisoning: When a hallucination makes it into the context

Prevention Strategies:

• Implement context validation layers
• Use source attribution and confidence scoring
• Regular context auditing and cleanup procedures

The Context Window Management Challenge

The problem happens because when information comes in stages, the assembled context contains early attempts by the model to answer questions before it has all the information.

Solution Framework:

1. Priority-based Context Ranking
2. Dynamic Context Compression
3. Staged Information Assembly
4. Conflict Resolution Protocols

Memory Selection Gone Wrong

At the AIEngineer World's Fair, Simon Willison shared an example of memory selection gone wrong: ChatGPT fetched his location from memories and unexpectedly injected it into a requested image.

Best Practices:

• Implement explicit memory consent mechanisms
• Use context relevance scoring
• Provide memory transparency to users

The Future of Context Engineering

Emerging Trends for 2025 and Beyond

Autonomous Context Optimization Context learning systems that adapt context strategies automatically represent the next frontier, where systems learn optimal context patterns from successful interactions.

Multi-Modal Context Integration Future systems will seamlessly blend text, images, audio, and structured data into coherent context frameworks that mirror human multi-sensory understanding.

Context Sovereignty I've been working through the idea of context engineering lately (as well as the related context sovereignty) — ensuring users maintain control over their contextual information and how it's utilized.

The Strategic Imperative

LLM applications are evolving from single prompts to more complex, dynamic agentic systems. As such, context engineering is becoming the most important skill an AI engineer can develop.

Key Investment Areas:

• Infrastructure: Context storage and retrieval systems
• Tooling: Context engineering development environments
• Talent: Engineers skilled in information architecture
• Processes: Context testing and optimization workflows

Getting Started: Your Context Engineering Journey

Week 1: Assessment & Foundation

1. Audit existing AI implementations for context usage patterns
2. Identify failure modes related to insufficient or poor context
3. Map information sources available in your organization
4. Establish baseline metrics for current AI performance

Week 2: Template Development

1. Create context templates for your most common use cases
2. Implement basic RAG integration for knowledge retrieval
3. Design context versioning system for A/B testing
4. Set up monitoring for context effectiveness

Week 3-4: Advanced Implementation

1. Deploy dynamic context assembly pipelines
2. Integrate multiple information sources (databases, APIs, documents)
3. Implement context compression and optimization techniques
4. Launch pilot applications with enhanced context engineering

Beyond: Continuous Optimization

The agentic future will be built one context at a time. Engineer them well.

Essential Resources & Tools

Frameworks & Platforms

• LlamaIndex: Context-aware AI application framework
• LangChain: Multi-modal context orchestration
• Anthropic Claude: Advanced context window management
• OpenAI API: System message and context integration

Learning Resources

• Context Engineering Guide by LangChain
• Awesome Context Engineering GitHub Repository
• DataCamp Context Engineering Tutorial

Community & Discussion

• AI Engineer World's Fair presentations on context engineering
• Industry blogs from practitioners at Manus, Anthropic, and OpenAI
• Research papers on retrieval-augmented generation and memory systems

Conclusion: The Context Engineering Imperative

Context engineering represents more than just an evolution in AI development—it's a fundamental paradigm shift that separates production-grade AI systems from experimental prototypes. Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates. It is about the engineering of context and providing the right information and tools, in the right format, at the right time.

The organizations and individuals who master context engineering today will have a decisive advantage in the AI-driven economy of tomorrow. They'll build more reliable systems, deliver better user experiences, and solve more complex problems with greater efficiency and accuracy.

As we move forward, remember: the future belongs to those who can architect intelligence, not just prompt it.Context engineering is your blueprint for that architectural mastery.

Ready to transform your AI implementations? Start with a single use case, apply these context engineering principles, and experience the difference that systematic context design can make. The age of "prompt and hope" is over—welcome to the era of engineered intelligence.

A deep dive into creating intelligent agents that can reason, act, and operate within their computational constraints

Introduction

As AI agents become increasingly sophisticated, developers are discovering that building truly effective autonomous systems requires more than just powerful language models. The key lies in understanding three fundamental principles: how to structure multi-step reasoning and action cycles, how agents perceive and work within their computational boundaries, and why specifications trump creative prompting.

In this post, we'll explore how to build "manus-like" agents—systems capable of complex, multi-round interactions—while understanding their inherent limitations and designing robust specification frameworks.

Understanding the ReAct Paradigm

What is ReAct?

ReAct (Reasoning and Acting) represents a fundamental shift in how we architect AI agents. Unlike simple request-response systems, ReAct agents can:

• Reason about problems step by step
• Act by invoking tools and gathering information
• Reflect on results and adjust their approach
• Iterate through multiple cycles until completion

The Multi-Round Action Architecture

A manus-like agent operates through discrete cycles, each containing three phases:

Phase 1: Observation & Analysis

Agent receives input → Analyzes current state → Identifies required actions

Phase 2: Tool Invocation & Execution

Selects appropriate tools → Executes actions → Gathers results

Phase 3: Synthesis & Planning

Processes results → Updates mental model → Plans next actions

Implementing the ReAct Loop

Here's how to structure a robust ReAct agent:

Step 1: Define the Reasoning Framework

Your agent needs explicit instructions for each phase:

• How to analyze problems systematically
• When to gather more information vs. when to act
• How to evaluate the success of each action
• When to conclude the task is complete

Step 2: Create Action Boundaries

Establish clear rules for:

• Maximum number of iterations per task
• Fallback strategies when tools fail
• Escalation paths for complex problems
• Resource usage limits

Step 3: Build Reflection Mechanisms

After each action, the agent should:

• Evaluate whether the action achieved its intended goal
• Identify any new information that changes the problem space
• Adjust the strategy if the current approach isn't working
• Maintain a running summary of progress

Example ReAct Structure

TASK: Research competitor pricing and create analysis report

ITERATION 1:
Thought: I need to gather pricing data from multiple competitors
Action: web_search("competitor A pricing 2024")
Result: Found pricing page with basic tier info
Reflection: Good start, but need more comprehensive data

ITERATION 2: 
Thought: Need to get detailed pricing for all tiers from competitor A
Action: web_fetch("https://competitor-a.com/pricing")
Result: Complete pricing matrix obtained
Reflection: Have comp A data, need competitors B and C

ITERATION 3:
Thought: Should search for competitor B pricing information  
Action: web_search("competitor B enterprise pricing")
...

The Context Window Constraint: An Agent's Limited Worldview

Understanding Agent Perception

One of the most crucial concepts for agent developers is understanding that agents can only work with information present in their context window. This creates fundamental limitations that shape how we must design agent systems.

What Agents Can and Cannot See

Agents CAN access:

• The initial prompt and instructions
• Previous messages in the current conversation
• Results from tool calls within the current session
• Data retrieved from MCP servers during the current execution
• Their own reasoning and reflection from previous iterations

Agents CANNOT access:

• Information from previous conversations or sessions
• Data not explicitly retrieved through tool calls
• Real-time information unless specifically fetched
• Cached results from previous tool invocations
• "Hidden" or implied context not present in the window

Designing for Context Limitations

Strategy 1: Comprehensive Information Gathering

Since agents forget everything outside their context, ensure they gather all necessary information within each session:

Instead of assuming: "The user mentioned their company's Q3 goals yesterday"
Design for: "Let me search for this user's Q3 goals in their documents"

Strategy 2: Explicit State Management

Build agents that maintain explicit state within their context:

CURRENT SESSION STATE:
- Task: Create marketing analysis
- Data gathered: [list of sources]
- Analysis completed: [list of sections] 
- Remaining work: [list of pending items]

Strategy 3: Tool-First Information Architecture

Design your agent to immediately gather context through tools rather than making assumptions:

WRONG: "Based on our previous discussion about your company..."
RIGHT: "Let me search your documents to understand your company context..."

Context Window Best Practices

Optimize for Retention:

• Summarize key findings after each major tool call
• Maintain running lists of important data points
• Use structured formats that are easy to parse later in the conversation

Plan for Forgetting:

• Design tools that can re-establish context quickly
• Build redundancy into information gathering
• Create clear handoff mechanisms when context limits are reached

Specifications Over Prompts: The Foundation of Reliable Agents

Why Specifications Matter More Than Creative Writing

Many developers approach agent design like creative writing—crafting clever prompts with personality and flair. However, production-ready agents require engineering precision, not creative prose.

The Specification-First Approach

Define Before You Write:

Instead of starting with "You are a helpful AI assistant who...", begin with:

• Input specifications: Exactly what data formats the agent should expect
• Output specifications: Precise formatting requirements for responses
• Behavioral specifications: Clear rules for decision-making
• Error handling specifications: How to respond to edge cases
• Performance specifications: Success criteria and quality metrics

Building Robust Specifications

Input Specification Example:

INPUT FORMAT:
- Task type: [research|analysis|creation|review]
- Scope: [specific parameters and boundaries]
- Required outputs: [list of deliverable formats]
- Constraints: [time, resources, quality requirements]
- Context: [relevant background information]

VALIDATION RULES:
- All required fields must be present
- Task type must match supported capabilities
- Scope must be achievable within context limits

Behavioral Specification Example:

DECISION FRAMEWORK:
1. If task requires external data → use tools to gather information
2. If multiple sources conflict → present options with analysis
3. If insufficient information → explicitly list missing data
4. If task is beyond scope → explain limitations and suggest alternatives
5. If tools fail → implement fallback strategy [defined procedure]

ERROR HANDLING:
- Tool timeout → retry once, then fallback to alternative approach
- Invalid data → clean and validate, or request clarification  
- Partial results → clearly indicate completeness level

Specification Categories

Functional Specifications:

• What the agent must do
• Required capabilities and tools
• Success criteria for each task type

Interface Specifications:

• How users interact with the agent
• Expected input/output formats
• Communication protocols

Quality Specifications:

• Accuracy requirements
• Response time limits
• Error rate tolerances

Operational Specifications:

• Resource usage limits
• Security and privacy requirements
• Integration requirements with other systems

Testing Your Specifications

Specification Validation Process:

1. Edge Case Testing: Create scenarios that push boundary conditions
2. Ambiguity Detection: Identify areas where multiple interpretations are possible
3. Completeness Checking: Ensure all scenarios have defined behaviors
4. Consistency Verification: Check that specifications don't contradict each other

Putting It All Together: A Complete Agent Architecture

The Integrated Approach

A production-ready agent combines all three principles:

ReAct Foundation: Multi-round reasoning and action capabilities Context Awareness: Design that works within computational limits
Specification-Driven: Precise behavioral definitions

Implementation Checklist

ReAct Implementation:

• [ ] Multi-phase reasoning structure defined
• [ ] Tool invocation patterns established
• [ ] Iteration limits and fallbacks configured
• [ ] Reflection and adjustment mechanisms built

Context Management:

• [ ] Information gathering strategies implemented
• [ ] State management within context window
• [ ] Tool-first architecture established
• [ ] Context optimization techniques applied

Specification Framework:

• [ ] Comprehensive input/output specifications
• [ ] Behavioral rules clearly defined
• [ ] Error handling procedures documented
• [ ] Quality and performance criteria established

Advanced Patterns and Considerations

Compound Agent Architectures

For complex tasks, consider breaking work across multiple specialized agents:

• Research Agent: Focused on information gathering
• Analysis Agent: Specialized in data interpretation
• Synthesis Agent: Expert in combining insights into final deliverables

Dynamic Specification Adjustment

Build agents that can adjust their specifications based on:

• Task complexity assessment
• Available time and resources
• User expertise level
• Context richness

Monitoring and Improvement

Implement logging and analytics to track:

• Tool usage patterns and success rates
• Context window utilization efficiency
• Specification adherence and violation patterns
• User satisfaction with agent performance

Conclusion

Building sophisticated AI agents requires moving beyond prompt engineering toward systems engineering. The combination of ReAct reasoning patterns, context-aware design, and specification-driven development creates agents that are both powerful and reliable.

Remember: great agents aren't just well-prompted—they're well-specified, context-aware, and designed to operate effectively within their computational boundaries. Focus on these engineering fundamentals, and you'll build agents that deliver consistent value in production environments.

The future of AI agents lies not in more creative prompts, but in more rigorous engineering. Start with specifications, design for constraints, and implement systematic reasoning—your users will thank you for the reliability.

This post represents current best practices in agent development. As the field evolves rapidly, continue to validate these approaches against your specific use cases and emerging research.

• The tsvector type is used to represent a document in a way that’s fine-tuned for this kind of search.
• The tsquery type is used to structure the text query itself.

Using TypeORM in Node.js application gives you nice support of Typescript for ORM models.
Lets see how can we setup PostgreSQL and tsvector with TypeORM

Background

Lets assume we have product and categories tables and corresponding models. We want to search products by product name, ean and category name.

Product model:

@Entity()
export class Product {
  @PrimaryGeneratedColumn()
  id: number;

  @Column({
    type: 'varchar',
  })
  name: string;

  @Column({
    type: 'varchar',
  })
  ean: string;

  @Column({
    type: 'int',
  })
  categoryId: number;
}

Category model:

@Entity()
export class Category {
  @PrimaryGeneratedColumn()
  id: number;

  @Column({
    type: 'varchar',
  })
  name: string;
}

Setup tsvector

We need to create a column in the table where we want to store the search vectors and retrieve search results from.

Add full text search column to Product model

  @Column({
    type: 'tsvector',
    nullable: true,
  })
  fullTextSearch: string;

Create migration

import { MigrationInterface, QueryRunner } from 'typeorm'

export class SearchVector1724418715424 implements MigrationInterface {
  name = 'SearchVector1724418715424'

  public async up(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(`ALTER TABLE "product" ADD "fullTextSearch" tsvector`)

    await queryRunner.query(`
        CREATE INDEX "IDX_fullTextSearch" 
        ON "product" 
        USING GIN ("fullTextSearch");
    `)

    // Populate the fullTextSearch column for existing records
    await queryRunner.query(`
        UPDATE "product" p
        SET "fullTextSearch" = 
            setweight(to_tsvector(coalesce(p."ean", '')), 'A') || 
            setweight(to_tsvector(coalesce(p."name", '')), 'B') ||
            setweight(to_tsvector(coalesce(c."name", '')), 'C')
        FROM "category" c
        WHERE p."categoryId" = c."id";
    `)
  }

  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(`
            DROP INDEX "IDX_fullTextSearch";
        `)
    await queryRunner.query(`ALTER TABLE "product" DROP COLUMN "fullTextSearch"`)
  }
}

Updating fullTextSearch column with vectors

This can be done 2 ways - database trigger and using TypeORM subscribers.

Database trigger - can be added within migration

await queryRunner.query(`
    CREATE OR REPLACE FUNCTION update_fullTextSearch_trigger()
    RETURNS TRIGGER AS $$
    BEGIN
        NEW."fullTextSearch" :=
            setweight(to_tsvector(coalesce(NEW."gtin", '')), 'A') ||
            setweight(to_tsvector(coalesce(NEW."name", '')), 'B') ||
            setweight(to_tsvector(coalesce((SELECT "name" FROM "category" WHERE "id" = NEW."categoryId"), '')), 'C');
        RETURN NEW;
    END;
    $$ LANGUAGE plpgsql;

    CREATE TRIGGER "update_fullTextSearch"
    BEFORE INSERT OR UPDATE ON "product"
    FOR EACH ROW EXECUTE FUNCTION update_fullTextSearch_trigger();
`);

TypeORM subscribers

Using TypeORM subscribers offers a more controlled and maintainable way to update the search vector. This approach integrates directly with your application code, making it easier to manage, test, and version-control the logic.

• Create subscriber class

@EventSubscriber()
export class ProductSubscriber implements EntitySubscriberInterface<Product> {
  listenTo() {
    return Product
  }

  async afterInsert(event: InsertEvent<Product>) {
    await this.updateSearchVector(event)
  }

  async afterUpdate(event: UpdateEvent<Product>) {
    await this.updateSearchVector(event)
  }

  private async updateSearchVector(event: UpdateEvent<Product> | InsertEvent<Product>) {
    if (!event.entity) return
    await event.manager.query(
      `
        UPDATE product
        SET "fullTextSearch" = 
            setweight(to_tsvector(coalesce(product.gtin, '')), 'A') || 
            setweight(to_tsvector(coalesce(product.name, '')), 'B') || 
            setweight(to_tsvector(coalesce(category.name, '')), 'C')
        FROM category
        WHERE product."categoryId" = category.id
        AND product.id = $1
        `,
      [event.entity.id],
    )
  }
}

• Add subscriber to TypeORM DataSource

{
    type: 'postgres',
    host: process.env.DB_HOST,
    port: process.env.DB_PORT as number | undefined,
    username: process.env.DB_USER,
    password: process.env.DB_PASS,
    database: process.env.DB_NAME,
    entities: [
      Category,
      Product,
    ],
    subscribers: [ProductSubscriber],
  }

Apply migration to your database

Now we all set to use tsvector for full text search!

Usage

The usage is very straightforward with tsquery:

const queryBuilder = this.createQueryBuilder('product')

queryBuilder.where('product.fullTextSearch @@ plainto_tsquery(:value)', { value })


const result = await queryBuilder.getManyAndCount()

Summary

In this article, we’ve demonstrated how to integrate PostgreSQL’s full-text search capabilities using the tsvector and tsquery data types within a Node.js application using TypeORM. We walked through setting up the tsvector column in the Product model, creating a migration to populate and index this column, and explored methods to keep the search vector updated—whether through a database trigger or a TypeORM subscriber. By the end, you can now efficiently search across multiple fields like product name, EAN, and category name using full-text search, making it a powerful tool for optimizing search functionality in your applications.

In the fast-paced world of development, data integrity and reliability are paramount. Robust data validation and efficient handling of user data can make the difference between a smooth experience and an inconsistent application state.

George Fuechsel’s quote below summarizes what this article is about.

“Garbage in, garbage out.” — George Fuechsel

In this article, we will dive into data validation in NestJS. We will explore some complex use cases of class-validator and class-transformer to ensure data are valid and properly formatted. Along the way, we will discuss best practices, some advanced techniques, and common pitfalls to take your skills to the next level. My motive is to equip you to build more resilient and error-proof applications with NestJS.

Whiles we go through this journey together, keep in mind that we should never trust any inputs submitted by a user or client external to the application, regardless of whether it is part of a larger service(micro-service).

Table Of Contents

• Introduction
• Data Transfer Object (DTO). What is it?
• Initial Configuration: Setting Up Your NestJS Project
• Creating User DTOs
• Adding Class Validators to Fields
• Validating Nested Objects
• Using Transform() and Type() from Class-transformer
• Conditional Validation
• Handling Validation Errors
• Understanding Pipes
• Setting Up a Global Validation Pipe
• Formatting Validation Errors
• Creating Custom Validators
• Custom Password Validator
• Asynchronous Custom Validator With Custom Validation Options
• Common Pitfalls and Best Practices
• Conclusion
• Additional Resources

Data Transfer Object (DTO). What is it?

DTO is a pattern we can leverage to encapsulate data and transfer it to different layers of the application. They are useful for managing the data that flows in(request) and out(response) of the app.

Immutable DTOs

As we have already established, the main idea for using DTOs is to transfer data and as such, the data should not be changed after they have been created. In general, DTOs are designed to be immutable, meaning that once they are created their properties can not be modified. Some benefit that comes with this include but not limited to:

• Predictable behavior: The confidence that its data remains unchanged.
• Consistency: Once it is created, its state remains unchanged throughout its lifecycle until it’s garbage collected.

JavaScript does not have a built-in type for creating immutable types like we have record types in Java and C#. We can achieve similar behavior by making our fields readonly.

Initial Configuration: Setting Up Your NestJS Project

We will start with a mini user management project, which will include basic CRUD operations to manage users. If you’d like to explore the full source code, you can click here to access the project on GitHub.

Install NestJS CLI

$ npm i -g @nestjs/cli
$ nest new user-mgt

Install class-validator and class-transformer

npm i --save class-validator class-transformer

Generate the user module

$ nest g resource users
? What transport layer do you use? REST API
? Would you like to generate CRUD entry points? No

Create an empty DTO and entities folder. After everything, you should have this structure

Creating User DTOs

Let’s begin by creating the necessary DTOs. This tutorial will focus on only two actions, creating and updating a user. Create two files in the DTO folder

• user-create.dto.ts

export class UserCreateDto {
public readonly name: string;
public readonly email: string;
public readonly password: string;
public readonly age: number;
public readonly dateOfBirth: Date;
public readonly photos: string[];
}

• user-update.dto.ts

import { PartialType } from '@nestjs/mapped-types';
import { UserCreateDto } from './user-create.dto';

export class UserUpdateDto extends PartialType(UserCreateDto) {}

UserUpdateDto extends UserCreateDto to inherit all properties, the PartialType ensures that all fields are optional allowing for partial update. This saves us time so we don’t have to repeat it.

Adding Class Validators to Fields

Let’s break down how to add validation to the fields. Class-validator provides us with a lot of already-made validation decorators to which we can apply these rules to our DTOs. For now, we will use a few to validate UserCreateDto. click here for the full list.

import {
IsString,
IsEmail,
IsInt,
Min,
Max,
Length,
IsDate,
IsArray,
ArrayNotEmpty,
ValidateNested,
IsUrl,
} from 'class-validator';
import { Transform, Type } from 'class-transformer';

export class UserCreateDto {
@IsString()
@Length(2, 30, { message: 'Name must be between 2 and 30 characters' })
@Transform(({ value }) => value.trim())
public readonly name: string;

@IsEmail({}, { message: 'Invalid email address' })
public readonly email: string;

@IsString()
@Length(8, 50, { message: 'Password must be between 8 and 50 characters' })
public readonly password: string;

@IsInt()
@Min(18, { message: 'Age must be at least 18' })
@Max(100, { message: 'Age must not exceed 100' })
public readonly age: number;

@IsDate({ message: 'Invalid date format' })
@Type(() => Date)
public readonly dateOfBirth: Date;

@IsArray()
@ValidateNested()
@ArrayNotEmpty({ message: 'Photos array should not be empty' })
@IsString({ each: true, message: 'Each photo URL must be a string' })
@IsUrl({}, { each: true, message: 'Each photo must be a valid URL' })
public readonly photos: string[];
}

Our simple class has grown in size, we have annotated the fields with decorators from Class-Validator. These decorators apply validation rules to the fields. You may have questions about the decorators if you are new to this. For example, what do they mean? Let’s break down some of the basic validators we have used.

• IsString() → This decorator ensures that a value is a string.
• Length(min, max) → This ensures that the string has a link within the specified range.
• IsInt() → This decorator checks if the value is an integer.
• Min() and Max() → This ensures that a numeric value falls between the range
• IsDate() → This ensures that the value is a valid date
• IsArray() → Validates that the value is an array
• IsUrl() → Validate the value is a valid URL
• Transform() → Change the data into a different format

Decorator Parameters

The UserCreateDto fields validator contains additional properties passed into it. These allow you to:

• Customize validation rules
• Provide values
• Set validation options
• Provide messages when the validation fails etc.

Validating Nested Objects

Unlike normal fields validating nested objects requires a bit of extra processing, class-transformer together with class-validator allows you to validate nested objects.

We did a little bit of nested validation in UserCreateDto when we validated the photos field.

@IsArray()
@IsUrl({}, { each: true, message: 'Each photo must be a valid URL' })
public readonly photos: string[];

Photos are an array of strings. To validate the nested strings, we added ValidateNested() and { each: true } to ensure that, each link is a valid URL.

Let’s update photos a some-what complex structure. create a new file in DTO folder and name it user-photo.dto.ts

import { IsString, IsInt, Min, Max, IsUrl, Length } from 'class-validator';

export class UserPhotoDto {
@IsString()
@Length(2, 100, { message: 'Name must be between 2 and 100 characters' })
public readonly name: string;

@IsInt()
@Min(1, { message: 'Size must be at least 1 byte' })
@Max(5_000_000, { message: 'Size must not exceed 5MB' })
public readonly size: number;

@IsUrl(
{ protocols: ['http', 'https'], require_protocol: true },
{ message: 'Invalid URL format' },
)
public readonly url: string;
}

Now let’s update the photos section of UserCreateDto

export class UserCreateDto {
// Other fields

@IsArray()
@ArrayNotEmpty({ message: 'Photos array should not be empty' })
@ValidateNested({ each: true })
@Type(() => UserPhotoDto)
public readonly photos: UserPhotoDto[];
}

The ValidateNested() decorator ensures that each element in the array is a valid photo object. The most important thing to be aware of when it comes to nested validation is that the nested object must be an instance of a class else ValidateNested() won’t know the target class for validation. This is where class-transformer comes in.

Using Transform() and Type() from Class-transformer

Class-transformer provides us with the @Type() decorator. Since Typescript doesn’t have good reflection capabilities yet, we use @Type(() => UserPhotoDto) to give an instance of the class.

We can also utilize the Type() decorator for basic data transformation in our DTO. The dateOfBirth field in UserCreateDto is transformed into a date object using @Type(() => Date).

For complex DTO fields transformation, the Tranform() decorator handles this perfectly. It allows you to access both the field value and the entire object being validated. Whether you’re converting data types, formatting strings, or applying custom logic, @Transform() gives you the control to return the exact version of the value that your application needs.

@Transform(({ value, obj }) => {
// perform additional transformation
return value;
})

Conditional Validation

Most often, some fields need to be validated based on some business rules, we can use the ValidateIf() decorator, which allows you to apply validation to a field only if some condition is true. This is very useful if a field depends on other fields like multi-step forms.

Let’s update the UserPhotoDto to include an optional description field, which should only be validated if it is provided. If the description is present, it should be a string with a length between 10 and 200 characters.

export class UserPhotoDto {
// Other fields

@ValidateIf((o) => o.description !== undefined)
@IsString({ message: 'Description must be a string' })
@Length(10, 200, {
message: 'Description must be between 10 and 200 characters',
})
public readonly description?: string;
}

Handling Validation Errors

Before we dive into how NestJS handles validation errors, let’s first create simple handlers in the user.controller.ts. We need a basic route to handle user creation.

import { Body, Controller, Post } from '@nestjs/common';
import { UserCreateDto } from './dto/user-create.dto';

@Controller('users')
export class UsersController {
@Post()
createUser(@Body() userCreateDto: UserCreateDto) {
// delegating the creation to a service
return {
message: 'User created successfully!',
user: userCreateDto,
};
}
}

Trying this endpoint on Postman with no payload gives us a successful response.

NestJS has a good integration with class-validator for data validation. Still, why wasn’t our request validated? To tell NestJS that we want to validate UserCreateDto we have to supply a pipe to the Body() decorator.

Understanding Pipes

Pipes are flexible and powerful ways to transform and validate incoming data. Pipes are any class decorated with Injectable() and implement the PipeTransform interface. The usage of pipe we are interested is its ability to check that an incoming request meets a certain criteria or throw errors if otherwise.

The most common way to validate the UserCreateDto is to use the built-in ValidationPipe. This pipe validates rules in your DTO defined with class-validator

Now we pass a validation pipe to the Body() to validate the DTO

import { Body, Controller, Post, ValidationPipe } from '@nestjs/common';
import { UserCreateDto } from './dto/user-create.dto';

@Controller('users')
export class UsersController {
@Post()
createUser(@Body(new ValidationPipe()) userCreateDto: UserCreateDto) {
// delegating the creation to services
return {
message: 'User created successfully!',
user: userCreateDto,
};
}
}

With this small change, we get the errors below if we try to create a user with no payload.

Awesome right :)

Setting Up a Global Validation Pipe

To ensure that all requests are validated across the entire application. We have to set up a global validation pipe so that we don’t have to pass validation pipe to every Body() decorator.

Update main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { ValidationPipe } from '@nestjs/common';

async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.useGlobalPipes(
new ValidationPipe({
whitelist: true,
transform: true,
}),
);
await app.listen(3000);
}

bootstrap();

The built-in validation pipe uses class-transformer and class-validator, we can pass validations options to be used by these underlying packages. whitelist: true automatically strips any properties that are not defined in the DTO.transform: true automatically transforms the payload into the appropriate types defined in your DTO.

ValidationPipe({
whitelist: true,
transform: true,
}),

With this, we can remove the pipe we passed to createUser endpoint and it will still be validated. Passing it to parameters helps us fine-tune the validation we need for specific endpoints.

@Post()
createUser(@Body() userCreateDto: UserCreateDto) {
// ...
}

Formatting Validation Errors

The default validation errors format is not bad, we get to see all the errors for the validations that failed, Some frontend developers will scream at you though for mixing all the errors, I have been there😂. Another reason to separate it is when you want to display errors under the fields that failed on the UI.

For nested objects, we also need to retrieve all the errors recursively for a smooth experience. We can achieve this by passing a custom exceptionFactory method to format the errors.

Update main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import {
BadRequestException,
ValidationError,
ValidationPipe,
} from '@nestjs/common';

async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.useGlobalPipes(
new ValidationPipe({
transform: true,
whitelist: true,
exceptionFactory: (validationErrors: ValidationError[] = []) => {
const getPrettyClassValidatorErrors = (
validationErrors: ValidationError[],
parentProperty = '',
): Array<{ property: string; errors: string[] }> => {
const errors = [];

const getValidationErrorsRecursively = (
validationErrors: ValidationError[],
parentProperty = '',
) => {
for (const error of validationErrors) {
const propertyPath = parentProperty
? `${parentProperty}.${error.property}`
: error.property;

if (error.constraints) {
errors.push({
property: propertyPath,
errors: Object.values(error.constraints),
});
}

if (error.children?.length) {
getValidationErrorsRecursively(error.children, propertyPath);
}
}
};

getValidationErrorsRecursively(validationErrors, parentProperty);

return errors;
};

const errors = getPrettyClassValidatorErrors(validationErrors);

return new BadRequestException({
message: 'validation error',
errors: errors,
});
},
}),
);
await app.listen(3000);
}

bootstrap();

This looks way better. Hopefully, you don’t go through what I went through with the front-end developers to get here 😅. Let’s go through what is happening.

We passed an anonymous function to exceptionFactory. The functions accept the array of validation errors. Diving into the validationError interface.

export interface ValidationError {
target?: Record<string, any>;
property: string;
value?: any;
constraints?: {
[type: string]: string;
};
children?: ValidationError[];
contexts?: {
[type: string]: any;
};
}

For example, if we apply IsEmail() on a field and the provided value is not valid. A validation error is created. We also want to know the property where the error occurred. We need to keep in mind that, we can have nested objects for example the photos in UserCreateDto and therefore we can have a parent property let’s say, photos where the error is with the url in the UserPhotoDto.

We first declare an inner function, that takes the errors and sets the parent property to an empty string since it is the root field.

const getValidationErrorsRecursively = (
validationErrors: ValidationError[],
parentProperty = '',
) => {

};

We then loop through the errors and get the property. For nested objects, I prefer to show the fields as photos.0.url. Where 0 is the index of the invalid photo in the array.

The error messages are stored in the constraints field as it’s in the validationError interface. We retrieve these errors and store them under a specific field.

if (error.constraints) {
errors.push({
property: propertyPath,
errors: Object.values(error.constraints),
});
}

For nested objects, the children property of a validation error contains an array of validationError for the nested objects. We can easily get the errors by recursively calling our function and passing the parent property.

if (error.children?.length) {
getValidationErrorsRecursively(error.children, propertyPath);
}

Creating Custom Validators

While Class-validator provides a comprehensive set of built-in validators, there are times when your requirements exceed the standard validation rules or the standard validation doesn’t fit what you want to do. Custom validators are useful when you need to enforce rules that aren’t covered by the standard validators. Examples:

• We can create a custom validator to enforce a specific rule on what a valid password should be.
• We can create another to ensure that the username is unique.

To create a custom validator, we have to define a new class that implements the ValidatorConstraintInterface from class-validator. This requires us to implement two methods:

• validate → Contains your validation logic and must return a boolean
• defaultMessage → Optional default message to return when the validation fails.

Custom Password Validator

Create a new folder in users module named validators. Create two files, is-valid-password.validator.ts and is-username-unique.validator.ts. It should look like this.

A valid password in our use case is very simple. it should contains

• At least one uppercase letter.
• At least one lowercase letter.
• At least one symbol.
• At least one number.
• Password length should be more than 5 characters and less than 20 characters.

Update is-valid-password.validator.ts

import {
ValidatorConstraint,
ValidatorConstraintInterface,
ValidationArguments,
} from 'class-validator';

@ValidatorConstraint({ name: 'IsStrongPassword', async: false })
export class IsValidPasswordConstraint implements ValidatorConstraintInterface {
validate(password: string, args: ValidationArguments) {
return (
typeof password === 'string' &&
password.length > 5 &&
password.length <= 20 &&
/[A-Z]/.test(password) &&
/[a-z]/.test(password) &&
/[0–9]/.test(password) &&
/[!@#$%^&*(),.?":{}|<>]/.test(password)
);
}

defaultMessage(args: ValidationArguments) {
return 'Password must be between 6 and 20 characters long and include at least one uppercase letter, one lowercase letter, one number, and one special character';
}
}

IsValidPasswordContraint is a custom validator because it is decorated with ValidatorConstraint(), we provide our custom validation rules in the validate method. If the validate function returns false, the error message in the defaultMessage will be returned. Providing these methods implements the ValidatorContraintInterface. To use isValidPasswordContraint, update the password field in UserCreateDto. For ValidatorConstraint({ name: ‘IsStrongPassword’, async: false }), we provided the constraint name that will be used to retrieve the error and also, since all actions in the validate are synchronous, we set async to false.

import { Validate } from 'class-validator';

export class UserCreateDto {
// other fields

@Validate(IsValidPasswordConstraint)
public readonly password: string;
}

Now, if we try again with an invalid password, we get this result indicating our custom validator is working.

We can go further and create a decorator for the validator so that we can decorate the password field without using the Validate.

Update is-valid-password.validator.ts

import {
ValidatorConstraint,
ValidatorConstraintInterface,
ValidationArguments,
registerDecorator,
ValidatorOptions,
} from 'class-validator';

@ValidatorConstraint({ name: 'IsStrongPassword', async: false })
class IsValidPasswordConstraint implements ValidatorConstraintInterface {
// removing the implementation so that we focus on IsPasswordValid function
}

export function IsValidPassword(validationOptions?: ValidatorOptions) {
return function (object: NonNullable<unknown>, propertyName: string) {
registerDecorator({
target: object.constructor,
propertyName: propertyName,
options: validationOptions,
constraints: [],
validator: IsValidPasswordConstraint,
});
};
}

Creating custom decorators makes working with validators a breeze, NestJs gives us registerDecorator to create our own. we provide it with the validator which is the IsValidPasswordContraint we created. We can use it like this

export class UserCreateDto {
// other fields

@IsValidPassword()
public readonly password: string;
}

Asynchronous Custom Validator With Custom Validation Options

It is common to encounter scenarios where you need to validate against external systems. Let’s assume that the username in UserCreateDto is unique across the various servers.

Update is-unique-username.validator.ts

import {
ValidatorConstraint,
ValidatorConstraintInterface,
ValidationArguments,
registerDecorator,
ValidationOptions,
} from 'class-validator';

interface IsUsernameUniqueOptions {
server: string;
message?: string;
}

@ValidatorConstraint({ name: 'IsUsernameUnique', async: true })
export class IsUsernameUniqueConstraint
implements ValidatorConstraintInterface
{
async validate(username: string, args: ValidationArguments) {
const options = args.constraints[0] as IsUsernameUniqueOptions;
const server = options.server;

// server check, let assume username exist
return !(await this.checkUsernameOnServer(username, server));
}

defaultMessage(args: ValidationArguments) {
const options = args?.constraints[0] as IsUsernameUniqueOptions;
return options?.message || 'Username is already taken';
}

async checkUsernameOnServer(username: string, server: string) {
return true;
}
}

export function IsUsernameUnique(
options: IsUsernameUniqueOptions,
validationOptions?: ValidationOptions,
) {
return function (object: object, propertyName: string) {
registerDecorator({
target: object.constructor,
propertyName: propertyName,
options: validationOptions,
constraints: [options],
validator: IsUsernameUniqueConstraint,
});
};
}

Usage

export class UserCreateDto {
@IsString()
@Length(2, 30, { message: 'Name must be between 2 and 30 characters' })
@Transform(({ value }) => value.trim())
@IsUsernameUnique({ server: 'east-1', message: 'Name already exists' })
public readonly name: string;

// other fields
}

We created a simple interface to show the possible options we can pass to the decorator. These options are constraints that will be used by IsUsernameUniqueConstraint, we can get them through the validation arguments . const options = args.constraints[0] as IsUsernameUniqueOptions;

Logging options give us { server: ‘east-1’, message: ‘Name already exists’ }, We then called the required service and passed the server name and username to validate the uniqueness of the name.

Also, async is set to true to allow asynchronous operations inside the validate function; ValidatorConstraint({ name: ‘IsUsernameUnique’, async: true }).

Common Pitfalls and Best Practices

It is necessary to be aware of common pitfalls to ensure robust and maintainable code.

• Avoid direct use of entities. One common mistake is using entities directly. Entities are typically used for database interactions and may contain fields or relationships that shouldn’t be exposed or validated on incoming requests.
• Test Custom Validators Extensively. Validation logic is a critical part of your application’s security and data integrity. Ensure they are well-tested.
• Be Explicit with Error Messages. Provide error messages that are informative and user-friendly. It should communicate what the user should do to correct it.
• Leverage Built-in and Custom Validators Together. Our IsUniqueUsername validator still uses IsString() on the name field. We don’t have to reinvent everything if it is already available.

Conclusion

There is so much to add like validation groups, using service containers, etc, but this article is getting way longer than I anticipated 😅. As you continue developing with NestJS, I encourage you to explore more complex use cases and scenarios and share your experiences to keep the learning journey going.

Data validation is crucial in ensuring data integrity within any application and the principles covered here will serve as a strong foundation for further growth and mastery in building secure and efficient applications.

This is my very first article, and I’m eager to hear your thoughts! 😊 Please feel free to leave any feedback in the comments.

If you’d like to connect and stay updated on future content, you can find me on LinkedIn

favicon generator: https://www.favicon-generator.org/

ascii banner generator: https://manytools.org/hacker-tools/ascii-banner/

GitHub - public-apis/public-apis: A collective list of free APIs

A collective list of free APIs. Contribute to public-apis/public-apis development by creating an account on GitHub.

GitHubpublic-apis

Introduction:
In the dynamic realm of technology, Application Programming Interfaces (APIs) play a pivotal role in connecting various software systems, enabling them to communicate and share data seamlessly. While there are countless APIs available, the focus of this blog is on the fascinating world of free open source APIs – the digital building blocks that empower developers to create innovative applications without breaking the bank.

What are Free Open Source APIs?
Free open source APIs are software interfaces that are not only accessible without cost but also come with the added benefit of being open source. This means that the source code is available for anyone to view, modify, and distribute. These APIs adhere to the principles of collaboration and transparency, fostering a community-driven approach to development.

Advantages of Free Open Source APIs:

1. Cost-Efficiency:
   • One of the primary advantages of using free open source APIs is the elimination of licensing fees. Developers can leverage these APIs without worrying about financial constraints, making them an attractive choice for individuals, startups, and even larger organizations looking to reduce expenses.
2. Customization and Flexibility:
   • With access to the source code, developers can customize and tailor the functionality of the API to suit their specific needs. This flexibility empowers developers to create unique solutions that align perfectly with their project requirements.
3. Community Support:
   • The open source community is known for its collaborative spirit. Free open source APIs often have an active community of developers who contribute to ongoing improvements, troubleshoot issues, and share knowledge. This collective effort results in robust and reliable APIs.
4. Security and Trust:
   • Open source APIs are subject to peer review, which enhances their security. The transparency of the source code allows developers to scrutinize the implementation for vulnerabilities, contributing to a more secure overall ecosystem. Trust is built through transparency and collaboration.

Popular Free Open Source APIs:

1. OpenWeatherMap API:
   • Access real-time weather data for any location on Earth. This API is widely used in applications ranging from mobile weather apps to smart home systems.
2. OpenStreetMap API:
   • Harness the power of crowd-sourced mapping data with OpenStreetMap API. Developers can integrate maps, geocoding, and routing features into their applications.
3. GitHub REST API:
   • GitHub's API allows developers to interact with repositories, issues, and other GitHub features programmatically. This is invaluable for automating workflows and integrating GitHub functionalities into custom tools.
4. REST Countries API:
   • Retrieve information about countries, including details such as currency, population, and time zones. This API is a valuable resource for applications requiring country-specific data.

Conclusion:
Free open source APIs are the unsung heroes of the digital age, democratizing access to powerful tools and fostering a collaborative environment for innovation. As developers continue to explore and contribute to these open source projects, the possibilities for creating impactful and cost-effective applications are truly boundless. Embrace the world of free open source APIs, and unlock the potential for transformative and inclusive technological solutions.