← home

From Rigid to Robust: Revamping Our CI Templates for a Modern Workflow

tl;dr: Our old CI/CD templates were repetitive, inflexible, and hard to maintain. We rebuilt them using a modular, language-agnostic approach with reusable components, smart conditional logic, and automated tooling (like versioning and security scanning). This resulted in faster, more consistent, secure, and maintainable pipelines adaptable to various project needs.

Table of contents

Continuous Integration and Continuous Deployment (CI/CD) pipelines are the backbone of modern software development. However, the templates governing these pipelines can often become a source of frustration – rigid, repetitive, difficult to maintain, and struggling to keep up with diverse tech stacks. We faced these exact issues with our internal CI template repository. This is the story of how we transformed our CI architecture from a maintenance headache into a flexible, modular, and powerful asset.

The Challenge: When Templates Become Obstacles

Our previous CI template setup suffered from several growing pains:

  • Code Duplication: Similar logic, like setting up specific language environments or deployment steps, was copied across numerous templates.
  • Limited Flexibility: Customizing pipelines often meant duplicating large chunks of template code just to make small modifications.
  • Poor Modularity: A change, for instance, to a scanning tool configuration, required hunting down and updating multiple files.
  • Hardcoded Assumptions: Templates often made rigid assumptions about project structure or the specific language used, making them difficult to adapt. Example: A build job might only work if requirements.txt exists, failing for Node.js projects.

These issues led to slower development cycles and increased the risk of inconsistencies. It was clear a major overhaul was needed.

The Transformation: Introducing a Modular, Language-Agnostic Architecture

We embarked on a significant refactoring effort, guided by principles of modularity, flexibility, and maintainability. Here are the key technical improvements:

1. Modular Design with include

We heavily leveraged GitLab CI’s include: keyword. Instead of one monolithic .gitlab-ci.yml, projects now include relevant template parts. We created a common/ directory for shared logic (e.g., security scanning steps) and a languages/ directory for specific configurations (Node.js, Python).

Example Structure:

# .gitlab-ci.yml in a user's project
include:
  - project: 'your-group/your-templates-repo'
    ref: main # Or a specific version tag
    file:
      - '/common/security.yml'
      - '/languages/python.yml' # Include if it's a Python project
      - '/pipeline-template/lambda.yml' # If deploying a Lambda

variables:
  PYTHON_APP_DIR: "src" # Tell the template where the code lives

2. Language-Agnostic Approach & Smart Detection

The new architecture intelligently detects the project’s language and structure, often using predefined CI/CD variables (like PYTHON_APP_DIR or NODE_APP_DIR). Conditional logic (rules:) within the templates ensures jobs run only when needed.

Example Conditional Rule (Conceptual):

# From languages/python.yml
.python_default_rules: &python_default_rules
  rules:
    - if: '$PYTHON_APP_DIR && $PYTHON_APP_DIR != "."' # Run if Python dir is set
      exists:
        - "$PYTHON_APP_DIR/pyproject.toml" # And a key file exists
      when: always
    - when: never # Don't run otherwise

python-lint:
  <<: *python_default_rules # Apply the rule
  script:
    - echo "Running Python linting in $PYTHON_APP_DIR..."
    # - python lint commands...

3. Unified Pipeline Stages

Common stages like install-dependencies can now have language-specific implementations triggered conditionally, feeding into subsequent shared stages.

Example Unified Stage (Conceptual):

# From common/pipeline.yml
install-dependencies:
  stage: install
  script:
    - echo "All dependencies installed successfully"
  needs: # Optional dependencies based on detected language
    - job: python-install-dependencies # Runs if python rules match
      optional: true
    - job: node-install-dependencies # Runs if node rules match
      optional: true
  rules:
    # Rule to run if *any* language was detected
    - if: '$NODE_APP_DIR || $PYTHON_APP_DIR'
      when: always

4. Advanced CI/CD Capabilities

  • Automated Semantic Versioning: We implemented a system using commit message conventions (e.g., prefixing commits with #major, #minor, #patch). CI automation detects these prefixes on merge to the main branch, calculates the next semantic version, creates a tag, and generates a changelog entry.
  • Enhanced Security Scanning: Integrated Trivy for comprehensive container image and filesystem vulnerability scanning. Findings can be automatically pushed to AWS Security Hub for centralized visibility and tracking.
  • Multi-Language CDK Support: Deployment templates now better handle AWS CDK projects written in different languages (like TypeScript, Python), running the appropriate cdk commands based on detected project files (package.json, requirements.txt, etc.).
  • Improved Caching: Implemented more intelligent caching strategies, particularly for Kubernetes scanning steps, to speed up pipeline execution.

The Benefits: A More Efficient and Secure Future

This redesigned architecture yielded significant advantages:

  • Simplified Updates: Modifying a common task (like a scanner update) requires changes in only one central place.
  • Faster Development: Developers configure pipelines faster by including relevant blocks and setting a few variables.
  • Increased Consistency: Reduced duplication ensures builds and deployments are more reliable and predictable.
  • Enhanced Security: Automated, integrated scanning improves the overall security posture.
  • Greater Flexibility: Easily onboard new languages or adapt to different project structures without massive template rewrites.

Conclusion

Refactoring CI/CD templates is a worthwhile investment. By embracing modularity (include:), conditional logic (rules:), automation (versioning, scanning), and a language-agnostic design, we transformed our internal templates from a bottleneck into a powerful accelerator. If your CI/CD setup feels more like a collection of scripts than a cohesive system, consider adopting these principles for a more maintainable, flexible, and efficient future.