Skip to content

CI_TESTING_IMPLEMENTATION.md

Latest commit

 

History

History
238 lines (185 loc) · 7.95 KB

CI_TESTING_IMPLEMENTATION.md

File metadata and controls

238 lines (185 loc) · 7.95 KB

CI/CD Testing Implementation for QtMeshEditor

Overview

This document outlines the comprehensive testing infrastructure updates made to the QtMeshEditor project's GitHub Actions CI/CD pipeline. The implementation ensures that the new MaterialEditorQML unit tests are properly executed and their coverage is accurately reported to both SonarQube and CodeClimate.

Updated Components

1. GitHub Actions Workflow (.github/workflows/deploy.yml)

New Test Jobs Added:

Windows Unit Tests (unit-tests-windows)

  • Runs comprehensive MaterialEditorQML tests on Windows platform
  • Uses MinGW compiler with Qt 6.9.1
  • Executes all test categories: unit, QML integration, performance, and component tests
  • Uploads test results as artifacts for analysis

Enhanced Linux Unit Tests (unit-tests-linux)

  • Updated to run comprehensive test suite with proper coverage collection
  • Integrated with SonarQube and CodeClimate reporting
  • Uses gcov/lcov for coverage data generation
  • Supports X11 virtual display for QML tests

Key Improvements:

  1. Multi-Platform Testing: Tests now run on both Windows and Linux

  2. Comprehensive Test Execution: All test categories are executed:

    • MaterialEditorQML_test (C++ unit tests)
    • MaterialEditorQML_qml_test (QML integration tests)
    • MaterialEditorQML_perf_test (Performance tests)
    • MaterialEditorQML_qml_test_runner (QML component tests)

  3. Enhanced Coverage Collection:

    • Proper gcov flags for C++ coverage: --coverage -fprofile-arcs -ftest-coverage
    • Improved gcovr configuration with exclusions for test files
    • Better lcov filtering for CodeClimate integration

  4. Test Result Artifacts:

    • XML test results uploaded for all platforms
    • Coverage reports in multiple formats (XML, HTML, LCOV)
    • Separate artifacts for Windows and Linux test results

2. SonarQube Configuration (sonar-project.properties)

Updated Settings:

# Main source directories
sonar.sources=src/
sonar.tests=src/,tests/

# Test file patterns - include all our new test files
sonar.test.inclusions=**/*_test.cpp,**/test_*.cpp,tests/**/*.cpp,tests/**/*.qml

# Enhanced exclusions
sonar.exclusions=**/OgreXML/**,**/dependencies/**,**/*_autogen/**,**/CMakeFiles/**,**/ui_files/**,**/moc_*,**/_deps/**

# Coverage exclusions - exclude test files from coverage calculation
sonar.coverage.exclusions=**/*_test.cpp,**/test_*.cpp,tests/**/*.cpp,tests/**/*.qml,**/*_autogen/**

# C++ specific optimizations
sonar.cfamily.compile-commands=compile_commands.json
sonar.cfamily.cache.enabled=true
sonar.cfamily.threads=4

Key Improvements:

  • Proper test file classification for better analysis
  • Enhanced exclusion patterns to focus on production code
  • Optimized C++ analysis settings for performance
  • Better separation of source and test code

3. CodeClimate Configuration (.codeclimate.yml)

Enhanced Exclusions:

exclude_patterns:
  # ... existing patterns ...
  # Exclude all test files from code quality analysis
  - "**/*_test.cpp"
  - "**/test_*.cpp"
  - "tests/**/*.cpp"
  - "tests/**/*.qml"
  - "tests/**/*.h"
  - "**/*_test_runner.cpp"
  - "**/*_perf_test.cpp"
  - "**/*_qml_test.cpp"

# Test coverage settings
prepare:
  fetch:
  - url: https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64
    path: ./cc-test-reporter

Benefits:

  • Test files excluded from maintainability analysis
  • Focus on production code quality metrics
  • Proper test reporter integration

4. Test Build Configuration (tests/CMakeLists.txt)

Comprehensive Test Executable Setup:

# Helper function to create test executables
function(create_test_executable target_name source_files link_libraries)
    add_executable(${target_name} ${source_files})
    
    target_include_directories(${target_name} PRIVATE
        ${CMAKE_CURRENT_SOURCE_DIR}/../src
        ${CMAKE_CURRENT_SOURCE_DIR}/../ui_files
    )
    
    target_link_libraries(${target_name} ${link_libraries})
    
    # Add the test to CTest
    add_test(NAME ${target_name} COMMAND ${target_name})
endfunction()

Test Targets Created:

  1. MaterialEditorQML_test - C++ unit tests
  2. MaterialEditorQML_qml_test - QML integration tests
  3. MaterialEditorQML_perf_test - Performance tests
  4. MaterialEditorQML_qml_test_runner - QML component tests

Features:

  • Automatic Google Test discovery for detailed reporting
  • Proper library linking including Ogre3D and Assimp
  • CTest integration for CI execution
  • QML test file deployment to runtime directories

Test Execution Flow

Linux Pipeline:

  1. Build Phase: Configure with coverage flags, build all test executables
  2. Test Execution: Run comprehensive test suite with XML output
  3. Coverage Generation:
    • Generate gcov files for all object files
    • Create gcovr reports (XML, HTML)
    • Process lcov data for CodeClimate
  4. Reporting:
    • Upload to SonarQube with proper test/source separation
    • Submit coverage to CodeClimate
    • Archive test results and coverage reports

Windows Pipeline:

  1. Build Phase: Configure and build test executables
  2. Dependency Setup: Copy required DLLs for test execution
  3. Test Execution: Run all test categories with XML output
  4. Artifact Upload: Store test results for analysis

Coverage Metrics

Included in Coverage:

  • All production source files in src/ (excluding test files)
  • MaterialEditorQML.cpp and related components
  • Qt integration code
  • Ogre3D integration code

Excluded from Coverage:

  • All *_test.cpp files
  • Test runner and performance test files
  • Auto-generated MOC files
  • Dependencies (OgreXML, ogre-procedural)
  • CMake generated files

Quality Gates

SonarQube Integration:

  • Test Coverage: Tracked for production code only
  • Code Quality: Maintainability, reliability, security analysis
  • Test Detection: Proper identification of test vs. source files
  • Performance: Optimized analysis with caching and threading

CodeClimate Integration:

  • Maintainability: Focus on production code complexity
  • Test Coverage: Accurate coverage reporting via lcov
  • Duplication: Detection excluding test code patterns

Usage Instructions

Running Tests Locally:

mkdir build
cmake -S . -B build -DBUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug
cmake --build build
ctest --test-dir build --verbose

Generating Coverage Reports:

# With coverage flags
cmake -S . -B build -DBUILD_TESTS=ON -DCMAKE_CXX_FLAGS="--coverage"
cmake --build build
ctest --test-dir build
gcovr --root . --html --html-details -o coverage.html

CI Triggers:

  • Pull Requests: Full test suite runs on Linux and Windows
  • Master Branch: Complete pipeline with coverage reporting
  • Releases: All platforms including macOS

Benefits Achieved

  1. Comprehensive Coverage: 50+ individual test cases across multiple categories
  2. Cross-Platform Validation: Tests run on Windows and Linux
  3. Quality Assurance: Integrated SonarQube and CodeClimate reporting
  4. Performance Monitoring: Dedicated performance test suite
  5. Regression Detection: Automated test execution on every change
  6. Documentation: Clear separation of test and production code
  7. Maintainability: Organized test structure with helper functions

Troubleshooting

Common Issues:

  1. QML Test Failures: Ensure X11 virtual display is properly configured
  2. Coverage Gaps: Verify gcov flags are applied to all source files
  3. Missing Dependencies: Check that all required Qt modules are linked
  4. Test Discovery: Confirm Google Test executables are properly built

Debug Commands:

# Check test executables
ls -la build/bin/*test*

# Verify coverage files
find build -name "*.gcno" -o -name "*.gcda"

# Test individual components
./build/bin/MaterialEditorQML_test --gtest_list_tests

This implementation provides a robust, scalable testing infrastructure that ensures code quality while supporting the comprehensive MaterialEditorQML test suite.