Integration Testing Guide

This guide outlines the methodology for writing and maintaining integration tests in TAVI. Integration tests verify how multiple components work together as a system, complementing unit tests that validate individual components.

Overview

Integration tests in TAVI are located in tests/integration/ and serve to:

Verify complete workflows end-to-end
Test interaction between multiple components (backend, frontend, storage, etc.)
Ensure configuration and initialization work correctly in a realistic environment
Catch issues that unit tests cannot detect

Integration tests are marked with the @pytest.mark.integration decorator and can be run separately from unit tests using:

pixi run integration-test

Running All Tests (Including Integration)

To run both unit and integration tests:

pytest -m ""

Test Structure

Integration tests inherit from IntegrationTest base class provided by the neutrons_standard library and use pytest with Qt support via pytest-qt.

Basic Test Template

import pytest
from neutrons_standard.test.integration.test_base import IntegrationTest
from neutrons_standard.test.integration.test_summary import TestSummary

@pytest.mark.integration
class TestMyWorkflow(IntegrationTest):

    @pytest.fixture(scope="function", autouse=True)
    def setup(self):
        """Initialize state for each test function."""
        pass

    def test_workflow_name(self, qtbot, qapp, monkeypatch):
        """Test a complete workflow with multiple steps."""
        self.test_summary = (
            TestSummary.builder()
            .step("Initialize components")
            .step("Perform action")
            .step("Verify result")
            .build()
        )

        # Test implementation here

        self.test_summary.SUCCESS()

Key Principles

Reusable Subtest Methods

Break down test workflows into reusable subtest methods that can be composed in different ways. Each subtest method encapsulates a specific action or verification step and can execute independently given the appropriate component state.

Good Example:

@pytest.mark.integration
class TestLoadRawScans(IntegrationTest):

    def _subtest_load_folder(self, state, qtbot):
        """Subtest: Load a folder via file dialog."""
        def auto_accept():
            for w in QApplication.topLevelWidgets():
                if isinstance(w, QFileDialog):
                    w.selectFile(Resource.getPath("/inputs/integration/load"))
                    w.accept()
                    break

        QTimer.singleShot(0, auto_accept)
        qtbot.mouseClick(state.load_button)
        qtbot.waitUntil(lambda: len(state.project_model.scans) > 0)

    def _subtest_verify_scans_visible(self, state):
        """Subtest: Verify loaded scans appear in the project view."""
        assert len(state.project_model.scans) > 0
        assert state.gui.scan_list_view.count() > 0

    def test_load_and_display_scans(self, qtbot, qapp):
        """Main test: Load scans and verify they're displayed."""
        state = ApplicationState(qtbot, qapp)
        state.gui.show()

        self._subtest_load_folder(state, qtbot)
        self._subtest_verify_scans_visible(state)

    def test_load_multiple_folders(self, qtbot, qapp):
        """Main test: Load multiple folders sequentially."""
        state = ApplicationState(qtbot, qapp)
        state.gui.show()

        self._subtest_load_folder(state, qtbot)
        self._subtest_load_folder(state, qtbot)  # Reuse the same subtest
        assert len(state.project_model.scans) > 1

Benefits:

Subtests are composable and reusable across multiple test methods
Main test methods orchestrate which subtests run based on the scenario
State is explicitly managed by the main test method
Tests become more modular and easier to maintain
Reduces code duplication for common operations

Test Isolation

Each integration test should be independent:

Use fixtures with appropriate scope to initialize state fresh for each test
Clean up GUI widgets and temporary resources after each test
Monkeypatch external interactions (file dialogs, message boxes) to avoid GUI popups

def test_workflow(self, qtbot, qapp, monkeypatch):
    def fail_on_exec(*args, **kwargs):
        raise AssertionError("Unexpected message box exec() called")

    monkeypatch.setattr(TaviMessageBox, "exec", fail_on_exec)

    # Test implementation

    file_menu_view.close()  # Always clean up

Simulate User Interactions

Use pytest-qt features to simulate realistic user behavior:

qtbot.mouseClick() for button clicks
qtbot.waitUntil() for waiting on asynchronous events
QTimer.singleShot() to schedule operations
File dialog auto-acceptance via monkeypatching

def auto_accept_file_dialog():
    for w in QApplication.topLevelWidgets():
        if isinstance(w, QFileDialog):
            w.selectFile(Resource.getPath("/inputs/integration/load"))
            w.accept()
            break

QTimer.singleShot(0, auto_accept_file_dialog)
qtbot.mouseClick(button)

Use TestSummary for Documentation

Document test steps using TestSummary to make test intent clear:

self.test_summary = (
    TestSummary.builder()
    .step("Open Folder Browser")
    .step("Load Raw Scans into ProjectView")
    .step("Verify data loaded correctly")
    .build()
)

Wait for Asynchronous Operations

Integration tests often need to wait for backend processing:

# Wait for a condition to be true (with timeout)
qtbot.waitUntil(lambda: file_menu_view.isVisible())

# Wait for a fixed duration (use sparingly)
qtbot.wait(50)  # 50 milliseconds

Test Data Management

Integration tests use test data stored in test_data/ directory. This includes:

Sample HDF5 files (e.g., test_data/tavi_test_exp1031.h5)
Configuration and macro files
Input directories for file loading tests

When adding new integration tests that require test data:

Store minimal, representative data files in test_data/
Document the data format and contents
Use neutrons_standard.config.Resource.getPath() to reference test data
Keep file sizes small to maintain fast test execution

Git-LFS Consideration

As test data grows, consider using Git LFS (Git Large File Storage):

Beneficial when: binary test files exceed 50-100 MB collectively
Advantages: keeps Git repository lightweight, maintains full history
Implementation: add patterns to .gitattributes for large files
Example: *.h5 filter=lfs diff=lfs merge=lfs -text

Future plans may include:

Migration of large HDF5 test files to LFS
Automated test data generation from smaller base files
Docker-based test environment with pre-built test data

Common Patterns

Loading Application Components

def test_application_startup(self, qtbot, qapp):
    # Initialize project and application models
    tavi_project_model = TaviProjectModel()
    filestore = LocalFileStore()
    application_model = ApplicationModel(filestore)

    # Wire up models through a presenter
    dict_of_model = {
        "TaviProjectProxy": TaviProjectProxy(tavi_project_model),
        ApplicationModelInterface.__name__: ApplicationModelProxy(application_model),
    }

    main_presenter = MainPresenter(dict_of_model)
    gui = main_presenter._view
    gui.show()
    qtbot.addWidget(gui)

    # Now test interactions with the initialized GUI

Testing File Operations

def test_file_loading(self, qtbot, qapp, monkeypatch):
    # Set up the state
    state = ApplicationState(qtbot, qapp)

    # Mock file dialog to automatically accept and select a file
    def auto_accept():
        for w in QApplication.topLevelWidgets():
            if isinstance(w, QFileDialog):
                w.selectFile(Resource.getPath("/inputs/integration/load"))
                w.accept()
                break

    QTimer.singleShot(0, auto_accept)

    # Trigger the file open action
    action = state.presenter.file_menu_presenter._view.load_folder_action
    qtbot.mouseClick(state.gui, Qt.LeftButton, pos=...)

    # Wait and verify
    qtbot.waitUntil(lambda: len(state.project_model.scans) > 0)
    assert UUID(value="expected_uuid") in state.project_model.scans

Best Practices

✓ Do:

Create reusable state-based components for common setup
Document test steps using TestSummary
Keep individual tests focused on a single workflow
Use monkeypatching to control external dependencies
Wait for conditions rather than using fixed delays
Clean up GUI resources after each test
Add comments explaining non-obvious GUI interactions

✗ Don’t:

Create monolithic tests that test multiple unrelated workflows
Use fixed delays (time.sleep()) for waiting on conditions
Leave GUI windows open after tests complete
Import test utilities directly from integration modules (use fixtures)
Write tests that depend on the order of execution
Hardcode file paths; use Resource.getPath() instead

Debugging Integration Tests

Running with Verbose Output

pytest tests/integration/test_load_raw_scans.py -v -s

Running a Single Test

pytest tests/integration/test_load_raw_scans.py::TestLoadRawScans::test_load_ORNL_Spice -v -s

Capturing Screenshots

For debugging GUI issues, pytest-qt can capture screenshots:

def test_with_screenshot(self, qtbot, qapp):
    # ... test code ...
    qtbot.screenshot()  # Creates a screenshot in the current directory

Troubleshooting

Test hangs waiting for dialogs: - Ensure file dialogs are being mocked with monkeypatch.setattr() - Check that QTimer.singleShot(0, ...) is scheduled before triggering the action - Use qtbot.waitUntil() with timeout instead of infinite waits

GUI elements not found: - Add qtbot.wait(50) to allow layout to complete after showing widgets - Use qtbot.waitUntil() to wait for elements to become visible - Check that widgets are being properly initialized in the fixture

Singleton state pollution: - Integration tests bypass singleton reset (see conftest.py) - Ensure proper cleanup in test fixtures if singletons are used - Use separate test classes for tests with conflicting singleton state

Integration Test Examples

See tests/integration/test_load_raw_scans.py for a complete example of:

Setting up application state with multiple models
Mocking file dialogs and message boxes
Simulating user interactions with the GUI
Using TestSummary to document test steps
Verifying application state changes

Contributing Integration Tests

When adding a new integration test:

Create a new test file in tests/integration/ named test_<feature_name>.py
Inherit from IntegrationTest base class
It will inherit the @pytest.mark.integration decorator this way.
Extract reusable state setup into separate classes
Document test steps with TestSummary
Add comments for non-obvious GUI manipulations
Ensure tests clean up after themselves
Run tests locally: pixi run integration-test
Verify unit tests still pass: pixi run test

Future Enhancements

Planned Improvements:

Git-LFS Integration: Large binary test files will be managed with Git LFS for faster clones and reduced repository size
Test Data Factories: Utilities to programmatically generate test data instead of storing large files
Snapshot Testing: Visual regression testing for GUI components
Performance Baselines: Track integration test execution time to catch performance regressions
CI/CD Optimization: Parallel execution of independent integration tests
Test Video Recording: Automated recording of test execution for debugging failed tests