unraid-mcp/.plan.md

# Implementation Plan: mcporter Integration Tests + Destructive Action Gating

**Date:** 2026-02-15
**Status:** Awaiting Approval
**Estimated Effort:** 8-12 hours

## Overview

Implement comprehensive integration testing using mcporter CLI to validate all 86 tool actions (after removing 4 destructive array operations) against live Unraid servers, plus add environment variable gates for remaining destructive actions to prevent accidental operations.

## Requirements

1. **Remove destructive array operations** - start, stop, shutdown, reboot should not be exposed via MCP
2. **Add per-tool environment variable gates** - UNRAID_ALLOW_*_DESTRUCTIVE flags for remaining destructive actions
3. **Build mcporter test suite** - Real end-to-end testing of all 86 actions against live servers (tootie/shart)
4. **Document all actions** - Comprehensive action catalog with test specifications

## Architecture Changes

### 1. Settings Infrastructure (Pydantic-based)

**File:** `unraid_mcp/config/settings.py`

- Migrate from simple `os.getenv()` to Pydantic `BaseSettings`
- Add 7 destructive action gate flags (all default to False for safety):
  - `allow_docker_destructive` (docker remove)
  - `allow_vm_destructive` (vm force_stop, reset)
  - `allow_notifications_destructive` (delete, delete_archived)
  - `allow_rclone_destructive` (delete_remote)
  - `allow_users_destructive` (user delete)
  - `allow_keys_destructive` (key delete)
  - `allow_array_destructive` (REMOVED - no longer needed after task 1)
- Add `get_config_summary()` method showing gate status
- Maintain backwards compatibility via module-level exports

**Dependencies:** Add `pydantic-settings` to `pyproject.toml`

### 2. Tool Implementation Pattern

**Pattern for all tools with destructive actions:**

```python
from ..config.settings import settings

# In tool function:
if action in DESTRUCTIVE_ACTIONS:
    # Check 1: Environment variable gate (first line of defense)
    if not settings.allow_{tool}_destructive:
        raise ToolError(
            f"Destructive {tool} action '{action}' is disabled. "
            f"Set UNRAID_ALLOW_{TOOL}_DESTRUCTIVE=true to enable. "
            f"This is a safety gate to prevent accidental operations."
        )

    # Check 2: Runtime confirmation (second line of defense)
    if not confirm:
        raise ToolError(f"Action '{action}' is destructive. Set confirm=True to proceed.")
```

**Tools requiring updates:**
- `unraid_mcp/tools/docker.py` (1 action: remove)
- `unraid_mcp/tools/virtualization.py` (2 actions: force_stop, reset)
- `unraid_mcp/tools/notifications.py` (2 actions: delete, delete_archived)
- `unraid_mcp/tools/rclone.py` (1 action: delete_remote)
- `unraid_mcp/tools/users.py` (1 action: delete)
- `unraid_mcp/tools/keys.py` (1 action: delete)

### 3. mcporter Integration Test Suite

**New Directory Structure:**

```
tests/integration/
├── helpers/
│   ├── mcporter.sh          # mcporter wrapper (call_tool, call_destructive, get_field)
│   ├── validation.sh        # Response validation (assert_fields, assert_equals, assert_success)
│   └── reporting.sh         # Test reporting (init_report, record_test, generate_summary)
├── tools/
│   ├── test_health.sh       # 3 actions
│   ├── test_info.sh         # 19 actions
│   ├── test_storage.sh      # 6 actions
│   ├── test_docker.sh       # 15 actions
│   ├── test_vm.sh           # 9 actions
│   ├── test_notifications.sh # 9 actions
│   ├── test_rclone.sh       # 4 actions
│   ├── test_users.sh        # 8 actions
│   ├── test_keys.sh         # 5 actions
│   └── test_array.sh        # 8 actions (after removal)
├── run-all.sh               # Master test runner (parallel/sequential)
├── run-tool.sh              # Single tool runner
└── README.md                # Integration test documentation
```

**mcporter Configuration:** `config/mcporter.json`

```json
{
  "mcpServers": {
    "unraid-tootie": {
      "command": "uv",
      "args": ["run", "unraid-mcp-server"],
      "env": {
        "UNRAID_API_URL": "https://myunraid.net:31337/graphql",
        "UNRAID_API_KEY": "${UNRAID_TOOTIE_API_KEY}",
        "UNRAID_VERIFY_SSL": "false",
        "UNRAID_MCP_TRANSPORT": "stdio"
      },
      "cwd": "/home/jmagar/workspace/unraid-mcp"
    },
    "unraid-shart": {
      "command": "uv",
      "args": ["run", "unraid-mcp-server"],
      "env": {
        "UNRAID_API_URL": "http://100.118.209.1/graphql",
        "UNRAID_API_KEY": "${UNRAID_SHART_API_KEY}",
        "UNRAID_VERIFY_SSL": "false",
        "UNRAID_MCP_TRANSPORT": "stdio"
      },
      "cwd": "/home/jmagar/workspace/unraid-mcp"
    }
  }
}
```

## Implementation Tasks

### Task 1: Remove Destructive Array Operations

**Files:**
- `unraid_mcp/tools/array.py`
- `tests/test_array.py`

**Changes:**
1. Remove from `MUTATIONS` dict:
   - `start` (lines 24-28)
   - `stop` (lines 29-33)
   - `shutdown` (lines 69-73)
   - `reboot` (lines 74-78)
2. Remove from `DESTRUCTIVE_ACTIONS` set (line 81) - set becomes empty `{}`
3. Remove from `ARRAY_ACTIONS` Literal type (lines 85-86)
4. Update docstring removing these 4 actions (lines 105-106, 115-116)
5. Remove tests for these actions in `tests/test_array.py`

**Acceptance:**
- ✅ Array tool has 8 actions (down from 12)
- ✅ `DESTRUCTIVE_ACTIONS` is empty set
- ✅ Tests pass for remaining actions
- ✅ Removed mutations are not callable

### Task 2: Add Pydantic Settings with Destructive Gates

**Files:**
- `unraid_mcp/config/settings.py`
- `pyproject.toml`
- `.env.example`

**Changes:**

1. **Add dependency:** `pydantic-settings>=2.12` in `pyproject.toml` dependencies

2. **Update settings.py:**
   - Import `BaseSettings` from `pydantic_settings`
   - Create `UnraidSettings` class with all config fields
   - Add 6 destructive gate fields (all default to False):
     - `allow_docker_destructive: bool = Field(default=False, ...)`
     - `allow_vm_destructive: bool = Field(default=False, ...)`
     - `allow_notifications_destructive: bool = Field(default=False, ...)`
     - `allow_rclone_destructive: bool = Field(default=False, ...)`
     - `allow_users_destructive: bool = Field(default=False, ...)`
     - `allow_keys_destructive: bool = Field(default=False, ...)`
   - Add `get_config_summary()` method including gate status
   - Instantiate global `settings = UnraidSettings()`
   - Keep backwards compatibility exports

3. **Update .env.example:** Add section documenting all destructive gates

**Acceptance:**
- ✅ `settings` instance loads successfully
- ✅ All gate fields default to False
- ✅ `get_config_summary()` shows gate status
- ✅ Backwards compatibility maintained (existing code still works)

### Task 3: Update Tools with Environment Variable Gates

**Files to update:**
- `unraid_mcp/tools/docker.py`
- `unraid_mcp/tools/virtualization.py`
- `unraid_mcp/tools/notifications.py`
- `unraid_mcp/tools/rclone.py`
- `unraid_mcp/tools/users.py`
- `unraid_mcp/tools/keys.py`

**Pattern for each tool:**

1. Add import: `from ..config.settings import settings`
2. Add gate check before confirm check in destructive action handler:
   ```python
   if action in DESTRUCTIVE_ACTIONS:
       if not settings.allow_{tool}_destructive:
           raise ToolError(
               f"Destructive {tool} action '{action}' is disabled. "
               f"Set UNRAID_ALLOW_{TOOL}_DESTRUCTIVE=true to enable."
           )
       if not confirm:
           raise ToolError(f"Action '{action}' is destructive. Set confirm=True to proceed.")
   ```
3. Update tool docstring documenting security requirements

**Acceptance (per tool):**
- ✅ Destructive action fails with clear error when env var not set
- ✅ Destructive action still requires confirm=True when env var is set
- ✅ Both checks must pass for execution
- ✅ Error messages guide user to correct env var

### Task 4: Update Test Suite with Settings Mocking

**Files:**
- `tests/conftest.py`
- `tests/test_docker.py`
- `tests/test_vm.py`
- `tests/test_notifications.py`
- `tests/test_rclone.py`
- `tests/test_users.py`
- `tests/test_keys.py`

**Changes:**

1. **Add fixtures to conftest.py:**
   ```python
   @pytest.fixture
   def mock_settings():
       # All gates disabled

   @pytest.fixture
   def mock_settings_all_enabled(mock_settings):
       # All gates enabled
   ```

2. **Update each test file:**
   - Add `mock_settings` parameter to fixtures
   - Wrap tool calls with `with patch("unraid_mcp.tools.{tool}.settings", mock_settings):`
   - Add 3 destructive action tests:
     - Test gate check (env var not set, confirm=True → fails)
     - Test confirm check (env var set, confirm=False → fails)
     - Test success (env var set, confirm=True → succeeds)

**Acceptance:**
- ✅ All 150 existing tests pass
- ✅ New gate tests cover all destructive actions
- ✅ Tests verify correct error messages
- ✅ Tests use mocked settings (don't rely on actual env vars)

### Task 5: Create mcporter Configuration

**Files:**
- `config/mcporter.json` (new)
- `tests/integration/README.md` (new)

**Changes:**

1. Create `config/mcporter.json` with tootie and shart server configs
2. Document how to use mcporter with the server in README
3. Include instructions for loading credentials from `~/workspace/homelab/.env`

**Acceptance:**
- ✅ `mcporter list unraid-tootie` shows all tools
- ✅ `mcporter call unraid-tootie.unraid_health action=test_connection` succeeds
- ✅ Configuration works for both servers

### Task 6: Build mcporter Helper Libraries

**Files to create:**
- `tests/integration/helpers/mcporter.sh`
- `tests/integration/helpers/validation.sh`
- `tests/integration/helpers/reporting.sh`

**Functions to implement:**

**mcporter.sh:**
- `call_tool <tool> <action> [params...]` - Call tool via mcporter, return JSON
- `call_destructive <tool> <action> <env_var> [params...]` - Safe destructive call
- `get_field <json> <jq_path>` - Extract field from JSON
- `is_success <json>` - Check if response indicates success
- `get_error <json>` - Extract error message

**validation.sh:**
- `assert_fields <json> <field>...` - Verify required fields exist
- `assert_equals <json> <field> <expected>` - Field value equality
- `assert_matches <json> <field> <pattern>` - Field matches regex
- `assert_success <json>` - Response indicates success
- `assert_failure <json> [pattern]` - Response indicates failure (negative test)

**reporting.sh:**
- `init_report <tool>` - Initialize JSON report file
- `record_test <report> <action> <status> [error]` - Record test result
- `generate_summary` - Generate console summary from all reports

**Acceptance:**
- ✅ Helper functions work correctly
- ✅ Error handling is robust
- ✅ Functions are reusable across all tool tests

### Task 7: Implement Tool Test Scripts

**Files to create:**
- `tests/integration/tools/test_health.sh` (3 actions)
- `tests/integration/tools/test_info.sh` (19 actions)
- `tests/integration/tools/test_storage.sh` (6 actions)
- `tests/integration/tools/test_docker.sh` (15 actions)
- `tests/integration/tools/test_vm.sh` (9 actions)
- `tests/integration/tools/test_notifications.sh` (9 actions)
- `tests/integration/tools/test_rclone.sh` (4 actions)
- `tests/integration/tools/test_users.sh` (8 actions)
- `tests/integration/tools/test_keys.sh` (5 actions)
- `tests/integration/tools/test_array.sh` (8 actions)

**Per-script implementation:**

1. Source helper libraries
2. Initialize report
3. Implement test functions for each action:
   - Basic functionality test
   - Response structure validation
   - Parameter validation
   - Destructive action gate tests (if applicable)
4. Run all tests and record results
5. Return exit code based on failures

**Priority order (implement in this sequence):**
1. `test_health.sh` - Simplest (3 actions, no destructive)
2. `test_info.sh` - Large but straightforward (19 query actions)
3. `test_storage.sh` - Moderate (6 query actions)
4. `test_docker.sh` - Complex (15 actions, 1 destructive)
5. `test_vm.sh` - Complex (9 actions, 2 destructive)
6. `test_notifications.sh` - Moderate (9 actions, 2 destructive)
7. `test_rclone.sh` - Simple (4 actions, 1 destructive)
8. `test_users.sh` - Moderate (8 actions, 1 destructive)
9. `test_keys.sh` - Simple (5 actions, 1 destructive)
10. `test_array.sh` - Moderate (8 actions, no destructive after removal)

**Acceptance:**
- ✅ Each script tests all actions for its tool
- ✅ Tests validate response structure
- ✅ Destructive action gates are tested
- ✅ Scripts generate JSON reports
- ✅ Exit code indicates success/failure

### Task 8: Build Test Runners

**Files to create:**
- `tests/integration/run-all.sh`
- `tests/integration/run-tool.sh`

**run-all.sh features:**
- Load credentials from `~/workspace/homelab/.env`
- Support sequential and parallel execution modes
- Run all 10 tool test scripts
- Generate summary report
- Return exit code based on any failures

**run-tool.sh features:**
- Accept tool name as argument
- Load credentials
- Execute single tool test script
- Pass through exit code

**Acceptance:**
- ✅ `run-all.sh` executes all tool tests
- ✅ Parallel mode works correctly (no race conditions)
- ✅ Summary report shows pass/fail/skip counts
- ✅ `run-tool.sh health` runs only health tests
- ✅ Exit codes are correct

### Task 9: Document Action Catalog

**File to create:**
- `docs/testing/action-catalog.md`

**Content:**
- Table of all 86 actions across 10 tools
- For each action:
  - Tool name
  - Action name
  - Type (query/mutation/compound)
  - Required parameters
  - Optional parameters
  - Destructive? (yes/no + env var if yes)
  - Expected response structure
  - Example mcporter call
  - Validation criteria

**Acceptance:**
- ✅ All 86 actions documented
- ✅ Specifications are detailed and accurate
- ✅ Examples are runnable
- ✅ Becomes source of truth for test implementation

### Task 10: Integration Documentation

**Files to create/update:**
- `tests/integration/README.md`
- `docs/testing/integration-tests.md`
- `docs/testing/test-environments.md`
- `README.md` (add integration test section)

**Content:**
- How to run integration tests
- How to configure mcporter
- Server setup (tootie/shart)
- Environment variable gates
- Destructive action testing
- CI/CD integration
- Troubleshooting

**Acceptance:**
- ✅ Clear setup instructions
- ✅ Examples for common use cases
- ✅ Integration with existing pytest docs
- ✅ CI/CD pipeline documented

## Testing Strategy

### Unit Tests (pytest - existing)
- **150 tests** across 10 tool modules
- Mock GraphQL responses
- Fast, isolated, offline
- Cover edge cases and error paths

### Integration Tests (mcporter - new)
- **86 tests** (one per action)
- Real Unraid server calls
- Slow, dependent, online
- Validate actual API behavior

### Test Matrix

| Tool | Actions | pytest Tests | mcporter Tests | Destructive |
|------|---------|--------------|----------------|-------------|
| health | 3 | 10 | 3 | 0 |
| info | 19 | 98 | 19 | 0 |
| storage | 6 | 11 | 6 | 0 |
| docker | 15 | 28 | 15 | 1 |
| vm | 9 | 25 | 9 | 2 |
| notifications | 9 | 7 | 9 | 2 |
| rclone | 4 | (pending) | 4 | 1 |
| users | 8 | (pending) | 8 | 1 |
| keys | 5 | (pending) | 5 | 1 |
| array | 8 | 26 | 8 | 0 |
| **TOTAL** | **86** | **~150** | **86** | **8** |

## Validation Checklist

### Code Changes
- [ ] Array tool has 8 actions (removed start/stop/shutdown/reboot)
- [ ] Settings class with 6 destructive gate flags
- [ ] All 6 tools updated with environment variable gates
- [ ] All 6 tool tests updated with gate test cases
- [ ] All existing 150 pytest tests pass
- [ ] `pydantic-settings` added to dependencies
- [ ] `.env.example` updated with gate documentation

### Integration Tests
- [ ] mcporter configuration works for both servers
- [ ] All 3 helper libraries implemented
- [ ] All 10 tool test scripts implemented
- [ ] Test runners (run-all, run-tool) work correctly
- [ ] All 86 actions have test coverage
- [ ] Destructive action gates are tested
- [ ] Reports generate correctly

### Documentation
- [ ] Action catalog documents all 86 actions
- [ ] Integration test README is clear
- [ ] Environment setup documented
- [ ] CI/CD integration documented
- [ ] Project README updated

## Success Criteria

1. **Safety:** Destructive actions require both env var AND confirm=True
2. **Coverage:** All 86 actions have integration tests
3. **Quality:** Clear error messages guide users to correct env vars
4. **Automation:** Test suite runs via single command
5. **Documentation:** Complete action catalog and testing guide

## Risks & Mitigations

### Risk: Breaking existing deployments
**Impact:** HIGH - Users suddenly can't execute destructive actions
**Mitigation:**
- Clear error messages with exact env var to set
- Document migration in release notes
- Default to disabled (safe) but guide users to enable

### Risk: Integration tests are flaky
**Impact:** MEDIUM - CI/CD unreliable
**Mitigation:**
- Test against stable servers (tootie/shart)
- Implement retry logic for network errors
- Skip destructive tests if env vars not set (not failures)

### Risk: mcporter configuration complexity
**Impact:** LOW - Difficult for contributors to run tests
**Mitigation:**
- Clear setup documentation
- Example .env template
- Helper script to validate setup

## Dependencies

- `pydantic-settings>=2.12` (Python package)
- `mcporter` (npm package - user must install)
- `jq` (system package for JSON parsing in bash)
- Access to tootie/shart servers (for integration tests)
- Credentials in `~/workspace/homelab/.env`

## Timeline Estimate

| Task | Estimated Time |
|------|---------------|
| 1. Remove array ops | 30 min |
| 2. Add settings infrastructure | 1 hour |
| 3. Update tools with gates | 2 hours |
| 4. Update test suite | 2 hours |
| 5. mcporter config | 30 min |
| 6. Helper libraries | 1.5 hours |
| 7. Tool test scripts | 4 hours |
| 8. Test runners | 1 hour |
| 9. Action catalog | 2 hours |
| 10. Documentation | 1.5 hours |
| **Total** | **~12 hours** |

## Notes

- Integration tests complement (not replace) existing pytest suite
- Tests validate actual Unraid API behavior, not just our code
- Environment variable gates provide defense-in-depth security
- mcporter enables real-world validation impossible with mocked tests
- Action catalog becomes living documentation for all tools

---

**Plan Status:** Awaiting user approval
**Next Step:** Review plan, make adjustments, then execute via task list