Iteration Plan v4

Project Status Summary 

Current version: v0.9.0

Codebase metrics:

Source code: ~4,900 lines across 23 modules
Test code: 830 tests, 99% coverage (2750/2789 statements, 39 uncovered lines)
CI: 9-matrix (3 Python versions × 3 OS)
Documentation: Sphinx + 8 guides + 3 iteration plans
Mypy: 0 errors (strict_optional, disallow_incomplete_defs enabled)
Ruff: 0 lint errors

Architecture (5 mixins + 1 core client):

MT5WebClient (524 lines)
├── _MarketDataMixin      (845 lines)  — symbols, ticks, bars, book, subscriptions
├── _TradingMixin         (698 lines)  — positions, orders, trade execution
├── _OrderHelpersMixin    (502 lines)  — buy/sell/close/modify convenience
├── _AccountMixin         (488 lines)  — account info, OTP, verification
└── _PushHandlersMixin    (334 lines)  — push notification handlers

+ Infrastructure:
  ├── Transport            (227 lines)  — WebSocket, state machine, rate limiter
  ├── Protocol codec       (270 lines)  — dispatch-table serialization
  ├── Schemas             (1076 lines)  — 24 binary protocol schemas
  └── Support modules      (340 lines)  — events, logging, metrics, etc.

Completed in previous iterations:

✅ Phase 1-5: Core quality, transport, observability, API, protocol
✅ Phase v3: Rate limiter fix, exception narrowing, env log level

Gap Analysis 

Coverage gaps (39 uncovered lines):

transport.py:72,119,166,175-176,202,209    — metrics callbacks, state edges
exceptions.py:47-50                         — TradeError optional attrs
_market_data.py:200-202,398-399,406-407     — symbol resolution error paths
_push_handlers.py:200,283-284              — handler dispatch edges
protocol.py:246,251,256,260                 — variable-length field errors
_parsers.py:82,455-457,494                  — timestamp/tick edge cases
client.py:272,301                           — reconnection metric hooks
_logging.py:32                              — structlog import
__init__.py:10-11                           — ImportError fallback

Functional gaps:

No symbol cache TTL — stale data risk in long-running sessions
No typed push handler callbacks — raw dicts instead of dataclasses
No connection health monitoring — no latency or heartbeat metrics
No callback error isolation — one bad callback kills the connection
_market_data.py at 845 lines (over 800-line guideline)

Infrastructure gaps:

No integration test framework — only offline unit tests
No performance benchmarks — no regression detection
No fuzz testing — protocol parser not stress-tested
No PyPI release automation — manual publishing
No protocol reference docs — only code comments

Advanced feature gaps:

No protocol version tracking — hardcoded OUTER_PROTOCOL_VERSION
No order lifecycle manager — no automatic state tracking
No connection pool — single account per client

Phase 15: Coverage & Code Health (v0.9.0)

Goal: Close coverage gaps, extract oversized module, harden error paths.

15.1 Close coverage to 99.5%+

Priority: HIGH

Target the 39 uncovered lines with focused tests.

transport.py (7 lines):

Line 72: connect() when ws is not None — triggers close-then-reconnect
Lines 119, 166, 202, 209: metrics callback hooks — test with MetricsCollector mock
Lines 175-176: timeout future cleanup when future already removed from dict

exceptions.py (4 lines):

Lines 47-50: TradeError with retcode, symbol, action keyword args

_market_data.py (6 lines):

Lines 200-202: get_full_symbol_info() symbol not in cache after fetch
Lines 398-399, 406-407: secondary exception paths in symbol resolution

protocol.py (4 lines):

Lines 246, 251, 256, 260: malformed variable-length fields (short buffer)

_parsers.py (5 lines):

Line 82: timestamp coercion with zero/negative value
Lines 455-457: batch tick parsing empty/malformed input
Line 494: copy tick record edge case

_push_handlers.py (3 lines):

Line 200: dispatch with no registered handlers
Lines 283-284: book push handler with empty book data

client.py (2 lines):

Lines 272, 301: reconnection success/failure metric callbacks

_logging.py (1 line):

Line 32: structlog import available path

__init__.py (2 lines):

Lines 10-11: importlib.metadata fallback

15.2 Extract currency conversion mixin 

Priority: MEDIUM

Move currency conversion methods from _market_data.py (845 lines) to _currency.py:

currency_rate_get()
_resolve_conversion_rates()
_calc_profit_raw()
_calc_margin_raw()
Related helper methods

Reduces _market_data.py from ~845 to ~695 lines (within 800-line guideline).

15.3 Symbol cache TTL 

Priority: MEDIUM

Add optional cache invalidation to _MarketDataMixin:

symbol_cache_ttl: float = 0 constructor parameter (0 = no expiry)
Store _symbols_loaded_at: float timestamp on load_symbols()
load_symbols() checks TTL: skips if not expired, reloads if expired
invalidate_symbol_cache() manual reset method
Tests for TTL logic (expired, not expired, manual invalidation)

Phase 16: Resilience & Typed Events (v0.10.0)

Goal: Production-ready reliability features and type-safe event system.

16.1 Typed push handler callbacks 

Priority: HIGH

Add typed callback registration using events.py dataclasses:

# New typed handlers (supplement existing untyped ones)
client.on_tick_event(lambda event: ...)           # event: TickEvent
client.on_book_event(lambda event: ...)           # event: BookEvent
client.on_trade_result_event(lambda event: ...)   # event: TradeResultEvent
client.on_account_event(lambda event: ...)        # event: AccountEvent

Implementation:

Add _typed_tick_handlers, _typed_book_handlers, etc. lists to _PushHandlersMixin
Create event objects from raw push data in handler dispatch
Keep existing untyped on_tick() etc. for backward compatibility
Return SubscriptionHandle from typed registrations for easy cleanup

16.2 Callback error isolation 

Priority: HIGH

Prevent a single bad callback from killing the connection:

Wrap each callback invocation in try/except in _dispatch()
Log callback errors with full traceback via get_logger()
Add on_callback_error(callback) registration for error reporting
Continue processing remaining callbacks and messages after error
Track callback error count in metrics

16.3 Connection health monitoring 

Priority: MEDIUM

Add health check mechanism to transport:

@dataclass(frozen=True, slots=True)
class HealthStatus:
    state: TransportState
    ping_latency_ms: float | None
    last_message_at: float | None
    uptime_seconds: float
    reconnect_count: int

health = await client.health_check()
client.on_health_degraded(lambda status: ...)  # latency > threshold

Implementation:

Measure ping round-trip time in heartbeat loop
Track _last_message_at timestamp in recv loop
Expose client.health_check() async method
Optional health_threshold_ms parameter (default 5000)
Emit on_health_degraded when threshold exceeded

Phase 17: Testing Infrastructure (v1.0.0)

Goal: Comprehensive testing beyond unit tests. Mark as v1.0.0 release candidate.

17.1 Integration test framework 

Priority: HIGH

Create tests/test_integration.py gated by PYMT5_INTEGRATION=1:

Bootstrap handshake test (connect, receive bootstrap, disconnect)
Login/logout cycle test
Symbol load test (verify schema parsing against live data)
Tick subscription test (subscribe, receive ≥1 tick, unsubscribe)
Heartbeat round-trip test
Order placement test (demo account, minimal lot)

Implementation:

pytest.mark.integration marker
conftest.py fixture for credentials from env vars
CI job on schedule (not per-commit), skip if env vars missing
Timeout guards on all integration tests (30s max)

17.2 Performance benchmarks 

Priority: MEDIUM

Create tests/test_benchmarks.py using pytest-benchmark:

SeriesCodec.serialize() throughput (ops/sec)
SeriesCodec.parse() throughput
pack_outer() / unpack_outer() throughput
AESCipher.encrypt() / decrypt() throughput
Rate limiter acquire latency under load
Symbol cache lookup throughput (name → ID)

Implementation:

Add pytest-benchmark to dev dependencies
Gate behind pytest.mark.benchmark marker
Store baselines in benchmarks/ directory
Compare in CI to detect regressions (±10% threshold)

17.3 Fuzz testing for protocol parser 

Priority: LOW

Create tests/test_fuzz.py using hypothesis:

Random byte sequences → parse_response_frame() should not crash
Random field values → SeriesCodec.serialize() round-trip
Malformed frames → graceful ProtocolError, no memory leaks
Truncated messages → proper error handling

Phase 18: Documentation & Release (v1.0.0)

Goal: Community-ready documentation and automated release pipeline.

18.1 Protocol reference documentation 

Priority: HIGH

Create docs/protocol_reference.rst:

Frame format specification (outer header, inner header, body)
Key exchange and AES encryption flow diagram
Session lifecycle (bootstrap → login → authenticated → close)
Command ID catalog with request/response schemas
Field type encoding reference (PROP_* types, endianness, padding)
Tick/book push notification format
Error response codes and meaning

18.2 PyPI release automation 

Priority: HIGH

Create .github/workflows/release.yml:

Trigger on git tag v*.*.*
Run full test suite (3 Python × 3 OS)
Run mypy + ruff checks
Build sdist and wheel (python -m build)
Publish to PyPI via trusted publisher (OIDC)
Create GitHub Release with auto-generated changelog
Publish docs to GitHub Pages

18.3 API reference with Sphinx autodoc 

Priority: MEDIUM

Enhance Sphinx documentation:

Autodoc for all public methods
Usage examples per category (trading, market data, account, events)
Error handling guide with exception hierarchy diagram
Configuration reference (rate_limit, reconnect, metrics, TTL)
Migration guide from v0.x to v1.0

Phase 19: Advanced Features (v1.1.0)

Goal: Production trading enhancements post-v1.0.

19.1 Protocol version tracking 

Priority: HIGH

Track MT5 server build number from bootstrap response:

Extract server build from bootstrap body in transport.py
Add server_build: int property
Log warning for unknown build versions
Expose via client.server_build

19.2 Order lifecycle manager 

Priority: MEDIUM

Create pymt5/_order_manager.py with OrderManager class:

Track pending/active orders by ID with automatic updates from trade pushes
Order lifecycle events: created → filled/partially_filled/canceled
Position aggregation per symbol
PnL tracking per position
Export position snapshot to DataFrame

19.3 Connection pool 

Priority: LOW

Support multiple concurrent MT5 connections:

pool = MT5ConnectionPool(accounts=[
    {"server": "...", "login": 123, "password": "..."},
    {"server": "...", "login": 456, "password": "..."},
])
async with pool:
    await pool.broadcast_subscribe_ticks([symbol_id])

19.4 Dev-mode schema validation 

Priority: LOW

Enable runtime validation when PYMT5_DEBUG=1:

Assert field count matches schema after parsing
Log warning on unparsed trailing bytes
Validate parsed values against expected ranges
Zero performance impact when disabled

Implementation Roadmap 

Phase	Version	Scope	Status
15	v0.9.0	Coverage 99.5%, extract currency mixin, symbol cache TTL	NEXT
16	v0.10.0	Typed events, callback isolation, health monitoring	Planned
17	v1.0.0-rc	Integration tests, benchmarks, fuzz testing	Planned
18	v1.0.0	Protocol docs, PyPI automation, API docs	Planned
19	v1.1.0	Protocol versioning, order manager, connection pool	Planned

Priority Summary 

Priority	Item	Phase
HIGH	Close coverage to 99.5%	15.1
HIGH	Typed push handler callbacks	16.1
HIGH	Callback error isolation	16.2
HIGH	Integration test framework	17.1
HIGH	Protocol reference docs	18.1
HIGH	PyPI release automation	18.2
HIGH	Protocol version tracking	19.1
MEDIUM	Extract currency mixin	15.2
MEDIUM	Symbol cache TTL	15.3
MEDIUM	Connection health monitoring	16.3
MEDIUM	Performance benchmarks	17.2
MEDIUM	API reference docs	18.3
MEDIUM	Order lifecycle manager	19.2
LOW	Fuzz testing	17.3
LOW	Connection pool	19.3
LOW	Dev-mode schema validation	19.4

Verification Checklist 

After each phase:

python -m pytest tests/ -v --tb=short — all tests pass
ruff check pymt5/ tests/ — no lint errors
python -m mypy pymt5/ --ignore-missing-imports — no type errors
cd docs && python -m sphinx . _build/html -W --keep-going — docs build
python -m pytest tests/ -v --cov=pymt5 --cov-report=term-missing — coverage target met