refactor: implement structured event models and improve trace parsing

- Add Pydantic models for syscall events (ExecveEvent, ForkEvent, CloneEvent, ConnectEvent)
- Introduce UnparsedEvent model for raw trace lines that cannot be parsed
- Update TraceReader to yield structured event models instead of raw strings
- Refactor tests to handle new event model structure
- Update documentation to reflect new event model architecture
- Improve README with examples of structured event data
- Clean up project structure and remove legacy code markers

This change improves type safety and validation of event data while maintaining
backward compatibility through the UnparsedEvent model for unparseable lines.

Author: scc-tw

README.md

@@ -66,7 +66,7 @@ This is useful for troubleshooting or understanding what data is being collected
 
 ## Data Structure
 
-Linux EDR groups execve events by process name and maintains the full command line for context:
+Linux EDR captures various syscall events and represents them using structured Pydantic models. Here's an example of an `ExecveEvent` within a report:
 
 ```json
 {
@@ -75,14 +75,30 @@ Linux EDR groups execve events by process name and maintains the full command li
   "window_end": "2023-01-01T12:00:00+00:00",
   "total": 150,
   "command_counts": {"ls": 50, "cat": 30, "bash": 70},
-  "process_events": {
-    "ls": ["ls -la /tmp", "ls /home", "ls -l /var/log"],
-    "cat": ["cat /etc/passwd", "cat /var/log/syslog"],
-    "bash": ["bash -c 'whoami'", "bash /tmp/script.sh"]
-  }
+  "events": [
+    {
+      "timestamp": "12345.67890",
+      "pid": 1001,
+      "command": "ls",
+      "args": ["-la", "/tmp"]
+    },
+    {
+      "timestamp": "12346.00000",
+      "pid": 1002,
+      "child_pid": 1003
+    },
+    {
+      "timestamp": "12347.11111",
+      "pid": 1004,
+      "fd": 3,
+      "address": "1.1.1.1:443"
+    }
+  ]
 }
 ```
 
+The previous grouping by process name (`process_events`) in Cell reports might change based on how these structured events are aggregated. The core reporting hierarchy remains.
+
 ## Configuration
 
 Linux EDR can be configured using a `config.ini` file with the following options:
@@ -229,11 +245,17 @@ linux-edr/
 │   ├── cli.py            # Typer-based CLI interface
 │   ├── app.py            # Core application logic
 │   ├── config.py         # Configuration management
-│   ├── trace.py          # Non-blocking trace reader
+│   ├── trace.py          # Non-blocking trace reader & parser
 │   ├── aggregator.py     # Thread-safe event buffering
-│   ├── summary.py        # Report generation
 │   ├── reporter.py       # OpenAI integration and output
 │   ├── report_manager.py # Hierarchical report handling
+│   ├── domain/
+│   │   └── models/
+│   │       └── events/   # Pydantic models for syscall events
+│   │           ├── base.py
+│   │           ├── execve.py
+│   │           ├── fork.py
+│   │           └── ... # other event types
 │   └── models.py         # Pydantic data models
 ├── tests/                # Comprehensive test suite
 ├── docs/                 # Documentation

docs/api/domain/models.md

@@ -6,19 +6,62 @@ The domain layer contains the core business logic and entities of the Linux EDR
 
 ### Event Models
 
-Domain models for system events captured from the Linux kernel.
+Domain models for system events captured from the Linux kernel. These models are based on Pydantic for validation and type safety.
+
+All syscall events inherit from a `BaseSyscallEvent`:
 
 ```python
-# Example usage
-from linux_edr.domain.models import Event, EventType, ProcessEvent
+from linux_edr.domain.models.events import BaseSyscallEvent
+
+# Base class structure (simplified)
+class BaseSyscallEvent(BaseModel):
+    timestamp: str
+    pid: int
+```
+
+Specific syscall events extend this base class, adding relevant fields.
+
+#### Execve Event
 
-# Create an event instance
-event = ProcessEvent(
+```python
+from linux_edr.domain.models.events import ExecveEvent
+
+# Example usage
+event = ExecveEvent(
+    timestamp="12345.67890",
     pid=1234,
     command="ls",
     args=["-la", "/home"],
-    timestamp=datetime.now()
 )
+print(event.model_dump())
+```
+
+#### Fork/Clone Events
+
+```python
+from linux_edr.domain.models.events import ForkEvent, CloneEvent
+
+# Example usage
+fork_evt = ForkEvent(timestamp="12346.00000", pid=100, child_pid=101)
+clone_evt = CloneEvent(timestamp="12347.00000", pid=200, child_pid=201, flags="CLONE_FS")
+
+print(fork_evt)
+print(clone_evt)
+```
+
+#### Connect Event
+
+```python
+from linux_edr.domain.models.events import ConnectEvent
+
+# Example usage
+connect_evt = ConnectEvent(
+    timestamp="12348.00000",
+    pid=500,
+    fd=3,
+    address="192.168.1.1:80"
+)
+print(connect_evt)
 ```
 
 ### Report Models

docs/architecture/overview.md

@@ -17,7 +17,7 @@ For more details, see the [Clean Architecture](clean-architecture.md) page.
 
 ### Domain Layer
 
-- **Models**: Defines the structure of events and reports using Pydantic, ensuring data consistency and validation.
+- **Models**: Defines the structure of events and reports using Pydantic, ensuring data consistency and validation. Includes a base `BaseSyscallEvent` and specific models for traced syscalls (e.g., `ExecveEvent`, `ForkEvent`).
 
 ### Application Layer
 
@@ -45,15 +45,16 @@ For more details, see the [Clean Architecture](clean-architecture.md) page.
 
 ## Data Flow
 
-1. The `TraceReader` continuously reads `execve` events from the kernel trace pipe.
-2. Events are passed to the `Aggregator`, which buffers them in a thread-safe manner.
-3. A background scheduler triggers the appropriate use case at the configured interval.
-4. The use case retrieves a snapshot of events from the `Aggregator`.
-5. The service creates a Level 1 `Cell` report from the event snapshot.
-6. The `Cell` is passed to the `ReportManager`.
-7. The `ReportManager` saves the `Cell` and checks if enough Cells exist to create a Level 2 `Block`. This process continues up the hierarchy (Daily, Weekly, Monthly).
-8. The `Reporter` can optionally save the initial `Cell` report to a JSON file and send it to OpenAI for analysis.
-9. Higher-level reports (Blocks, etc.) can also be configured for AI analysis via the `ReportManager` interacting with the `Reporter`.
+1. The `TraceReader` continuously reads raw event strings from the kernel trace pipe.
+2. The `TraceReader` attempts to parse known syscall event lines (e.g., execve, fork, connect) into corresponding Pydantic models (`ExecveEvent`, `ForkEvent`, etc.). Unparsed lines are yielded as raw strings.
+3. Parsed event models (or raw strings if parsing fails) are passed to the `Aggregator`, which buffers them.
+4. A background scheduler triggers the report generation process at the configured interval.
+5. The reporting process retrieves a snapshot of buffered events (now structured models) from the `Aggregator`.
+6. A Level 1 `Cell` report is created from the event snapshot.
+7. The `Cell` is passed to the `ReportManager`.
+8. The `ReportManager` saves the `Cell` and checks if enough Cells exist to create a Level 2 `Block`. This process continues up the hierarchy (Daily, Weekly, Monthly).
+9. The `Reporter` can optionally save the initial `Cell` report to a JSON file and send it to OpenAI for analysis.
+10. Higher-level reports (Blocks, etc.) can also be configured for AI analysis via the `ReportManager` interacting with the `Reporter`.
 
 ## Project Structure
 
@@ -62,6 +63,7 @@ linux-edr/
 ├── linux_edr/                   # Main source code package
 │   ├── domain/                  # Core business logic
 │   │   └── models/              # Domain entities and value objects
+│   │       └── events/          # Pydantic models for specific syscall events
 │   ├── application/             # Application-specific business rules
 │   │   ├── services/            # Stateless operations
 │   │   └── use_cases/           # Business processes
@@ -71,12 +73,12 @@ linux-edr/
 │   │   └── controllers/         # Input adapters (CLI, API controllers)
 │   ├── app.py                   # Core application logic (legacy)
 │   ├── config.py                # Configuration management (legacy)
-│   ├── trace.py                 # Non-blocking trace reader (legacy)
-│   ├── aggregator.py            # Thread-safe event buffering (legacy)
+│   ├── trace.py                 # Non-blocking trace reader & parser
+│   ├── aggregator.py            # Thread-safe event buffering
 │   ├── summary.py               # Initial report generation (legacy)
-│   ├── reporter.py              # OpenAI integration (legacy)
+│   ├── reporter.py              # OpenAI integration & report output
 │   ├── report_manager.py        # Report management (legacy)
-│   ├── models.py                # Pydantic data models (legacy)
+│   ├── models.py                # Pydantic data models (legacy - to be removed/merged)
 │   └── cli.py                   # CLI interface (legacy)
 ├── tests/                       # Comprehensive test suite
 ├── docs/                        # Documentation source files

linux_edr/domain/models/events/__init__.py

@@ -3,11 +3,13 @@
 from .fork import ForkEvent
 from .clone import CloneEvent
 from .connect import ConnectEvent
+from .unparsed import UnparsedEvent
 
 __all__ = [
     "BaseSyscallEvent",
     "ExecveEvent",
     "ForkEvent",
     "CloneEvent",
     "ConnectEvent",
+    "UnparsedEvent",
 ] 

linux_edr/domain/models/events/unparsed.py

@@ -0,0 +1,5 @@
+from pydantic import BaseModel
+
+class UnparsedEvent(BaseModel):
+    """Represents a raw line from the trace pipe that could not be parsed."""
+    raw_line: str 

linux_edr/trace.py

@@ -20,7 +20,9 @@
 CLONE_PATTERN = re.compile(r"(\S+)\s+\[(\d+)\]\s+.*clone.*child_pid=(\d+)\s+flags=(\S+)")
 CONNECT_PATTERN = re.compile(r"(\S+)\s+\[(\d+)\]\s+.*connect.*fd=(\d+)\s+addr=(.+)")
 
-from .domain.models.events import ExecveEvent, ForkEvent, CloneEvent, ConnectEvent, BaseSyscallEvent
+from .domain.models.events import (
+    ExecveEvent, ForkEvent, CloneEvent, ConnectEvent, BaseSyscallEvent, UnparsedEvent
+)
 
 class TraceReader:
     """
@@ -156,12 +158,15 @@ def _parse_line(self, line: str) -> Optional[BaseSyscallEvent]:
 
         return None
 
-    def __iter__(self) -> Generator[Union[str, BaseSyscallEvent], None, None]:
+    def __iter__(self) -> Generator[Union[BaseSyscallEvent, UnparsedEvent], None, None]:
         """
-        Iterate over lines from the trace pipe.
+        Iterate over events from the trace pipe.
+
+        Attempts to parse known syscall events into Pydantic models.
+        If a line cannot be parsed, it yields an `UnparsedEvent` model containing the raw line.
 
         Yields:
-            Lines from the trace pipe, one at a time
+            A `BaseSyscallEvent` subclass if parsed successfully, otherwise an `UnparsedEvent`.
         """
         if self.fd is None:
             logger.error("Cannot iterate: file descriptor is not open")
@@ -207,8 +212,13 @@ def __iter__(self) -> Generator[Union[str, BaseSyscallEvent], None, None]:
                                     continue
 
                                 parsed_evt = self._parse_line(line)
-                                # Yield the parsed object if recognized, else the raw line for backward-compat.
-                                yield parsed_evt if parsed_evt else line
+                                # Yield the parsed object if recognized, else yield an UnparsedEvent
+                                if parsed_evt:
+                                    yield parsed_evt
+                                else:
+                                    # Optionally log the unparsed line here if needed
+                                    # logger.debug(f"Unparsed trace line: {line}")
+                                    yield UnparsedEvent(raw_line=line)
                         except OSError as e:
                             if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
                                 continue

tests/test_trace.py

@@ -6,6 +6,7 @@
 import errno
 from unittest.mock import patch, MagicMock, mock_open
 from linux_edr.trace import TraceReader
+from linux_edr.domain.models.events import UnparsedEvent
 
 
 class TestTraceReader(unittest.TestCase):
@@ -129,7 +130,14 @@ def test_iteration(self, mock_selector, mock_read, mock_open):
                 break
 
         # Verify lines were read correctly
-        self.assertEqual(lines, ["line1", "line2", "line3"])
+        self.assertEqual(
+            lines,
+            [
+                UnparsedEvent(raw_line="line1"),
+                UnparsedEvent(raw_line="line2"),
+                UnparsedEvent(raw_line="line3"),
+            ],
+        )
 
     @patch("os.open")
     @patch("os.read")
@@ -163,7 +171,9 @@ def test_unicode_decode_error(self, mock_selector, mock_read, mock_open):
 
         # Verify line was read and invalid characters were replaced
         self.assertEqual(len(lines), 1)
-        self.assertIn("Invalid UTF-8", lines[0])
+        # Check the raw_line attribute of the UnparsedEvent
+        self.assertIsInstance(lines[0], UnparsedEvent)
+        self.assertIn("Invalid UTF-8", lines[0].raw_line)
 
     @patch("os.open")
     @patch("os.read")
@@ -201,7 +211,7 @@ def test_eagain_handling(self, mock_selector, mock_read, mock_open):
             break
 
         # Verify line was read after EAGAIN
-        self.assertEqual(lines, ["line1"])
+        self.assertEqual(lines, [UnparsedEvent(raw_line="line1")])
 
     @patch("os.open")
     @patch("os.read")

tests/test_trace_errors.py

@@ -6,6 +6,7 @@
 from linux_edr.trace import TraceReader
 from itertools import islice
 import logging
+from linux_edr.domain.models.events import UnparsedEvent
 
 
 class TestTraceReaderErrors(unittest.TestCase):
@@ -62,7 +63,7 @@ def test_open_retry_on_ebadf(self, mock_logger, mock_selector, mock_open):
                 mock_reopen.assert_called_once()
 
                 # Verify data after reopen was read
-                self.assertEqual(lines, ["data after reopen"])
+                self.assertEqual(lines, [UnparsedEvent(raw_line="data after reopen")])
 
                 # Verify warning was logged about bad descriptor
                 mock_logger.warning.assert_any_call("Bad file descriptor, reopening trace pipe")