Filtering Architecture

Design

currentFilters is a flat object mapping column names to values:

let currentFilters = {};
// Example: { "Dest Port": "80", "Source IP": "10.0.0.1" }

Filters are global — they apply across all tabs (Alert, DNS, HTTP, All Events, etc.). When a filter is set on one tab, it persists when switching to another tab.

Search (currentSearch) is a separate server-side full-text filter using SQLite FTS5. It is an array of terms — typing tcp 80 creates two chips [🔍 "tcp" ×] [🔍 "80" ×]. Each chip is independently removable. Quoted phrases like "GET /login" become a single chip. Search terms AND together on the backend. Search complements column filters: you can search for 192.168.1.1 and then further narrow with aggregation filters. Search is cleared when loading a new PCAP or when clicking Clear All.

Scalable Fetch vs. Full-Batch Fallback

With no column filter active, the table, aggregations, and Sankey diagram all use lightweight, unbounded server-side queries - fetchEventsPage() (SQL LIMIT/OFFSET, always exactly CONFIG.TABLE_PAGE_SIZE rows per page), fetchAggregationData(), and fetchSankeyData() (both server-side GROUP BY over the full matching set). None of these are bounded by the query-limit setting described below - they work the same regardless of dataset size.

The moment a column filter is applied (applyFilter/applyFilters, which sets currentFilters[column] = value), everything falls back to full-batch client-side mode: canUseScalableFetch(), canUseServerAggregation(), and canUseServerSankey() all check Object.keys(currentFilters).length === 0 and become false. This is because arbitrary column-value filtering has no server-side WHERE-clause equivalent - only search (q=) and type= are implemented server-side. applyFilters() then calls ensureCappedBatch(eventType), which fetches up to getUserQueryLimit() raw rows (default 75,000, user-adjustable up to the server ceiling MAX_QUERY_LIMIT = 100,000 - see GET /api/limits in api.md) into tabDataCache/allEvents, and filtering, sorting, table pagination, and (for log/sigmaalert/binary, which have no server-side aggregation at all) aggregation all happen in JS over that in-memory array.

log/sigmaalert/binary event types always need the full batch for their aggregation view regardless of filters, since their columns are dynamic/data-dependent or (for binary) have no scalable path at all - see needsFullBatch().

If the true match count exceeds what was fetched, ensureCappedBatch() marks the type in truncatedTypes, and isCurrentTabTruncated() drives a visible warning in the filter bar - filtered/aggregated results are never silently incomplete without an indicator, but they can still be a partial view of a very large, heavily-filtered dataset.

Key Functions

Function	Role
`performSearch()`	Splits `searchInput` into terms, pushes to `currentSearch` array, clears input, refreshes data
`clearSearchTerm(index)`	Removes single term at index from `currentSearch`, refreshes data
`applyFilter(sectionId, columnName, value)`	Sets `currentFilters[columnName] = value`, rebuilds table + agg grid + stats + sankey
`clearFilter(columnName)`	Deletes `currentFilters[columnName]`, rebuilds
`clearAllFilters()`	Clears both `currentSearch` and `currentFilters`, refreshes data from server
`getFilteredEvents(sectionId, events, eventType)`	Returns events matching all active column filters
`buildSection(eventType, events)`	Builds the main data table, applies `currentFilters` internally
`buildAggregationsSection(eventType, events)`	Builds agg grid; expects pre-filtered events
`buildAllEvents()`	Builds "All Events" table, applies `currentFilters` internally
`buildAggregationsSectionAll()`	Builds agg grid for "All Events"; applies `currentFilters` internally
`updateFilterBarVisibility()`	Shows/hides the filter bar when search or filters exist
`buildStats(filteredStats)`	Rebuilds stats cards with filtered/total counts when constraints are active
`canUseScalableFetch()` / `canUseServerAggregation(eventType)` / `canUseServerSankey(eventType)`	`true` when no column filter is active (and, for scalable fetch, no client-only sort) - gates server-side pagination/aggregation/Sankey vs. the full-batch fallback
`ensureCappedBatch(eventType)`	Fetches up to `getUserQueryLimit()` raw rows into `tabDataCache`/`allEvents`; no-op if already cached
`needsFullBatch(eventType)`	`true` when the aggregation view or Sankey diagram can't use its server-side fetch (log/sigmaalert always; any type when a column filter is active)
`getUserQueryLimit()`	Reads/validates the user's configured full-batch row cap from `localStorage`, falling back to `CONFIG.DEFAULT_QUERY_LIMIT`
`computeFilteredStats()`	Counts events by type from the filtered subset
`getSankeyEvents()`	Returns filtered events for the currently visible tab for the Sankey diagram
`refreshCurrentView()`	Rebuilds the current tab's table, aggregations, stats, filter bar, and Sankey diagram
`updateSankeyDiagram()`	Rebuilds the Sankey diagram using events from `getSankeyEvents()`

Data Flow

buildSection / buildAllEvents apply currentFilters internally
Filter bar is rendered by updateFilterBarVisibility() into #filterBarContainer
buildAggregationsSection expects pre-filtered events from the caller
loadTabData must call getFilteredEvents() before passing data to buildAggregationsSection
buildAggregationsSectionAll applies currentFilters internally (exception to the pre-filtered rule)

Column Overlap

Per-event-type columns differ from "All Events" columns:

Shared columns	Per-type only	All-events only
Time, Protocol, Source IP, Source Port, Dest IP, Dest Port	Everything else `getColumnsForType()` returns per event type - see below	Type (event type itself, e.g. `"DNS"`/`"ANOMALY"`), Detail

Per-type columns are not enumerated here, including their count. Every event type has its own real per-type columns (getColumnsForType() in static/socrates.js) - e.g. Alert/Category/Severity for alert, Query/Type for dns, JA3/JA3S for quic, Opcode/Fin/Payload for websocket, Operation/Message ID/Result Code for ldap. A prior version of this doc tried to keep a hardcoded copy of that list here (and later, just a hardcoded event-type/column count as a compromise); both went stale the same way, every time a new event type was added - even the count needed updating far more often than anyone remembered to. Treat getColumnsForType() as the only source of truth for what columns exist per type, including how many there are - don't re-add a number here either.

Two columns mean something different depending on context, and that distinction genuinely belongs in this doc (it's the one thing a fresh read of getColumnsForType() won't tell you by itself): - Type: on a per-type tab it's protocol-specific (a DNS/mdns record type, dnp3's own type field, anomaly's type field, ...). In "All Events" it means the event_type itself, uppercased (e.g. "HTTP"). extractAllValue() special-cases 'Type' for exactly this reason. - Detail: only exists in "All Events" - a one-line, event-specific summary (e.g. the DNS query name, the alert signature, the ldap operation). Each event type's Detail rendering lives in extractValue()'s shared 'Detail' case.

When filtering, both extractValue() and extractAllValue() handle every per-type column. extractAllValue() delegates everything except 'Type' straight to extractValue(), so it automatically supports the full, current set of per-type columns without needing its own copy of the list - this is also why a filter set on one tab (e.g., Alert on the Alerts tab) correctly matches events when switching to "All Events". Only truly unknown columns return ''.

REGRESSION, fixed: extractAllValue() used to also special-case 'Command' (hardcoded to e.ftp?.command, which went stale the moment pgsql/enip/pop3 gained their own real command fields - it silently returned '' for all three in the "All Events" view) and 'Message' (read e.anomaly?.message, a field that has never existed in Suricata's eve.json anomaly schema - the real field is event, and no column has been labeled 'Message' since anomaly gained real columns). Both overrides were removed; extractValue()'s own per-protocol handling was already correct.

Known Gotchas

1. `currentFilters` must stay flat

Nesting it as {sectionId: {columnName: value}} causes filters to disappear when switching tabs because each tab creates a new empty section entry. Tests enforce this: - test_currentFilters_is_flat_object_not_nested - test_all_filtering_functions_use_global_currentFilters

2. `buildAggregationsSection` expects pre-filtered events

When calling buildAggregationsSection(eventType, events) in aggregation mode, events must already be filtered through getFilteredEvents(). Passing raw events will show incorrect aggregation counts. Tests enforce this: - test_loadTabData_filters_agg_tables_in_advanced_mode_cached - test_loadTabData_filters_agg_tables_in_advanced_mode_fresh

3. Aggregation toggle must filter before building agg tables

When the user toggles Aggregation ON, the handler must call getFilteredEvents() before buildAggregationsSection(). See the advancedToggle change listener.

4. onclick quoting

Never use JSON.stringify() inside onclick attributes — it produces double-quoted strings that break inside double-quoted HTML attributes. Use single-quoted template expressions with escaped internal single quotes. Tests enforce this: - test_no_json_stringify_in_apply_filter_onclick - test_no_json_stringify_in_clear_filter_onclick - test_no_bare_json_stringify_in_onclick_templates

Running Tests

python3 -m unittest discover tests -v

All tests must pass. The filtering-related tests are in:

TestFilterOnclickQuoting (6 tests)
TestAdvancedModeFilterBar (25 tests)
TestSearchUI (30 tests)