ARRAY DEDUPLICATION MECHANICS, SORTING ALGORITHMS & PIPELINE SYNTAX
Technical architecture for hash-set deduplication, natural sort collation, string transformations, and airgapped client privacy.
1. Computational Complexity & Hash Set Deduplication
Array deduplication evaluates item uniqueness across discrete text sequences. Traditional naive nested comparison algorithms operate in quadratic time O(n²), creating browser freezing on lists containing tens of thousands of records. This utility implements an amortized linear time O(n) deduplication pipeline powered by JavaScript's native Set collection:
- Exact Hash Identity: Values are hashed into internal V8 buckets using SameValueZero equality, stripping duplicates in a single pass while preserving initial insertion order.
- Case-Folded Normalization: When case-insensitivity is active, keys are normalized to lowercase during lookup while preserving original casing upon output emission.
- Whitespace & Boundary Trimming: Leading and trailing carriage returns (
\r), newlines (\n), and tab stops are stripped prior to comparison to prevent false negatives caused by irregular formatting.
2. Collation & Natural Numeric Sorting
Standard ASCII lexicographical sorting sorts character codes sequentially, resulting in the notorious alphanumeric ordering problem where node10 precedes node2. Our engine executes Intl.Collator with numeric: true collation, correctly evaluating multi-digit numbers embedded in text tokens:
WHERE col IN ('a','b') clauses or valid JSON array literals with one click.