Search infrastructure that handles billions of documents, sub-second query response, and real-time log analytics. FreedomDev designs, deploys, and optimizes Elasticsearch clusters for enterprises that have outgrown basic database search — from shard architecture and mapping strategy to full ELK stack observability. 20+ years of database and infrastructure expertise, Zeeland, Michigan. Projects range from $25K to $250K+.
Elasticsearch is a distributed search and analytics engine built on Apache Lucene that powers search infrastructure for organizations including Netflix, Uber, GitHub, and Wikipedia. Elastic NV — the company behind it — carries a market cap north of $10 billion and serves over 20,000 subscription customers. The technology indexes structured and unstructured data across distributed clusters, returns full-text search results in milliseconds, and doubles as a real-time analytics engine for log data, metrics, and security events. When your PostgreSQL LIKE queries start taking seconds instead of milliseconds, when your application search returns irrelevant results because it cannot understand synonyms or typos, when your operations team drowns in logs they cannot correlate — that is when Elasticsearch becomes a necessity rather than a luxury.
Elasticsearch 8.x fundamentally changed the deployment and security model. TLS is enabled by default between nodes and clients. The Elastic Stack moved to a unified security layer that eliminates the old X-Pack licensing confusion. Vector search and kNN capabilities landed natively, making Elasticsearch a viable engine for semantic search and retrieval-augmented generation (RAG) pipelines without bolting on a separate vector database. The Elasticsearch Relevance Engine (ESRE) introduced reciprocal rank fusion for hybrid search — combining BM25 lexical scoring with vector similarity in a single query. These are not incremental patches. They represent Elastic's pivot from pure search infrastructure into an AI-era retrieval platform.
But the technology is only as good as the cluster architecture underneath it. A misconfigured Elasticsearch cluster is one of the most expensive infrastructure mistakes an engineering team can make. Shards sized above 50GB become unmergeable and degrade query performance. Mappings defined as dynamic with no explicit field types produce mapping explosions that consume heap memory. Index lifecycle management (ILM) policies that skip the warm and cold tiers waste SSD storage on data nobody queries. Cross-cluster search configured without proper remote cluster permissions opens security holes. These are not edge cases — they are the default failure modes we see in every Elasticsearch audit we perform.
FreedomDev has designed search infrastructure and database systems for over two decades. We understand Elasticsearch not as an isolated technology but as a component in a larger data architecture — sitting between your application layer and your primary database, fed by Logstash or Beats pipelines, visualized through Kibana dashboards, governed by index templates and ILM policies. We handle cluster design, shard strategy, mapping optimization, query tuning, ELK stack deployment, and the integration plumbing that connects Elasticsearch to your application. Whether you need product search that understands natural language, log analytics that correlates events across 50 microservices, or a search API that serves 10,000 queries per second, we build the infrastructure that makes it work.
Cluster architecture determines everything downstream — query latency, indexing throughput, storage cost, and failure recovery. We design clusters with explicit shard sizing strategies: primary shards capped at 50GB to maintain merge efficiency, shard count calculated against JVM heap (20 shards per GB of heap as the ceiling), and replica allocation spread across availability zones for fault tolerance. Node roles are separated — dedicated master-eligible nodes (3 minimum for split-brain prevention), dedicated data nodes tiered into hot/warm/cold for cost optimization, dedicated coordinating nodes for query routing under heavy search load, and dedicated ingest nodes when Logstash pipelines run transformations at the cluster level. We tune JVM heap to 50% of available RAM (never exceeding 31GB to stay within compressed oops), configure circuit breakers to prevent OOM crashes, and set up shard allocation awareness so your cluster survives an availability zone failure without losing data or serving stale results.
The Elastic Stack — Elasticsearch, Logstash, Kibana, and Beats — is the most widely deployed open-source observability platform in production today. We deploy full ELK stacks that ingest logs from Filebeat and Metricbeat agents across your infrastructure, transform and enrich them through Logstash pipelines with grok patterns and GeoIP lookups, index them into time-series indices with ILM policies that roll over daily, transition to warm tier after 7 days, cold tier after 30, and delete after 90. Kibana dashboards give your operations team real-time visibility into application errors, request latency percentiles, infrastructure metrics, and security events. We configure Elastic Alerts (formerly Watcher) for anomaly detection — PagerDuty when error rates spike, Slack when disk usage crosses 85%, email when a specific log pattern appears that indicates a known failure mode.
Elasticsearch is not your primary database — it is a search-optimized read layer that syncs from your source of truth. We build the integration plumbing: Change Data Capture (CDC) pipelines using Debezium or custom Logstash JDBC inputs that keep Elasticsearch indices synchronized with your PostgreSQL, MySQL, or SQL Server databases in near-real-time. Application-layer integration through the official Elasticsearch clients for Java, Python, Node.js, .NET, or PHP — with connection pooling, retry logic, bulk indexing batches (optimal at 5-15MB per bulk request), and circuit breakers that prevent Elasticsearch failures from cascading into your application. We implement search APIs with faceted filtering, autocomplete with edge n-gram tokenizers, fuzzy matching for typo tolerance, and highlighting that shows users exactly why a result matched.
Poor search relevance is almost always a mapping and analyzer problem, not an Elasticsearch limitation. We design explicit index mappings — no dynamic mapping in production — with field types chosen for their query behavior: keyword fields for exact-match filtering and aggregations, text fields with custom analyzers for full-text search, nested objects for array-of-objects that need independent querying, and flattened fields for high-cardinality dynamic metadata that would otherwise cause mapping explosions. Custom analyzers chain character filters (HTML stripping, pattern replacement), tokenizers (standard for prose, keyword for identifiers, path_hierarchy for file paths), and token filters (lowercase, synonym graphs, stemming, stop words, edge n-grams for autocomplete). We tune BM25 parameters when the default k1=1.2 and b=0.75 do not fit your content profile, implement function_score queries that blend text relevance with business signals like popularity or recency, and set up search relevance testing with rated search queries so you can measure improvements quantitatively.
Storage cost optimization through data tiering is one of the highest-ROI Elasticsearch improvements. Hot nodes use NVMe SSDs for data written and queried in the last 24-48 hours. Warm nodes use standard SSDs for data aged 2-30 days — still searchable but with relaxed latency requirements, force-merged to a single segment per shard to reduce overhead. Cold nodes use high-capacity HDDs or S3-backed searchable snapshots for data older than 30 days that must remain searchable for compliance or historical analysis. Frozen tier indices live entirely in S3 with a local cache, reducing storage cost by 90% compared to hot tier. We define ILM policies that automate rollover (by size or age), transition between tiers, force-merge warm indices, and delete expired data. For time-series data — logs, metrics, events — this architecture typically reduces Elasticsearch storage costs by 60-70% compared to keeping everything on hot-tier SSDs.
Elasticsearch 8.x enables TLS and authentication by default, but enterprises running clusters upgraded from 6.x or 7.x often have security configurations that are incomplete or misconfigured. We audit role-based access control (RBAC), configure document-level and field-level security for multi-tenant indices, set up API key management for service-to-service authentication, and integrate with your existing identity provider via SAML or OpenID Connect. For version upgrades — especially the 7.x to 8.x jump that introduces breaking changes in mapping types, security defaults, and Java API client — we run rolling upgrades with pre-upgrade deprecation audits, compatibility testing against your actual query patterns, and rollback plans at each node. For migrations from Solr, Amazon CloudSearch, or Algolia, we handle index schema translation, data migration, query DSL conversion, and performance benchmarking against your existing system.
Skip the recruiting headaches. Our experienced developers integrate with your team and deliver from day one.
Our product search was running against PostgreSQL and returning irrelevant results at 800ms per query. FreedomDev designed an Elasticsearch cluster with custom analyzers and synonym dictionaries — search latency dropped to 40ms, our conversion rate on search-initiated sessions increased 35%, and the hot-warm-cold architecture keeps our storage costs predictable as our catalog grows.
A product catalog with 500K+ SKUs where database queries cannot deliver the search experience customers expect. We index product data from your ERP or PIM into Elasticsearch with custom analyzers that handle product names, model numbers, and technical specifications. Faceted navigation (brand, price range, category, attributes) uses aggregations on keyword fields. Autocomplete suggestions use edge n-gram tokenizers that match partial input in under 50ms. Synonym dictionaries map customer language to product terminology — 'couch' finds 'sofa', 'TV' finds 'television'. Typo tolerance via fuzziness handles misspellings without returning garbage results. The search API serves results in under 100ms at 2,000+ concurrent queries per second.
An engineering team running 30-80 microservices across Kubernetes cannot debug production issues because logs are scattered across containers that restart and lose their local storage. We deploy Filebeat as a DaemonSet that ships container logs to Logstash, which enriches them with Kubernetes metadata (pod name, namespace, deployment, labels), parses structured fields from JSON logs, and routes them to date-stamped indices in Elasticsearch. Kibana dashboards show error rates by service, request latency distributions, and correlation views that trace a single request ID across all services it touched. ILM rolls indices daily, keeps 14 days searchable on hot nodes, 90 days on warm, and archives to S3 snapshots for compliance. Mean time to resolution drops from hours of SSH-ing into pods to minutes of Kibana filtering.
Organizations with large document repositories — contracts, medical records, internal knowledge bases, regulatory filings — that need full-text search across PDF, Word, and HTML content. We use the Elasticsearch ingest attachment plugin (Apache Tika) to extract text from binary documents at index time, then apply custom analyzers with domain-specific synonym dictionaries and stemming rules. Nested metadata fields enable filtering by author, department, date range, document type, and classification. Highlighting returns the exact paragraph and sentence that matched, not just a document link. For healthcare and legal, we configure field-level security so users only see documents matching their clearance level, and audit logging tracks every search query for compliance.
Elasticsearch powers Elastic Security (formerly Elastic SIEM) for organizations that need real-time threat detection without the cost of Splunk Enterprise Security. We deploy Elastic Agent across endpoints, ingest firewall logs via Syslog, pull cloud audit trails from AWS CloudTrail and Azure Activity Logs, and normalize everything into Elastic Common Schema (ECS). Detection rules run as Elasticsearch queries against incoming events — failed login brute force patterns, impossible travel anomalies, lateral movement indicators. Alerts route to your SOC team via PagerDuty or ServiceNow. Dashboards show attack surface visibility, threat hunt timelines, and compliance posture. Storage costs stay manageable through frozen-tier indices backed by S3 for the 12-month retention windows that compliance frameworks require.