3 reasons why your AWS Elasticsearch is powered up for 2019
AWS Elasticsearch Service (Amazon ES), is a fully managed service that makes it easy for you to deploy, secure, operate, and scale Elasticsearch to search, analyze, and visualize data in real-time. The service provides easy-to-use APIs and real-time analytics to power use-cases, whilst benefiting from enterprise-grade availability, scalability and security. Sporting over 20 rules to keep your Amazon Elasticsearch secure and compliant, Cloud Conformity is excited to see the continued support and development of this foundational AWS service.
Elasticsearch News 2019
Huge Cost Savings with Reserved Instances
Earlier this year, AWS introduced Reserved Instances for Amazon ES for the C4, M4, R4 and I3 instance families. There is a commitment involved of either one-year or three years, however, the flexible payment options along with varying discount rates ease this a bit — All Upfront, Partial Upfront or No Upfront. Well worth it if you know you have the hours to use!
Quicker, Easier Data Migration
Just as we come to the end of 2018, AWS brings to the field the AWS Database Migration Service (DMS) for Elasticsearch. By adopting NoSQL databases, users can now upload and retrieve bulk data more efficiently, as well as migrate any services supporting DMS to Elasticsearch. Another great benefit is that change data is replicated in near-real time with complete support for combining and mapping data in transit as needed.
No More Downtime for In-Version Upgrades
AWS asks for customer feedback and delivers updates quickly such as the new in-version upgrades. This new feature allows you to upgrade ES clusters without any downtime. No downtime! You’ll now no longer need to take manual snapshots, restore them and then update the endpoint references because Amazon will do this for you.
Safer Transport: Node-to-Node Encryption
In the fall, AWS heightened security for ES communication by supporting node-to-node encryption. Transport Layer Security (TLS) applies to all communications between ES instances in a single cluster and ensures that data sent over HTTPS stays encrypted during distribution and replication. Those ultra-sensitive, compliant heavy workloads are now much safer than before.
A couple of things to keep in check and to consider when it comes to best practice for Elasticsearch:
Sizing ES Domains
Whilst even Amazon says there is no straightforward method to this, there are a few things to consider to get the right size for your organization.
- Storage Requirements
- Will your workloads be more long-term or temporary?
- How many replicas will you be using? (Each ES has one replica, and AWS recommend at least one replica in case of data loss)
- Have you accounted for indexing overheads and the standard ES 20% reserve?
2. Number of Shards
- How many primary shards are needed for an index?
- Are the indexes evenly distributed across the nodes?
- Are you getting the right elasticache price, based on the of nodes?
- Has any individual shard exceeded the recommended 50Mb size?
3. Choosing Instance Types & Testing
- What is your estimated CPU size?
- Have you tried testing with example workloads?
- Have you tried adjusting the size and retested?
Dedicated Master Nodes
Dedicated master nodes are used for increased cluster stability and whilst performing management tasks they don’t hold or respond to data requests, Amazon recommends three master nodes for optimum security and cost efficiency.
Cloud Conformity’s Golden Top Tip
Never, ever, ever expose your ElasticSearch nodes to the Internet, deploy the service inside your VPC and control access using IAM.
Cloud Conformity currently has 21 rules for Elasticsearch and we add to this regularly. Our rules span the Well-Architected Framework Pillars of Security, Performance Efficiency, Cost Optimization, Reliability, and Operational Excellence. If you follow these best practices and make the changes in your CloudFormation templates, your ElasticSearch infrastructure will be run sweetly.