At PromptCloud, we deal with data at a massive scale on a daily basis, with nearly millions of records and logs being written into Elasticsearch in almost real time. For the scalable architecture we have, this amount of data is ever increasing . We’ve had to encounter many undesirable situations in the past – unwanted behavior from data, servers going down, exhausted disk space or extreme CPU load on the servers and so on. And for a distributed system of such scale, gradually it turns out to be nearly impossible to detect any kind of anomaly and inconsistency in the data in stipulated time. We then realized that we need to add a monitoring tool to supervise the massive amount of data stored in Elasticsearch. ElastAlert was included in our technical stack as an alerting tool to handle the data monitoring part.
ElastAlert is a nifty framework used for sending out alerts on anomalies, spikes, or other patterns of interest from data stored in Elasticsearch. If your system demands writing data into Elasticsearch in near real time and you want to be alerted based on preset rules, ElastAlert can be the best option for you. It even works with all versions of Elasticsearch.
Elastalert is very easy to set up, purely event-driven, modular and a highly reliable tool. A simple demonstration on how to configure ElastAlert is given here.
In simple words, ElastAlert’s job is to search for a particular pattern from the bulk of data being written into Elasticsearch and send out alerts if it can detect such pattern. That ‘pattern’ is written by the system administrator, which is termed as ‘Rule for Elastalert’.
A simple example of Elastalert rule for service monitoring is given below:
Say we like to know all the different URIs that took more than 20 seconds to serve the request over a period of 2 weeks.
# All requests in 2 weeks with more than 20 seconds to serve requests
There could be numerous other important use cases where we can use Elastalert for custom rule types.
Several rule types with common monitoring paradigms are pre-built into ElastAlert:
If the rules are written correctly and for genuine use cases, ElastAlert’s performance is great for any kind of unstructured data.
There are a host of other features that make Elastalert more useful:
As the quantity of data increases, so does the need for automation. It is practically impossible to manually search for patterns or anomalies in large chunks of data and there’s always the risk of missing something crucial. Monitoring the data available in Elasticsearch in near real time using Elastalert is the ultimate solution to this problem.
It makes for a great way to look through certain patterns to trigger alerts, keep duplicates at bay and prevent unnecessarily triggering some heavy weight service that could eat up resources.
While Kibana is exceptionally good for querying and visualizing data, a companion tool like Elastalert makes it possible to be alerted when inconsistencies or patterns are detected in the data.