As part of another work project I needed to install some central logging to run in AWS. Simple you may say, create an instance and use Puppet to install ELK server stack on there, or go back to basics and create a rsyslog server. Normally this would be well and good, however I needed to ensure that all parts solution were highly available and fully redundant, this meant that I would have engineer a lot of HA into my ELK or rsyslog servers.
This lead me to think I could configure rsyslog to ship its logs to a RDS instancee , however this could have a high cost associated with it. My next idea was to ship the logs to S3 , this sounds simpler than it is so I quickly scrapped writing my own script and looked at Fluentd , this acts as a director for logs and seemed to tick all the boxes, the only downside was that shipping the logs to S3 wasn’t done it real time, for most of the time this shouldn’t be a problem, however the solution could have load spikes and there are some quite aggressive auto-scaling rules in place meaning so it would be possible for instances to automatically spun up and then terminated before the logs had chance to be shipped to S3.
While I was looking into a way of getting around Amazon made an announcement ;
You can now use Amazon CloudWatch to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files. You can send your existing log files to CloudWatch Logs and monitor these logs in near real-time.
Ooooooooo, thats convenient. The great thing is that it just works and its really simple to install;
The first step is add a IAM user or role with the following permissions ……
…… once you have done this bit and launched an EC2 instance with the new role or have the user credentials its just a case of running the install script ……
and thats it, if you goto your AWS Console you should now see the /var/log/messages and /var/log/secure logs being captured.
Don’t you just love it when a new service is launched at the exact time when you need it most :)
For more on service please see here and for more in-depth reading here