Monitoring and Log analysis strategies for the Dynamic world of Cloud….

Monitoring and Log analysis strategies for the Dynamic world of Cloud….

Organizations are becoming increasingly interested in leveraging cloud computing services to improve flexibility and scalability of the IT services delivered to end-users. However, organizations using cloud computing services face the following challenge: decreased visibility into the performance of services being delivered to their end-users.

Many cloud providers offer dashboards for tracking availability of their services as well as alerting capabilities for identifying service outages in a timely manner, but these capabilities are not sufficient for end-users who need to have a full control of the performance of cloud services in use. More importantly, organizations cannot rely on monitoring capabilities offered by their cloud service providers, and they need to deploy third-party solutions that allow them to monitor the performance and levels of SLA achievements of cloud services.

When software delivered, the customer is responsible for monitoring the infrastructure and the application. The customer is also responsible for capacity planning to ensure that additional infrastructure is procured and ready in time when usage reaches certain thresholds. With the cloud model, the cloud vendor must perform these tasks in real time and instantly scale the system automatically when certain thresholds are hit. The best monitoring strategy for the cloud is a proactive strategy that detects problems before they have a broad impact on the overall system and on the user experience.

There are a number of categories that should be monitored:


1. VM Availability

2. Software/Service Availability

3. Application Availability

Performance Matrices

1. Throughput

2. Response Time

 Auto scaling and Capacity Planning

User Defined metrics

Log file analysis

Cloud service/delivery Models


So Monitoring required for the three different layers of the cloud environment:

  • Application Layer
  • Service/Software Layer
  • Infrastructure Layer

I have been working in cloud environment from last 3-4 year. I have extensive experience in VM provisioning, application management and configuration management.  We have evaluated and used number of Application Monitoring solution and log monitoring solution. I am sharing some sharing my experience with these tools.

Paid Solution

Splunk (

Splunk (the product) captures indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations.

Splunk aims to make machine data accessible across an organization and identifies data patterns, provides metrics, diagnoses problems and provides intelligence for busiatness operation. Splunk is a horizontal technology used for application management, security and compliance, as well as business and web analytics.


Splunk Enterprise performs three key functions as it moves data through the data pipeline.

  1. It consumes data from files, the network, or elsewhere through forwarder.
  2. Then it indexes the data through Indexer Component.
  3. Finally, it runs interactive or scheduled searches on the indexed data through a fancy Dashboard.

Splunk Server has various applications for the supporting it monitoring and log analysis. You can use them according to yours business needs. Following are the common applications of Splunk can be used for the monitoring IAAS and PAAS cloud provider like Azure, Rackspace, AWS and Terremark Etc..


AppDynamics (

This is very good monitoring solution for the IaaS and PaaS Provider

New Relic (

This is very good monitoring solution for the IaaS and PaaS Provider

 Open Source Solution

Hyperic HQ Opensource version(

We have used hyperic as monitoring solution in our one of application it is very good in providing VM stats , VM availability and service/Software stats and software availability.

Let me first give you brief about Open source Hyperic HQ designed to provide all fundamental management and monitoring capabilities for web applications and IT infrastructures.
Key Facts about the HQ Architecture

Hyperic HQ Architechture

This diagram is a simple illustration of the key HQ components and how they fit together. The diagram doesn’t reflect a real-world deployment, as it shows only a single HQ Agent. In a typical deployment, there are many agents – one on every machine you manage with HQ.

Logstash (

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.


Nagios (



About suniluiit

Technical Architect working in Big data and Cloud technologies for the last 5 years with overall software industry experience of around 9 years. Architected and Working on Impetus Workload Migration Product which allows organizations to saves 50%-80% manual offloading time and cost. It provides faster parallel & scalable data migration to Hadoop along with incremental data options. It also maximize the existing investments in code and reuse of SQL scripts. Architected and Developed cloud agnostic application for deployment and configuration management of the enterprise application including technologies stacks like CQ5, Cassandra, Solr, Application Server, Web server, Haproxy, F5 and messaging server. Experienced in working and leading R&D teams for building new expertise in fields such as Big data, ETL offloading to Big Data and Cloud computing. Worked on some of the impetus open source product around Big Data and Social Media Specialties: Big data, Hadoop, HIve, Sqoop, Spark, J2EE/ SOA, NoSQL, Cassandra, HBase, Cloud Computing (Private/Hybrid/Public- AWS, Google, Azure, Rackspace, Openstack, VMWare, Terremark, RabbitMQ, Kafka, Memcached, Puppet, HypericHQ, Splunk etc.
This entry was posted in Cloud Computing and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s