Monitoring and Log analysis strategies for the Dynamic world of Cloud….

Monitoring and Log analysis strategies for the Dynamic world of Cloud….

Organizations are becoming increasingly interested in leveraging cloud computing services to improve flexibility and scalability of the IT services delivered to end-users. However, organizations using cloud computing services face the following challenge: decreased visibility into the performance of services being delivered to their end-users.

Many cloud providers offer dashboards for tracking availability of their services as well as alerting capabilities for identifying service outages in a timely manner, but these capabilities are not sufficient for end-users who need to have a full control of the performance of cloud services in use. More importantly, organizations cannot rely on monitoring capabilities offered by their cloud service providers, and they need to deploy third-party solutions that allow them to monitor the performance and levels of SLA achievements of cloud services.

When software delivered, the customer is responsible for monitoring the infrastructure and the application. The customer is also responsible for capacity planning to ensure that additional infrastructure is procured and ready in time when usage reaches certain thresholds. With the cloud model, the cloud vendor must perform these tasks in real time and instantly scale the system automatically when certain thresholds are hit. The best monitoring strategy for the cloud is a proactive strategy that detects problems before they have a broad impact on the overall system and on the user experience.

There are a number of categories that should be monitored:

Availability

1. VM Availability

2. Software/Service Availability

3. Application Availability

Performance Matrices

1. Throughput

2. Response Time

 Auto scaling and Capacity Planning

User Defined metrics

Log file analysis

Cloud service/delivery Models

So Monitoring required for the three different layers of the cloud environment:

  • Application Layer
  • Service/Software Layer
  • Infrastructure Layer

I have been working in cloud environment from last 3-4 year. I have extensive experience in VM provisioning, application management and configuration management.  We have evaluated and used number of Application Monitoring solution and log monitoring solution. I am sharing some sharing my experience with these tools.

Paid Solution

Splunk (http://www.splunk.com/)

Splunk (the product) captures indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations.

Splunk aims to make machine data accessible across an organization and identifies data patterns, provides metrics, diagnoses problems and provides intelligence for busiatness operation. Splunk is a horizontal technology used for application management, security and compliance, as well as business and web analytics.

Splunk Enterprise performs three key functions as it moves data through the data pipeline.

  1. It consumes data from files, the network, or elsewhere through forwarder.
  2. Then it indexes the data through Indexer Component.
  3. Finally, it runs interactive or scheduled searches on the indexed data through a fancy Dashboard.

Splunk Server has various applications for the supporting it monitoring and log analysis. You can use them according to yours business needs. Following are the common applications of Splunk can be used for the monitoring IAAS and PAAS cloud provider like Azure, Rackspace, AWS and Terremark Etc..

Splunk is a Paid Solution. If yours monitoring and logging data is huge then Splunk costlier solution.

 

AppDynamics (http://www.appdynamics.com/)

This is very good monitoring solution for the IaaS and PaaS Provider

New Relic (http://newrelic.com/)

This is very good monitoring solution for the IaaS and PaaS Provider

Open Source Solution

Hyperic HQ Opensource version(http://www.hyperic.com/hyperic-open-source-download)

We have used hyperic as monitoring solution in our one of application it is very good in providing VM stats , VM availability and service/Software stats and software availability.

Let me first give you brief about Open source Hyperic HQ designed to provide all fundamental management and monitoring capabilities for web applications and IT infrastructures.
Key Facts about the HQ Architecture

This diagram is a simple illustration of the key HQ components and how they fit together. The diagram doesn’t reflect a real-world deployment, as it shows only a single HQ Agent. In a typical deployment, there are many agents – one on every machine you manage with HQ.

Logstash (http://logstash.net/)

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Advertisements

About suniluiit

Hello I am a R&D Engineer in R&D Lab of Impetus Infotech Pvt. Ltd. I did my B.Tech From UIIT, HPU shimla.
This entry was posted in J2EE technoloy, Monitoring. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s