Let’s understand Big Data

Of late, I have been working in the field of data science. I thought I would share with you all what I have been learning. I will start with the term Big Data.

What's Big Data?

During the last few years, you have been hearing everyone saying “Big Data”. It has become a buzz word in most of the discussions. You might be thinking what is this Big Data about.

• ·       Is it Huge data?
• ·       Is it Large Amounts of Data?
• ·       Is it meant for Big People?
• ·       What is it all about?
• ·       And, more importantly, how to identify the big data, and what is the use of it.

Today, I want to put the concept in clear and simple terms so that you can understand what it is and identify by yourself what the big data is in any context.

Let us define Big Data formally first –

Big Data is massive unstructured data that has variety, velocity, volume, and veracity, which can be used in decision making.

Massive Unstructured Data

Of late, lots and lots of Data is becoming available from various sources, in different formats.

Variety

When we say variety, we mean the different types of data sources that are becoming available – text, audio, video, click streams, log files and more.

Velocity

Data is frequently time-sensitive and must be used simultaneously with its stream into the enterprise to maximize its value.

Volume

Terabytes and even petabytes of information is becoming available.  It calls for scalable storage and a distributed approach to querying.  

Veracity

With information flowing in from so many different sources, at such a speed and in huge quantities that need to be used immediately, the important questions are - How much reliable the data is? How accurate the data is?

Data can be ambiguous, inconsistent, spam, that needs to be filtered out before taking major decisions.

Decision Making based on Data

The importance of data in decision-making is recognized worldwide. And, data is flowing in big volumes, with high velocity, from various sources of different types, bringing uncertainty alongside.  

Data Scientists – The most sought after people in the corporate

The task is

• ·       to identify what data is relevant,
• ·       what are the sources available,
• ·       how to gather the data as it flows in,
• ·       how to clean it, and shape it,
• ·       how to model it to draw data insights,
• ·       how to report the findings

Each of these is a big topic on its own.

Now, I think you can appreciate the need for data scientists in the corporates and how interesting and promising the career opportunity is.

Learn to Write DAX Online Training by Matt Allington

Learn to Write DAX Online is a combination of self-paced remote learning using the book “Learn to Write DAX“ by Matt Allington, weekly video based training with examples and demos to support the complex topics in the book, and weekly live screen sharing Q&A sessions with Matt Allington.

This training course has been specifically designed to provide the following benefits.

• The training is run in semesters so that you can be part of a class of people that are all learning together.
• You get to learn incrementally over a 5 week period in your own time.  Just find a couple of hours each week when you can spare the time to focus.
• There is a weekly video where Matt Allington explains the concepts that are harder to learn yourself.
• You read the chapters assigned for each week, do the exercises in those chapters and then watch the video to summarise the week.
• You can watch the weekly videos at any time in the week that suits you.
• You get direct access to Matt Allington to ask for clarification to any questions you have during the weekly Q&A session.  You can ask any questions you have at these sessions.
• You can purchase the course and start-off on the course at any time, and you will get access to the lessons and weekly videos immediately. However, you might have to wait for Q&A sessions until the next semester.
• You can choose the appropriate session times for the weekly Q&A sessions that suit your time zone.
• You will have access to the online course material (videos) for 12 months, so if you want to come back and watch them again then that is fine.  Also, you will receive the Q&A session recording through email and you can keep a copy for future reference.
• The cost is very affordable compared with a comparable live training course (about 80% cheaper)

If you are interested in enrolling for the course, please leave your contact details in the comments.

Financial Risks

Hi Friends,

Let’s recap the definition of Risk that I had given in my Blog Post – Risk Management.

“A Risk is any event that if happens can have a significant influence on execution”.

·         if happens – amounts to the probability of happening – meaning it may happen or may not happen, but there is a likelihood of happening.

·         Influence – amounts to impact on the execution.

Risk – An example –

In this blog, I will discuss Financial Risks.

What is a Financial Risk?

A financial risk is any risk associated with financing, with the potential for financial loss and uncertainty about its extent.

The term financial risk by itself is broad but can be understood better if we consider the different types of financial risks.

Types of Financial Risks

Financial risks can be categorized as follows –

Asset-backed Risk: The changes in one or more assets that support an asset-backed security will significantly impact the value of the supported security. For e.g. home loans. In order to finance home sales, banks issue bonds that serve as a debt obligation to its buyer. The buyer of the debt is essentially receiving the interest from the bank that the homebuyer is paying to it.

Credit Risk: A credit risk is the risk of default on a debt that may arise from a borrower failing to make required payments.

Liquidity Risk: The risk that a given security or asset cannot be traded quickly enough in the market to prevent a loss or make the required profit. 

Market Risk: Market risk is the risk of losses in positions arising from movements in market prices.

Operational Risk: The Basel II Committee defines operational risk as: "The risk of loss resulting from inadequate or failed internal processes, people, and systems or from external events."

Financial Risks – Sub-categories

The risk categories, in turn, have several sub-categories, as depicted in the picture below.

Asset-backed risks include interest rate, term modification, and prepayment risk.

*Prepayment risk is the risk that the buyer goes ahead and pays off the mortgage. Therefore, the buyer of the bond loses the right to the buyer's interest payments over time.

*Interest rate risk refers an asset whose terms can change over time, such as a Variable Rate Mortgage payment.

A Credit risk can be of the following types:

*Credit default risk – The risk of loss arising from a debtor being unlikely to pay the loan obligations in full or the debtor is more than 90 days past due on any material credit obligation. Default risk may impact all credit-sensitive transactions, including loans, securities, and derivatives.

*Concentration risk – The risk associated with any single exposure or group of exposures with the potential to produce large enough losses to threaten a bank's core operations. It may arise in the form of single name concentration or industry concentration.

*Country risk – The risk of loss arising from a sovereign state freezing foreign currency payments (transfer/conversion risk) or when it defaults on its obligations (sovereign risk). This type of risk is prominently associated with the country's macroeconomic performance and its political stability.

Liquidity risk can be of the following:

*Asset liquidity - An asset cannot be sold due to lack of liquidity in the market - essentially a sub-set of market risk. 

*Funding liquidity - Risk that liabilities:

- Cannot be met when they fall due

- Can only be met at an uneconomic price

- Can be name-specific or system

The most commonly used types of Market risk are:

*Equity risk - the risk that stock or stock indices prices or their implied volatility will change.

*Interest rate risk - the risk that interest rates or their implied volatility will change.

*Currency risk - the risk that foreign exchange rates or their implied volatility will change.

*Commodity risk - the risk that commodity prices or their implied volatility will change.

Official Basel II types of Operational risks are the following:

*Internal Fraud – misappropriation of assets, tax evasion, intentional mismarking of positions, bribery.

*External Fraud – theft of information, hacking damage, third-party theft, and forgery

*Employment Practices and Workplace Safety – discrimination, workers’ compensation, employee health, and safety.

*Clients, Products, and Business Practice – market manipulation, antitrust, improper trade, product defects, fiduciary breaches, account churning.

*Damage to Physical Assets – natural disasters, terrorism, vandalism.

*Business Disruption and Systems Failures – utility disruptions, software failures, hardware failures.

*Execution, Delivery, and Process Management – data entry errors, accounting errors, failed mandatory reporting, negligent loss of client assets.

Microsoft Excel – Spreadsheet to Business Intelligence

Hi Friends, we have been using Microsoft Office for our different purposes both at home and office. Though Excel made its entry as a substitute to the legacy spreadsheet applications, such as Lotus 1-2-3, it went a long way with versatility and user-friendliness enabling higher up data analysis.

At its core, Excel is widely used for data entry. From the novice users to Excel Pros, there is a large toolset that is available in the current versions of Excel.

Excel – Primary Data Tools

Let us start examining the basic tools that Excel provided for data processing –

• Data Ranges
• Data Tables
• Charts

You can enter data in rows and columns in a worksheet in an Excel workbook. You can either work on that data, considering it as a data range or convert it to a table for more sophisticated operations.

You can visualize the data patterns using Charts, and Excel has a Recommended Charts option that suggests you the appropriate chart types based on your data.

Next Level of Tools

When you have large data sets, you might want to aggregate and summarize the data. For this purpose, Excel introduced -

• PivotTables
• Pivot Charts

PivotTables, help you examine the different facets of the data and generate specific reports. Further, Excel allows you to change the PivotTables dynamically that facilitates portraying the data results during presentations and changing them on the fly to answer the questions that are raised.

Pivot Charts is a cousin of PivotTables and portrays the data in chart forms instead of tables.

VLOOKUP and HLOOKUP

Excel has 500+ built-in functions that enable you to perform the required operations on the raw data to produce the desired results. Of these, the LOOKUP functions got a wide usage. If your data is in two tables, and if you need to combine the data from both the tables, VLOOKUP function comes handy. For e.g. if you have a Products table and a Sales table, you can obtain the sales data for a particular product with VLOOKUP.

Though VLOOKUP was the most sought after Excel function, it has its drawbacks, which have been solved with the introduction of other tools in Excel.

Data Analysis Tools in Excel

This is where Excel began its superior performance. Excel crossed its spreadsheet boundaries by a realm of data analysis add-ins. You can find the following data analysis tools handy if your data sets are not too huge.

• What-if Analysis
• Forecasting (predicting data trends)
• Analysis Toolpak (for statistical analysis)
• Solver (for optimization and equation solving)

Big Data Evolution

Excel Pros were quite satisfied with the data analysis tools that are available, restraining to single tables of data, spanning around some hundreds of rows.

Then, Big Data has become a buzz word in the industry and anything and everything related to data started being called Big Data.

Let us pause here to understand the background of this commotion. The data sources have become vast – from a simple text message to databases containing millions of rows of data. Data is available through the web. The top managers of the companies have realized that if their decision making is based on this availability of data from various sources, not only the decisions would be fruitful, but also, they can have purview of data trends so that they can take immediate actions when necessary. This has raised the expectations of the management from the data analysts.

Data Science is the most highly paid domain now and several tools catering the needs of data analysts came into existence. But, if we observe closely, most of these tools are more into reporting with various visualizations and dashboards.

Yes, reporting is one of the tasks of a data analyst. But, before reporting, the data is to be analyzed and the key data insights are to be brought out so as to enable the decision makers focus on the appropriate and relevant information. Further, as the data is obtained from several sources, the data in its raw format might require cleaning and shaping before it is subjected to analysis.

So, the order is – Data Cleaning, Shaping, Analysis, and Reporting. This is where Microsoft paved its way through Excel.

Excel Power Tools

Microsoft introduced the Power Tools to handle the data analysis for decision making.

• Power Pivot
• Power Query
• Power View
• Power Map

Microsoft also came up with a standalone tool for Business Intelligence – Power BI.

I will cover more details on Power Tools in the next blog.

Agile Methodologies – Extreme Programming + Scrum – A Current Trend

Readers, of late Agile Software Development is gaining momentum with many organizations adapting the agile practices as made suitable to their requirements. Several Agile methodologies came into existence with each one focusing on a strategy that helps the organizations meet their objectives. Hence, the choice of an agile methodology should be at par with the organization’s goals. I will discuss the latest Scrum + Extreme Programming hybrid trend in this post.

Extreme Programming

Extreme Programming (XP) was developed by Kent Beck focusing on using the industry best practices and taking them to extreme. Extreme Programming is one of the earliest Agile Methodologies that is continuously evolving. Extreme Programming provides a flexible framework that can be fine-tuned to a specific methodology.

Extreme Programming Practices

Kent Beck defined the following Extreme Programming Practices in his Book – Extreme Programming Explained.

·         The Planning Game

·         Short Releases

·         Metaphor

·         Simple Design

·         Testing

·         Refactoring

·         Pair Programming

·         Collective Ownership

·         Continuous Integration

·         40 hour Week

·         On-site Customer

·         Coding Standards

These practices are interdependent and achieve the Extreme Programming objective – is any of the practices is weak, the strength of the other practices makes up for it.

Evolving Extreme Programming Practices

The Extreme Programming practices are continuously evolving and are found to be effective in the other agile methodologies also. Some examples of the evolved practices are the following:

·         On-Site Customer is evolving into Whole Team

·         The Planning Game is evolving into Release Planning and Iteration Planning

·         Testing is evolving into Acceptance Testing, Unit Testing and Test-Driven Development

·         Refactoring is evolving into Design Improvement

·         40-Hour Week is evolving into Sustainable Pace

Remember that any of the Extreme Programming Practices that you choose should be implemented be implemented without compromising on their values. Otherwise, you cannot claim that you are using Extreme Programming.

The recent trend in Extreme Programming is in the use is Scrum + Extreme Programming Hybrid.

Scrum

Scrum is a popular and widely used agile methodology. In Scrum, the development work is broken down into Releases that are accomplished in time-boxed short iterations called Sprints. In every Sprint, only the required and sufficient functionality that is prioritized by the Customer that you can deliver by the end of Sprint is taken up and is termed as Backlog Items. The developers and the testers are involved in every activity from the beginning to the end. Acceptance Criteria is defined as agreed with the customer at the beginning of the Sprint itself and continuous testing will be adapted. A working product increment is released at the end of every sprint.

Scrum, as with the case of any development methodology, is effective in certain situations, but has its own shortcomings.

·         The time-boxed Sprints will not allow any flexibility in the Release schedule that hampers both development and testing

·         Scrum, on its own do not give directions for development, and assumes that the agile manifesto is truly followed.

Hence, Scrum is usually combined with other Agile Methodologies that focus more on the Development Strategies.

Scrum + Extreme Programming Hybrid

Scrum is found to be more effective if incorporated with Extreme Programming Practices that are complimentary in nature. While Scrum focuses on the fixed scope for Sprints, Burn-Down charts, etc., Extreme Programming focuses on the aspects such as continuous communication, frequent feedback loops, refactoring, collective ownership, continuous integration, Test-Driven Development, etc. This hybrid is producing anticipated results, as

·         Scrum being a defined methodology, can be adapted from day-one of the project

·         Extreme Programming focusing on communication and team cohesion, enables the team to get more focused on the development

Scrum + Extreme Programming Hybrid Tools

SpiraTeam Tool and Rapise Tool can be used for Scrum + Extreme Programming hybrid projects.

For details, refer:

https://www.inflectra.com/SpiraTeam/

https://www.inflectra.com/Rapise/

Project Monitoring and Tracking

Readers, the core of any project is execution. A successful project is one that is executed within the stimulated time, within the budget and with the expected quality. Hence, project needs to be monitored on these aspects continuously and corrective actions need to be taken immediately. This would require gathering the following information periodically –

·         Scheduled Work vs. Completed Work

·         Time Elapsed vs. Remaining Time

·         Budget Expended vs. Remaining Budget

·         Resources Planned vs. Resources Availability

·         Estimated Effort vs. Actual Effort

·         Metrics on required parameters, such as Defects

There is a subtle difference between project monitoring and project tracking. Project tracking involves gathering the above information from time to time and taking actions accordingly. For e.g. if there is an expected schedule slippage, put more resources to complete that work on time. If any planned resource is unavailable due to unforeseen conditions, arrange for an alternative resource.

On the other hand, project monitoring involves holistic approach for project management. It involves the following activities –

·         Communication with the customer and stakeholders to discuss the project status and obtaining timely feedback

·         Managing any changes that occur in the work that is being executed

·         Identify, assess and manage the project risks

·         Ensure customer satisfaction throughout the project

·         Analyze the performance of resources

·         Arrange training and/or mentoring if and when necessary

·         Focus on team cohesion by team building, team motivation and team collaboration

·         Monitor quality aspects of the work

·         Take timely decisions as and when required.

Maintaining a project dashboard visible to the entire team is a good practice. Similarly, mail communication and periodic status reports to customer will enhance transparency.

Risk Management

Introduction

As promised in my earlier post, I will discuss in detail about Risk Management in this post. Let me start defining what we mean by a Risk –

“A Risk is any event that if happens can have a significant influence on execution”.

•         if happens – amounts to probability of happening – meaning it may happen or may not happen, but there is a likelihood of happening.
•         Influence – amounts to impact on the execution.

Thus, a Risk has two parameters – Probability and Impact.

A note here – by CMMI Institute’s definition of a Risk, Impact mentioned is negative. While, by PMI definition of a Risk, Impact can be either negative or positive.

In this blog, I go by CMMI Institute’s definition that if there is a Risk, it would mean that it can have negative impact if it occurs. This is because of two reasons –

1.   World renowned CMMI Lead Appraisers abide by this definition.

2.   Several organizations that I guided for quality implementation and sustenance, successfully managed their projects with Risk Management following this Risk definition.

Risk – An example –

With this introduction, we will get into the Risk Management activities.

Risk Management Activities

Risk Management involves the following activities –

1.   Identifying Risks

2.   Assessing Risks

3.   Planning Risk Mitigation

4.   Planning Risk Contingency

5.   Monitoring Risks

1.   Identifying Risks

This is the most important and necessary activity in Risk Management. And it is an ongoing activity and not a one-time activity. When you identify Risks, the statement of Risk as a likelihood event should be recorded. For e.g.

•         The incessant rains may damage crops.
•          The peak rainfall may cause a flood.

These are valid statements of Risks, whereas the following are not.

•          The incessant rains damaged crops.
•          The peak rainfall will cause a flood.

It is recommended to have a Risk database wherein you have a record of earlier Risks and unforeseen events. This would help you as a ready reckoner for identifying the Risks in a particular field.

It is important to record the identified Risks, preferably in a tracker to facilitate monitoring.

2.   Assessing Risks

Once a Risk is identified, it needs to be assessed.  You need to find the two parameters of the Risk - probability of occurrence of the Risk and the extent of the impact if the Risk occurs.

I suggested and found practical, the following quantification of the Risks –

•        Probability – An integer in the Scale of 1-5 (1-Lowest, 5-Highest)
•        Impact - An integer in the Scale of 1-5 (1-Lowest, 5-Highest)

A third parameter, Risk Exposure is obtained from these two –

Risk Exposure = Probability*Impact

Thus, Risk Exposure would be an integer in the Scale of 1-25.

The next step, would be to compare the Risk Exposure to Risk Exposure Threshold.

The Risk Exposure Threshold can be defined at an organization level, a business unit level or a project level.

•          If Risk Exposure of a Risk is more than the Risk Threshold, then you need to plan Risk Mitigation and Risk Contingency – steps 3 and 4 given below.
•          Otherwise, proceed to step 5, skipping steps 3 and 4.

This comparison is required because, it is not cost effective to plan mitigation/ contingency for every Risk identified, unless it is of significant magnitude.

3.   Planning Risk Mitigation

A plan to mitigate a Risk aims at decreasing the probability of occurrence of the Risk.

E.g. Mitigation Plan for the Risk given in the example above.

4.   Planning Risk Contingency

A contingency plan for a Risk aims at decreasing the impact of the Risk if it occurs.

E.g. Contingency Plan for the Risk given in the example above.

5.   Monitoring Risks

Risk monitoring involves the following tasks –

•          Tracking the status (open/closed) of the Risks. A Risk that is no more valid or a Risk that already happened and will not repeat in near future can be closed
•          Noticing symptoms for occurrence of the identified Risks so that action can be taken at appropriate time. Everyone involved in the work can contribute to this task as anyone can get feelers about the symptoms. Hence, it is a good practice to discuss Risks in group meetings
•          Execution of Risk mitigation plans when required
•          Execution of Risk contingency plans when required
•          Identifying Risks from time to time
•          Contributing identified Risks to the Risk database

Change Management

Readers, following my earlier post on the comparison between Change Management and Risk Management, received requests on more about these two Project Management disciplines. I will discuss Change Management in this post and Risk Management in a subsequent one.

Change Management as the name suggests is for managing changes. But, what changes? The changes that are considered for Change Management can be broadly classified as –

• Technology changes
• Domain changes
• Resource changes
• Location changes
• Sub-contractor changes
• Strategy changes

Steps in Change Management

1.   Identify the dependencies of the change and the dependencies on change. This is to quantify the change.

2.   Assess the impact of the change on the current scenario – this could be positive or negative immediately, but positive in the long run.

3.   If the change is regarding a product that is in market, conduct a customer survey to know the likely response to the change.

4.   Identify any risks that are foreseen for implementing the change.

5.   Estimate the duration and budget for change.

6.   Estimate the resource requirements for the change.

7.   Enquire on the availability of the resources.

8.   Take a decision on implementing the change.

•  The change can be accepted for immediate implementation.
• The change can be accepted but for a delayed implementation.
• The change can be put on hold for want of more information or for a clarity on the situation.
• The change can be rejected if it is not viable due to budget constraints or resource crunch or any other reasons.

Change Management Board

In an organization, the change management is normally taken up by a change control board comprising of the field personnel who would be accountable for the change, top management who can make business decisions, subject matter experts if the change is regarding technology and/or domain.

Based on the type of change and magnitude of change, change control board might have to meet once or several times and ensure that a decision on change is taken as soon as possible.