What is "Big Data"?

I get asked this often, usually by management level types who have heard it as a buzz word and really want to understand how it affects their business. Here's how I broke it down to my 14 year old.

If you look at the press (news, TV, commentators) big press means anything that has to do with the words, pictures and connections that people put on "social sites".

In the business world Big Data means ... well ... a lot of data. And I don't mean a little, I mean a ton. Think on the order of Billions of rows of data. A good rule of thumb would be any data that is over a Tera-byte in size. And that's the minimum.

The third thing "Big Data" could refer to is volume speed. It might not be huge in size, but how fast does the data get generated.

For example, let's say you record the date and time for any phone ring that happens within the United States in the next 30 min. And you had to pinpoint the busiest phone out of all the phones in the US. The data volume size is small. But the volume of data that comes in is crazy, your talking 2 million data points a second. And your "Big Data Solution" has to make some decision on that data within 60 seconds.

This is really the "new" and revolutionary part of Big Data as only recently (in the last 5 years) has the average computing power been strong enough to handle this volume at that speed.

The other reason "Big Data" is considered as new is because businesses have started to take new approaches to it. The goal is to figure out how all this data can be used to "hit" the bottom line.

Personal Opinion: 
The challenge that I see with Big Data is letting your solutions be driven by a measurable ROI or a measurable impact to customer.

The "Big Data" solutions that succeed are those that start with one of those two objectives in mind and move from there.

Nebulously built "repositories"/"reporting ware houses"/"business data troughs" don't really thrive in the long run. (Notice I said thrive, they serve a purpose and can be of value, but they don't generate that spark that makes the C-Level executives go nuts.)


So to put it concisely

1. Big Data is not social data
2. Big Data can refer to physical data size
3. Big Data can refer to the speed of incoming data
4. The goal is to use Big Data to affect the bottom line

Check my Facts:
IBM Article - Analytics-The real-world use of big data

On Leadership

Great Article - http://www.forbes.com/sites/mikemyatt/2013/07/01/the-4-pillars-of-stable-leadership/

If I might summarize...

True North: Stable leaders have an open mind, but they also have strong convictions and principles. While stable leaders listen to others, they are not prone to being wishy-washy. Their values drive their actions – not the court of public opinion. You might not always agree with stable leaders, but you’ll never have any doubt as to where they stand. An aligned vision based upon clearly stated values, and the character to hold people accountable to values over outcomes create a high-trust culture. Purpose and people matter more than process and short-term results.

You Play How You Practice: Whether it’s Nik Wallenda, a CEO, an educator, politician, or anyone else, your performance is always tied to your preparation. Training, development, and continuous life-long learning are the foundational cornerstones of stable leadership.

Lead With Compassion: The most stable leaders understand their success is rooted in the care and well-being of those they lead. Stable leaders have a natural bias toward empathetic and compassionate behavior. When those you lead know you care, it creates a sense of trust and stability not found in more mercenary and callous leaders.

Freedom To Fail: If the people you lead are afraid to make mistakes you’ll never see their best work – you will have led them to perpetual state of mediocrity. Smart leaders make it safe for people to think big, take risks, and try new and different things. Nothing creates stability more than a high-trust environment where people are rewarded for the right behaviors – not punished for them.

Let's Talk Salary

If you are in the data space, before you read the following, decide what percent of the market you fall in, in terms of performance. You in the top 10%, middle or just starting out.

OK, read on.


Management analysts use data to improve an organization’s profits and efficiency.

• Mean: $42.30/hr or $87,980
• Bottom 10%: $21.12/hr or $43,920
• Top 10%: $68.16 or $141,780
• Highest Sector: Petroleum and coal products

Financial analysts determine the best investments to sell or buy by evaluating economic data and business trends.

• Mean: $42.18 or $87,740
• Bottom 10%: $22.26 or $46,300
• Top 10%: $69.99 or $145,850
• Highest Paying Sector: Public Relations

Computer systems analysts study the needs of an organization so they can recommend hardware and software

• Mean: $39.58 or $82,230
• Bottom 10%: $23.74 or $49,370
• Top 10%: $57.72 or $120,060
• Highest Paying Sector: Animal Slaughtering (u read that right)

My Personal Take:

Bottom 10% all make nearly the same amount, but no one wants to be that guy. Financial Analysts have the potential to make the most money.

I don't know if this included bonuses but I believe that it does.

There is money to be made.

Sources:
Bureau of Labor Statistics
Salary Write Up
The Differences Between a Business Analyst and a Data Analyst

LogiAnalytics - 5 Steps to Masking Your Connection String

Hi All,

I was recently asked if there was a way to mask your connection string in Logi Studio. I thought I'd jot down some notes.


My thinking was why not put the connection string in a DLL. That will make it a bit harder for folks to find. Not exactly 128 bit encryption but each step helps. So that means I need a plugin that populates the connection string element in the Settings.xml file BEFORE the Logi report is processed.

This should be pretty straight forward.

First: There is an excellent plugin example already available on Dev-Net. But it reads from a text file which is still not masked. (Take a look)

Step 1: Fire up Visual Studio and create a new Class Library project

Step 2: Add a reference to Logi's DLL.

• Project Properties
• References
• Add
• Browse
• In your web root folder is a "bin" folder, in there is a file called rdPlugin.dll, add that
• Click the just added reference and in the Properties of it, set Copy Local to false. (This makes sure we don't overwrite the existing dll file when we deploy, this is imporant, do not skip)

I then used this code, it's fairly straight forward, if that doesn't makes sense to you, you probably shouldn't be going this route.

BTW: Do both of us a favor and check that your connection string is correct by putting it in a normal connection element in the Settings file.

Imports System.Xml
Imports System.Web
Imports System.IO
Public Class LogiPlugin
    Public Sub setConnectionString(ByRef rdObjects As rdPlugin.rdServerObjects)
        Dim sConnID As String = rdObjects.PluginParameters("ConnID")
        Dim xmlSettings As New XmlDocument()
        xmlSettings.LoadXml(rdObjects.CurrentDefinition)
        'locate your connection element from it's ID (this is passed in from the variable)
        Dim eleConn As XmlElement = xmlSettings.SelectSingleNode("//*[@ID='" & sConnID & "']")
        Dim sConnection As String = "Server=OneWickedMachine;Database=NorthWind;User Id=sa;Password=ucannothave;"
        eleConn.SetAttribute("ConnectionString", sConnection)
        rdObjects.CurrentDefinition = xmlSettings.OuterXml
    End Sub

End Class

Step 3: Deploy to the file

There are multiple ways to do this, the most stright forward...but not so formal approach

• Build your project.
• Browse to the Debug/Bin folder of your VS project
• Copy the SetConnection.dll to the wwwroot/YourSite/bin folder

Step 4: Call Plugin in Studio

Open up Logi Studio and alter the Settings file

Add a Connection Element, but place "NA" in all the required attributes.

Add a Plugin element

• Assembly Name: ..\bin\SetConnection.dll (referencial path, so you can deploy on Dev, QA and Prod
• Class Type Name: SetConnection.LogiPlugin.LogiPlugin
• Event: Load Definition
• Method: setConnectionString

Add a Plugin Parameter call and ConnID = NorthWind (or your database)

Step 5: Test it!

You should be good to go from this point.

Suggestion: If you have multiple environments (Dev/QA/Prod), I would add an additional parameter for which connection string to use. This can be done within the vb file and should be pretty straight forward.

Logi Analytics - jQuery Auto-complete

This would be the third time I've been asked to incorporate some type of auto-complete function using jQuery and Logi elements.

Here is my cheat sheet:

1. First create a report that returns the results you want in your auto-complete drop down:

• @Request.q~ will have what the user typed
• Keep it simple and quick as this will impact response times
• A single data layer and data table is typically enough

2. Include the jquery script files, I've not had a lot of issues with versioning, so anything over 9 should be fine.

• http://code.jquery.com/jquery-1.9.1.js
• http://code.jquery.com/ui/1.10.3/jquery-ui.js

3. Add a user input element with an ID of inptSearchText

4. Add the following script in a script tag, typically at the bottom of the page:


$(document).ready(function(){

  $("#iptAutoComplete").autocomplete({
minLength: 2,                       
delimiter: /(,|;)\s*/, // regex or character 
source:  function(request, response) 
{
                                    var matches = null;
var data = null;
                                    if(data != null )
{
                                          matches = $.map(data, function(tag) {                                         
                                                if ( tag.value.match(new RegExp("^" +request.term, "i"))){
                                                return tag;
                                                }
                                    });   
                                          response(matches);
                                    }

if (matches == null || matches.length == 0){
                                          $.ajax({
                                                      url: 'rdPage.aspx',
                                                      data: {rdReport:"jQuery.search",rdReportFormat:"DataLayerXml", q:request.term },
                                                      dataType: 'xml',
                                                      success: function( xmlResponse ) {
                                                            data = $( "dtSearchResults", xmlResponse ).map(
                                                                        function() {
                                                                              return {
                                                                                    value: $(this).attr("FORMATTED_NAME"),
                                                                                    id: $(this).attr("EMP_ID")
                                                                              };
                                                                        });
                                                                  response(data.slice(0, 10));   
                                                            }
                                          })
                                    }
                              },
select: function(event, ui)
{ 
alert("you have selected " + ui.item.value + " (" + ui.item.id + ")");
        }
});

});


5. Modify the following:

• "jQuery.search" - replace with the name of the report from step 1
• FORMATTED_NAME needs to be your displayed value from prior table
• EMP_ID needs to be the id you want triggered with the specific data set
• minLength:2 - Set this to the number of characters the user types before triggering the AJAX call
• alert(....) - Call your own javascript function or re-direction after selection. Typically you would store the ID in some hidden variable for future usage.