Posts

DSC_0511 Zach Doty Cover Photo Univariate Linear Regression

Univariate Linear Regression Concepts

Howdy, machine learning compatriots! Welcome back to our foray into getting started with machine learning. Previously, we covered some core machine learning concepts, namely supervised machine learning algorithms and unsupervised / deep learning. (For the full series to date, here’s our Machine Learning for Beginners page.)

 

Today we’re learning the concepts behind supervised machine learning algorithms. Specifically, we examine univariate (one variable) linear regression. Univariate linear regression is the beginner’s playpen in supervised machine learning problems. We endeavor to understand the “footwork” behind the flashy name, without going too far into the linear algebra weeds.

 

Quick Recap: Supervised Machine Learning Problems

If you’re just dropping into the series, we’ll quickly set today’s stage. Univariate linear regression falls under the category of regression algorithms, withing supervised learning machine learning problems.

 

2017-04-30-001-Machine-Learning-Algorithm-Types

  • Supervised learning: we provide the algorithm with pre-cleaned, pre-labeled data. The algorithm learns off the data we provide to classify or predict new data.
  • Regression: making a line of best fit.

 

When we first covered supervised machine learning concepts, regression was shown to make a line of best fit from existing data, so we could predict new data points. Below, we first used the example of an SEO team predicting how many unique linking domains a page would need to achieve a certain rank. (A supervised learning problem, using a regression algorithm for future predictions.)

 

2017-04-11-002-Regression-Problem-Linear-Quadratic-Comp

Important note: our graphic above is similar to linear, but is not quite, linear regression. Details, details. At any rate, this should take us in nicely to examining the inner workings of univariate linear regression.

 

A High Level Look at the Regression Problem Process

If I’m being brutally honest, the process of translating machine learning education to public-facing blog posts has been my toughest effort to date. In other words, I try to make my posts easy to follow, like dummy notes I’m taking as I learn.

That being said, 2 weeks into a machine learning course, and the content has already gone off the wheels, deep into linear algebra and so forth. So, instead of going into the weeds for publication, I’m trying to keep it snackable (buzzword bingo, drink!) and down to earth.

Let’s settle in slowly on the regression problem process. Also illustrated below, we need a few key steps:

  1. A cleaned training data set with correct labels
  2. A program (such as Matlab or Octave) with functionality and access to an appropriate univariate linear regression algorithm
  3. A hypothesis and prediction of new values

2017-04-30-002-Univariate-Machine-Learning-Regression-Process

 

Let’s peel back a layer and go slightly deeper. Since cleaning and correctly labeling training data is largely dependent on you & your domain, we’re skipping that step. ¯_(ツ)_/¯

 

Instead, let’s look closer at the algorithm & hypothesis portions! Our first stop is something called the cost function.

 

The Cost Function in Linear Regression Learning Problems: Squared Error

Before we jump into cost function, let’s turn over a new leaf in visual examples. Instead of our SEO example, let’s look at a problem that could be more linear-friendly. Below, let’s assume we have some data on a customer’s lifetime value plotted against the number of marketing touchpoints they’ve interacted with.

2017-04-30-003a-Univariate-Linear-Regression-Sample-Data-Viz

 

Okay, with the housekeeping complete, let’s remember our goal for linear regression: find the line of best fit. 

Let’s also tie this back to the real world. Perhaps we’re a marketing director or VP of marketing needing to convey the ideal number of marketing touchpoints to the CMO and CEO. Doing so could help guide budgeting, channel mix, and planning questions.

How do we find a line of best fit? Through linear algebra and programming, we can objectively determine the best fit by testing hypotheses and measure each hypothesis line against the actual data points for closeness of fit.

2017-04-30-004a-Univariate-Linear-Regression-Cost-Function-Hypothesis-A

Being frank, the material up to this point is pretty humdrum. However, when we start making hypotheses such as the above, things get interesting. The program “makes a guess” as to the line of best fit, perhaps like the illustration above. I’m no “eggspert”, but that doesn’t look like a great line of fit.

 

But have no fear dear reader, math/science comes to the rescue. The next portion of the algorithm calculates the distance (cost function / squared error) from the training data to the predicted line of best fit in a process called squared error function. When you plot the hypothesis against the squared error sum, you may get a distribution something like the below.

2017-04-30-005-Univariate-Linear-Regression-Cost-Function-Plot

Bare with me. Let’s say we plotted:

  1. Our illustrated hypothesis (teal plus sign)
  2. Other attempted hypotheses (tan x’s), and,
  3. The best fit hypothesis (green outlined star)

This renders a convex parabolic distribution. To get the line of best fit, we want to get as low on the X axis as possible. (Known as the global minimum.) The further magic in machine learning is how we move from a lame hypothesis (teal plus sign) to solution (green outlined star). Now, meet a technique called gradient descent. Sidebar: if we’re being more mathematical and technical about it, this really plots to a 3D conic distribution, but the above explanation should suffice for now if we’re not getting bogged down in the math.

 

Parameter Learning & Gradient Descent

Gradient descent is the iterative mathematical process of working our way down the squared error plot from a lousy hypothesis to a line of best fit. Again, we’re not delving into calculations and derivatives – there’s a TON of math that goes behind this material.

Gradient descent systematically tests increments of hypotheses against a specified learning rate. The learning rate is essentially the magnitude or speed with which you which try to move along the convex function down to zero.

2017-04-30-006-Univariate-Linear-Regression-Simplified-Gradient-Descent

 

Wrap Up

Did I mention this is one of the toughest posts for me to date? The other contender is my DIY Alexa Raspberry Pi article. It’s now 3:20 a.m. on a Saturday night/Sunday morning as I type this conclusion. (Insert horror emoji.)

So, if we were to break down all of the above into a short bulleted list:

  1. Univariate linear regression takes sample data to make a line of best fit
  2. “Best fit” is objectively measured by a squared error function, or the summed distances of the hypothesis line from the actual data points
  3. The hypothesis and squared error function plot roughly a convex parabolic graph
  4. Gradient descent is an algorithm that systematically reduces the squared error hypothesis, guided in part by the learning rate
  5. The gradient descent iteratively seeks the global minimum on the convex function, AKA the line of best fit
  6. The line of best fit is determined, (and Teh Lurd of Teh Rings finishes on your second monitor to Herb Alpert’s Spanish Flea.)

 

Next up, we’ll be installing some machine learning software (Matlab & Octave) and diving into multivariate regression. Look after each other.

DSC_0045 Zach Doty Cover Photo Group By SQL Statement Function

GROUP BY SQL Statement

Introducing the GROUP BY SQL Statement in PostgreSQL

‘Ello SQL geeks! Welcome back to our SQL learning journey. We left off with a beginner SQL skills challenge and the aggregate SQL functions: MIN, MAX, AVG and SUM. Today we’re looking at the GROUP BY statement. We’ll learn about this function in PostgreSQL and walk through usage of this handy SQL statement.

 

About the GROUP BY SQL Statement/Clause

From my simple understanding, GROUP BY functions like a hybrid of the following:

  • SELECT DISTINCT keyword (If used without an aggregate function like SUM), and,
  • An Excel pivot table, rolling up aggregate figures (Count, Sum, Average, etc.) into unique rows

If you’re familiar with Excel Pivot tables, then you’ll recognize here the power of this function.

 

Let’s take a look at some examples to clarify.

 

First Look at Using the GROUP BY Function

To better illustrate the power of GROUP BY, we’ll first show its usage without aggregate functions. Consider the following:

If we query the address table of our sample database with a generic SELECT * FROM address; we get back an atrocious 605 rows of data. Aggregate and useless!

2017-04-27-001-GROUB-BY-Generic-SELECT-Query-Pitfall

In contrast, if we call the GROUP BY function, we’ll get back a cleaner output, with fewer rows – only unique values returned. While this is an incremental improvement to analyzing data, there’s much left to be desired.

2017-04-27-002-GROUP-BY-Without-Aggregate-Function

 

What’s missing here? How about that pivot table-esque functionality? This is where the power of using GROUP BY with aggregate functions gets awesome.

Using the GROUP BY SQL Statement with Aggregate Functions

As with most analysis, a single data point or data series rarely holds significant insight value on its own. Let’s drive home that point by leveraging the GROUP BY statement with the SUM aggregate function. Below, we compare the ratings of films in our sample database in aggregate by replacement cost. Perhaps this could serve in-store strategies for loss prevention.

SELECT rating, SUM(replacement_cost)
FROM film
GROUP BY rating;

2017-04-27-003-GROUP-BY-SUM-Aggregate-Function

If we extend this functionality to more real world examples, we could use the following for GROUP BY:

  • Grouping page-level / URL data to roll up clickstream analytics data
  • Large scale analysis of CRM data for customer segmentation analysis
  • Analyzing returns for financial data

The list could (and I’m sure it does) go on.

Let’s take this one step further and reduce potential future workload, by building sort functionality into our query. Below, we add a line to get most expensive ratings to least.

SELECT rating, SUM(replacement_cost)
FROM film
GROUP BY rating
ORDER BY SUM(replacement_cost) DESC;

2017-04-27-004-GROUP-BY-SUM-ORDER-BY-SUM

By the looks of this, no need to guard Land Before Time 8. 🙂

Extending our lesson: you can use the COUNT, AVG, and other aggregate functions to analyze as desired.

 

Wrap Up

Alright, this was a relatively gentle introduction into more advanced SQL functions. GROUP BY is a rather critical function, so in our next article, we’ll be doing yet another skills challenge. Joy!

Feel free to catch up on our other articles that help you learn PostgreSQL. Also, check out some how-to’s on developing Amazon Alexa skills, and a new series on getting started with Machine Learning. As always, please share with your colleagues and share thoughts in the comments below. Cheers.

SQL Aggregate Functions: Min, Max, Avg and Sum

Welcome back, SQL nerds! We’re back in action on the journey of learning SQL, after a beginner PostgreSQL skills challenge.

We’re reaching the end of basic functions and queries with this article. Based on what we’ve learned so far, we can do basic counting, filtering, sorting and pattern matching against PostgreSQL databases.

If you’re just tuning in, here’s the page on how to learn SQL, and the previous SQL article on the LIKE Statement.

Okay, enough jabber. Let’s jump in. The aggregate functions of MIN, MAX, AVG and SUM are our turning point into more complex SQL queries that involve concepts such as GROUP BY, among others.

At the same time, the functions on their own aren’t super complicated. Because we’re ass-u-ming you’re familiar with the general concepts of minimum, maximum, etc., we’re going to forgo conceptual and syntax explanations for demos.

 

AVG Aggregate Function

As we level up in SQL, we’re going to do less and less explaining / handholding / screenshots. That said, we’re going to explore our DVD rental data set for a table with a nice numerical component that would make good use of the functions.

We did a SELECT * FROM film LIMIT 15; to get a peek at the columns. For the purposes of this exercise, the replacement_cost column will do nicely.

To get the AVG:

SELECT AVG(replacement_cost) FROM film;

2017-04-17-001-AVG-Aggregate-PostgreSQL-Function

Using ROUND for Decimal Place Control

You’ll notice in the example above that we got 3 decimal places on what’s supposed to be a dollar amount. How do we fix that? Glad you asked. Meet the ROUND function.

We pass in the target value (average of replacement_cost) and mandate the number of decimal places we’d like returned. Below:

SELECT ROUND( AVG(replacement_cost), 2) FROM film;

2017-04-17-002-ROUND-for-AVG-Aggregate-PostgreSQL-Function

 

MIN Aggregate Function

To find the minimum value in a given column:

SELECT MIN(replacement_cost) FROM film;

2017-04-17-004-MIN-Aggregate-PostgreSQL-Function

 

MAX Aggregate Function

To find the maximum value in a given column:

SELECT MAX(replacement_cost) FROM film;

2017-04-17-003-MAX-Aggregate-PostgreSQL-Function

 

SUM Aggregate Function

To find the maximum value in a given column:

SELECT SUM(replacement_cost) FROM film;

2017-04-17-005-SUM-Aggregate-PostgreSQL-Function

 

Wrap-Up

Alright, that was a relatively quick article! Hopefully this was a reprieve from more involved sections in the past. We should find that as we continue to strengthen our core PostgreSQL  capabilities, these articles and our SQL queries should be easier and easier. If you found this article interesting, you might enjoy a new section on how to get started in machine learning. Cheers.

 

 

 

DSC_0007 Zach Doty Cover Photo for Beginner PostgreSQL Skills Challenge

Beginner SQL Skills Challenge!

Howdy, SQL geeks. Hope this post finds you swell!

Over the past few months, we’ve gained a ton of ground in learning SQL, or at least I have. 🙂

Let’s take a moment to:

  1. Test our knowledge of SQL skills learned thus far
  2. Start seeing SQL queries less as statements of code, and more as real world business challenges

In this article, we’ll have a recap plus three sections:

  • Recap of the training database we’ve been working with
  • General statements of each business problem
  • Hints and thoughts about how to approach each problem
  • Solutions to each problem

 

Recap: Our Training Database

Our training database is a best/old faithful. We’ve been using the surprisingly popular DVD rental database in a .tar file for our practice database.

Contained within this databases are various tables with fictitious information, including: customer information, film production information, business/pricing information and so forth.

In our challenges, we’ll execute various SQL queries to extract pieces of insight for business tasks. It’s assumed in this article that you’ve installed PostgreSQL via pgAdmin and have followed this article series so far, using the DVD rental training database.

Without further ado, let’s begin.

 

The SQL Challenges

Alright, here we go:

  1. How rentals were returned after July 17, 2005?
  2. How many actors have a last name that starts with the letter A?
  3. How many unique districts are represented in the customer database?
  4. Can you return the actual list of districts from challenge #3?
  5.  How many films have a rating of R and a replacement cost between $5 and $15?
  6. How many films have the word Truman somewhere in the title?

 

How to Approach the Challenges

Right, then. In this section, we’ll add some color commentary (read: hints) to our challenges. This should help you understand the mechanics of the solution, while ensuring you can’t see the answers all in one screen. 😉

 

Challenge 1: Rentals Returned After July 17, 2005

As with all challenges, a problem well stated is half (or more) solved. So let’s look at the high levels of the ask, and work our way down. We need information on rentals, so this means we’ll probably be querying the rentals table.

We would want to first examine the rentals table in a concise manner by doing:

SELECT * FROM rental LIMIT 5;

Once you’ve surveyed the table, we really only need one column returned (pun not intended) and we only want the sum figure of returns, where (HINT!!) the return date was after (think a logical operator here) July 17, 2005.

 

Challenge 2: Actors That Have a Last Name Beginning with “A”

Like our first challenge, let’s work from the “top down”. We need actor information, so querying the actor table would be a great place to start. Similar to last time, we need a count of values matching a condition. The difference versus challenge #1  is we need to find match a pattern like or such as an actor’s last name that begins with the letter A.

 

Challenge 3: Number of Unique Districts in the Customer Database

The title and description could cause some confusion here. You may need to do some basic SELECT * FROM table_name LIMIT X; queries to make sure you’ve got the right table. Once you do, we’re looking for an amount of distinct values in the database. Order of operations matters.

 

Challenge 4: Returning the Actual Lists of Districts from Challenge #3

Not much to hint at here – getting challenge #3 is the key here. You’ll really only be simplifying the correct query in challenge #3 to get the correct answer here.

 

Challenge 5: Cheap, (Mildly) Naughty Films

This one might be the longest query yet in this challenge. So we’re looking up film information, thus should know which table to query. We’re returning a value where a certain rating must be returned, and (HINT) we need to layer in one more lens of qualification. That lens dictates we specify a range between (cough, hint!) two values.

 

Challenge 6: Where in the World are Films Containing “Truman”?

This challenge is more of a recency test than retention of older concepts. You’ll need to employ pattern matching again for this business challenge/query to find titles that have some match like Truman in the title.

 

 

Challenge Solutions

Is that your final answer? Below are the queries, with screenshots of what I did.

 

Solution 1: Rentals Returned After July 17, 2005

SELECT COUNT(return_date) FROM rental
WHERE return_date > ‘2005-07-17’;

2017-04-15-001-Challenge-1-Sol-Count-From-Where

 

Solution 2: Actors That Have a Last Name Beginning with “A”

SELECT COUNT(*) FROM actor
WHERE last_name LIKE ‘A%’;

2017-04-15-002-Challenge-2-Sol-Count-From-Where-Like

 

Solution 3: Number of Unique Districts in the Customer Database

SELECT COUNT(DISTINCT(district)) FROM address;

2017-04-15-003-Challenge-3-Sol-Count-Distinct

 

Solution 4: Returning the Actual Lists of Districts from Challenge #3

SELECT DISTINCT(district) FROM address;

2017-04-15-004-Challenge-4-Sol-Select-Distinct

 

Solution 5: Cheap, (Mildly) Naughty Films

SELECT COUNT(*) FROM film
WHERE rating = ‘R’
AND replacement_cost BETWEEN 5 AND 15;

2017-04-15-005-Challenge-4-Sol-Count-From-Where-Between-And

 

Solution 6: Where in the World are Films Containing “Truman”?

SELECT COUNT(*) FROM film
WHERE title LIKE ‘%Truman%’;

2017-04-15-006-Challenge-4-Sol-Count-Where-Like-Wildcard

 

Wrap Up

Well done for completing these challenges! You shall indeed pass. 🙂

Soon, we’ll be covering aggregate SQL functions, such as MIN, MAX, AVG and SUM.

If you’re just joining us, here’s a running list of our articles so to date (4/16/2017):

 

DSC_0064 Zach Doty Unsupervised Machine Learning Intro Cover Photo

Unsupervised Learning Introduction: Machine Learning Essentials

Howdy, machine learning students! Today we’re going to introduce the concept of unsupervised machine learning algorithms.

Quick Recap: Supervised Learning

Before we jump in, let’s quickly recap our last article introducing supervised machine learning algorithms. This will give us the appropriate context for unsupervised learning.

In supervised machine learning problems, we supply pre-labeled data to the algorithm. By supplying data that’s already correctly labeled, we ask the algorithm to further predict (regression) or label (classification) new data.

2017-04-11-004-Multiple-Input-Classication-Machine Learning

 

Unsupervised Machine Learning = Unlabeled Data

The most immediate and prominent difference  for unsupervised learning is the data. Above, we gave the algorithm “a boost” by supplying the intended “right” answers in the data. Below, in an unsupervised machine learning problem, there are no right answers…yet.

2017-04-15-001M-Unsupervised-Machine-Learning-Problem-Data

We’ve supplied the algorithm with data in the problem, but it’s provided without labels or “answers”. We are mandating that the algorithm discover structure and infer patterns/labels on its own. We could also compare the above example to a clustering problem.

So in unsupervised learning, we supply a large amount of unlabeled data, without explicitly identified form or structure. We ask the algorithm to come up with ideas of structure and segmentation on its own.

Some additional applications of unsupervised learning could include:

  • Market segmentation of massive transaction data
  • Large scale social networking data
  • Astronomical data analysis
  • Large scale market data
  • Mass audio/voice analysis
  • Large scale gene clustering

 

Wrap Up

That was a bit of a quick one! The challenge with some these technical subject matter areas is sometimes we have limited room to run before going off into the technical weeds. This is one of those areas. Next, we’ll be covering some key concepts in the areas of machine learning model representation, cost function and parameter learning. Don’t worry too much about those yet, we’ll take it step by step. 🙂
As always, feel free to follow my other journeys of learning PostgreSQL, learning how to develop Amazon Alexa Skills, learning how to get started in algorithmic trading, JavaScript for beginners…and more to come soon! Cheers.

DSC_0024 Zach Doty Cover Photo for What is Machine Learning

What is Machine Learning?

Hello there, fellow Machine Learning (ML) students! Welcome back to our crash course in starting machine learning from an absolute beginner’s perspective.

In our previous article, we covered an introduction to Machine Learning, answering several key questions:

  • Where is machine learning used in our lives?
  • Where did machine learning come from?
  • Where is machine learning headed?

Forging ahead in our learning journey, we’ll introduce some definitions of machine learning and look at the major types of machine learning applications.

 

Machine Learning (ML), A Casual Definition by Arthur Samuel

Our first definition, teased in the last article, follows:

Machine learning is the practice of giving computers the ability to learn without being explicitly programmed to do so.

 

More on Arthur Samuel & Why His Definition on ML Matters

If you’re like me, you might not have heard of Arthur Samuel. Who is he, and why does his opinion matter in the fields of artificial intelligence and machine learning?

Arthur Samuel was a pioneer in artificial intelligence and computer gaming fields. In 1959, he coined the term “machine learning” as a founding father in the field. That’s why he’s important! Let’s also look at a more formal / scientific definition.

 

A More Formal Machine Learning Definition

Tom Mitchell, of Carnegie Mellon, offers a definition with more structure.

  • A well defined learning problem follows
  • E * T = P
    • Note: His definition does not include mathematical operators. I’m taking a large liberty to insert them myself. ¯\_(ツ)_/¯
  • Experience (E) placed against Task (T) is measured by Performance (P)

2017-04-06-001-Machine-Learning-Definition-ETP-Framework

Here’s a further example:

Example: playing Go.

E = the experience of playing many games of Go.

T = the task of playing and winning Go.

P = the probability that the program will win the next game.

 

Major Categories of Machine Learning Algorithms

If you judge by press coverage of ML as I have, it appears to be a nebulous field. (In all fairness, it may still be.) However, there is structure we can take in learning ML. There are two types of machine learning algorithms:

  • Supervised learning algorithms
  • Unsupervised learning algorithms

There are a couple of other prominent types of machine learning algorithms as well: reinforcement learning and recommender systems.

 

 Wrap-Up

Congratulations, we’ve cleared a very gentle introduction to machine learning, and it’s novice/high level definitions. I look forward to learning more with you, dear reader! Our next articles will cover a bit more detail about the two major ML algorithm types: supervised learning, and unsupervised learning. Until then, look after each other.

DSC_0013 Zach Doty Cover Photo for Introduction to Machine Learning

An Introduction to Machine Learning

Howdy! This is an abrupt interruption to our regularly scheduled programming of SQL lessons, Amazon Alexa Skill development and Algorithmic trading. For those readers who don’t personally know me, I’m on a quest/streak to level up as a technologist.

Getting around to the point, I’ve been taking self-paced courses in varying forms to learn, apply and share new skills. However, I’ve heard the Coursera Stanford class in Machine Learning taught by Andrew Ng recommended so widely, I’m just going to doggedly sprint a marathon. I’m starting almost a week behind, working a busy job, trying to have a social life, and many other things…but darnit, I’m going to give this class my best shot. Hopefully I finish. 🙂

Okay, over-sharing complete. Let’s jump into a brief introduction of Machine Learning.

Machine learning, according to Andrew Ng (Chief Scientist at Baidu), is the science of getting computers to learn without being explicitly programmed.

Where is machine learning used in our lives?

Machine learning is employed a large number of actors. Here are a few examples:

  • Search engines, such as Bing, use machine learning to process MASSIVE amounts of data to quickly rank web pages in order of relevance, with limited human intervention.
  • Social networks, such as Facebook, use machine learning to recognize your friend’s faces for auto-tagging capabilities.
  • Email providers such as Apple Mail may employ spam filters that continuously learn to protect your inbox, your computer, and most importantly, your sanity.
  • Tech companies such as Amazon use natural language processing (NLP) to create conversational experiences and transactions with skills and services.
  • Entertainment companies such as Netflix use self-learning algorithms to recommend compelling new films and TV shows for those of you who binge watch

Where did machine learning come from?

Machine learning originated from a computer science field known as artificial intelligence. Long seen as a pipe dream from Star Trek (crass and careless reference, I know), machine learning is a practical and attainable segue to artificial intelligence, or machines and programs that contain some degree of self awareness.

This capability is a relatively new, yet a rapidly exploding field that grows as mathematical, statistical, hardware and software capabilities continue to compound and improve.

What follows is a better question still.

Where is machine learning headed?

Machine Learning in the future could look like a few different things (but not limited to this list, obviously!):

  • Predictive and preventative applications in engineering, medicine, and security
  • “Load bearing” performance in complex tasks, such as architecting, coding and programming self-driving cars
  • Coordinate machines and programs that study our behavior at our request and perform tasks, such as performing spring cleaning
  • Assistants or programs that are intelligent – able to optimize and independently solve problems on our behalf

Wrap-Up on the Machine Learning Introduction

That wasn’t so bad was it? We’ll follow soon with a more formal definition of Machine Learning and its various tranches of study. Cheers all.

DSC_0104 Zach Doty Cover Photo for Learning LIKE SQL Statement

The LIKE Statement: SQL Statement Fundamentals

Howdy folks! We are overdue for another installation of SQL learning. I’ve slept a few times since the past couple of articles on how to learn SQL. Previously, we talked about the IN Statement, BETWEEN statement, and ORDER BY clause.

In this article, we’ll learn how to execute the LIKE statement in SQL queries. Let’s jump in.

 

About the LIKE Statement & Why it’s Important

Have you ever worked with a data set that’s overwhelmingly large  or complex? Or overwhelmingly large and complex? Sometimes, you need to find data, but can’t recall the exact string or values for a lookup. Or, perhaps, you’re working with a messy data output from say, Google Keyword Planner that groups a range of close variants into one value?

Say you’re looking for values related to designer clothing and designer clothes. Without a better solution, the most probable solution for you is to do a bunch of sorting, filtering, classifying and other data sleuthing at great expense to time and sanity.

The LIKE statement exists to help with the debacle of only having / knowing  part of the lookup criteria you need, courtesy of pattern matching.

PostgreSQL and many other SQL engines/platforms support the LIKE statement, which functions a bit like the below. (Pun not intended.)

 

SELECT keyword, search volume

FROM table

WHERE keyword LIKE ‘cloth%’

 

The above tells pgAdmin / PostgreSQL to get the keyword and search volume columns from table, where the keyword values match values that begin with ‘cloth’ and are followed by anything else, the percent sign. The combination of calling out a text string with an operate is known as a pattern.

When you execute the LIKE statement in a SQL query, pgAdmin will begin reading through the table rows to see if the pattern you’ve specified returns any matches. For the season marketing technology folks, this functionality sure does resemble regex in some ways.

However, there are some differences.

  • Here, instead of the * character being wildcard, the % sign serves as a wildcard matching all characters.
  • If you want to match a single character, the underscore character is used.

 

LIKE Statement Syntax & Examples

Let’s try some examples. Below, we’ll call on the faithful DVD rental practice database, and run a query for customers that have first names like Jen. (Jennifer, Jenny, etc.) Our below code produces the following result.

SELECT first_name,last_name

FROM customer

WHERE first_name LIKE ‘Jen%’;

2017-03-30-001-LIKE-SQL-Statement-Example

 

That being considered, there are other ways we can use the like statement. Above, we used a wildcard to match any endings to a particular string.

Conversely, we could execute a SQL query that specifies a certain ending value, with the wildcard preceding. If we extend that example to such a query below, we should see the following result:

SELECT first_name,last_name

FROM customer

WHERE first_name LIKE ‘%y’;

2017-03-30-002-LIKE-SQL-Statement-Example

 

Above, we’ve flipped the tables so we capture every possible beginning condition under this pattern. Also important to note, the patterns aren’t limited beginnings or ends. You can use this wildcard in the middle, etc. Consider the following:

SELECT first_name,last_name

FROM customer

WHERE first_name LIKE ‘%er%’;

2017-03-30-003-LIKE-SQL-Statement-Example

 

Now, on top of all this awesomeness, realize we’ve been using it as a matching filter. We can also employ the NOT LIKE syntax to exclude values meeting the pattern we’ve specified.

Let’s mix things up a bit. We mentioned a second type of pattern matching character that we haven’t used yet: the underscore.

SELECT first_name,last_name

FROM customer

WHERE first_name LIKE ‘_her%’;

2017-03-30-004-LIKE-SQL-Statement-Example

 

Above, we’ve made a similar ask. However, instead of requesting all possible matches, we’ve mandated SQL only return the ‘er’ string that begins with ‘h’.

To throw you a quick curveball, it’s worth noting the LIKE statement is case sensitive in its matching. Could be bad, could be good, could be neither. However, there’s a way for you to force case insensitivy on the queries. The difference in your statement appears relatively minor. Instead of using a function that calls LIKE or NOT LIKE, you’ll use ILIKE.

 

Wrap-Up

Hope you found this useful! Stay tuned for more SQL learnings and application. If you’re new here, visit the page on how to learn SQL. If you’re interested in more educational material, check out our ongoing series of how to develop Amazon Alexa voice search skills, and getting started with algorithmic trading. Cheers!

Building an Interactive Alexa Quiz Skill, Part 2

Disclaimer: this was typed late at night on a tired mind. Please excuse typos, convention errors, and generally poor writing. 🙂

Howdy, Alexa nerds! Welcome back to our journey in learning Amazon Alexa Skill Development. Quick funny aside, would you care to guess my most common use of the Echo? It’s to play looong Spotify playlists that are basically background noise to help our new dog when Hannah & go to the gym or meet friends. Anyway!

Let’s jump back in. The previous article covered setting up an AWS Lambda function for the Alexa Skill Service. Now, we’ll be working more with the Skill interface. See below for the conceptual overview, or an early article on building Alexa skill interfaces for a basic fact skill.

2017-03-07-001-Alexa-Skill-Interface-Develpoment-Framework

 

Working in the Amazon Developer Console: Alexa Skills Kit

You probably know the drill now, log in to the Amazon Developer console. Once you’ve logged in, select the “Alexa” menu item from the home screen, then choose the “Alexa Skills Kit” Option.

2017-03-27-B-001-Alexa-Quiz-Skill-Create-Alexa-Skill-Interface-Start

If you’ve previously published or started development of skills, you should see them listed on this screen. Now, click, “Add a New Skill”. We should be looking at a very familiar screen here. 🙂

Add/edit the following:

  • Language (assuming you’ll leave the English US default here)
  • Name of the Skill displayed in the Alexa app and store
  • Invocation name users will speak to start your skill

Click Save and Next to proceed.

2017-03-27-B-002-Alexa-Quiz-Skill-Create-Alexa-Skill-Information

 

Working with the Interaction Model

Here comes the tough part, more copy and paste! Okay, sarcasm and humor doesn’t always translate well via text. We’re going to continue to lean fairly heavily on Amazon’s examples here to get ourselves familiarized with the more advanced concepts of intent schema and slot types.

That caveat aside, head back to the files we originally downloaded, but this time, we’re interested in the speechAssets folder and its contents:

  • json
  • Sample utterances (text document)

First, let’s open up the Intent Schema JSON file in our text editor of choice. Below, a look at what you should approximately be seeing. Copy and paste the entirety of the JSON file into the Intent Schema field of the Interaction Model tab.

2017-03-27-B-003-Alexa-Quiz-Skill-Create-Alexa-Skill-Intent-Schema-JSON

Audible: Our First Encounter with Custom Slot Types

Alright, no smooth segue here. We’re having the first encounter with what’s known as custom slot types.

If you were to try and save the skill progress so far, you’ll receive an error message from the developer console that says something like, “Error: There was a problem with your request: Unknown slot type ‘LIST_OF_ANSWERS’ for slot ‘Answer’. Why is that?

2017-03-27-B-004-Alexa-Quiz-Skill-Create-Alexa-Skill-Custom-Slot

If you take a closer look at the Intent Schema JSON file, you’ll notice that  most of the intents are built-in Amazon intents. E.g., “intent”: “AMAZON.RepeatIntent”. The “AnswerIntent” looks nothing like the built-in Amazon intents. Instead, we see a name, “Answer” and type, “LIST_OF_ANSWERS” that was so delicately referenced in the error message.

So how do we remedy this situation? We use the information presented to us in the error message and the JSON file to work our way over this issue. You’ll likely note under the custom slot types mentions “Enter Type”.

2017-03-27-B-005-Alexa-Quiz-Skill-Create-Alexa-Skill-Custom-Slot-Values

Match that information up with our error message and the JSON code, and we’ll enter, “LIST_OF_ANSWERS”. In the values section, we’ll enter on separate lines: 1, 2, 3, and 4. I’ll note here for clarity, that this essentially corresponds to the A/B/C/D multiple choice functionality of the quiz. We’ll see this in greater detail in a bit.

Okay, click “Add” as highlighted above, then click “Save”. Next, return to your files and open up the Sample Utterances text file. You should see something like the below.

2017-03-27-B-006-Alexa-Quiz-Skill-Create-Alexa-Skill-Sample-Utterances

You’ll note it’s quite a bit different than the previous, simple, fact-dispensing Skill we previously built. Take note of the {Answer} sample utterance. These are the pieces of dynamic input and interaction coming together into an Alexa Skill. We’ve defined a custom interaction outside of Amazon’s standard functions, and specified a range of acceptable answers the user can give us. That whole structure meets the user experience here, called in by the {Answer} slot name and custom slot type.

Enough conceptual babble. Copy and paste the sample utterances text into the developer console! Click Save beneath the sample utterances, and click Next.

2017-03-27-B-007-Alexa-Quiz-Skill-Create-Alexa-Skill-Sample-Utterances-List

 

Continuing the Skill Interface Build-out

Alright, so far, we’ve accomplished the following:

  • Provided basic skill information about our new skill
  • Specified details about the interaction model, including;
    • Intent Schema
    • Custom Slot Type
    • Acceptable/specified values for the custom slot type
    • Sample utterances

Next, we need to fill in some simple but crucial configuration details. Remember the ARN we generated by setting up the AWS Lambda function in the previous article? You need it here. Below you can see:

  1. I’ve selected the recommended endpoint type of AWS Lambda ARN
  2. Selected my geographic region of North America and,
  3. Pasted in the full ARN

I’m not going to work with account linking yet, because honestly, it looks really darn complicated and its well past midnight as I type this. Soon. 🙂 Click next and proceed to the testing tab!

In the testing tab, you should first see that the skill is enabled for testing on your account. You can:

  • play back responses from Alexa in the voice simulator to test pronunciation, etc
  • More importantly, use the service simulator to run a sample utterance, and see if your skill actually works.

2017-03-27-B-008-Alexa-Quiz-Skill-Testing-Alexa-Skill

Above, we can see the response to our sample utterance asking SEO Quiz returns as expected. Woohoo! Also, did you know the Alexa voice simulator automatically bleeps out most curse words? Did you know you can kind of work around that by putting extra vowels in the word? I digress. (It’s almost 1 am writing this now, productivity on the rise!) When you’re satisfied, click Next.

We’re getting close! Time to enter some publishing information. I’ll leave the first few sections to you: Category, Sub-Category, Testing Instructions, Country/Region availability, Short and Full skill descriptions.

Now, in the example phrases, I provided some updates to the sample utterances, namely to the starting Intent. Below, see the example phrases of “Alexa open SEO Quiz” and so forth. The “gotcha” here that set me back on my first skill is that the example phrases must be derived from your sample utterances.

2017-03-27-B-009-Alexa-Quiz-Skill-Testing-Alexa-Skill

Upload your 108×108 and 512×512 pixel icon images, click Next and submit the requisite privacy & compliance information. Done!

 

Wrap-Up

So, we’re mostly done, not completely done. The part for usto do now is customizing the template code in your AWS index.js file. Ideally, I would prefer a more eloquent closing, but it’s late, will have to wait for another time. Look after each other.

 

The BETWEEN Statement: SQL Statement Fundamentals

Hey there, folks! Welcome back to our journey in learning SQL. Our last few of posts covered the ORDER BY clause, LIMIT statement and the COUNT function. (See full list of SQL tutorials here.)

Today, we’re going to cover the BETWEEN statement. This is the start of some deeper material. In addition to the BETWEEN statement, we’ll also soon be covering IN and LIKE statements.

 

About the BETWEEN Statement

The BETWEEN statement (rather, operator) is used to match a value against a specified range of values. Maybe we want to get transactions between a certain dollar amount.

For example, value BETWEEN low AND high;

More about the BETWEEN statement. If the value is greater than or equal to the low value and  less than or equal to the high value, the expression returns true, or vice versa. Also, the BETWEEN operator can be rewritten by using the greater than or equal to ( >=) or less than or equal to ( <=) operators as the following statement.

value >= low and value <= high;

One other way to think about the BETWEEN statement is that takes two WHERE statements and lumps them into one. Think back to the WHERE statement, as we would say first:

SELECT column1 FROM table WHERE column1 >= 2 AND column1 <= 7;

 

To BETWEEN or NOT BETWEEN

Conversely, you can extend the usage of the original BETWEEN operator to NOT BETWEEN. This is similar in concept, except working toward exclusion, instead of inclusion. So, if we want to check if a given value is outside of a range, we can use the NOT BETWEEN operator as below.

value NOT BETWEEN low AND high;

Again, similarly to the double WHERE statements, NOT BETWEEN simplifies the burden into a single statement. For sanity, we’ll skip the SELECT WHERE example.

 

Using BETWEEN in pgAdmin / PostgreSQL

Below, we’ve taken the BETWEEN statement for a road test on our address table.

SELECT address_id,address FROM address

WHERE address_id BETWEEN 10 AND 20;

2017-02-18-001-BETWEEN-SQL-Statement-Example

How sweet was that? Syonara, comparison operators. Conversely, if we convert the above query into a NOT BETWEEN operator, we should see the below statement return the following results.

SELECT address_id,address FROM address

WHERE address_id NOT BETWEEN 10 AND 20;

2017-02-18-002-NOT-BETWEEN-SQL-Statement-Example

Okay, that was cool. Let’s hold up for a quick second. What about non-integer columns, such as data? Caveat: there’s a lot more to data types than our humble example below. For the purposes of this article, we can take a YYYY-MM-DD date and place the values into strings.

SELECT rental_id,rental_date,inventory_id FROM rental

WHERE rental_date BETWEEN ‘2005-05-24’ AND ‘2005-05-27’;

2017-02-18-003-BETWEEN-Non-Integer-SQL-Statement-Example

Voila!

Wrap-Up

Alright, for those of you reading this later, it’s 2 a.m. on a Friday night / Saturday morning, and I’m about ready to call it a night. Sorry I’m not providing a better conclusion for you today. I’m excited to delve further into more complex SQL queries. Keep track of my shared journey of a beginner learning SQL. Cheers!