Looker might slow you down

At the beginning of this year the data analysts and engineers at my company undertook two large migration feats. They moved our data warehouse into Snowflake and secondly, they moved our analytics tooling away from Periscope and into Looker.

Looker is heralded as a way for non-SQL savvy users to self serve and build their own data reports and tables without leaning on a Data Analyst who knows SQL to do it for them.

On one hand Looker has delivered on that promise; people in the business are increasingly getting used to building their own Dashboards. On the other hand, I’ve noticed my Data Analysts are as busy as they’ve ever been, if not more. What gives?

Looker creates busywork for your Analysts

Looker is not a zero-config plug and play service. Its not the case that you can just hook it up to your data warehouse and your tables are auto discovered and configured automatically.

In fact, what I’ve noticed is that Data Analysts in Looker shops are spending a solid chunk of their time writing out LookML; a Domain-specific-language (read Vendor Lockin) that Looker requires Analysts to use to redefine the schema of the underlying data models in a format that Looker understands. These schemas are called views and you need at least one view for each table in your database (a huge task if you are an enterprise company with hundreds of tables). In addition to defining views, explores (another custom Looker concept) have to be defined by the Analyst before an end user can start building off the data.

Looker hinders rapidly iterating product development

Looker is not a great place for dashboards of experimental work or rapidly evolving datasets and isn’t great for getting insight into new features that were built and deployed within the past few days or weeks.

Looker is best for building dashboards on top of your more mature corners of the business. The idea is that your data analyst does an initial up front concerted effort to diligently sort, organize and label your data schema. For them to do this, it implies your data schema should be relatively stable. This might suit some of the more traditional, stable parts of your company like tracking User Sign Ups, or Orders.

My current team’s culture is to iterate fast, to try something, measure the results, adopt it and move on if necessary. Ideally we’d like to add a new production table, start amassing data in it, and then quickly build dashboards (that might not last longer than a week or two) to measure the result or inspect the contents. Every time we start collecting a new piece of data and want to report it in Looker for our business partners to play with, we have to do grunt work to get it to appear in Snowflake, and once in Snowflake we have to do grunt work to get it readily available in Looker. Admittedly this could be mitigated if we were flush with Data Analysts embedded in our team who could immediately start defining the LookML schema the moment each migration lands but the reality is that they are a shared resource and you have to get in line and wait your turn if you need something to appear in Looker.

In Summary

Looker has value for sure, but it doesn’t mean you can hire fewer analysts or engineers, if anything you should hire more, and if you’re a fast moving business with an ever-evolving product, your team’s ability to make data driven decisions is still constrained by the capacity of your Data Analytics team and if you consider yourselves rapid iterators making data driven decisions you may find traditional SQL-based analytics tools are better suited for the job.

Protect your Python app from Timeouts

In this post, I show how to better protect your python service when making http calls to other services by guaranteeing a timeout by monkey-patching the requests library.

Services tend to talk to other services. These services might be owned by you, another team at work, or made available by a third party (e.g. Google Maps API).

How tolerant is your service to the failure of one of these upstream services? Have you tested what will happen when one of these services isn’t responding to your requests they way you expected?

I found this out the hard way recently when I deployed a new feature to my python service that required talking to another service. This upstream service was taking a long time to respond to my high volume of requests.

I use the requests library to call the APIs of other services. This library, has no default timeout value for requests that are taking too long. The consequence is that all of my service’s resources were quickly exhausted and waiting for responses from this upstream service that were never going to arrive.

The result? My service pretty much ground to a halt.

Other similar libraries written for other languages do have a default timeout setting in place. For instance, my ruby services use excon which has a default timeout of 60 seconds. This is probably still too high for most cases but its very straightforward to set your own default

# config/initializers/excon.rb
Excon.defaults[:read_timeout] = ENV.fetch('EXCON_DEFAULT_TIMEOUT_SECONDS', 10).to_i
Excon.defaults[:write_timeout] = ENV.fetch('EXCON_DEFAULT_TIMEOUT_SECONDS', 10).to_i
Excon.defaults[:connect_timeout] = ENV.fetch('EXCON_DEFAULT_TIMEOUT_SECONDS', 10).to_i

As for the requests library…

Most requests to external servers should have a timeout attached, in case the server is not responding in a timely manner. By default, requests do not time out unless a timeout value is set explicitly. Without a timeout, your code may hang for minutes or more. link

Eeek!

This means if you or another contributor forgets to add a timeout paramter to your requests library call you run this risk of a request never timing out (and hogging resources in doing so)!

Ensuring a default timeout whenever the requests library is used can be only be achieved (as of requests v.2.17.3 at least) using monkey patching.

I try to only use monkey patching as a last resort as it is brittle and harder than normal to debug but since there is no global constant or environment variable that can be overwritten we’ll have to make do with monkey patching!

import requests
from requests.adapters import TimeoutSauce
import configuration


class GlobalDefaultTimeoutSauce(TimeoutSauce):
    # A subclass of TimeoutSauce that will use a
    # default timeout setting when overrides are not already specified

    def __init__(self, *args, **kwargs):
        default_timeout_seconds = configuration.REQUESTS_DEFAULT_TIMEOUT_S
        connect = kwargs.get('connect') or default_timeout_seconds
        read = kwargs.get('read') or default_timeout_seconds
        super(GlobalDefaultTimeoutSauce, self).__init__(connect=connect, read=read)


def monkey_patch_requests_timeout_strategy():
    # Subsequent usages of the requests library will use a default
    # timeout if none is specified by the caller.
    # Call me once during your app's init phase, before any requests are made.
    requests.adapters.TimeoutSauce = GlobalDefaultTimeoutSauce

To see how this is used take a look at the adapters module in the requests project. link

Now I can boost the resiliency of my service from degredations of the services I depend on! Hope this helps someone else out!

Hillbilly Eligy

The Hillbilly Eligy by J. D. Vance is a memoir of a boy growing up in Ohio in a [sic] white working class family that had its fair share of hardship and troubles with addiction. Its written very straightforwardly and conjures a vivid picture of the author’s upbringing. I read this book because it was being billed online as a good insight into a part of America that had been out of sight and out of mind. I don’t think this wholly explains why the ‘16 election swung the way it did but nevertheless a worthwhile read.

Tidbits I took away from this book were:

  • Be wary of the perils of living in an area that has one large main employer.
  • If an area suddenly has no jobs, the bottom of the housing market can fall out and only the relatively rich can afford to bite the cost and move away.
  • The author notes that the most stable/happy members of his family were those that married other halves from happier/stabler backgrounds.
  • Adverse childhood experiences can rear their ugly head in adulthood.
  • Addiction is the worst.
  • Its very hard to resist a community’s weariness and resignation about class immobility.

Ornaments

The morning of my flight home for the holidays I hastily came up with something for my family’s arts’n’crafts gift exchange.

Luckily the Techshop was pretty quiet so I could get some time on the Laser Cutter to draw up these fellas

It took three iterations to get it right, the first was too big, the second was good but needed handles. Once you’ve got it though, its just a question of copy and paste to make lots!

Lessons learned

  • The Laser Cutter slices through the MDF sheet like its butter

Equipment and Materials used

References

Danish Christmas Tree

Late November I made this Wooden Dowel Christmas Tree with advent-style felt ornaments (with some talented friends helping with all the ornaments).

The tree is collapsible (you just pull the branches out) which is useful for the rest of the year when its not Christmas!

Lessons learned

  • Dowel is cheap
  • 25 hand-stiched ornaments can take a while
  • Once you’re half way through you have to finish!

Resources and References

Even Steven - available for download

Even Steven is now available as a free download on the App Store

Feedback welcomed via Twitter

Thanks to people that helped test it!

Even Steven - a group expenses app for iOS

Update Now available on the App Store!

Even Steven is a handy app that records group spending for events such as holidays and suggests who needs to pay what to whom in order to get even!

Even Steven is designed for iPhone and:

  • requires no signing up
  • does not require an internet connection (ideal when holidaying abroad and not having data)

Behind the scenes, Even Steven is a pure Swift3 iOS app and uses no third party libraries.

Even Steven 1.0 is still undergoing beta testing amongst friends but will hopefully be available for free to all before Christmas!

Walnut Tealight Holder

Walnut Tealight Holder

Still horrendous at making things with my hands but will keep trying! This is a tealight holder that is basically three bits of wood, two drill holes and glue.

  • Band Saw
  • Pillar Drilling Machine
  • 1 12” Forstner Bit
  • Sand Paper
  • Wood Glue
  • Circular Saw

Lessons learnt

Buying Wood

  • Walnut is a beautiful dark hardwood
  • Walnut is expensive
  • A DIY Store will sell wood at fixed lengths, but will cut it into pieces for you.
  • My car can fit about 10feet of wood just!

Prepping

  • Make a detailed plan beforehand.
  • If it doesn’t work on paper it won’t work in the shop (this applies to coding too btw). Draw out everything to scale.
  • Use SketchUp or some other CAD software if you want to have reusable plans.
  • Itemize the equipment you’ll need. Nothing more frustrating than being halfway through and realising you don’t have any sanding paper!
  • Use Painters Tape or Super77 glue to stick your paper plans onto real wood.

In the Shop

  • Sanding concave curves is hard without a spindle sander
  • Always sand with the grain
  • Wood Glue needs clamps and needs to sit clamped for at least 30mins
  • Have a Pencil and a backup!

Tensorflow 101

Tensorflow seems like a pretty exciting alternative to scikit-learn that works just as effectively for beginners as it does for pros.

For my team’s monthly drink’n’sync I wrote an interactive worksheet on using TensorFlow to build a handwritten number detector.

Check it out on github!

Pong in < 250 lines of JS!

This is a crude example of a playable game that can be expressed in less than 250 lines of code!

Demo

Code

  1<!DOCTYPE html>
  2<html>
  3<body>
  4  <canvas id="game"> </canvas>
  5  <script type="text/javascript">
  6    var game = (function() {
  7      var Width = 800, Height = 450;
  8      var canvas = document.getElementById("game");
  9      var FPS = 1000 / 60;
 10      canvas.width = Width;
 11      canvas.height = Height;
 12      canvas.setAttribute('tabindex', 1);
 13      var ctx = canvas.getContext("2d");
 14
 15      var BG = {
 16        Color: '#333',
 17        Paint: function() {
 18          ctx.fillStyle = this.Color;
 19          ctx.fillRect(0, 0, Width, Height);
 20        }
 21      };
 22
 23      var Ball = { Radius: 5, Color: '#999', X: 0, Y: 0, VelX: 0, VelY: 0 };
 24      Ball.Paint = function() {
 25        ctx.beginPath();
 26        ctx.fillStyle = this.Color;
 27        ctx.arc(this.X, this.Y, this.Radius, 0, Math.PI * 2, false);
 28        ctx.fill();
 29        this.Update();
 30      };
 31      Ball.Update = function() {
 32        this.X += this.VelX;
 33        this.Y += this.VelY;
 34      };
 35      Ball.Reset = function() {
 36        this.X = Width / 2;
 37        this.Y = Height / 2;
 38        this.VelX = (!!Math.round(Math.random() * 1) ? 1.5 : -1.5);
 39        this.VelY = (!!Math.round(Math.random() * 1) ? 1.5 : -1.5);
 40      };
 41
 42      function Paddle(position) {
 43        this.Color = '#999';
 44        this.Width = 5;
 45        this.Height = 100;
 46        this.X = 0;
 47        this.Y = Height / 2 - this.Height / 2;
 48        this.Score = 0;
 49        if (position == 'left')
 50          this.X = 0;
 51        else this.X = Width - this.Width;
 52        this.Paint = function() {
 53          ctx.fillStyle = this.Color;
 54          ctx.fillRect(this.X, this.Y, this.Width, this.Height);
 55          ctx.fillStyle = this.Color;
 56          ctx.font = "normal 10pt Calibri";
 57          if (position == 'left') {
 58            ctx.textAlign = "left";
 59            ctx.fillText("score: " + Player.Score, 10, 10);
 60          } else {
 61            ctx.textAlign = "right";
 62            ctx.fillText("score: " + Computer.Score, Width - 10, 10);
 63          }
 64        };
 65        this.IsCollision = function() {
 66          if (Ball.X - Ball.Radius > this.Width + this.X || this.X > Ball.Radius * 2 + Ball.X - Ball.Radius)
 67            return false;
 68          if (Ball.Y - Ball.Radius > this.Height + this.Y || this.Y > Ball.Radius * 2 + Ball.Y - Ball.Radius)
 69            return false;
 70          return true;
 71        };
 72      };
 73
 74      window.requestAnimFrame = (function() {
 75        return window.requestAnimationFrame || window.webkitRequestAnimationFrame || window.mozRequestAnimationFrame || window.oRequestAnimationFrame || window.msRequestAnimationFrame || function(callback) {
 76          return window.setTimeout(callback, FPS);
 77        };
 78      })();
 79      window.cancelRequestAnimFrame = (function() {
 80        return window.cancelAnimationFrame || window.webkitCancelRequestAnimationFrame || window.mozCancelRequestAnimationFrame || window.oCancelRequestAnimationFrame || window.msCancelRequestAnimationFrame || clearTimeout
 81      })();
 82
 83      var Computer = new Paddle();
 84      var Player = new Paddle('left');
 85
 86      function Paint() {
 87        ctx.beginPath();
 88        BG.Paint();
 89        Computer.Paint();
 90        Player.Paint();
 91        Ball.Paint();
 92      }
 93
 94      function MouseMove(e) {
 95        Player.Y = e.pageY - Player.Height / 2;
 96      }
 97      canvas.addEventListener("mousemove", MouseMove, true);
 98
 99      function Loop() {
100        init = requestAnimFrame(Loop);
101        Paint();
102        if (Player.IsCollision() || Computer.IsCollision()) {
103          Ball.VelX = Ball.VelX * -1;
104          Ball.VelX += (Ball.VelX > 0 ? 0.5 : -0.5);
105          if (Math.abs(Ball.VelX) > Ball.Radius * 1.5)
106            Ball.VelX = (Ball.VelX > 0 ? Ball.Radius * 1.5 : Ball.Radius * -1.5);
107        }
108        if (Ball.Y - Ball.Radius < 0 || Ball.Y + Ball.Radius > Height)
109          Ball.VelY = Ball.VelY * -1;
110        if (Ball.X - Ball.Radius <= 0) {
111          Computer.Score++;
112          Ball.Reset();
113        } else if (Ball.X + Ball.Radius > Width) {
114          Player.Score++;
115          Ball.Reset();
116        }
117        if (Computer.Score === 10)
118          GameOver(false);
119        else if (Player.Score === 10)
120          GameOver(true);
121        Computer.Y = (Computer.Y + Computer.Height / 2 < Ball.Y ? Computer.Y + Computer.Vel : Computer.Y - Computer.Vel);
122      };
123
124      function GameOver(win) {
125        cancelRequestAnimFrame(init);
126        BG.Paint();
127        ctx.fillStyle = "#999";
128        ctx.font = "bold 40px Calibri";
129        ctx.textAlign = "center";
130        ctx.fillText((win ? "YOU WON!" : "GAME OVER"), Width / 2, Height / 2);
131        ctx.font = "normal 16px Calibri";
132        ctx.fillText("refresh to replay", Width / 2, Height / 2 + 20);
133      }
134      return {
135        Start: function() {
136          Ball.Reset();
137          Player.Score = 0;
138          Computer.Score = 0;
139          Computer.Vel = 1.25;
140          Loop();
141        }
142      };
143    })();
144    game.Start();
145  </script>
146</body>
147</html>

About

Oliver Wilkie is a software engineer in San Francisco.

Oli spends a lot of time writing ruby code at Flexport to automate and scale tasks such as boat tracking, boat timetables and cost prediction. Before that Oli spent several years down the road at Square working on logistics problems of a different kind. Oli’s primary focus wherever he goes is getting stuff done by writing software efficiently and applying it wisely to areas that haven’t fully benefited from it so far. A believer in technology, but also digital minimalism, Oli is passionate about finding that balance between the two.

Oil owes his start to loving parents, supportive teachers, a Randy Pausch lecture and the earnestness of youth which carried him to London to study Computing at Imperial College, then France to intern (and frankly have the summer of a lifetime) and finally a pilgrimage to San Francisco which he has happily called home for the past six years.

Reroku (Heroku on a Raspberry Pi)

My team recently switched our applications from Heroku to barebones AWS. The experience made me truely appreciate just how effective Heroku is and how it has made deploying apps as painfree as could be. After all, what good is making an app if nobody can see it!

I’ve also been a huge fan of my Raspberry Pi. Every time I look at it, I feel challenged to think of a cool use for it.

So with that in mind, I decided to take a shot of making a proof-of-concept of a Heroku-like service that can run on a raspberry pi. It was a great exercise in Unix scripting and gluing a few things together. Plus I got to learn more about the innards of git and what happens when you do a git push!

Kudos to the Dokku project which heavily inspired me.

Its a proof-of-concept but I’m reluctant to invest time making a stable version since I think something like docker+kubernetes might be just around the corner for the Pi and I’d rather invest time figuring out how to make an accessible interface for that!

Check it out on github!