Code, ideas, things that I'm thinking about

Building a Raspberry Pi internet speed checker with NodeJS and GoogleDocs

March 18, 2017 0 Comments nodejs, RaspberryPi, speed-test

I frequently experience periods of internet drop outs and wanted to build something to monitor my internet without any manual intervention. I wanted this to create an automated report for me so I could then email it off to my ISP and complain.

So I set about looking on the internet for something that would do this. There are some options and there are a lot of home baked solutions in many different languages.

After a while I remembered I had a Raspberry Pi gathering dust so I dug it out and set off.

Here's how I did it...

Things you'll need:

  • Raspberry Pi
  • SD card
  • Ethernet cable
  • Google account
  • IFTTT account

Setting up your Raspberry Pi

First of all, install Raspbian or some other linux disto on your Raspberry Pi. The simplist way to do this is using the NOOBS method which is described in detail here. --> NOOBS

Make sure you've connected your Raspberry Pi to your router via ethernet cable to get the best results.

Then make sure you've got NodeJS installed on the Pi.

Open the terminal and type

$ node --version

If node is installed you should get a result with a version number similar to below.

v6.9.5  

If you don't get a version number back, you can download and install NodeJs from https://nodejs.org/en/download/.

Set up IFTTT trigger

The results of each speed tests need to be pushed into the cloud, the easiest place for the results to end up is a google docs spreadsheet. We can leverage IFTTT's Maker Webhooks channel to get an endpoint which we can post to which will insert the results into a google doc.

You should go to https://ifttt.com/maker_webhooks and create a trigger and configure it to add a row to a spreadsheet.

If  
    Maker Event "speed-test", 
then  
    add row to spreadsheet in xxxxx@gmail.com’s Google Drive

This will create a unique key that looks something like aslkjfadnadsknadsjknad. Keep note of this, we'll use it soon.

Write your node app

So now we've got the Raspberry Pi set up. We have to write the program which will run on a specified interval to do the speed test.

We want this program to do 3 things.

  • Run on a specified interval
  • Do a internet speed test
  • Post results to the cloud

I'm going to use the node-cron package to run the speed test on a specified interval. For the speed test, I'm going to use speedtest-net.

Now, create a node application, and install the dependencies you'll need.
(I've included isomorphic-fetch so we can easily make a request to our IFTTT endpoint)

$ mkdir node-speed-checker
$ cd node-speed-checker
$ npm init -y
$ npm install node-cron speedtest-net isomorphic-fetch

Lets create a function that runs a speed test and then plucks the correct information from the returned data. We then want to create a querystring that we can append to our IFTTT trigger url. I've added an environment variable here called IFTTT_KEY so that we can push the key in when we run up the app.

function runSpeedTest() {  
    speedTest({maxTime: 5000})
    .on('data', data => {
        console.dir(data);
        const meta = {
            client: data.client,
            server: data.server
        };
        const queryString = `value1=${data.speeds.download}&value2=${data.speeds.upload}&value3=${JSON.stringify(meta)}`;
        const key = process.env.IFTTT_KEY;
        fetch(`https://maker.ifttt.com/trigger/speed-test/with/key/${key}/?${queryString}`);
    }).on('error', err => {
        console.error(err);
    });
}

Now that we have our function that will run the speed test, we need to set up the scheduler.

cron.schedule('0 * * * *', runSpeedTest);  

This will run runSpeedTest every hour on the hour.

You can check out this full code at https://github.com/madole/node-cron-speed-checker

Now, you just need to put your code onto your Raspberry Pi and run the app. The easiest way to do this is to push your code to a github repo, SSH into the Pi and clone the codebase.

When you've done all this this, install the dependencies, set the IFTTT_KEY environment variable and run the app.

$ git clone https://github.com/madole/node-cron-speed-checker.git
$ cd node-cron-speed-checker
$ npm install
$ IFTTT_KEY=aslkjfadnadsknadsjknad node index.js

This will now run a speed-test from your Raspberry Pi and log the results to your Google Spreadsheet. After a few days, you'll start to build up a result set and over a few weeks you'll have enough data to create some graphs and see your peak and low times.

Whitelisting multiple domains with kcors in Koa

February 17, 2017 0 Comments nodejs, Javascript, koa, cors, security, kcors

Kcors

The Koa CORs library kcors has documentation which is a bit light on how to actually whitelist more than one domain for your node server.

It has a decent JSDoc which tells you the the origin option can be a string, function or generator function. But it doesn't tell you how to use this to whitelist more than one origin.

/**
 * CORS middleware
 *
 * @param {Object} [options]
 *  - {String|Function(ctx)|GeneratorFunction(ctx)} origin `Access-Control-Allow-Origin`, default is '*'
 *  - {String|Array} allowMethods `Access-Control-Allow-Methods`, default is 'GET,HEAD,PUT,POST,DELETE,PATCH'
 *  - {String|Array} exposeHeaders `Access-Control-Expose-Headers`
 *  - {String|Array} allowHeaders `Access-Control-Allow-Headers`
 *  - {String|Number} maxAge `Access-Control-Max-Age` in seconds
 *  - {Boolean} credentials `Access-Control-Allow-Credentials`
 *  - {Boolean} keepHeadersOnError Add set headers to `err.header` if an error is thrown
 * @return {Function}
 * @api public
 */

The normal way to use this function when only whitelisting one domain is something like this:

 app.use(cors({ origin: 'http://mysafeorigin.com' }));

But what if you want to connect to your API from two origins? Well the function overload on this module option is how to do it.

First we create an array of origins you want to whitelist.

const whitelist = ['http://mysafeorigin.com', 'http://myothersafeorigin.com'];  

Then we create a function which takes the the koa context as a param, plucks the origin off the request and compares it to the whitelist, if it's not in the whitelist, chuck an error.

 function checkOriginAgainstWhitelist(ctx) {
     const requestOrigin = ctx.accept.headers.origin;
     if (!whitelist.includes(requestOrigin) {
         return ctx.throw(`πŸ™ˆ ${requestOrigin} is not a valid origin`);
     }
     return requestOrigin;
 }

Then just hook it up to your cors call, I'm using koa2 so I'll use koa-convert to convert it from the koa1 syntax to koa2.

 app.use(convert(cors({ origin: checkOriginAgainstWhitelist })));

Altogether that should look like:

const whitelist = ['http://mysafeorigin.com', 'http://myothersafeorigin.com'];

function checkOriginAgainstWhitelist(ctx) {  
    const requestOrigin = ctx.accept.headers.origin;
    if (!whitelist.includes(requestOrigin) {
        return ctx.throw(`πŸ™ˆ ${requestOrigin} is not a valid origin`);
    }
    return requestOrigin;
 }
app.use(convert(cors({ origin: checkOriginAgainstWhitelist })));  

Now any request coming from an origin in your whitelist will pass it's pre-flight request and your browser will allow the request to take place, any that don't satisfy the whitelist check will chuck and error to be handled by your error handler middleware and your browser will give you feedback similar to this:

Fetch API cannot load http://myunsafeorigin.com/stuff
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://myunsafeorigin.com' is therefore not allowed access.
The response had HTTP status code 500. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Mocking relative dependencies in Jest with jest.mock()

December 07, 2016 0 Comments Javascript, Testing, Jest

If you're like me and have just entered the world of Jest for testing your React application. You might be familiar with a mocking library called Proxyquire.

Proxyquire - "nodejs require in order to allow overriding dependencies during testing."

An example of using proxyquire to mock an import would be something like this.

/* Source File - src/doSomething.js */
import dependencyOne from '../utils/dependencyOne';

export default () => dependencyOne();

/* Test File - test/doSomething.spec.js */
import proxyquire from 'proxyquire';

const doSomething = proxyquire('../src/doSomething', {  
    '../utils/dependencyOne': cb => cb()
}); 

doSomething() //calls our mocked dependencyOne instead of the real one  

You can see here that when we mock dependencyOne, we use the same exact path that the source file uses to import the relative dependency.

In Jest however, this same functionality is delivered with a slight change in usage.

Lets take the above example now in Jest's syntax.

/* Source File - src/doSomething.js */
import dependencyOne from '../utils/dependencyOne';

export default () => dependencyOne();

/* Test File - test/doSomething.spec.js */
import doSomething from '../src/doSomething';

jest.mock('../src/utils/dependencyOne', cb => cb());

doSomething() //calls our mocked dependencyOne instead of the real one  

You can see that we use the path relative to the test file this time to mock the dependency. The mock gets hoisted so it's evaluated before the import statement so this allows us to have the same functionality and control the dependencies of the file we're trying to test.

This isn't well documented so I thought I'd write this down to save someone else the headache I had trying to figure this out.

Putting Yarn Package Manager to the test

November 22, 2016 0 Comments yarn, Javascript, packages, package manager

What is Yarn?

FAST, RELIABLE, AND SECURE DEPENDENCY MANAGEMENT. - https://yarnpkg.com/

How does it achieve all of this?

It uses caching to speed up installs by only ever downloading them once and parallelizes operations to maximise resource utilisation.

It uses checksums to verify every package before installing.

It is deterministic so if you check in your lockfile, you can guarantee every developer who yarn installs will get the exact same dependencies as you.

How do I get Yarn?

According to the yarn installation website you can install yarn with the standard OS package managers. I use a mac so I tried to install it with brew (brew update && brew install yarn) but I had issues with this particular installation.

I then discovered you can install yarn via npm, which on some level feels like cheating on npm but anyway... $ npm i -g yarn will install yarn globally. You can check this was successful by $ yarn --version and making sure you get a version number back. This time the installation was fine and I was ready to get down to testing.

Is this really as good as they say?

I wanted to test against a well known project so I cloned the React git repository.

Before each test I ran $ rm -rf node_modules to remove my node modules and start with a clean slate each time. And just to be fair and make sure I really start the yarn installs with a cold cache, I ran $ yarn cache clean before starting the yarn tests.


Test one - NPM

$ time npm install

First time round:

npm i 48.39s user 15.08s system 106% cpu 59.727 total

Second time round:

npm install 44.50s user 13.40s system 106% cpu 54.331 total


Test Two - Yarn

yarn install

First time round - cold cache

✨ Done in 29.48s.

Second time round - warm cache

✨ Done in 14.05s.


Conclusion

With a warm cache, yarn installs are around 3 times faster than npm on average from my tests. That's an incredible savings when you have a project with a huge dependency list.

I've started using Yarn on the project I work on. It's brought the local install time down from ~123s to ~40s with a warm cache.

I've not put it into production just yet but I think there are a lot of gains to be made using Yarn as your package manager.

A cool feature of yarn is yarn upgrade-interactive, just merged in last week. It allows you to interactively upgrade packages in a nice command line tool.

Bonus coolness: They have managed to get /u/Shitty_Watercolour from Reddit to do all the illustrations for the yarnpkg website http://shittywatercolour.com/

My favourite VSCode plugins

November 06, 2016 0 Comments VSCode, Code, packages, plugins

A while ago I published a comprehensive list of my favourite Atom Editor plugins, I've since moved to VSCode for the majority of my development so I thought I'd gather together a similar list for it.


Auto-close-tag

Automatically add HTML/XML close tag, same as Visual Studio IDE does.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install auto-close-tag

Features

  • Automatically add closing tag when you type in the closing bracket of the opening tag
  • After closing tag is inserted, the cursor is between the opening and closing tag
  • Set the tag list that would not be auto closed
  • Automatically close self-closing tag
  • Support auto close tag as Sublime Text 3
  • Use Keyboard Shortcut or Command Palette to add close tag manually

auto-close-tag


Autoimport

Automatically finds, parses and provides code actions and code completion for all available imports. Works with Typescript and TSX.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install autoimport

autoimport


Code Settings Sync

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install code-settings-sync

Features

  • Use your github account token and GIST.
  • Easy to Upload and Download on one click.
  • Upload Key : Shift + Alt + U
  • Download Key : Shift + Alt + D
  • Use Other User's Public GIST to completely sync your settings with them.
  • Show a summary page at the end with details.
  • Auto Download Latest Settings on Startup.
  • Auto upload Settings on file change.

It Syncs

  • Settings File
  • Keybinding File
  • Launch File
  • Snippets Folder
  • VSCode Extensions

Document This

"Document This" is a Visual Studio Code extension that automatically generates detailed JSDoc comments for both TypeScript and JavaScript files.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install docthis

Document This


Flow Language Support

This extension adds Flow support for VS Code. Flow is a static type checker, designed to find type errors in JavaScript programs.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install flow-for-vscode


Git Blame

See Git Blame information in the status bar for the currently selected line.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install gitblame

git blame


Git Lens

Provides Git information (most recent commit, # of authors) in CodeLens, on-demand inline blame annotations, a blame explorer, and commands to compare changes w/ the working tree or previous versions

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install gitlens

GitLens

Provides Git information (most recent commit, # of authors) in CodeLens, on-demand inline blame annotations, status bar blame information, a blame explorer, and commands to compare changes with the working tree or previous versions.

Features

  • Provides CodeLens on code blocks:

    • Recent Change - author and date of the most recent check-in for that block

      Clicking on the CodeLens opens a Blame history explorer with the commits and changed lines in the right pane and the commit (file) contents on the left

    • Authors - number of authors of a block and the most prominent author (if there are more than one)

      Clicking on the CodeLens toggles Git blame annotations on/off

  • Provides on-demand inline blame annotations with multiple styles

  • Provides Git blame information about the selected line in the status bar
  • Provides a Git blame history explorer to visualize the history of a file or block
  • Provides ability to compare diffs with the working tree as well as with previous versions
  • Provides many configuration settings to allow the customization of almost all Features

git lens


Git History

View git log, file or line History

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install githistory

Features

  • View Git History with graph and details (latest feature)
  • View the details of a commit, such as author name, email, date, committer name, email, date and comments.
  • View a previous copy of the file or compare it against the local workspace version or a previous version.
  • View the changes to the active line in the editor (Git Blame).
  • Configure the information displayed in the list
  • Use keyboard shortcuts to view history of a file or line

git history


Go

Rich Go language support for Visual Studio Code

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install Go

Features This extension adds rich language support for the Go language to VS Code, including:

  • Completion Lists (using gocode)
  • Signature Help (using godoc)
  • Snippets
  • Quick Info (using godef)
  • Goto Definition (using godef)
  • Find References (using guru)
  • File outline (using go-outline)
  • Workspace symbol search (using go-symbols)
  • Rename (using gorename)
  • Build-on-save (using go build and go test)
  • Lint-on-save (using golint or gometalinter)
  • Format (using goreturns or goimports or gofmt)
  • Generate unit tests squeleton (using gotests)
  • Add Imports (using gopkgs)
  • [partially implemented] Debugging (using delve)

go


Javascript ES6 Snippets

Code snippets for JavaScript in ES6 syntax

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install JavaScriptSnippets

Supported languages (file extensions) JavaScript (.js)
TypeScript (.ts)
JavaScript React (.jsx)
TypeScript React (.tsx)


Language Stylus

Stylus language support

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install language-stylus

Features

  • Syntax highlighting
  • Symbols provider
  • Completion for selectors, properties, values, variables, functions etc.

stylus


NPM Intellisense

Visual Studio Code plugin that autocompletes npm modules in import statements

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install npm-intellisense

npm intellisense

Path Intellisense

Visual Studio Code plugin that autocompletes filenames

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install path-intellisense

Path Intellisense


React Code Snippets

Code snippets for Reactjs development in ES6 syntax

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install ReactSnippets

Supported languages (file extensions)

  • JavaScript (.js)
  • TypeScript (.ts)
  • JavaScript React (.jsx)
  • TypeScript React (.tsx)
  • JSX after installing the corresponding extension

React Code Snippets


Seti icons

Icons for VS Code

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install seti-icons

seti icons


Sort Lines

Sorts lines of text

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install sort-lines

Features

Sort lines of text in Visual Studio Code. The following types of sorting are supported:

  • Sort lines - Regular character code based sort (F9)
  • Sort lines (case insensitive) - Case insensitive sort
  • Sort lines (reverse) - Reverse character code based sort
  • Sort lines (unique) - Regular character code keeping only unique items

Sort Lines


Firewatch Theme

A Firewatch theme for Visual Studio Code ported from Sublime Text Port

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install theme-firewatch

Firewatch


Babel ES6/ES7

Adds JS Babel es6/es7 syntax coloring

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install vscode-babel-coloring

Babel ES6/ES7


Color Highlight

Highlight web colors in your editor

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install color-highlight

Color highlight


ESLint

Integrates ESLint into VS Code.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install vscode-eslint


Jumpy

Jumpy provides fast cursor movement, inspired by Atom's package of the same name.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install vscode-jumpy

jumpy


SVGViewer

SVG Viewer for VSCode.

Installation

Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.

ext install vscode-svgviewer

SVGViewer SVGViewer2

Webpack-dev-server: Where does your CSS go?

September 14, 2016 0 Comments mini-post, webpack, webpack-dev-server, css

You might have noticed when using webpack-dev-server that the CSS seems to appear on the page as if by magic. If you have a look at the build manifest, you'll see your main.js file listed but no sign of the CSS bundle.

This may lead you to think that webpack-dev-server somehow magically inserts your CSS, meaning that if you wanted to check what CSS was built, you'd have to build the actual static file.

Infact, webpack inserts your CSS as a blob in a link tag in the head. It should look something like this:

<link rel="stylesheet" href="blob:http://localhost:3000/3432f4f3-4a5e-4c50-9b3c-01de2dfc5227">

This is an acutal link and if you click on it, you'll be able to see all the compiled CSS just as if it was a static file.