Warrant Canaries

Wikipedia defines a warrant canary as:

... a method by which a communications service provider informs its users that the provider has not been served with a secret United States government subpoena.

Practically, this ends up being a file or web location that states something to the effect of, "As of date, we have not received a subpoena." The notice usually includes a disclosure stating that no warrants have been served to the entity or its employees and no searches or seizures have been performed on the entity or the employees of the entity's assets. It will also include a date as to when the notice was updated and may also include links to external websites with time-relevant information such as news articles, major headlines, tweets, etc.

The most important part of the warrant canary is the signature and signed content. All of the above information is cryptographically signed, and the public key made available to verify the signature. The act of signing the notice increases the difficulty in forging a warrant canary.

There are many cases where warrant canaries exist and are used by commercial and non-commercial entities. One of the oldest and well-known instances is the rsync.net warrant canary. Other examples include:

There is, however, speculation that warrant canaries have questionable legal ground or could be used as an effective way to indirectly communicate said legal action by a government agency or court. At this time, there have been no cases where warrant canaries have been upheld. For more information, see the EFF Warrant Canary FAQ.

Creating A Warrant Canary

Creating a warrant canary is a fairly simple process. It requires just a small amount of time to become familiar with tools like GPG. After creating a warrant canary notice, it can be published by anyone with access to your website.

Before You Begin

To begin, you will need to install GPG and create a signing key. Create the signing key by following the official GPG Getting Started guide. A key is only created once and will be used to update your canary in the future; it is crucial that the same key be used for subsequent canary updates.

Creating The Notice

As with previous examples, the notice should contain the disclosures that are most relevant to your needs as well as information and data that can sufficiently be determined as both accurate and time relevant. This often includes the current date, sports scores, weather information, etc. For example:

It is Friday, December 26th, 2014 at 4:50 pm EST.

To this date no warrants, searches or seizures of any kind have ever been performed on my assets or any assets belonging to members of my household.

Headlines from http://www.npr.org/sections/news/archive?date=12-31-2014
Body Of Catholic Priest Found In Southern Mexico
Businesses Buzz With Anticipation In Wake Of U.S.-Cuba Thaw
Military Policy Impedes Research On Traumatic Brain Injuries
In The Nation's Capital, A Signature Soup Stays On The Menu
Already Bleak Conditions Under ISIS Deteriorating Rapidly

Week 16 NFL Scores
Giants 37 Rams 27
Cols 7 Cowboys 42
Bills 24 Raiders 26
Seahawks 35 Cardinals 6

You can verify this document using the public key 953023D848C35059A2E2488833D43D854F96B2E4.

With your notice saved as warrant_canary.txt, sign it with your GPG key.

$ gpg --clearsign warrant_canary.txt

Running this command will create a file named warrant_canary.txt.asc.

Hash: SHA256

It is Friday, December 26th, 2014 at 4:50 pm EST.

To this date no warrants, searches or seizures of any kind have ever been performed on my assets or any assets belonging to members of my household.

Headlines from http://www.npr.org/sections/news/archive?date=12-31-2014
Body Of Catholic Priest Found In Southern Mexico
Businesses Buzz With Anticipation In Wake Of U.S.-Cuba Thaw
Military Policy Impedes Research On Traumatic Brain Injuries
In The Nation's Capital, A Signature Soup Stays On The Menu
Already Bleak Conditions Under ISIS Deteriorating Rapidly

Week 16 NFL Scores
Giants 37 Rams 27
Cols 7 Cowboys 42
Bills 24 Raiders 26
Seahawks 35 Cardinals 6

You can verify this document using the public key 953023D848C35059A2E2488833D43D854F96B2E4.
Comment: GPGTools - https://gpgtools.org


That file is your current warrant canary and should be made available as you see fit. The most common url used to present your canary is "/canary". In this case, the canary is available at http://ngerakines.me/canary.

Next Steps

With your canary online and available, you'll need to be sure that the signing key used to sign the notice is also available. Please refer to the Exchanging Keys and Distributing Keys documentation to export your key to share with others and make available through a GPG key server.

It may also be in your interest to have third parties verify your key and identity. This allows other key owners to demonstrate trust. More information can be found on the GPG documentation: Validating other keys on your public keyring.

Securing Your Canary

When a canary is not updated or is removed, it means that several things may have happened.

The first is simply human error. For safety and security purposes, the act of signing a warrant canary is a manual process. That means that a human has to be at a computer and run the commands to create and sign the warrant canary notice. There are plenty of reasons from sickness to changing companies and even simply forgetfulness that could be the reason why a canary is not updated.

It could also mean that the the entity no longer wants to include the notice. A change in management or ownership could result in the canary is neglected or removed.

Lastly, it could mean that harm, detention or a lack of control is preventing the canary from being updated.

A watchdog or Dead man's switch can be used mitigate damage or loss of data or reputation.

Dead Man Switch: Revocation

Using a revocation certificate, a trusted third party can publicly revoke the key used to cryptographically sign canaries.

A core component of the warrant canary is the signature. The signature is used to determine that the contents of the canary have not been tampered with and provide a way to identify the owner of the signing key through the web of trust. When the GPG key used to sign the canary is created, a revocation certificate should be created along with it.

If you forget your passphrase or if your private key is compromised or lost, this revocation certificate may be published to notify others that the public key should no longer be used. -- The GNU Privacy Handbook

A revocation certificate can be securely given to a trusted third party responsible for publishing the revocation certificate under certain conditions.

Conditions could range from:

  • The warrant canary not being updated after a certain period of time.
  • Unusual behavior or contact with the company.
  • A cue or hint that it should be done so through information contained in a dead drop or press release.

Multiple Signers

As a way to reduce the risk of human error from raising false concern, multiple signers can sign a canary or multiple canaries can be published used. The most common way to do this would be to have two or more members of an organization create signatures of the canary and append it to the notice.

This can be done by creating one or more detached signatures along with the canary.

$ gpg --output canary.sig1 --detach-sig canary

When the above command is run, a file named canary.sig1 is created that contains a signature of the canary file. You can publish these additional signatures along-side the canary or append them to the bottom of the canary file.

IOT and Home automation, 10 months later

In December of 2013 I was given a SmartThings kit and that kicked off a home automation project. I didn't go all-out and try to automate all the things, but instead tackled a single area, specifically my home office. Nearly 10 months later, I hardly think about it but use it every single day.

  • If the office lights are on and I leave the house, the lights are automatically turned off.
  • When I enter the house from being away, turn on the lights in the office.
  • When I'm at home, but there isn't any movement in my office after 15 minutes, turn the lights off automatically.
  • When I'm home, there is movement in my office and the lights are off then turn the lights on.

This started with some z-wave light switches and the SmartThings kit. I use a motion/presence detector (battery powered) that detects movement in the office. The z-wave light switches are used to software control the lights and then my phone is connected to the SmartThings system to determine when I'm nearby.

That isn't the only system in place, but it is the one that I use the most. Additionally, I've got motion sensors in other parts of the house, including on the garage door, and presense fobs in the cars. I also have a nest installed as well as door locks that support z-wave.

So what is next? I'd like to get the rest of the light switches in the house replaced with z-wave switches and find a way to automate the garage door. Having an "away mode" able to turn off all of the lights, lock the doors and set the nest tempature would be nice too. Maybe someday.


Clean Build Versions With DocOpt

With some recent Go projects, I've been using docopt.go for command line argument parsing. It greatly reduces the complexity of dealing with arguments and options. On it's own, arguments can be processed without much work:

package main

import (

var (
    githash string = ""

func main() {
    usage := `Awesome

Usage: awesome [--help --version --config=<file>]
       awesome daemon [--help --version --config <file>]
       awesome thing [--verbose... ] <with> <more>...

  --help     Show this screen.
  --version  Show version.
  --verbose  Verbose

    arguments, _ := docopt.Parse(usage, nil, true, version(), false)
    // ... do something with arguments

func version() string {
    previewVersion := "1.0.0"
    if len(githash) > 0 {
        return previewVersion + "+" + githash
    return previewVersion

What I've also been doing is using a small Makefile to add extra information to the version:

    go build -ldflags "-X main.githash `git rev-parse --short HEAD`"

If the go build command is used, then the version given to docopt is just 1.0.0, but if the main.githash is set as it is in the Makefile, then the version ends up being something like "1.0.0+b74276b".

$ ./awesome --version

You can take it one step further and have the version set as a var that can easily be updated with sed or awk. An example would look something like:

package main

var (
    AWESOME_VERSION = "1.0.0"

Using awk to update the version would look like this:

$ awk '/AWESOME_VERSION/ { sub($3, "\"2.0.0\""); print; next}1' version.go
package main

var (
    AWESOME_VERSION = "2.0.0"

Secure Email On OSX

There are a few really good articles out there on how to send and receive secure email using Thunderbird and GPG. This small guide will show you how you can use Mail app along with GPG Tools for the same result.

Install GPG Tools

First, head over to https://gpgtools.org/ and download the latest release. At the time of this article, the latest stable release is 2.1.

To start off on the right foot, before you install it, open up terminal and verify that you've downloaded a package that matches the following signature: ac7a636bfee1027d8f43a12a82eea54e7566dcb8. This can be accomplished with the following commands:

$ cd ~/Downloads
$ shasum GPG\ Suite\ -\ 2013.10.22.dmg
ac7a636bfee1027d8f43a12a82eea54e7566dcb8  GPG Suite - 2013.10.22.dmg

Once you can verify that the dmg file that you've downloaded hasn't been tampered with during transfer, go ahead and open it and run through the install process. This will install the base GPG tools, a graphical key manager and a plugin for Mail.

Create A Key

Next you'll want to create a key. Open the newly installed "GPG Keychain Access" application and click the "New" button to create a key. You'll be prompted for your full name and email address, which you should fill in. Be sure to also check the box to have the public key uploaded once generated. Having accurate information is vital and if this is your first time going through this process. I strongly recommend setting the comment to your website or twitter under the advanced options.

Next, you'll be prompted to set a password for your key. Choose a strong password. Depending on your system, it may take a few moments for the key to be generated after your password is accepted. Don't be alarmed.

Configure Mail

There shouldn't be anything extra needed to send and receive encrypted and or signed email through the Mail app now. In the Mail app preferences is a "GPGMail" section that should indicate that GPGMail is ready for use. It is set to encrypt/sign drafts and sign all new messages by default.

Test Sending Signed Mail

From Mail, create a new message to send to a loved one, friend, coworker or the like. Once you fill in the To, Subject, and Body, ensure that the message is signed by clicking the checkmark box button within the new mail window. If you have Mail configured to sign by default, you may be prompted within a few seconds to give the password for the key.

It is important to note that you can sign outbound email to anyone, but you can only encrypt email messages to people who have given you their public key. This is where the GPG Keychain Access app comes into play.

With the GPG Keychain Access app you can also import key files given to you and search for keys for people you may know. If someone sends you their public key you can use the "import" feature to load the key into your keyring. Alternatively, if you know the email address or name, you can attempt to search for keys associated with them on public key servers.

When composing emails to addresses that have public keys associated with them, you'll have the option of encrypting the email messages being sent. If you don't have any other public keys in your key ring, you can test this by sending an encrypted email to yourself.


Guard your private key. It is critical that you ensure your private key is safe and secure. For everyday use, keeping it on a personal, non-public computer is probably enough. If you feel that a computer that has your private key on it has been compromised, infected by a virus or malware, etc then you revoke the key and create a new one.

Find a thumbdrive that you don't use and back up your keyring to it. This should also include a revocation certificate. A revocation certificate will allow you to revoke the key if the key is lost or compromised.

When backing up your private key, consider using symmetric encryption using a password to encrypt the backup file. This can be done with GPG using the following command:

$ cd /Volumes/thumbdrive
$ gpg --output backup.zip.gpg --symmetric backup.zip

When you need to decyrpt your backup, you can use the following command:

$ gpg --output backup.zip -d backup.zip.gpg

When publishing your key on your blog or website, you can export a plain text version of your key that can be read as text and imported easily using the following command:

gpg --armor --export person@wherever.place

The output of that command can be placed inside of a pre block as-is. It is the most direct way to share your key wih someone viewing your blog or website. An alternative would be to create a small signature block telling people how to find your key.

$ gpg --fingerprint nick@gerakines.net
pub   4096R/4F96B2E4 2013-06-15
      Key fingerprint = 9530 23D8 48C3 5059 A2E2  4888 33D4 3D85 4F96 B2E4
$ gpg --clearsign

You need a passphrase to unlock the secret key for
user: "Nick Gerakines (http://ngerakines.me/) <nick@gerakines.net>"
4096-bit RSA key, ID 4F96B2E4, created 2013-06-15

9530 23D8 48C3 5059 A2E2  4888 33D4 3D85 4F96 B2E4

Hash: SHA256

9530 23D8 48C3 5059 A2E2  4888 33D4 3D85 4F96 B2E4

Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org


Chef Application Cookbooks

This is a follow up to the blog post Creating A Chef Cookbook. Since writing that blog post, I created the preview project and several cookbooks for it. With it, I've done a few things differently and believe that they represent some notable trends in the chef community.

Embed Application Cookbooks in Application Repositories

Instead of having a separate git repository for each cookbook, all of the cookbooks for the preview application are in the preview git repository in the 'cookbooks' directory.

Practically, this doesn't change anything for the cookbook itself. In projects that reference this cookbook that use berkshelf, I have to update the Berksfile to point to the project repository and the subdirectory that contains the cookbook. When a cookbook is uploaded to a chef server, the location is irrelevant.

There is one thing that I do want to make very clear: The application build process doesn't build or prepare the cookbooks. The cookbooks are independently built, tested and released. I've seen projects where the cookbook is "generated" as part of the build process for the application and I feel strongly against that.

Create Environment Cookbooks

To learn more about the environment cookbook pattern, read the The Environment Cookbook Pattern written by Jamie Winsor.

In addition to the primary "preview" application cookbook, I created an environment cookbook called "preview_prod". This cookbook is used to represent the default configuration, files and actions needed to release the preview application into a production-like environment.

When looking at environment cookbooks, it is really important to note that these don't contain node information and attributes, but rather represent what a configuration of the application cookbook looks like in a given environment.

In the case of the preview_prod/metadata.rb file, I list several attributes that are required by the cookbook:

name             'preview_prod'
maintainer       'Nick Gerakines'
maintainer_email 'nick@gerakines.net'
license          'MIT'
description      'Installs/Configures preview_prod'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          '0.2.2'

depends 'preview'

supports 'centos'

recipe 'preview_prod::node', 'Configures and prepares a preview application node.'
recipe 'preview_prod::storage', 'Configures and prepares a storage node.'

attribute 'preview_prod/node_id',
  :display_name => 'The id of the preview node.',
  :required => 'required',
  :type => 'string',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/cassandra_hosts',
  :display_name => 'The cassandra hosts used by the preview node.',
  :required => 'required',
  :type => 'array',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/edge_host',
  :display_name => 'The base url used to request assets from the cluster.',
  :required => 'required',
  :type => 'string',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/s3Key',
  :display_name => 'The S3 key used to store generated assets.',
  :required => 'required',
  :type => 'string',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/s3Secret',
  :display_name => 'The S3 secret key used to store generated assets.',
  :required => 'required',
  :type => 'string',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/s3Host',
  :display_name => 'The S3 host used to store generated assets.',
  :required => 'required',
  :type => 'string',
  :recipes => ['preview_prod::node']

attribute 'preview_prod/s3Buckets',
  :display_name => 'The S3 buckets used to store generated assets.',
  :required => 'required',
  :type => 'array',
  :recipes => ['preview_prod::node']

Even though the preview_prod::node and preview_prod::storage recipes describe how to create production-like preview cluster nodes separately, the preview_prod::default exists to allow engineers to deploy to a single, full-stack node. This follows the idea that the default recipe's purpose should be to represent the most common and simple use for engineers that are new to the cookbook.

In the preview_prod::node recipe, we are using the preview_prod required and unsatisfied attributes to override attributes that have default values in the preview cookbook:

node.override[:preview][:config][:common][:nodeId] = normal[:preview_prod][:node_id]
node.override[:preview][:config][:storage][:engine] = 'cassandra'
node.override[:preview][:config][:storage][:cassandraKeyspace] = 'preview'
node.override[:preview][:config][:storage][:cassandraKeyspace] = normal[:preview_prod][:cassandra_hosts]
node.override[:preview][:config][:simpleApi][:edgeBaseUrl] = normal[:preview_prod][:edge_host]
node.override[:preview][:config][:uploader][:engine] = "s3"
node.override[:preview][:config][:uploader][:s3Key] = normal[:preview_prod][:s3Key]
node.override[:preview][:config][:uploader][:s3Secret] = normal[:preview_prod][:s3Secret]
node.override[:preview][:config][:uploader][:s3Host] = normal[:preview_prod][:s3Host]
node.override[:preview][:config][:uploader][:s3Buckets] = normal[:preview_prod][:s3Buckets]

include_recipe 'preview::default'

Your mileage may vary in terms of what a production cookbook should look like. The preview project is open source and public, but for internal environment cookbooks you may have default values or databag references for attribute values.

Build Cookbook

This is another pattern that I'm using at work and really like: Creating a cookbook to bootstrap a development environment. Again, this is another take on the environment cookbook pattern

Specifically for this project, this cookbook installs the version of the golang compiler required to build the preview application as well as tools git. In the preview_build/recipes/default.rb file, this looks like:

include_recipe 'golang::default'

node.default['go']['packages'] = ['github.com/gpmgo/gopm']

include_recipe 'golang::packages'

The preview project is open source and public, so I'm using travis-ci (https://travis-ci.org/ngerakines/preview) to compile the application and run the short tests. The build cookbook pattern is useful if you've got a build environment and CI that has a build agent. The cookbook would be applied to the build agent and the chef-client command executed at the beginning of the build agent run to ensure that it is up to date.

For a disposable build environment, we can use environment variables to create a GOPATH dynamically:

GOPATH=gopath-`date +%s`
echo "export GOPATH=$GOPATH" > env-gopath

Then, your commands would look like:

$ . path/to/env-gopath
$ go get ./...
$ go build
$ go test ./... -test.short
$ rm -rfv $GOPATH && env-gopath

Practically, it makes sense to use something like gopm to fetch specific versions of the packages used. The above script could be updated to use gopm instead

Application Integration Test Cookbook

For this project, I took it one step further and created an additional cookbook called preview_test that contains recipes, configuration and files to run integration tests. This cookbook is still heavily in development as I'm using it to learn how to effectively use chef-metal and kitchen-metal. I'll put up another blog post when I've got something demonstrable.