I've been an ELK Stack (Elasticsearch, Logstash, Kibana, and Beats) user for quite some time, using exclusively the open source version fo the stack. Generally it's works well and, with some exceptions, supports our mostly-Solaris based environment (using LX zones to host most of the beefier components, and using custom-built …

After 18 months of using Bacula and sending copies of my data to the cloud (in this case, cloud I operate in another location) using an S3-compatible storage mechanism, I noticed I had a lot of data sitting around on my current server for backups. When I set out to …

Back in September of last year, I wrote in Bacula: 6 months on that cloud backups required part.0 in order to be recognized for automatic part retrieval. While this was mostly accurate, the critical file is actually part.1. As such, when referencing my own blog post when trimming …

In June, I mentioned in an article about Docker on SmartOS that we are doing some work with GitLab these days as a replacement for my venerable Gitolite server (and, to an increasing extent Jenkins). Deploying from Pelican I'm likely going to write more on GitLab in the near future …

This spring, there was a some movement on the Illumos/SmartOS front in implementing features to better support running LX zones with Linux variants. Since Docker images (generally) run on Linux underpinnings, support for running Docker images on SmartOS are dependent upon this support working correctly. For those familiar with …

As I've possibly mentioned here before, ClueTrust is using Elasticsearch to perform analysis of log information. Recently, I finally decided to take some our telemetry inforamtion and pull it in to Elasticsearch as a data exploration and statistical tool. Importing structured XML data into Elasticsearch Although there are some filters …

Quite a while back, RS wrote a comprehensive ansible role for handling Let's Encrypt certificate issuance and renewal. We both use this role extensively, which is why it was a significant issue when it suddenly started throwing type errors deep inside of the dnspython library during an nsupdate call in …

Originally this was going to contain a brief Bacula, 6 months on section at the start. Of course, that became much too detailed, so I split them up, however I would encourage you to read it. Restore Testing Backup is the most obvious part of doing backups. Almost everyone's aware …

It's been about six months since I originally wrote Welcome Bacula, describing our transition to Bacula from our previous solution (and a bit of history even before that). If you haven't read it, it might be worth a read. Although not quite 6 months since I wrote the first piece …

We've heard it all before: AWS is expensive, and you need to watch out for the hidden sharp edges of their pricing model. Today I provide a small lesson in that concept. History ClueTrust has run through a number of backup methodologies over the year, originally using Retrospect (when they …

The background Many years ago, all of my systems were pets. I tried to make them easier to manage by standardizing on a single operating system (MacOS X Server at the time) and used management tools that were part of that suite. As time moved forward, Apple decided to concentrate …

I wasn't originally going to write this up on the blog, but considering that we've just finished our transition from our old backup software (BRU, no link) to Bacula community edition and considering that it's World Backup Day, it seemed like it would make sense. As many of you are …

This weekend, Rob and I had been testing the use of hardware keys to secure ssh sessions, especially for back-end console access and certain administrative functions. Since the hardware keys are a special case, and cannot be added to the ssh-agent, we were slinging around a fair number of command …

I've been doing a bunch of maintenance on my two blogs (company and personal) and one purpose has been to track down malformed and mis-mapped URLs on the site. Since both have been through changes in the underlying blog engine a couple of times, there are multiple sets of URLs …

Why client certificates? RS wrote about Preventing drive-bys with client certs and although we'd discussed this method for some time, I hadn't deployed it yet. However, some recent log-spelunking had led me to determine that I liked the idea of a second layer of protection on some of my sites …

OK, now that I have your attention with the catchy title, let me get right into the reason behind this post. Rob has been doing a lot of work lately on a set of roles to provision raspberry pi systems. I'm grateful for the work in this area, because frankly …

At ClueTrust, we use a lot of automation to run our systems. It's mostly how just a couple of us can manage hundreds of virtual servers and keep them up-to-date and operational. A few years back, I moved from using Puppet to Ansible, mostly at the suggestion of RS, who …

Over the past few weeks, RS and I have noticed an increasing number of unexplained failures logging in over SSH with both manual and automated means. On most of the servers it was just an inconvenience, but as it started to become more frequent, it became a significant issue for …

When I set out to move Gaige's Pages to a static web generator, chronicled in Gaige's Pages moves to static generation, I stated one of the reasons that I favored Pelican was because it is written in python, which is a language that I'm intimately familiar with. Not surprisingly, that …

As part of completing our shut-down of 2007-vintage Xserves at the hosting center, we're moving a lot of servers to SmartOS (or at least SmartOS-hosted VMs). We've been really happy with the system so far. Here's a quick story of the power of this environment. As part of the transition …