Using Ansible to build a high availablity Nzbget usenet downloader

I’m limited to about 80MB/s on downloads on my VPC at Digital Ocean, but I run Nzbget for downloading large files from usenet. It doesn’t take long to download at all, but out of curiosity I wanted to see if I could parallelize this and download multiple files at the same. I use Sonarr for searching usenet for freely distributable training videos which then sends them to NZBget for downloading. Since Sonarr can send multiple files to nzbget which get queued up, I figured I can reduce the queue by downloading them at the same time.

Using Ansible and Terraform (devops automation tools), I can spin up VPC on demand, provision them, configure them as nzbget download nodes and then destroy the instances when complete.

The instances all run the same nzbget config and the instances use haproxy for round-robin distribution. I will probably change this to Consul, but I just wanted something quick so I used a basic haproxy config.

Terraform builds 4 nzbget, 1 haproxy, and 1 ELK instance. It configures a VIP which I point Sonarr to. Here’s the terraform config that builds an nzbget server:

resource "digitalocean_droplet" "nzbget1" {s
    image = "centos-7-x64"
    name = "nzbget1"
    tags = ["nzbget"]
    region = "nyc1"
    size = "2gb"
    private_networking = true
    ssh_keys = [
      "${var.ssh_fingerprint}"
    ]
connection {
      user = "root"
      type = "ssh"
      private_key = "${file(var.pvt_key)}"
      timeout = "2m"
}
provisioner "remote-exec" {
    inline = [
      "export PATH=$PATH:/usr/bin",
      "sudo yum -y install epel-release"
    ]
  }
}

Here is a terraform play to provision 6 new hosts (1 Elasticsearch, 1 HAproxy and 4 Nzbget nodes):

I run a script which takes the IPs/node names from the terraform output and updates my local /etc/hosts file, my ansible hosts file, the haproxy.cfg.j2 ansible template and my ssh_config file (optional, for convenience).

Secrets, API keys and Passwords
I keep my API keys and passwords in my ansible vault. This keeps them AES encryped but still allows Ansible to put them into the config file templates for nzbget. The API keys are for the Usenet providers and the passwords are for accessing the Web UI.

Here is the ansible playbook that configuress the nzbget instances:

#!/usr/bin/env ansible-playbook
---
- hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: TASK | Create usenet group
      group:
        name: usenet
        state: present

    - name: TASK | Create usenet user
      user:
        name: usenet
        shell: /bin/bash
        groups: usenet
        append: yes
        password: *** REMOVED ***
        update_password: on_create

    - name: TASK | Create /apps directory
      file:
        path: /apps
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Create /apps/data/nzbget directory
      file:
        path: /apps/data/nzbget
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Install EPEL Repo
      action: >
        yum name=epel-release state=present update_cache=yes

    - name: TASK | Install packages
      action: >
        yum name={{ item }} state=present update_cache=yes
      with_items:
        - vim
        - git
        - mlocate
        - par2cmdline
        - p7zip
        - unzip
        - tar
        - gcc
        - python-feedparser
        - python-configobj
        - python-cheetah
        - python-dbus
        - python-devel
        - screen
        - vim
        - htop
        - iftop
        - bind-utils
        - tree
        - jq
        - telnet
        - lsof
        - tcpdump
        - nload
        
    - name: TASK | Install unrar
      yum:
        name: ftp://rpmfind.net/linux/dag/redhat/el7/en/x86_64/dag/RPMS/unrar-5.0.3-1.el7.rf.x86_64.rpm
        state: present

    - name: TASK | Install yenc
      yum:
        name: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/22/Everything/x86_64/os/Packages/p/python-yenc-0.4.0-4.fc22.x86_64.rpm
        state: present

    - name: TASK | Change owner of Nzbget to usenet
      file:
        path: /apps
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Install systemd unit file for Nzbget
      template: src="templates/nzbget.service.j2" dest="/etc/systemd/system/nzbget.service" mode=0644


    - name: TASK | Copy Nzbget config
      template: src="templates/nzbget.conf.j2" dest="/apps/nzbget/nzbget.conf" mode=0644

    - name: TASK | Enable and start Nzbget
      systemd:
        name: nzbget
        enabled: yes
        state: started

    - name: TASK | Install filebeat
      yum: 
        name: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-x86_64.rpm
        state: present
        update_cache: yes

    - name: TASK | Copy filebeat config file
      template: src="templates/filebeat.yml.j2" dest="/etc/filebeat/filebeat.yml" mode=0644

    - name: TASK | Enable and start filebeat
      systemd:
        name: filebeat
        enabled: yes
        state: restarted

Next steps
The next part of this post will be automatically provisioning a Digital Ocean volume and setting it up as an NFS share. Then I can mount the volume on all of the nzbget nodes for centralized storage. Since I pay by the minute for the VPC instances, I want to be able to quickly provision them, download as much as I can as quickly as I can and then destroy the infrastructure. I’ll keep the volume around, but I can destroy the nzbget downloaders once they’ve finished their queues.

Technology used: