Remote read-only storage. NGINX + Furumi

Prerequisites

  • Linux server with data. I use Ubuntu 20.04
  • Linux box as a client. I use Arch Linux

Server setup

Server uses NGINX to index and serve data. Config looks like this:

server {
    listen 80;
    listen [::]:80;
    server_name music;
    root /storage/Torrents/MUS;
    location / {
        include custom/auth.conf;
        autoindex on;
        autoindex_format json;
        try_files $uri $uri/ =404;
    }
}

Client setup

Furumi mount FUSE over HTTP. Get Arch package or binary on Release page. Install FUSE support:

Ubuntu: sudo apt install fuse

Arch: sudo pacman -S fuse

Uncomment user_allow_other in /etc/fuse.conf. Update server and auth in /etc/furumi.yml If using binary release copy Systemd unit manually from latest release. Arch pkg installs everything automatically.

Music Library

Deprecated and discontinued. Use NGINX + Furumi HTTP Fuse

Preface

I’ve used Goolge Music, Youtube Music, Yandex.Music & Spotify. The last one is the best, but i can’t control it so I’ve made self hosted service. First of all I managed to setup NFS share on my Linux Server and share it between my workstations, it’s quite simple and reliably but it’s hard to secure and can’t stand network changes. I come into two solutions. Player way 1. Make an API that can handle my music library and serve clients who want it. 2. Make a player or plugin for existent player as a client for my API. Filesystem way 1. Same as previous #1 2. Make a FUSE app to mount my library over HTTPS securely.


API was made by @nixargh. I’ve tried player way first, you can find it in GitHub taihen player. Player works somehow but i do not support it anymore. Later I’ve made FUSE app mus-fuse it works exactly i want it to. With any player, i use Cmus btw.

Prerequisites

  • Linux server with music. I use Ubuntu 18.04
  • Linux box as a client. I use Arch Linux

Setup server

Precompiled binary is available on release page playongo. Also you can compile it according to README.md. Scan music library, it takes some time:

$ ./playongo -musicDir ~/Music -scan

Create user for running server

$ sudo useradd -d /storage/MUS/ -c "playongo server" -U -r playongo 

Create systemd unit /lib/systemd/system/playongo.service for managing API app. Unit file assumes that playongo binary located in /opt/playongo/playongo and music is stored in /storage/MUS/.

[Unit]
Description=Playongo music library service
[Service]
Type=simple
User=playongo
Group=playongo
Restart=always
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
WorkingDirectory=/opt/playongo
ExecStart=/opt/playongo/playongo -musicDir /storage/MUS/
ExecStop=/usr/bin/killall -TERM /opt/playongo/playongo
[Install]
WantedBy=multi-user.target

Enable and start unit:

$ sudo systemctl enable playongo.service
$ sudo systemctl enable playongo.service
# Check if it works. It should show your music library as JSON.
$ curl http://localhost:12345/songs | less 

Also it’s a good idea to run Nginx as a proxy and SSL terminator, here is an example of setup, make sure you do have SSL certificates. If don’t so create a new certificate from Let’s Encrypt immediately. Also mind that auth_basic is enabled. It’s strongly recommend to use it but you can disable it by commenting any auth_basic* directives. How to create .htpasswd Do not use CloudFlare because low latency is a decisive factor here.

server {
    listen 80;
    listen [::]:80;

    server_name mus.hexor.ru;

    gzip on;
    gzip_min_length 256;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_vary on;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://127.0.0.1:12345;
        proxy_redirect off;
        auth_basic           "Administrator’s Area";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }

### SSL PART
    listen [::]:443 ssl ipv6only=on; 
    listen 443 ssl; 
    ssl_certificate /etc/letsencrypt/live/mus.hexor.ru/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/mus.hexor.ru/privkey.pem; 
    include /etc/letsencrypt/options-ssl-nginx.conf; 
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 
### END OF SSL PART

}

Client side

As FUSE is using for mus-fuse it should be installed, fuse package for Arch Linux and Ubuntu. Precompiled binary is available on release page mus-fuse. Also you can compile it according to REAMDE.md.

Mus-fuse can be run like that:

$ export HTTP_USER=user
$ export HTTP_PASS=pass
$ export RUST_LOG=info
$ ./mus-fuse <address> <mountpoint>
# address is your server with playongo with protocol (https|http)

Systemd unit file may be like this:

[Unit]
Description=Mount mus-fuse

[Service]
User=ab
Type=simple
Environment=HTTP_USER=user
Environment=HTTP_PASS=pass
ExecStart=/usr/bin/mus-fuse /MUS https://mus.hexor.ru
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

#linux #golang #rust #music #cmus #selfhosted

Calendar server

Preface

Remote calendar applications use CalDav protocol to sync events, it work over HTTP. I use Radicale server.

Prerequisites

  • Linux server with python3-pip, Nginx installed.

Installing

Use pip for fetch Radicale server. Create user for service and grant permissions.

$ sudo python3 -m pip install --upgrade radicale
$ sudo useradd -d /srv/cal/ -c "Radicale server" -U -r radicale 
$ sudo mkdir -p /srv/cal
$ sudo chown -R radicale:radicale /srv/cal

Configuration

Main config file: /etc/radicale/config

[server]
# Bind localhost only
hosts = 127.0.0.1:5232

[auth]
type = None
htpasswd_filename = /path/to/users
htpasswd_encryption = bcrypt
[storage]
filesystem_folder = /srv/cal

I turned off internal auth in favor of Basic Auth in Nginx Proxy.

Simple systemd unit file my be like this: /lib/systemd/system/radical-server.service

[Unit]
Description=Radicale CalDav server for self-hosted calendars.
[Service]
Type=simple
User=radicale
Group=radicale
Restart=always
StartLimitInterval=60s
StartLimitBurst=3
WorkingDirectory=/srv/cal
ExecStart=/usr/bin/python3 -m radicale --config /etc/radicale/config
ExecStop=/usr/bin/killall -TERM /usr/bin/python3 -m radicale --config /etc/radicale/config
[Install]
WantedBy=multi-user.target

Enable and run server:

$ sudo systemctl enable radical-server.service
$ sudo systemctl start radical-server.service

Nginx Proxy Settings

Nginx config file:

server {
    listen 0.0.0.0:80;
    listen [::]:80;

    server_name cal.hexor.ru;

    gzip on;
    gzip_types
      application/javascript
      application/x-javascript
      application/json
      application/rss+xml
      application/xml
      image/svg+xml
      image/x-icon
      application/vnd.ms-fontobject
      application/font-sfnt
      text/css
      text/plain;
    gzip_min_length 256;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_vary on;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://127.0.0.1:5232;
        proxy_redirect off;
        auth_basic           "Administrator’s Area";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

Don’t forget to change server_name and restart Nginx.

How to create /etc/nginx/.htpasswd

Create calendar

Open https://<server_name> in browser using credentials from .htpasswd and create a new calendar.

Setting up Android calendar

I use OpenSync app. Install app and create account like this: app-screenshot

And it works

Cloud Torrent

Steps

  • Install Transmission sudo apt install transmission-daemon
  • Update config
cd /var/lib/transmission-daemon/.config/
sudo vim transmission-daemon/settings.json

Pay attention to: download-dir rpc-host-whitelist – specify domain via you going to access Cloud Torrent

{                                                                       
    "alt-speed-down": 50,                                               
    "alt-speed-enabled": false,                                         
    "alt-speed-time-begin": 540,                                        
    "alt-speed-time-day": 127,                                          
    "alt-speed-time-enabled": false,                                    
    "alt-speed-time-end": 1020,                                         
    "alt-speed-up": 50,                                                 
    "bind-address-ipv4": "0.0.0.0",                                     
    "bind-address-ipv6": "::",                                          
    "blocklist-enabled": false,                                         
    "blocklist-url": "http://www.example.com/blocklist",                
    "cache-size-mb": 4,                                                 
    "dht-enabled": true,                                                
    "download-dir": "/storage/MUS",                                     
    "download-queue-enabled": true,                                     
    "download-queue-size": 5,                                           
    "encryption": 1,                                                    
    "idle-seeding-limit": 30,                                           
    "idle-seeding-limit-enabled": false,                                
    "incomplete-dir": "",                                
    "incomplete-dir-enabled": false,                                    
    "lpd-enabled": true,                                                
    "message-level": 1,                                                 
    "peer-congestion-algorithm": "",                                    
    "peer-id-ttl-hours": 6,                                             
    "peer-limit-global": 200,                                           
    "peer-limit-per-torrent": 50,                                       
    "peer-port": 50100,                                                 
    "peer-port-random-high": 65535,                                     
    "peer-port-random-low": 49152,                                      
    "peer-port-random-on-start": false,                                 
    "peer-socket-tos": "default",                                       
    "pex-enabled": true,                                                
    "port-forwarding-enabled": true,                                    
    "preallocation": 1,                                                 
    "prefetch-enabled": true,                                           
    "queue-stalled-enabled": true,                                      
    "queue-stalled-minutes": 30,                                        
    "ratio-limit": 2,                                                   
    "ratio-limit-enabled": false,                                       
    "rename-partial-files": true,                                       
    "rpc-authentication-required": false,                               
    "rpc-bind-address": "127.0.0.1",                                    
    "rpc-enabled": true,                                                
    "rpc-host-whitelist": "*hexor.ru",                                  
    "rpc-host-whitelist-enabled": true,                                 
    "rpc-password": "",
    "rpc-port": 9091,                                                   
    "rpc-url": "/transmission/",                                        
    "rpc-username": "",                                                 
    "rpc-whitelist": "127.0.0.1",                                       
    "rpc-whitelist-enabled": true,                                      
    "scrape-paused-torrents-enabled": true,                             
    "script-torrent-done-enabled": false,                               
    "script-torrent-done-filename": "",                                 
    "seed-queue-enabled": false,                                        
    "seed-queue-size": 10,                                              
    "speed-limit-down": 100,                                            
    "speed-limit-down-enabled": false,                                  
    "speed-limit-up": 100,                                              
    "speed-limit-up-enabled": false,                                    
    "start-added-torrents": true,                                       
    "trash-original-torrent-files": false,                              
    "umask": 18,                                                        
    "upload-slots-per-torrent": 14,                                     
    "utp-enabled": true                                                 
}
server {
    listen [::]:443 ssl http2;
    listen 443 ssl http2;
    server_name <HOSTNAME>;
    include security.conf;
    include letsencrypt.conf;
    location / {
        proxy_pass http://127.0.0.1:9091;
        include proxy.conf;
        auth_basic           "Administrator’s Area";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
    access_log /var/log/nginx/<HOSTNAME>.access.log custom;
    error_log /var/log/nginx/<HOSTNAME>.error.log;         
}
  • It works. But standard web gui is awful, so I use Custom WEB GUI. It installs via downloading shell script and executing it. Usually it’s bad, so it’s a good idea to revise this script and make decision execute it or not.
chown -R debian-transmission:debian-transmission \
/usr/share/transmission
sudo -u debian-transmission \
wget https://github.com/ronggang/transmission-web-control/raw/master/release/install-tr-control.sh \
--no-check-certificate -O /tmp/install-tr-control.sh
sudo -u debian-transmission \
bash /tmp/install-tr-control.sh

KVM/QEMU self hosted hypervisor

Requirements:

  • Ubuntu Linux server
  • CPU with virtualisation support

Configuring

Install vt staff

$ sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils

Bridging

In order for your virtual machines to access your network interface and be assigned their own IP addresses, we need to configure bridged networking on our system. Let’s utilise netplan utility. /etc/netplan/00-installer-config.yaml

network:
  ethernets:
    enp2s0f0:
      dhcp4: false
      dhcp6: false
  bridges:
    br0:
      interfaces: [enp2s0f0]
      dhcp4: true
      dhcp6: true
  version: 2

Generate and apply network config:

$ sudo netplan generate
$ sudo netplan --debug apply

# Check configuration
$ sudo networkctl
IDX LINK       TYPE     OPERATIONAL SETUP
  1 lo         loopback carrier     unmanaged
  2 enp2s0f0   ether    enslaved    configured
  3 br0        bridge   routable    configured
  4 virbr0     bridge   no-carrier  unmanaged
  5 virbr0-nic ether    off         unmanaged

$ sudo ip a 
2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether xxx brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xxx brd ff:ff:ff:ff:ff:ff
    inet 192.168.88.28/24 brd 192.168.88.255 scope global dynamic br0
       valid_lft 535sec preferred_lft 535sec

It works from now, let’s grant permissions to manage virtmanager to user:

$ sudo adduser $USER libvirt-qemu
$ sudo adduser $USER libvirt

Managing VMs

Use virt-manager GUI utility on client or virsh CLI tool for managing VMs and data pools.

#linux #virtualisation #kvm #qemu #selfhosted

Arch Linux Repository

Preface

It’s extremely convenient to have self-hosted repository for your own packages and deliver it to your machines.

Prerequisites

  • Ubuntu Server with Nginx and Docker

Creating repository

Repository database is managed via repo-add script bundled with Arch Linux pacman package manager. Since pacman is not available in Ubuntu repository I use docker archlinux image for managing repository. This guide assumes that repository located in /srv/arch-repo. First of all move all your packages into /srv/arch-repo. Following command will create or update repository database.

REPO_URL=repo.sun.hexor.ru
REPO_PATH=/srv/arch-repo
docker run -v ${REPO_PATH}:/repo --rm archlinux \
bash -c "repo-add /repo/${REPO_URL}.db.tar.gz /repo/*pkg.tar.zst"

Important aspect

Name of the database should be REPO_URL.db.tar.gz, in this case REPO_URL is repo.sun.hexor.ru.

Sharing repository via HTTPS

Nginx config. (See how to setup Nginx)

server {
    server_name repo.sun.hexor.ru;
    listen [::]:443 ssl;
    listen 443 ssl;
    include security.conf;
    include letsencrypt.conf;
    root /srv/arch-repo;
    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
    access_log /var/log/nginx/logs/repo.sun.hexor.ru.access.log custom;
    error_log /var/log/nginx/logs/repo.sun.hexor.ru.error.log;
}

Configure repo on yours machines

Add your repo to /etc/pacman.conf:

[repo.sun.hexor.ru]
Server = https://repo.sun.hexor.ru

Periodically database update

I use systemd timer for that. Service:

[Unit]
Description=Updating arch linux repository database for %I
Requires=docker.service

[Service]
ExecStart=/usr/bin/docker run -v /srv/arch-repo:/repo --rm archlinux bash -c "repo-add /repo/%i.db.tar.gz /repo/*pkg.tar.zst"

[Install]
WantedBy=multi-user.target

Timer:

[Unit]
Description=Schedule arch repo database update for %I

[Timer]
# every 15 minutes
OnCalendar=*:0/15

[Install]
WantedBy=timers.target

Activate timer:

REPO_URL=repo.sun.hexor.ru
systemctl enable [email protected]${REPO_URL}.timer