Random Musings on Technology

Using Spare Disk Space as One Large Disk


This is a neat trick in Linux for when you have a bit (some GB) in different disks and want to use it to save large files. I use this for mining XCH (Chia Cryptocoin).

  1. First check the available space on each disk using df -h:
Filesystem      Size  Used Avail Use% Mounted on  
/dev/sdg        4.6T  4.5T   84G  99% /media/omar/SEA_5T_B
/dev/sdp2        30T   30T   68G 100% /media/omar/seagate1

Here we have 2 disks, one with 84G and another with 68G. The size of the file we want to store is more than 100GB. To do that, we will first create "image" files using the remaining space in each disk:

truncate -s67G /media/omar/seagate1/67G.img  
truncate -s83G /media/omar/SEA_5T_B/83G.img  

Then, we create a loopback device with losetup for each disk:

losetup --show --find /media/omar/seagate1/67G.img  
losetup --show --find /media/omar/SEA_5T_B/83G.img  

We must pay attention to the data returned by those two previous commands, as they will tell us the name of the loopback device (e.g. /dev/loop1).

As a next step, we are going to use btrfs to create a single filesystem from those 2 devices:

sudo mkfs.btrfs -d single /dev/loop1 /dev/loop2 (Replace /dev/loop1 and /dev/loop2 with whatever you got from the losetup step)

Finally, we can mount our btrfs filesystem in some directory like:

sudo mount -t btrfs -o ssd,nodatacow,nodatasum,discard,nodiratime,discard=async,noatime /dev/loop1 loops/

In my case I added all those options to make access faster. With btrfs you can mount a several-disk filesystem by referring to one of the devices (like /dev/loop1 in this case)

After mounting this device, we do a df -h again and should see our new disk mounted in the specified directory with the summed amount of space:

Filesystem      Size  Used Avail Use% Mounted on  
/dev/loop1      150G  3.8M  148G   1% /media/omar/loops

And that's it! we've got 150G to use for large files!

Some references:

The Path to Kueski 2.0


Back in the day (5 years ago) I wrote a good blog post related to a migration the Engineering team did from a monolithic infrastructure into a "microservice based" architecture.

The original URL is this, but I've seen it has gone down a couple of times in the last days so I saved a copy of it in archive.is:

The Path to Kueski 2.0 (Archive).

Another good blog post that is worth saving, is the Better Logging (Archive) post by Jorge del Rio which has good ideas about logging in production environments.

Web Sequence Digrams


A nice script to generate sequence diagrams in an HTML page:

Taken from: https://unpkg.com/js-sequence-diagrams-autorenderer@1.0.1/index.js

// JS-sequence-diagrams auto-renderer; renders a document using JS-sequence-diagrams (https://bramp.github.io/js-sequence-diagrams/)
// See readme.md for usage instructions
// Pretty inefficient, but it works:
const a = document.createElement('script'); a.setAttribute('src', 'https://cdnjs.cloudflare.com/ajax/libs/webfont/1.6.28/webfontloader.js');  
a.onload = () => {  
  const b = document.createElement('script'); b.setAttribute('src', 'https://cdnjs.cloudflare.com/ajax/libs/raphael/2.3.0/raphael.min.js');
  b.onload = () => {
    const c = document.createElement('script'); c.setAttribute('src', 'https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.5.2/underscore-min.js');
    c.onload = () => {
      const d = document.createElement('script'); d.setAttribute('src', 'https://cdnjs.cloudflare.com/ajax/libs/js-sequence-diagrams/1.0.6/sequence-diagram-min.js');
      d.onload = () => {
        const diagram = Diagram.parse(document.body.textContent);
        document.body.textContent = "";
        const x = document.createElement('div');
        diagram.drawSVG(x, {theme: 'simple'});

As an example usage, paste this into a file test.html and open it in your browser:

Jon->Ygritte: Do you remember the cave?  
Note right of Ygritte: Long silence...  
Ygritte-->Jon: We should have stayed in that cave  
Jon->Ygritte: We'll go back there.  
Ygritte-->Jon: You know nothing Jon Snow

# Rendered using JS-sequence-diagrams (https://bramp.github.io/js-sequence-diagrams/) by the line below:
# <script src="https://unpkg.com/js-sequence-diagrams-autorenderer@1.0.1/index.js"></script>