<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Lubos' Blog]]></title><description><![CDATA[Random rumbling.]]></description><link>https://lubos.sk/</link><image><url>https://lubos.sk/favicon.png</url><title>Lubos&apos; Blog</title><link>https://lubos.sk/</link></image><generator>Ghost 3.40</generator><lastBuildDate>Thu, 30 Jan 2025 22:37:51 GMT</lastBuildDate><atom:link href="https://lubos.sk/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Wireguard: The Road warrior]]></title><description><![CDATA[Tunnels are laid, sites are connected. Now, it's your time to roam! Become the road warrior with Wireguard.]]></description><link>https://lubos.sk/wireguard-the-road-warrior/</link><guid isPermaLink="false">5ff20b144163230011475ef7</guid><category><![CDATA[guide]]></category><category><![CDATA[vpn]]></category><category><![CDATA[wireguard]]></category><category><![CDATA[technology]]></category><dc:creator><![CDATA[Lubos Babjak]]></dc:creator><pubDate>Mon, 04 Jan 2021 23:04:54 GMT</pubDate><media:content url="https://lubos.sk/content/images/2021/01/road_warrior_1920_650.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://lubos.sk/content/images/2021/01/road_warrior_1920_650.jpg" alt="Wireguard: The Road warrior"><p>We were slowly, but surely laying down the foundation for our final setup in the Wireguard VPN series. We setup the<a href="https://lubos.sk/site-to-site-vpn-with-wireguard/"> site to site connection</a>, we <a href="https://lubos.sk/persistent-routing-for-your-site-to-site-wireguard-vpn/">made it persistent</a>. Now we put the last piece together; the on-the-go VPN on your smartphone! <em>This is where you shine in eyes of your significant other. Think of all that UK Netflix you will watch.</em></p><p>There are native Android &amp; iOS applications for Wireguard, in this article, I will be focusing on Wireguard on iOS. Having said that let's jump in and prepare.</p><h1 id="requirements">Requirements</h1><p>You will need to download the official Wireguard application which can be fetched from here: <a href="https://apps.apple.com/us/app/wireguard/id1441195209">App store</a>. Now we can move to the actual setup.</p><p>There are 3 ways how to configure your VPN connection in Wireguard app:</p><ol><li>Create from QR code.</li><li>Create from scratch.</li><li>Create from file.</li></ol><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2021/01/Image-from-iOS--1-.jpg" class="kg-image" alt="Wireguard: The Road warrior" srcset="https://lubos.sk/content/images/size/w600/2021/01/Image-from-iOS--1-.jpg 600w, https://lubos.sk/content/images/size/w1000/2021/01/Image-from-iOS--1-.jpg 1000w, https://lubos.sk/content/images/2021/01/Image-from-iOS--1-.jpg 1125w" sizes="(min-width: 720px) 720px"><figcaption>Add VPN connection to Wireguard</figcaption></figure><p>I prefer the first option, as it's quick to use, once you setup all the necessary tooling for it. Here you can get creative.</p><h2 id="the-setup">The setup</h2><p>First off, I assume you already have a working Wireguard setup, including working NAT rules in place, if not feel free to refer to the linked guide in the header of this article.</p><h3 id="generate-wireguard-peer-configuration">Generate Wireguard peer configuration</h3><p>Your smartphone will act as an another peer in Wireguard network, therefore we will need to configure public &amp; private keys for it. This can be done via following command:</p><figure class="kg-card kg-code-card"><pre><code>wg genkey | tee iphone_lubos.key | wg pubkey &gt; iphone_lubos.pub
</code></pre><figcaption>Public &amp; private keys being generated by wg tool.</figcaption></figure><h3 id="add-peer-s-public-key-into-your-wireguard-config">Add peer's public key into your Wireguard config</h3><p>Once you get your keys setup, it's time to add the peer's public key into your Wireguard configuration, so the peer gets to access your VPN.  To add the peer's public key into your configuration simply append following line into your main <code>wg0.conf</code> file:</p><figure class="kg-card kg-code-card"><pre><code>[Peer]
PublicKey = 9fC02smHeQ79BMhWcxlHtAXMm5ptvBYL8A2oOWyVDzw=
AllowedIPs = 0.0.0.0/0
</code></pre><figcaption>Wireguard peer configuration part</figcaption></figure><p>Make sure that the <code>AllowedIPs</code> is set to <code>0.0.0.0/0</code>; this is necessary as our VPN server has to accept connections from any IPs from this peer. Remember in this case the <code>AllowedIPs</code> acts as a sort of access control list for the peer and you won't know ahead of time what external IP your peer(smartphone) will have when connecting to the VPN. Now apply the updated Wireguard configuration file to your Wireguard interface via following command:</p><figure class="kg-card kg-code-card"><pre><code>wg setconf wg0 wg0.conf
</code></pre><figcaption>Applying new Wiregruad configuration to wg0 interface.</figcaption></figure><p>With all this in place, your VPN instance is now ready to accept connection from your peer, so let's move to the peer configuration.</p><h3 id="peer-configuration">Peer configuration</h3><p>As mentioned in the beginning of this article, I said that I prefer the configuration via QR code, however before we can generate a QR code, we should prepare the configuration file itself. The file is also useful, if you go with the setup method 2 and 3.</p><p>In the second case you can use the prepared file as a reference for typing down the configuration manually; that takes ages, but who am I to judge. In the third case you can simply share the file with the peer either via email/slack/etc and the peer can just load it and call it a day.</p><p>Back to our road warrior VPN configuration for the peer. It should look like this:</p><figure class="kg-card kg-code-card"><pre><code>[Interface]
Address = 10.170.1.3/32
DNS = 192.168.1.122
PrivateKey = ch0R012Uq/IKAKQwTfGCuqCnrQ4BcTnYimOFpsz23Wk=

[Peer]
PublicKey = IYF1xaxijfg0f05y3fR4NIOzFJrmkPFkB5Y52yWW2AE=
Endpoint = x.x.x.x:51871
AllowedIPs = 0.0.0.0/0
</code></pre><figcaption>Wiregurad peer configuration for iPhone peer.</figcaption></figure><p>Let's quickly walk through the parameters that we are setting in the <code>Interface</code> section we have two parameters:</p><ul><li><code>Address</code> this should be an IP from our VPN network range that gets assigned to this peer, i.e in my case I went with <code>10.170.1.0/24</code> range, therefore I gave <code>10.170.1.3</code> to this peer.</li><li><code>DNS</code> sets the IP of the DNS server that our peer will use to resolve hostnames. Very crucial parameter, we need to make sure that we set this in order not to leak our IP by not making DNS resolution via our public IP.</li><li><code>PrivateKey</code> pretty self explanatory, it's the value of the private key that we generated at the start of this guide.</li></ul><p>In the <code>Peer</code> section we 3 parameters that have to be set:</p><ul><li><code>PublicKey</code> it's the value of public key of our VPN server. Not your peer's!</li><li><code>Endpoint</code> is the external IP and port of your VPN server that is reachable from the Internet.</li><li><code>AllowedIPs</code> this is the part where you either make it or break it, set it to <code>0.0.0.0/0</code> nothing else.</li></ul><p>Remember, on the peer side the <code>AllowedIPs</code> parameter acts as a routing table for the peer. It tells Wireguard application to route traffic for all the IPs via your VPN tunnel, this is where the magic is.</p><p>We are almost done! Now you have 2 options how to get the QR code; the nerd way via command line or the boring way googling on how to generate QR code on the web and then leaking your private key to the Internet.</p><h3 id="generating-qr-code">Generating QR code</h3><p>If you don't want to install any fancy binaries on your VPN server, then just search the web for QR code generator and paste your peer configuration file in it. For example this one does the trick for your: <a href="https://www.the-qrcode-generator.com/">Web based QR code generator</a>. Please note, that using this method you might be sending sensitive stuff to a 3rd party server, which introduces security risk!</p><p>Now, if you take your security and safety seriously you should generate the QR in a safe manner. This can be done via command line tool such as <code>qrencode</code>. On Debian based systems the installation is as simple as running:</p><figure class="kg-card kg-code-card"><pre><code>apt install qrencode
</code></pre><figcaption>Casual installation of qrencode package on a Debian based system.</figcaption></figure><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2021/01/eOclpVf3Cu.gif" class="kg-image" alt="Wireguard: The Road warrior"><figcaption>Generating QR codes like it's nothing.</figcaption></figure><p>Getting a QR code in a secure manner is as easy as this, as demonstrated in the above gif. You can accomplish the same by running the command:</p><figure class="kg-card kg-code-card"><pre><code>qrencode -t ansiutf8 &lt; peer.cfg
</code></pre><figcaption>Command for generating QR code.</figcaption></figure><p>If you wish to send the generated QR code as an image to the peer, you need to generate the image. This is done by running the following command:</p><figure class="kg-card kg-code-card"><pre><code>qrencode -t png -o qrcode.png &lt; peer.cfg
</code></pre><figcaption>Command for generating QR code and saving it as PNG.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2021/01/qrcode-1.png" class="kg-image" alt="Wireguard: The Road warrior"><figcaption>Generated QR code for the VPN.</figcaption></figure><h3 id="the-payday-adding-the-connection-to-wireguard-application">The payday; adding the connection to Wireguard application</h3><p>With all the hard work we have done so far; we are getting there. Our sweet forbidden UK Netflix is almost within our grasp. Last thing we need to do is to scan the QR code with our Wireguard application and we are all set.</p><p>Wireguard application is very easy to use, you just click the blue plus icon in the top right corner of UI and a menu will pop up. Select <em>Create from QR code </em>and scan the QR code. Name your VPN connection and you are done.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2021/01/out.gif" class="kg-image" alt="Wireguard: The Road warrior"><figcaption>Quickly add Wireguard connection via QR code.</figcaption></figure><h2 id="life-behind-20-vpns">Life behind 20 VPNs</h2><p>As you can see, the setup was again straightforward with Wireguard, we managed to quickly generate a peer configuration for our smartphone, add it to the smartphone via a hefty QR code that we also generated. Now all that is left is to enjoy your secure connection via VPN.</p><p>Now, we can take our VPN experience one more level further! We can configure Wireguard application in such a way, that it will automatically enforce VPN connection based on our connection type. For example you can force it to use VPN whenever you are connected to the internet via cellular or you can also set it up to connect to VPN, whenever you are connecting via unknown WiFi. This is especially useful, if you don't want your IP to be leaked on public WiFis, or you don't want your background traffic being sniffed prior turning VPN on manually.</p><p>To set it up open up Wireguard application:</p><ul><li>Click your VPN connection.</li><li>In the top right corner click <em>Edit</em></li><li>Scroll all the way down to the <em>On-demand activation</em></li><li>Configure your rules for On-demand.</li><li>In my case I have VPN always on, when on cellular and Wifi expect of my home Wifi.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2021/01/out-1.gif" class="kg-image" alt="Wireguard: The Road warrior"><figcaption>Setting up on demand Wireguard</figcaption></figure><p>Aaaand done! Cheers, enjoy watching unlimited UK Netflix without any limitation! Assuming your VPN server is UK based.</p>]]></content:encoded></item><item><title><![CDATA[Persistent routing for your site to site Wireguard VPN]]></title><description><![CDATA[Make it last!]]></description><link>https://lubos.sk/persistent-routing-for-your-site-to-site-wireguard-vpn/</link><guid isPermaLink="false">5fcd052f4f4f9d001084bb9d</guid><category><![CDATA[wireguard]]></category><category><![CDATA[vpn]]></category><category><![CDATA[technology]]></category><category><![CDATA[linux]]></category><category><![CDATA[guide]]></category><dc:creator><![CDATA[Lubos Babjak]]></dc:creator><pubDate>Mon, 07 Dec 2020 00:30:47 GMT</pubDate><media:content url="https://lubos.sk/content/images/2020/12/persistent_routing_wireguard.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://lubos.sk/content/images/2020/12/persistent_routing_wireguard.jpg" alt="Persistent routing for your site to site Wireguard VPN"><p>Building on top of what we have setup in the last article:<a href="https://lubos.sk/site-to-site-vpn-with-wireguard/"> Site to Site VPN guide with Wireguard</a> we should probably make our routing setup persist in between restarts. We don't want our work to vanish after a reboot and render our infrastructure unavailable, are we?</p><p>Now with our objective defined, how do we tackle this? First off, as mentioned in the previous article I run Debian stable which means that the networking stack isn't managed via <strong><em>systemd-networkd</em>, </strong>but rather via <strong><em>/etc/network/. </em></strong>Therefore this configuration will be assuming you run the same.</p><figure class="kg-card kg-image-card kg-width-wide"><img src="https://lubos.sk/content/images/2020/12/s2s_vpn-4.png" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN" srcset="https://lubos.sk/content/images/size/w600/2020/12/s2s_vpn-4.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/s2s_vpn-4.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/s2s_vpn-4.png 1600w, https://lubos.sk/content/images/2020/12/s2s_vpn-4.png 1661w" sizes="(min-width: 1200px) 1200px"></figure><p>To refresh your memory in my setup, I have only 2 peers:</p><ul><li>Peer 1 in home site acts as a gateway to my LAN.</li><li>Peer 2 acts just as a standard Wireguard peer.</li></ul><h2 id="configuration">Configuration</h2><p>Most of the configuration will be added to the interfaces configuration file that can be found at <code>/etc/network/interfaces</code><br>One small change will be added to your sysctl configuration file, generally found at <code>/etc/sysctl.conf</code><br></p><h3 id="peer-1-home-site-vpn-m8-sk-">Peer 1 - <strong>Home Site (vpn.m8.sk)</strong></h3><p>This is where I keep main part of the network configuration as this peer is our entry gateway to my LAN, therefore it acts as a router for any incoming traffic to LAN from my VPN network device.</p><p>This router functionality is achieved by running a NAT, that enables 2 networks( <code>10.170.1.0/24</code>  &amp; <code>192.168.0.0/16</code>) to interact between each other.</p><p>The specific configuration how to achieve this functionality is following:</p><figure class="kg-card kg-code-card"><pre><code># VPN Tunnel
auto wg0
iface wg0 inet static
   address 10.170.1.1/24
   netmask 255.255.255.0
   pre-up ip link add dev wg0 type wireguard
   pre-up wg setconf wg0 /data/containers/conf/vpn.m8.sk/wg0.conf
   pre-up iptables -A FORWARD -o wg0 -j ACCEPT
   pre-up iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
   post-down ip link delete dev wg0
   post-down iptables -D FORWARD -o wg0 -j ACCEPT
   post-down iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE</code></pre><figcaption>/etc/network/interfaces</figcaption></figure><p>As you can see there is nothing fancy going on, Wireguard accepts simple configuration, therefore we don't do any real magic in here. Let's go through what is being set:</p><ul><li>We specify what interface we want to configure.</li><li>We assign IP address and broadcast to the interface.</li><li>Before we bring up the network device, we create it.</li><li>Once the network interface is created, we assign Wireguard configuration to it.</li><li>Now, the main part! We setup NAT via iptables. Thing to watch out for is that you need to specify your main network interface that has access to your LAN in here. In my case this is <code>enp1s0</code> and that's it!</li><li>Last few bits are just cleanup commands for when we bring down the interface. In short, it deletes the network device and removes the setup NAT.</li></ul><p>Last bit is to make sure that our Linux kernel knows how to forward ip packets. This can be achieved by adding following line into your <code>/etc/sysctl.conf</code></p><figure class="kg-card kg-code-card"><pre><code>net.ipv4.ip_forward = 1</code></pre><figcaption>/etc/sysctl.conf</figcaption></figure><p>With all this said, that's it. We don't need to do any further configuration on <strong>Peer 1. </strong>We can now verify that the setup works by rebooting our <strong>Peer 1</strong></p><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/NQLL7VYIL1.gif" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN"><figcaption>Reboot test of Peer 1</figcaption></figure><h3 id="peer-2-remote-site-hertzner-box-"><strong>Peer 2 - Remote Site (hertzner-box)</strong></h3><p>Peers that will be connecting to your LAN via VPN, don't need any fancy NATs. The only thing that we need to add to the standard network interface configuration is a <strong>static route</strong> to our LAN.</p><p>We apply somewhat similar configuration, again to the same file located at <code><code>/etc/network/interfaces</code></code>:</p><figure class="kg-card kg-code-card"><pre><code># VPN Tunnel
auto wg0
iface wg0 inet static
   address 10.170.1.2/24
   netmask 255.255.255.0
   pre-up ip link add dev wg0 type wireguard
   pre-up wg setconf wg0 /data/containers/conf/vpn-2.m8.sk/wg0.conf
   up ip route add 192.168.0.0/16 dev wg0
   post-down ip link del dev wg0</code></pre><figcaption>/etc/network/interfaces</figcaption></figure><p>This configuration is not that different than the first one, so let's go through the differences:</p><ul><li>The only difference is that after we bring the interface up, we setup a static route to my LAN. In this case <code>192.168.0.0/16</code> via <code>wg0</code> VPN interface.</li></ul><p>Setup wise we are done, let's test and see, if our configuration works!</p><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/H0xnh1Mdqc.gif" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN"><figcaption>ifdown &amp;&amp; ifup test on Peer 2</figcaption></figure><p>We can see that the configuration works, and once the interface is brought up it gets setup properly and it automatically connects to <strong>Peer 1. </strong>In case you would have more peers, you just repeat the above configuration as many times as you have peers. Assuming your peers are setup properly for Wireguard in the first place.</p><h2 id="reverse-route-bonus-">Reverse route bonus!</h2><p>Now that our remote site can access our LAN in our home site and the configuration is persisted, we should maybe look into the way, how we can enable our LAN clients to access VPN network without the need to install VPN!</p><p>Say no more! This part is specific for EdgeRouters, if you have other router taking care of your LAN, you will have to find the appropriate settings yourself.</p><p>Login to EdgeRouter WebUI, navigate to <strong>Routing </strong>section in the upper right menu bar. Once there, click the <strong>Add Static Route</strong> button.</p><p>Following window will pop up, here you will have to fill it based on your network configuration. In my case, my VPN network subnet is <code>10.170.1.0/24</code> and my VPN instance in home site that handles the VPN connections has an address <code>192.168.1.170/32</code></p><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/chrome_wtGHBISZCk.png" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN" srcset="https://lubos.sk/content/images/size/w600/2020/12/chrome_wtGHBISZCk.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/chrome_wtGHBISZCk.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/chrome_wtGHBISZCk.png 1600w, https://lubos.sk/content/images/2020/12/chrome_wtGHBISZCk.png 1885w" sizes="(min-width: 1200px) 1200px"><figcaption>Static route dialog</figcaption></figure><p>Once set, your configuration should look like something around these lines:</p><figure class="kg-card kg-image-card kg-width-wide"><img src="https://lubos.sk/content/images/2020/12/K4aPTJtFiX.png" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN" srcset="https://lubos.sk/content/images/size/w600/2020/12/K4aPTJtFiX.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/K4aPTJtFiX.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/K4aPTJtFiX.png 1600w, https://lubos.sk/content/images/2020/12/K4aPTJtFiX.png 1885w" sizes="(min-width: 1200px) 1200px"></figure><p>Now, it's testing time! Let's SSH into some of my machines in home site and test pinging one of our clients in VPN network.</p><figure class="kg-card kg-image-card kg-width-wide"><img src="https://lubos.sk/content/images/2020/12/QMNURSacWs.gif" class="kg-image" alt="Persistent routing for your site to site Wireguard VPN"></figure><p>Bingoo, everything works as a charm. Now I can reach my remote sites from all my devices on LAN and all my servers in remote sites can reach LAN without an issue. This opens up a lot of possibilities for the future projects! Let's see what will be our next project</p>]]></content:encoded></item><item><title><![CDATA[Site to site VPN with Wireguard]]></title><description><![CDATA[Connect your local and remote site via nifty WireGuard VPN tunnel in just 2 quick steps! Or maybe not...]]></description><link>https://lubos.sk/site-to-site-vpn-with-wireguard/</link><guid isPermaLink="false">5fcaa018ea4ea70010dd58e3</guid><category><![CDATA[technology]]></category><category><![CDATA[vpn]]></category><category><![CDATA[wireguard]]></category><category><![CDATA[linux]]></category><category><![CDATA[guide]]></category><dc:creator><![CDATA[Lubos Babjak]]></dc:creator><pubDate>Sun, 06 Dec 2020 00:53:00 GMT</pubDate><media:content url="https://lubos.sk/content/images/2020/12/wireguard_blog_logo.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://lubos.sk/content/images/2020/12/wireguard_blog_logo.jpg" alt="Site to site VPN with Wireguard"><p>Connecting your remote locations with your home network has never been so easy!</p><p>For quite a long time I've been running 2 sites, my home lab running multiple docker containers, VMs and many more devices that utilized LAN. Then for heavy duty stuff I pay for dedicated servers hosted at commercial datacenters, these run ElasticSearch clusters, Grafana/Prometheus stack and this website for example.</p><p>Whenever I needed to connect my "on-premise" stuff with remote infrastructure, I've always done it in a dirty way via SSH tunnels, this most of the time worked fine thanks to combination of <a href="https://www.harding.motd.ca/autossh/">autossh</a> running in Docker, but it involved intermediate step of setting up the tunnel, whenever I wanted to connect some service between the sites, which at times could be cumbersome. I always wanted to have a proper site to site connection, but I've never put effort into setting it up, so here we are!</p><h2 id="why-wireguard">Why Wireguard?</h2><p>I had prior experience with OpenSSL, but it just seemed too complex for this task. In the past I've also run <a href="https://www.tinc-vpn.org/">tinc VPN</a> which I really liked, but I felt like trying something new, so with a bit of searching around I stumbled upon <a href="https://www.wireguard.com/">Wireguard</a>.</p><h3 id="some-of-the-things-i-like-about-wireguard">Some of the things I like about Wireguard</h3><ul><li>Runs in Linux kernel, instead of userspace. This gives it the speed performance over OpenVPN or any other VPN that runs in userspace.</li><li>Very simple configuration, you will see later down ;)</li><li>Uses simple network interface, no need to play with taps and tuns!</li></ul><p>Now without further ado, let's jump to mine setup.</p><h1 id="target-setup">Target Setup</h1><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/s2s_vpn-3.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/s2s_vpn-3.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/s2s_vpn-3.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/s2s_vpn-3.png 1600w, https://lubos.sk/content/images/2020/12/s2s_vpn-3.png 1661w" sizes="(min-width: 1200px) 1200px"><figcaption>Site to site VPN infrastructure</figcaption></figure><p>What I need to achieve is to be able to call my home site network(<code>192.168.0.0/16</code>) from my remote site. For example Dedicated server in remote site needs to be able to call a service on my home network directly via it's IP, let's say <code>192.168.1.200</code>. Here is what I've done to make it work.</p><h2 id="preparation">Preparation</h2><p>First off, we have to install Wireguard kernel module and the tools. I run most of my infrastructure under Debian Stable and at the time of writing this article Wireguard is not in the stable repository therefore I will have to use <a href="https://backports.debian.org/Instructions/">Backports</a>. Luckily the process is pretty straightforward. If you use anything else refer to: <a href="https://www.wireguard.com/install/">Link</a></p><!--kg-card-begin: markdown--><ol>
<li>Add following line into your <code>/etc/apt/sources.list</code>:</li>
</ol>
<pre><code>echo 'deb http://deb.debian.org/debian buster-backports main' &gt;&gt; /etc/apt/sources.list
</code></pre>
<ol start="2">
<li>Afterwards update the list of available packages from the new repository you have added:</li>
</ol>
<pre><code>apt update
</code></pre>
<ol start="3">
<li>Install Wireguard:</li>
</ol>
<pre><code>apt-get -t buster-backports install &quot;wireguard&quot; -y
</code></pre>
<!--kg-card-end: markdown--><h2 id="configuration">Configuration</h2><p>Now that we have installed all the dependencies let's set this up!<br>The setup is fairly easy and this is the beauty of using Wireguard, however there are few things that we will cover, as it was not that clear to me from the documentation because I didn't read it thoroughly and I struggled little bit because of it.</p><p>As you can see in the diagram I show above, we will have 2 Wireguard peers. But before we proceed to exact configuration steps, we should specify baseline of our setup.</p><p>VPN interface: <code>wg0</code><br>VPN network: <code>10.170.1.0/24</code><br>VPN port: <code>51871</code></p><p>Peers should be able to handle traffic for the VPN network itself, but also for my LAN network. Therefore we can say that allowed networks should be: <code>10.170.1.0/24, 192.168.0.0/16</code>. We will get to why this is necessary to specify later when we go through the configuration file.</p><p>As we have sorted out what we want to setup, we can now move to the specific configuration steps. I usually like to keep configuration files together therefore I've created following folder structure on my <strong>Peer 1</strong> VM.</p><pre><code>root@vpn:/data/containers/conf/vpn.m8.sk# tree
.
├── certs
│   ├── hertzner-box
│   │   ├── hertzner.key
│   │   └── hertzner.pub
│   └── vpn.m8.sk
│       ├── vpn.m8.sk.key
│       └── vpn.m8.sk.pub
├── peers
│   └── wg0_iphone_lubos.conf
└── wg0.conf

5 directories, 8 files
</code></pre><h3 id="peer-1-home-site-vpn-m8-sk-">Peer 1 - Home Site (vpn.m8.sk)</h3><p>This peer is located in my home site and this is the configuration I decided to go with:</p><p>First off every single Wireguard peer needs to have its set of keys, this consist of a public and private key that is used for encryption and decryption of transmitted/received data and it is also used for the peer authentication.</p><p>Assuming we are in the folder <code>certs/vpn.m8.sk</code> we can run following command to generate private and public key for my <strong>Peer 1</strong></p><pre><code>wg genkey | tee vpn.m8.sk.key | wg pubkey &gt; vpn.m8.sk.pub
</code></pre><p>This will create 2 files that will have private and public key in it respectively:</p><pre><code>root@vpn:/data/containers/conf/vpn.m8.sk# cat *.pub *key
mdv9qFDGcfG5ec2Opk/fNhz84albmcdS2dlKoOeSEQg=
yNdDrdkpcIdTyI11fqPDEY2lMk09I+kYbRKjA9yG320=
</code></pre><p>We repeat the same command, but this time we change to folder <code>certs/hertzner-box</code> and we also adjust the output file names to <code>hertzner.key</code> and <code>hertzner.pub</code>.</p><p><em>In my case I have only 2 peers, therefore I don't need any additional key pairs, however, if you would have more peers you just repeat the commands, you can also run the commands on the any of the peers, or even your machine, if you have <code>wg</code> installed.</em></p><p>Now that we got our key pair we can create our initial configuration, which should look like this:</p><pre><code>[Interface]
PrivateKey = yNdDrdkpcIdTyI11fqPDEY2lMk09I+kYbRKjA9yG320=
ListenPort = 51871

[Peer]
PublicKey = FqB69gnqdHtsST7R3QKVMZGNfUTFGFltqrKL343xdy5=
AllowedIPs = 10.170.1.2/32, 192.168.0.0/16
Endpoint = xxx.xxx.xxx.xxx:51871
</code></pre><p>I store this main configuration inside a file called <code>wg0.conf</code> which is in the base of my main configuration folder in this case <code>/data/containers/conf/vpn.m8.sk</code></p><p>The file follows INI configuration format and consists of two sections: <code>[Interface]</code> and <code>[Peer]</code>.</p><h4 id="interface">Interface</h4><p>The interface section contains configuration properties for your VPN interface:</p><ul><li>The <code>PrivateKey</code> property takes the value of your private key that is stored in <code>certs/vpn.m8.sk/vpn.m8.sk.key</code></li><li><code>ListenPort</code> property is the UDP port where you wish to receive VPN traffic.</li></ul><p>If you follow guides on the Internet some of them also include configuration properties like:</p><ul><li><code>Address</code></li><li><code>PreUp</code>, <code>PostUp</code>, <code>PreDown</code>, <code>PostDown</code></li></ul><p><strong>These properties are used by the <code>wg-quick</code> tool, but are not valid for the <code>wg</code> command!</strong></p><h4 id="peer">Peer</h4><p>The peer section contains configuration properties for your VPN peers, i.e the VPN clients that will connect to this VPN instance. For every single peer(client) that should connect to this specific VPN instance you have to set the <code>[Peer]</code> section.</p><p>Peer section holds following configuration properties:</p><ul><li><code>PublicKey</code> holds the value of public key of the peer, in this case that would be <code>certs/hertzner-box.pub</code></li><li><code>AllowedIPs</code> in the context of Peer section acts as a sort of ACL, i.e it tells our VPN instance what packets it should accept. In this case packets coming for <code>10.170.1.2/32, 192.168.0.0/16</code> are accepted by the instance.</li><li><code>Endpoint</code> property is access interface of your peer, i.e IP and a port combination where VPN instance can send the traffic to.</li></ul><p><code>AllowedIPs</code> property has also a second use, when sending the data from the VPN instance to it's peers, in this case the property is used as a sort of routing table, telling our VPN instance that the peer will accept the traffic for networks <code>10.170.1.2/32, 192.168.0.0/16</code> This is very crucial to understand when setting up Wireguard as you might run into issues, when <code>AllowedIPs</code> are not set properly.<br>For more examples I recommend reading the official documentation: <a href="https://www.wireguard.com/#cryptokey-routing">Link</a>. This was the part I skipped over initially when reading the documentation and it cost me some trial and error until I fixed it.</p><h4 id="network-device">Network device</h4><p>Now that we have our Wireguard configuration ready, we can move to setup of our network device that will be used by Wireguard. As promised by Wireguard, this is very simple and all you need is the <code>ip</code> command.</p><p>First off, you want to create a new network interface:</p><pre><code>ip link add dev wg0 type wireguard
</code></pre><p>Then you need to assign it an IP address, we decided to use the range <code>10.170.1.0/24</code>, therefore I will just use the first IP in the range for the <strong>Peer 1</strong></p><pre><code>ip address add dev wg0 10.170.1.1/24
</code></pre><p>Once we assign the IP, it's time to load our Wireguard configuration that we have prepared in the last step. Assuming we are in the configuration folder we setup earlier we run following command:</p><pre><code>wg setconf wg0 wg0.conf
</code></pre><p>With all this in place we are now ready to bring up the <code>wg0</code> interface up on <strong>Peer 1</strong>. You can do that via following command:</p><pre><code>ip link set up dev wg0
</code></pre><p>What surprised me was the status of the interface after I brought it up, however this is normal:</p><pre><code>root@vpn:/data/containers/conf/vpn.m8.sk# ip link show wg0
3: wg0: &lt;POINTOPOINT,NOARP,UP,LOWER_UP&gt; mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
</code></pre><p>We can now move on to the setup of <strong>Peer 2</strong>!</p><h3 id="peer-2-remote-site-hertzner-box-">Peer 2 - Remote Site (hertzner-box)</h3><p>I won't go into much details, as most of the details have been covered when we were preparing <strong>Peer 1</strong>. For <strong>Peer 2</strong> we only need to prepare the configuration file and setup the network interface. Key pair doesn't need to be generated as we have already did that on <strong>Peer 1</strong>, therefore we just need to copy over the values.</p><p>Configuration for our Peer 2 should look like this:</p><pre><code>[Interface]
PrivateKey = cCPxAr8vKhQtR4fFIgZF/j3cHXcW2oD8XytOHqwFlXo=
ListenPort = 51871

[Peer]
PublicKey = IYFNx4xijfW0805yqfR4NIOz5JrmkPFkB5Y02y6W2AE=
AllowedIPs = 10.170.1.1/32, 192.168.0.0/16
Endpoint = xxx.xxx.xxx.xxx:51871
</code></pre><p>I will now go briefly over the differences:</p><ul><li><code>PrivateKey</code> should contain the value of <code>certs/hertzner-box.key</code></li><li><code>PublicKey</code> should contain the value of <code>certs/vpn.m8.sk.pub</code></li></ul><p>As you can see the roles now switched and our home site acts as a Peer of remote site VPN instance. Make sure you don't mix up the keys in this configurations as then the authentication won't work.</p><p>Once you are done with the configuration file for the Peer 2 VPN instance we can move onto setting up the network interface.</p><h4 id="network-device-1">Network device</h4><p>We will repeat the same commands that we have used for <strong>Peer 1</strong></p><pre><code>ip link add dev wg0 type wireguard
ip address add dev wg0 10.170.1.2/24
wg setconf wg0 wg0.conf
ip link set up dev wg0
</code></pre><p>Please note that I have assigned different IP to the <strong>Peer 2</strong>! In this case second IP from our range, which is <code>10.170.1.2/24</code></p><h2 id="testing-the-vpn-tunnel">Testing the VPN tunnel</h2><p>With all the steps done on all of your Peers we can test the tunnel connectivity. I have no firewalls in place, therefore I just used simple <code>ping</code> from both sites.</p><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/WindowsTerminal_wmZYg6qE2L-1.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/WindowsTerminal_wmZYg6qE2L-1.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/WindowsTerminal_wmZYg6qE2L-1.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/WindowsTerminal_wmZYg6qE2L-1.png 1600w, https://lubos.sk/content/images/2020/12/WindowsTerminal_wmZYg6qE2L-1.png 2177w"><figcaption>Testing network tunnel between 2 clients</figcaption></figure><p>As you can see in the picture both sites respond to the ping test! This confirms that we have successfully connected remote and local site. However, we are not done yet! <strong>Remember, we wanted to access my LAN network from the Remote site!</strong> so let's try that now:</p><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/WindowsTerminal_mYti9Nyis0-1.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/WindowsTerminal_mYti9Nyis0-1.png 600w, https://lubos.sk/content/images/2020/12/WindowsTerminal_mYti9Nyis0-1.png 765w"><figcaption>LAN doesn't work</figcaption></figure><p>This doesn't work out of the box and we get to why in the next section.</p><h2 id="routing-from-vpn-network-to-lan-network-">Routing from VPN network to LAN network.</h2><p>In order to route traffic between our VPN network(<code>10.170.1.0/24</code>) and our LAN network(<code>192.168.1.0/16</code>) we need to setup few things:</p><ul><li>All remote peers need to have a static route to the LAN.</li><li>Our Wireguard peer in LAN has to act as a router for routing traffic between VPN network and LAN network.</li></ul><h2 id="routing">Routing</h2><p>Setting up a static route is very simple on all the remote peers run following command:</p><pre><code>ip route add 192.168.0.0/16 dev wg0
</code></pre><p>This will make sure that all the traffic for the <code>192.168.0.0/16</code> gets routed through <code>wg0</code></p><h2 id="nat">NAT</h2><p>In order to properly connect two networks together we will have to setup NAT on our VPN instance that is hosted in home site.<br>Add following iptables rules to enable the NAT:</p><pre><code>iptables -A FORWARD -i wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
</code></pre><p><code>wg0</code> is our VPN network interface, <code>enp1s0</code> is your main network interface that can reach your LAN.</p><p>Once that is set, you will need to tell the Linux kernel to forward ip packets:</p><!--kg-card-begin: markdown--><pre><code>sysctl -w net.ipv4.ip_forward=1
</code></pre>
<!--kg-card-end: markdown--><p>And with this in place we are done!</p><h1 id="wrap-up">Wrap up</h1><p>Now that we have managed to setup everything we can test our setup!</p><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/WindowsTerminal_TjrHdk9qCP-1.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/WindowsTerminal_TjrHdk9qCP-1.png 600w, https://lubos.sk/content/images/2020/12/WindowsTerminal_TjrHdk9qCP-1.png 673w"><figcaption>Site to site ping working</figcaption></figure><p>As you can see I am able to ping my LAN network from the remote site, I can even reach my IoT  network.</p><p>Please note that all the interface configurations that we setup are temporal, meaning that they will go away after you reboot your server. If you would like to see how to set this up permanently please read this article of mine: <a href="https://lubos.sk/persistent-routing-for-your-site-to-site-wireguard-vpn/">Persistent routing for your site to site Wireguard VPN</a></p><p>Also the sites are connected one way, e.g I am not able to call VPN network from inside my LAN network, this is something I want to setup in the future.</p><p>I've got a bonus!! I did brief tests with iperf3 to test out the network performance. My remote site has full 1gbit connection to the outside. My home site has 300/30Mbit.</p><h2 id="tcp-without-vpn">TCP without VPN</h2><p>I managed to push <strong>229Mbit/s</strong> from my remote site to my home site without VPN.</p><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/h1HI0DJvkO.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/h1HI0DJvkO.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/h1HI0DJvkO.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/h1HI0DJvkO.png 1600w, https://lubos.sk/content/images/2020/12/h1HI0DJvkO.png 2302w"><figcaption>Direct iperf test without VPN</figcaption></figure><h2 id="tcp-with-vpn-">TCP with VPN.</h2><p>Now the interesting part, tests via VPN. I hit whooping <strong>200Mbit</strong></p><figure class="kg-card kg-image-card kg-width-full kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/12/qagm3xjM5c.png" class="kg-image" alt="Site to site VPN with Wireguard" srcset="https://lubos.sk/content/images/size/w600/2020/12/qagm3xjM5c.png 600w, https://lubos.sk/content/images/size/w1000/2020/12/qagm3xjM5c.png 1000w, https://lubos.sk/content/images/size/w1600/2020/12/qagm3xjM5c.png 1600w, https://lubos.sk/content/images/size/w2400/2020/12/qagm3xjM5c.png 2400w"><figcaption>Iperf test via VPN</figcaption></figure>]]></content:encoded></item><item><title><![CDATA[ElasticSearch online cluster expansion]]></title><description><![CDATA[Expand your ElasticSearch cluster online without any downtime.]]></description><link>https://lubos.sk/elasticsearch-cluster-expansion/</link><guid isPermaLink="false">5ccf2850d8ea020011b4fc0b</guid><category><![CDATA[elasticsearch]]></category><category><![CDATA[expansion]]></category><category><![CDATA[online]]></category><category><![CDATA[upgrade]]></category><category><![CDATA[linux]]></category><category><![CDATA[technology]]></category><dc:creator><![CDATA[Lubos Babjak]]></dc:creator><pubDate>Sun, 05 May 2019 22:38:11 GMT</pubDate><content:encoded><![CDATA[<p>New day, new task as an operation guy. This time I have been assigned a task to expand one of our production ElasticSearch clusters. The target; perform it without the downtime of our application nor the cluster. Simple enough, but you never know what can happen down the road, so let's try to prepare the expansion and see what we can expect of this activity.</p><h2 id="current-cluster-setup">Current cluster setup</h2><p>At the moment our current cluster consist of 12 old nodes which we are going to decommission and replace with a shiny brand new 6 nodes, this should give us more breathing space for the future.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-05_20-39-59_o.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-05_20-39-59_o.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-05_20-39-59_o.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-05_20-39-59_o.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-05_20-39-59_o.png 1719w" sizes="(min-width: 720px) 720px"><figcaption>Current ElasticSearch cluster 12 data nodes + 1 master node</figcaption></figure><h2 id="the-plan">The plan</h2><p>Thanks to the flexibility of ElasticSearch, the expansion should be quite easy and almost fully handled by ElasticSearch itself due to its scalability, the high level steps should look as follow, we will get into the details of each step later:</p><!--kg-card-begin: markdown--><ol>
<li>Roll out ElasticSearch onto the new hardware.</li>
<li>Disable shard allocation in the cluster</li>
<li>Startup newly installed ElasticSearch nodes and check, if they have joined the cluster successfully</li>
<li>Enable shard allocation in the cluster and wait until the shards within the cluster get distributed to the new nodes</li>
<li>Start to decommission old nodes from the cluster one by one.</li>
</ol>
<!--kg-card-end: markdown--><h3 id="roll-out-elaticsearch-onto-the-new-hardware">Roll out ElaticSearch onto the new hardware</h3><p>I won't go into much details in this part as your ElasticSearch installation might differ, but you should make sure that the following is set properly for the new nodes:</p><!--kg-card-begin: markdown--><ul>
<li>The cluster name is the same as on the old nodes.</li>
<li>New data nodes can reach your master nodes, it is enough just to set master nodes into the zen discovery.</li>
<li>If you use ElasticSearch backup repositories, make sure that it's set on all the nodes, otherwire the node won't startup.</li>
</ul>
<p>No issue on our side as the nodes are provisioned via Ansible playbook, we can proceed further onto the next step.</p>
<!--kg-card-end: markdown--><h3 id="disable-shard-allocation-in-the-cluster">Disable shard allocation in the cluster</h3><p>Usually you don't have to do this step, but my plan is to join the nodes into the cluster all at once, but I don't want ElasticSearch to assign the shards to the nodes straight away as I want to check, if the nodes are fully operational, hence this step. Shard allocation can be disabled cluster wide via following API call:</p><!--kg-card-begin: markdown--><pre><code>root@es-master-1:~# curl -X PUT &quot;localhost:9200/_cluster/settings&quot; \
&gt; -H 'Content-Type: application/json' \
&gt; -d'{
&gt;   &quot;persistent&quot;: {
&gt;     &quot;cluster.routing.allocation.enable&quot;: &quot;none&quot;
&gt;   }
&gt; }'
{&quot;acknowledged&quot;:true,&quot;persistent&quot;:{&quot;cluster&quot;:{&quot;routing&quot;:{&quot;allocation&quot;:{&quot;enable&quot;:&quot;none&quot;}}}},&quot;transient&quot;:{}}
</code></pre>
<!--kg-card-end: markdown--><p>Once you disable the shard allocation we can move on and startup new ElasticSearch instances.</p><h3 id="startup-newly-installed-elasticsearch-nodes-and-check-if-they-have-joined-the-cluster-successfully">Startup newly installed ElasticSearch nodes and check, if they have joined the cluster successfully</h3><p>Startup your nodes via whatever method is most suitable for your ElasticSearch deployment; check, if the nodes have joined the cluster without any issues.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-05_21-40-37_o.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-05_21-40-37_o.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-05_21-40-37_o.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-05_21-40-37_o.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-05_21-40-37_o.png 1920w" sizes="(min-width: 720px) 720px"><figcaption>New 6 nodes added to the cluster</figcaption></figure><h3 id="enable-shard-allocation-in-the-cluster-and-wait-until-the-shards-within-the-cluster-get-distributed-to-the-new-nodes">Enable shard allocation in the cluster and wait until the shards within the cluster get distributed to the new nodes</h3><p>Time to enable shard allocation and wait until the cluster rebalances and some shards get moved to our new data nodes.</p><pre><code>root@es-master-1:/app/products/elasticsearch/logs# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
&gt; {
&gt;   "persistent": {
&gt;     "cluster.routing.allocation.enable": null
&gt;   }
&gt; }
&gt; '

{"acknowledged":true,"persistent":{},"transient":{}}</code></pre><figure class="kg-card kg-image-card"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-05_21-40-37_o-1.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-05_21-40-37_o-1.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-05_21-40-37_o-1.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-05_21-40-37_o-1.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-05_21-40-37_o-1.png 1920w" sizes="(min-width: 720px) 720px"></figure><h3 id="start-to-decommission-old-nodes-from-the-cluster-one-by-one">Start to decommission old nodes from the cluster one by one</h3><p>Now the fun begins, the shards have been rebelanced and we can start to decommission our old nodes. First we have to make sure that no shards are allocated to the node that we are about to decommission. To do that we have to move all shards away from it via following command:</p><pre><code>root@es-master-1:/app/products/elasticsearch/logs# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
&gt; {
&gt;   "transient" : {
&gt;     "cluster.routing.allocation.exclude._ip" : "10.135.136.62"
&gt;   }
&gt; }
&gt; '
{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"10.135.136.62"}}}}}}</code></pre><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-05_22-36-14_o.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-05_22-36-14_o.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-05_22-36-14_o.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-05_22-36-14_o.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-05_22-36-14_o.png 1920w" sizes="(min-width: 720px) 720px"><figcaption>Shards being moved from the old node</figcaption></figure><p>As you can see ElasticSearch is moving away all the <strong>primary shards</strong> from the node, once there are no shards on the node, we can switch off the node to decommission it from the cluster.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-05_22-49-18_o.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-05_22-49-18_o.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-05_22-49-18_o.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-05_22-49-18_o.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-05_22-49-18_o.png 1920w" sizes="(min-width: 720px) 720px"><figcaption>Empty node</figcaption></figure><p>Now that all the shards have been moved out of <strong>es-data-1 </strong>node we can switch it off and remove it from the cluster, rinse and repeat until you are left with only the new nodes in your cluster.</p><h2 id="finishing-up">Finishing up</h2><p>Once I've decommissioned all the old nodes I ended up with something like this:</p><figure class="kg-card kg-image-card"><img src="https://lubos.sk/content/images/2020/07/chrome_2019-05-06_00-32-02_o-1.png" class="kg-image" alt srcset="https://lubos.sk/content/images/size/w600/2020/07/chrome_2019-05-06_00-32-02_o-1.png 600w, https://lubos.sk/content/images/size/w1000/2020/07/chrome_2019-05-06_00-32-02_o-1.png 1000w, https://lubos.sk/content/images/size/w1600/2020/07/chrome_2019-05-06_00-32-02_o-1.png 1600w, https://lubos.sk/content/images/2020/07/chrome_2019-05-06_00-32-02_o-1.png 1920w" sizes="(min-width: 720px) 720px"></figure><p>Fully functional cluster running on brand new 6 nodes, no downtime needed at all, just make sure that you are of course not performing this during the peak time. :)</p>]]></content:encoded></item></channel></rss>