Jekyll2023-05-24T07:17:28+00:00https://vijaypathak.com.np/feed.xmlVijay PathakThis is a personal webpage of Vijay's where he share his research and interest in general.Vijay PathakDockerize Jekyll website2023-04-07T00:00:00+00:002023-04-07T00:00:00+00:00https://vijaypathak.com.np/2023/04/dockerize-jekyll<p>If you are running a static webpage generator that is built by Ruby and Jekyll, chances are you might have faced issues
installing gems and configuring the dev environment across various operating systems.</p>
<h1 id="what-is-docker-and-docker-container">What is Docker and Docker container?</h1>
<p>Docker is a platform that makes it simpler to build and ship software across many settings and infrastructures by
allowing developers to build, deploy, and operate programs within lightweight, portable containers and images.</p>
<h2 id="dockerize-your-jekyll-theme">Dockerize your Jekyll theme</h2>
<h3 id="create-dockerfile">Create Dockerfile</h3>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> ruby:3.0</span>
<span class="k">RUN </span>bundle config <span class="nt">--global</span> frozen 1
<span class="k">WORKDIR</span><span class="s"> /app/jekyll</span>
<span class="k">COPY</span><span class="s"> Gemfile Gemfile.lock ./</span>
<span class="k">RUN </span>bundle <span class="nb">install</span>
</code></pre></div></div>
<p>Run command <code class="language-plaintext highlighter-rouge">docker build -t jekyll .</code> to build a docker image.</p>
<h3 id="add-host-to-the-config">Add host to the config.</h3>
<p>Add variable host to the <code class="language-plaintext highlighter-rouge">_config.yml</code> file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>host: "0.0.0.0"
</code></pre></div></div>
<h3 id="create-docker-compose">Create Docker compose</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>version: "3.0"
services:
app:
image: jekyll
container_name: app-jekyll-container
command: [ "jekyll", "serve"]
ports:
- "4000:4000"
volumes:
- $PWD:/app/jekyll
</code></pre></div></div>
<p>Initiate the container by simply using <code class="language-plaintext highlighter-rouge">docker-compose up</code> and you have the local server running at <code class="language-plaintext highlighter-rouge">http://0.0.0.0:4000</code>.</p>Vijay PathakA guide on using Docker image and docker container for jekyll theme websites.Complete guide to automate deployment using Jenkins and Github webhooks in Ubuntu Server.2021-09-28T00:00:00+00:002021-09-28T00:00:00+00:00https://vijaypathak.com.np/2021/09/complete-guide-to-automate-deployment-using-jenkins<p>In this guide, we’ll learn to automate build, test, and deploy applications to the Ubuntu Server with minimal setup along with concepts of Continuos integration, Continuous delivery, and Continuos deployment.</p>
<p>This article will cover basic to advance processes in a fairly simple manner. We will automate Django deployment as an example.</p>
<p>Let’s learn the concepts first.</p>
<h1 id="what-is-devops">What is DevOps?</h1>
<p>We all have heard this subject many times in life especially if you are working as a software developer. In earlier days, Operations teams and Developers has to cooperatively work together to deliver products which created a waiting time to get feedback. <code class="language-plaintext highlighter-rouge">DevOps</code> is a combination of both roles to achieve the same business objective with automation.</p>
<p>There are three major phases to consider while creating CI/CDs;</p>
<h2 id="continues-integration">Continues Integration</h2>
<p>Continues Integration (CI) is the process where the test cases, build files are executed every time changes are detected in the codebase. This is one of the best practices in the Agile software development life cycle to know early risks. The failed builds are notified to the developer instantly.</p>
<p>This process can be done in two different ways in <a href="https://git-scm.com/">Git</a>.</p>
<ol>
<li>
<p>You can manually set up <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">Git hook</a> <code class="language-plaintext highlighter-rouge">post-receive</code>. You can check my previous <a href="https://vijaypathak.com.np/2020/10/complete-guide-on-deploying-django-application-using-chaussette-circus-in-aws-ec2-ubuntu.html#setup-git-repository-in-server">post</a> to learn more on the topic.</p>
</li>
<li>
<p>Use <a href="https://docs.github.com/en/github/extending-github/about-webhooks">Git webhooks</a>, both are invoked when doing git push and updates. I assume that your code is already hosted in Github, we’ll use GitHub webhooks to connect Jenkins.</p>
</li>
</ol>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-ci.jpeg" alt="Jenkins CI" class="full" /></p>
<p>The idea is to trigger scripts to auto-run tests, build files whenever a change is detected in the Github repository branch as shown in the above image. If a build is failed you can notify dev teams through standard emails or directly post on social platforms like Slack, through plugins.</p>
<h2 id="continuos-delivery-and-deployment">Continuos Delivery and deployment</h2>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-ci-cd.jpeg" alt="Jenkins CD" class="full" /></p>
<p>Continuous Delivery (CD) resumes from Continues Integration process, test/builds are verified and deployed to test or staging server as shown in the image above, generally manual/automate browser level tests are carried out by the QA/users at this stage. <a href="https://www.edureka.co/blog/what-is-smoke-testing/">Smoke tests</a> are done to ensure whether a software feature or bug build that has been delivered is stable.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-ci-cd-cd.jpeg" alt="Jenkins CD/CD" class="full" /></p>
<p>We’ll not cover much about Continuous Deployment (CD) in this guide but the idea is pretty straightforward; once the users (usually a client or beta users) test the build and no bugs are reported, the same code is pushed to the production server.</p>
<h2 id="git-flow">Git flow</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── Git
└── Branches
└── <span class="sb">`</span>Others<span class="sb">`</span>
└── Create branch from <span class="sb">`</span>develop<span class="sb">`</span>
└── Changes detected/Git webhook triggered to run Continuos integration <span class="k">if </span>enabled
└── Create merge request to <span class="s1">'develop'</span> <span class="o">[</span>manual process]
└── <span class="sb">`</span>Develop<span class="sb">`</span>
└── Changes detected/Git webhook triggered
└── Run Continuos integration
└── Run Continuos delivery to <span class="sb">`</span>staging server<span class="sb">`</span>
└── Create merge request to <span class="s1">'master'</span> <span class="o">[</span>manual process]
└── <span class="sb">`</span>Master/main<span class="sb">`</span>
└── Changes detected/Git webhook triggered
└── Auto-run Continuos deployment to <span class="sb">`</span>production server<span class="sb">`</span>
</code></pre></div></div>
<p>Here is one of the simple ways to visualize how the distributed version control system is structured to handle each cycle.</p>
<h1 id="jenkins">Jenkins</h1>
<p>It is an open-source automation tool to automate build, test, and deploy projects. This is all in one <code class="language-plaintext highlighter-rouge">DevOps</code> server that has to be installed manually and written in <a href="https://www.java.com/en/">Java</a>, which has many plugins to make automation smooth.</p>
<h2 id="configure-jenkins">Configure Jenkins</h2>
<blockquote>
<p><strong><em>It is recommended to create new server or instance for Jenkins only which will be the <code class="language-plaintext highlighter-rouge">master</code> server.</em></strong></p>
</blockquote>
<p>Let’s install Jenkins on Ubuntu20 LTS. Jenkins requires <code class="language-plaintext highlighter-rouge">Java</code> and <code class="language-plaintext highlighter-rouge">JDK</code> in order to compile Java code. There is <a href="https://openjdk.java.net/">Open Java Development Kit</a> (OpenJDK) that we will be using since it required no license for production uses.</p>
<p>Firstly, update the Ubuntu packages using;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt update && apt upgrade -y
</code></pre></div></div>
<h3 id="1-install-openjdk">1. Install OpenJDK</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install openjdk-11-jdk -y
</code></pre></div></div>
<h3 id="2-instal-the-latest-version-of-jenkins-and-enable-it-when-server-restarts">2. Instal the latest version of Jenkins and enable it when server restarts</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt update && sudo apt install jenkins
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo systemctl start jenkins && sudo systemctl enable jenkins
</code></pre></div></div>
<p>At last verify if it’s running using;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo systemctl status jenkins
</code></pre></div></div>
<h3 id="3-enable-port">3. Enable Port</h3>
<p>By default, Jenkins runs on port <code class="language-plaintext highlighter-rouge">8080</code>. You won’t be able to access it unless you enable the port <code class="language-plaintext highlighter-rouge">8080</code> from inbound ports settings.</p>
<p>If you are not using firewall. Enable the port using;</p>
<p><code class="language-plaintext highlighter-rouge">sudo ufw enable</code></p>
<p><code class="language-plaintext highlighter-rouge">sudo ufw allow OpenSSH</code></p>
<p><code class="language-plaintext highlighter-rouge">sudo ufw allow 8080</code></p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Hint:</strong> If you are using AWS cloud service, it is recommended to open inbound port directly from AWS security groups for any IP address.</p>
<p>You should be able to visit the default jenkins URL <code class="language-plaintext highlighter-rouge">http://YOUR_DOMAIN_OR_IP:8080/</code> and configure further.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/unlock-jenkins-server.png" alt="Unlock jenkins server" class="full" /></p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Hint:</strong> if your browser cant open Jenkins at this point, double check if port <code class="language-plaintext highlighter-rouge">8080</code> is enabled.</p>
<p>Similarly, copy and paste the default password from <code class="language-plaintext highlighter-rouge">sudo cat /var/lib/jenkins/secrets/initialAdminPassword</code> to unlock Jenkins.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-suggested-plugins-setup.png" alt="Jenkins suggested plugins setup" class="full" /></p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-add-user.png" alt="Jenkins add user" class="full" /></p>
<p>Next, simply select install suggested plugins and create a new username and password in order to log in.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-first-login.png" alt="Jenkins dashboard" class="full" /></p>
<p>The dashboard looks something like this after the first install. Yay! we have successfully configured and installed Jenkins.</p>
<h2 id="credentials">Credentials</h2>
<h3 id="connect-github-and-jenkins">Connect GitHub and Jenkins</h3>
<p>We’ll create new SSH keys with <code class="language-plaintext highlighter-rouge">sudo ssh-keygen</code> command in the server - I will name the key as <code class="language-plaintext highlighter-rouge">github-key</code>. If you navigate to <code class="language-plaintext highlighter-rouge">cd ~/.ssh</code> you will be able to see a list of keys. View the private key with <code class="language-plaintext highlighter-rouge">cat github-key</code> and copy.</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> The public key is used for encryption whereas private key is used for decryption process.</p>
<p>Next, we add private and public keys.</p>
<h3 id="1-public-key">1. Public key</h3>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Github-ssh.png " alt="Github add ssh" class="full" /></p>
<p>Navigate to Github settings and add public <a href="https://github.com/settings/keys">SSH</a> keys i.e.<code class="language-plaintext highlighter-rouge">github-key.pub</code>. This will give access to your Jenkins to pull latest changes.</p>
<h3 id="2-private-key">2. Private key</h3>
<p>The private keys are used for the decryption process so they must be securely stored in any environment. This is required for jenkins to pull from Github and push to the remote server.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-list-cred.png" alt="Jenkins list credentials" class="full" /></p>
<p>Go to <code class="language-plaintext highlighter-rouge">Manage Jenkins</code> > <code class="language-plaintext highlighter-rouge">Manage Credentials</code> > click <code class="language-plaintext highlighter-rouge">(global)</code>. You will see two credentials that I already added before on the right-hand side.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-add-private-keys.png" alt="Jenkins add private keys" class="full" /></p>
<p>At last, click <code class="language-plaintext highlighter-rouge">Add credentials</code>, select <code class="language-plaintext highlighter-rouge">SSH Username with private key</code> as <code class="language-plaintext highlighter-rouge">kind</code>, add server <code class="language-plaintext highlighter-rouge">username</code> (can be any name) and private keys (<code class="language-plaintext highlighter-rouge">github-key</code>) as shown in the image.</p>
<h3 id="connect-server-and-jenkins">Connect Server and Jenkins</h3>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-add-remote-server.png" alt="Jenkins add remote server" class="full" /></p>
<p>Jenkins will connect to the remote server for deployment which consists of build, run unit tests and deploy - we’ll use server private keys and <a href="https://searchsecurity.techtarget.com/definition/Secure-Shell">SSH</a>. Simply, add the server private key to connect Jenkins and remote server.</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> The private key is not same the as that we created <a href="#connect-github-and-jenkins">here</a>. The server private key is given when the instance is first created and usually named <code class="language-plaintext highlighter-rouge">id_rsa</code>.</p>
<h2 id="github-repository-webhooks">Github repository Webhooks</h2>
<p>Let’s create a basic private repository in GitHub and navigate to <code class="language-plaintext highlighter-rouge">Webhooks</code> section from settings. This process is vital since Jenkins will be only triggered through webhooks according to our previous configuration.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Github-webhooks.png" alt="Github webhooks" class="full" /></p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Github-add-webhook.png" alt="Github add webhooks" class="full" /></p>
<p>Click add webhooks, and provide the webhook URL.</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> You need to replace the <code class="language-plaintext highlighter-rouge">YOUR_SERVER_IP_OR_DOMAIN</code> according to your instance configs.</p>
<p>Great! At this point, we are done with adding the credentials required for server authentication. Let’s summarize what we have done so far; we understand the basics of CI/CDs, configure Jenkins, connect remote server and GitHub with SSH, and configured webhooks. I know it’s lots of things to take at all, but trust be everything will make sense when automation is created.</p>
<h2 id="setup-first-build-job">Setup first build job</h2>
<p>In Jenkins there are different ways to create a job, we’ll choose the Freestyle project. Jenkins cannot push changes to the remote server by default. Go to <code class="language-plaintext highlighter-rouge">Manage Jenkins</code> > <code class="language-plaintext highlighter-rouge">Manage Plugins</code> > <code class="language-plaintext highlighter-rouge">Available</code>, search and install <a href="https://plugins.jenkins.io/ssh/">SSH plugin</a> and restart the server. This is essential to run bash commands.</p>
<p>Next, we need to link private keys that were added <a href="#Connect-GitHub-and-Jenkins">earlier</a> to the server IP Address.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-add-server-ip.png" alt="Jenkins add server Ip" class="full" /></p>
<p>Go to <code class="language-plaintext highlighter-rouge">Manage Jenkins</code> > <code class="language-plaintext highlighter-rouge">Configure System</code> > search for SSH remote hosts and click add button located at the bottom section. You need to select Hostname (IP address), a port which is usually <code class="language-plaintext highlighter-rouge">22</code>, and private key of the remote server where we are deploying.</p>
<h3 id="configure-build">Configure build</h3>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-free-style-project.png" alt="Jenkins free style project" class="full" /></p>
<p>We will create a normal Freestyle project to configure automation and hit OK.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-add-project.png" alt="Jenkins build add project" class="full" /></p>
<p>In the next page, tick the project as <code class="language-plaintext highlighter-rouge">GitHub project</code>, copy the <code class="language-plaintext highlighter-rouge">HTTPs</code> url, and add some description to your project.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-build-source-code-management.png" alt="Jenkins build source code management" class="full" /></p>
<p>In the source code management section, add repository <code class="language-plaintext highlighter-rouge">SSH url</code> and the Github private key we created <a href="#2-private-key">here</a>. I choose <code class="language-plaintext highlighter-rouge">develop</code> as the build branch, leave blank for any branches.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-build-triggers-type.png" alt="Jenkins build trigger types" class="full" /></p>
<p>In the Build Triggers section, I will choose <code class="language-plaintext highlighter-rouge">GitHub hook trigger for GITScm polling</code> so that builds are triggered instantly when new code is pushed to Github. If you want to run the build after a certain time frame - <code class="language-plaintext highlighter-rouge">Poll SCM</code> can do it with <a href="https://opensource.com/article/17/11/how-use-cron-linux">Cron job</a> syntax.</p>
<p>There are two different ways to execute commands in a remote server from Jenkins where we build, test and deploy an application. They do the same thing in different ways.</p>
<h3 id="1-execute-shell-script-using-ssh">1. Execute shell script using SSH</h3>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-build-execute-remote-shell.png" alt="Jenkins execute remote shell" class="full" /></p>
<p>In the build environment section, click Execute shell script on a remote host using ssh option and select the private key we added <a href="#setup-first-build-job">earlier</a> from global configurations.</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> I showed the dummy ip as <code class="language-plaintext highlighter-rouge">0.0.0.0</code>, this will be your remote server IP address.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nb">cd </span>api/
<span class="nb">source </span>venv/bin/activate
git checkout develop
git pull origin develop
pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt
python manage.py migrate
<span class="c"># python manage.py collectstatic</span>
python manage.py <span class="nb">test
sudo </span>supervisorctl reload
</code></pre></div></div>
<p>Likewise, above is the sample <a href="https://www.djangoproject.com/">Django</a> application deployment shell script added in <code class="language-plaintext highlighter-rouge">pre-build-script</code> section to execute different commands for automation. You may modify it as your choice but for simplicity, we’ll keep it short for this guide.</p>
<h4 id="test-build">Test build</h4>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-shell-script-build.png" alt="Jenkins shell script build" class="full" /></p>
<p>Let’s push one change to the <code class="language-plaintext highlighter-rouge">develop</code> branch and wait for the auto-build, test, and deploy process. The build is completed as you can see in the image above.</p>
<h3 id="2-jenkins-pipelines">2. Jenkins Pipelines</h3>
<p>This is more mature and suitable for production uses. Each phase is called pipe or stage where one certain task or command is executed and the collection of such tasks is called pipelines. It is similar to running <a href="#1-execute-shell-script-using-ssh">bash script</a> but can organize complex activities effectively.</p>
<p>This guide will not cover in-depth for pipelines but you can have an idea from here.</p>
<p>The pipeline project has to be selected while creating a new job, and you need to write pipeline scripts to execute the same <a href="#1-execute-shell-script-using-ssh">bash script</a> commands in stages.</p>
<p>You can learn more about Jenkins pipelines <a href="https://www.jenkins.io/doc/book/pipeline/">here</a>. Here’s one sample pipeline script;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pipeline <span class="o">{</span>
agent any
stages <span class="o">{</span>
stage<span class="o">(</span><span class="s1">'Connect server'</span><span class="o">)</span> <span class="o">{</span>
steps <span class="o">{</span>
// connect server
sshagent<span class="o">([</span><span class="s1">'522v0e28-ffe0-4f20-bd0f-d7ef7834r38d7'</span><span class="o">])</span> <span class="o">{</span> // jenkins credential ID
sh <span class="s1">'''#!/bin/bash
// test connect server status
'''</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
stage<span class="o">(</span><span class="s1">'Build and Test'</span><span class="o">)</span> <span class="o">{</span>
steps <span class="o">{</span>
// start build and run unit <span class="nb">test </span>cases
<span class="o">}</span>
<span class="o">}</span>
stage<span class="o">(</span><span class="s1">'Deploy'</span><span class="o">)</span> <span class="o">{</span>
steps <span class="o">{</span>
// deploy changes
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p><a href="https://plugins.jenkins.io/ssh-agent/">SSH agent</a> plugin is required to connect the remote server with pipelines. The SSH agent will take credential ID as the first parameter for authenticating the remote server.</p>
<p>Moreover, you can also notify the team with email using <code class="language-plaintext highlighter-rouge">post-build-action</code> and by selecting when any builds pass or fails.</p>
<h4 id="test-build-1">Test build</h4>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Jenkins-pipelines-build.png" alt="Jenkins pipelines build" class="full" /></p>
<p>After successfully build you will see the changes in Pipelines structure.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>We learned to automate deployment with Continuous interaction and continuous delivery from scratch. Let’s summarize what we have learned so far. We understand the basics of CI/CDs, configure Jenkins, connect remote server and GitHub with SSH and configured webhooks, learned about credentials and how they work, and finally automate deploy to the server by running bash script and pipelines using CI/CD.</p>
<p>If you would like to see similar articles related to DevOps or you face any issues, don’t forget to add comments.</p>Vijay PathakA guide to continuos integration and continuos delivery project using Jenkins, GitHub and webhooks in Ubuntu server LTS 20. Learn how to automate Django in Jenkins.Recover AWS instance when SSH private key is lost or changed.2021-09-05T00:00:00+00:002021-09-05T00:00:00+00:00https://vijaypathak.com.np/2021/09/recover-aws-instance-when-ssh-private-key-is-lost-or-changed<p>A hard day in my work taught me to be extra careful when storing private key pairs and not to lose or replace them accidentally. This can be really frustrating situation and I feel like my experience can help someone someday. Luckily if you lose access to your private key or somehow it got replaced, this guide is for you and can be restored with an AWS console without much trouble.</p>
<h1 id="how-to-recover">How to recover?</h1>
<p>There are different ways to recover your instance private key. When you log in to the console of your EC2 instance, you can right-click and connect in different alternative ways.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/Instance-connect-type.png" alt=" Instance connect types" class="full" /></p>
<h2 id="scenario">Scenario</h2>
<p>SSH client is the most used to directly connect your instance remotely from any computer, and because we lost the private key - we can’t use this anymore. There are other ways like Session Manager and EC2 serial console, which will be a little complicated for this guide so we skip it.</p>
<p>However, one possible way out is to use <code class="language-plaintext highlighter-rouge">EC2 Instance Connect</code> directly from a web browser.</p>
<h3 id="instance-connect">Instance Connect</h3>
<p>If you are an admin or have privileges to add a policy to IAM users, simply add <code class="language-plaintext highlighter-rouge">AmazonConnect_FullAccess</code> permission for your group. You click connect and you are inside your server.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/ec2-instance-connect.jpg" alt=" EC2 Instance connect" class="full" /></p>
<p>Next create a new private key and use this from now.</p>
<p>Sometimes cases are not so simple and user cannot access other options due to lack of permission for the role or some services like <code class="language-plaintext highlighter-rouge">EC2 connect</code> is not available for certain regions.</p>
<h2 id="manual-recover---the-hard-way">Manual recover - the hard way</h2>
<p>The current running instance has to be selected and stopped before proceeding forward - the standard way. We can also keep it as a running state if it’s a test server or no users/low loads are in this server.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/instance-image-create.jpg" alt=" EC2 Instance image create" class="full" /></p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/create-instance-image.png" alt=" Instance image create setting" class="full" /></p>
<p>Next, click create an image (the current is already selected) by adding a name, description. It will take some time to clone the image from the current instance, when it’s done you can view it under <code class="language-plaintext highlighter-rouge">Images > AIM</code> section.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/launch-instance.png" alt="Launch instance" class="full" /></p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/instance-launch-config.png" alt="Launch instance config" class="full" /></p>
<p>You need to select your just created Image and click the launch instance button. The new page will be opened and the AIM image will be autoselected, click next. On this page you need to review our instance size and other configuration, you can click review and lunch if you want to keep it as previous instance settings.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/private-keys.jpg" alt="Private keys setup" class="full" /></p>
<p>Finally, a new window will pop up and ask you to either select the old key pair or add a new one. If you have access to the private key but it got replaced, you can keep using an old key. Likewise, if you don’t have access to the old keys - you need to create new SSH key pair and click on lunch instances. It will take a couple of minutes to start the new instance when it is done you will be able to view it in the Instance list. At this point, the restoration is almost completed.</p>
<h2 id="change-old-instance-elastic-ip">Change old instance elastic IP</h2>
<p>You need to connect to the remote instance and a static IP is essential to do this as you already know. We replaced the instance by following the above method but the IP is still pointing to the old server and we need to change it if you still want to keep the service intact as previously working. For me, I want to be able to use the same previous private key and IP so my team members don’t have to change anything.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2021/elastic-ip-disassociate.png" alt="Elastic IP disassociate" class="full" /></p>
<p>Go to the Elastic IP page and you should be able to see the old instance IP, right-click and disassociate the elastic IP from that instance. Once this is done. Select the new instance and click allocate elastic Ip address, you should be able to change the IP. That’s all.</p>
<h1 id="at-last">At last</h1>
<p>Congratulation, you just learned how to recover an instance that is almost unusable to fully working intact with previous configurations. This is one way I recover it with the available feature available during writing this guide. Please leave a comment if you need help or suggestions.</p>Vijay PathakA guide on recovering AWS instance lost or changed private key using AWS console.Load, performance analysis of APIs with Locust and load balancing.2021-04-12T00:00:00+00:002021-04-12T00:00:00+00:00https://vijaypathak.com.np/2021/04/load-testing-and-balancing-APIs-with-locust-and-nginx<p>I have always wondered how the system performs especially when thousand or even millions of users starts using it. I have now realized it’s necessary to have a fair idea of how many users your app or APIs endpoint can handle that is freshly built and how to scale over time.</p>
<p>It is also important to understand what types of request are successful and which are failing. To understand this we will be using a lite framework called <a href="https://locust.io/">Locust</a>, even users who are not much familiar with Python language will be able to learn and use it.</p>
<h1 id="what-is-locust">What is Locust?</h1>
<p>Locust is a testing framework that uses python Scripts that is scalable and widely used as a performance testing/analysis tool. It is powerful and simple to use - you define what type of users, how many and when to swarm your system.</p>
<h2 id="what-we-plan-on-learning">What we plan on learning?</h2>
<p>One of the great features of <a href="https://locust.io/">Locust</a> is that you write all your code in Python. We’ll be writing simple test cases to understand REST API that uses JSON Web Token; simulating users in a different time interval and find some interesting results and find a way to tackle the issue.</p>
<p>Let’s fork one of my project <a href="https://github.com/hbvj99/market-api">Market API</a> for testing.</p>
<h1 id="setting-it-up">Setting it up</h1>
<p>Let’s create a virtual environment first so that we isolate the dependencies;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- mkdir locust
- cd locust && pip install virtualenv
- virtualenv venv -p python3.8 && source venv/bin/activate
- pip install locust
</code></pre></div></div>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>More info:</strong> Locust uses Flask, a light-weight Python framework to run the testing in the browser where you can view the real-time analysis visualization and even export the result after test completion.</p>
<h1 id="writing-basic-tests">Writing basic tests</h1>
<h2 id="1-authenticaitons">1. Authenticaitons</h2>
<p>Let’s create <code class="language-plaintext highlighter-rouge">locustfile.py</code> and write tests for different endpoints.</p>
<ul>
<li>touch locustfile.py credentails.py</li>
</ul>
<p>Add the login credentails in <code class="language-plaintext highlighter-rouge">credentails.py</code> file.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">email</span> <span class="o">=</span> <span class="s">'YOUR_EMAIL_ADDRESS'</span>
<span class="n">password</span> <span class="o">=</span> <span class="s">'YOUR_SECRET_PASSWORD'</span>
</code></pre></div></div>
<p>Next, we write some real tests inside <code class="language-plaintext highlighter-rouge">locustfile.py</code> file. Lets test autentications.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">locust</span> <span class="kn">import</span> <span class="n">HttpUser</span><span class="p">,</span> <span class="n">task</span><span class="p">,</span> <span class="n">between</span>
<span class="kn">from</span> <span class="nn">credentails</span> <span class="kn">import</span> <span class="n">email</span><span class="p">,</span> <span class="n">password</span>
<span class="k">class</span> <span class="nc">MarketAPI</span><span class="p">(</span><span class="n">HttpUser</span><span class="p">):</span>
<span class="n">wait_time</span> <span class="o">=</span> <span class="n">between</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">1.9</span><span class="p">)</span>
<span class="n">base_api_version</span> <span class="o">=</span> <span class="s">'api/v1'</span>
<span class="k">def</span> <span class="nf">on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">login</span><span class="p">()</span>
<span class="o">@</span><span class="n">task</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">login</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_api_version</span><span class="si">}</span><span class="s">/auth/token/"</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s">"email"</span><span class="p">:</span> <span class="n">email</span><span class="p">,</span> <span class="s">"password"</span><span class="p">:</span> <span class="n">password</span><span class="p">})</span>
<span class="k">if</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">().</span><span class="n">get</span><span class="p">(</span><span class="s">'errors'</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">access_token</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'access'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">refresh_token</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'refresh'</span><span class="p">]</span>
</code></pre></div></div>
<p>In the above code, we write the user login test case for token-based authentication by providing email and password. We need to include the <code class="language-plaintext highlighter-rouge">login</code> function inside the <code class="language-plaintext highlighter-rouge">on_start</code> function so that this executed first every time. As described in the official <a href="https://docs.locust.io/en/stable/quickstart.html">doc</a>, When a test starts, <code class="language-plaintext highlighter-rouge">HttpUser</code> class will create an instance of this class for every user that it simulates, and each of these users will create a micro-thread that will call those methods.</p>
<p>The <code class="language-plaintext highlighter-rouge">@task</code> is a decorator that will wait for users between the time defined in <code class="language-plaintext highlighter-rouge">wait_time</code> attribute for each task - we defined to wait between 0.8 and 1.9 seconds. The task decorator is defined and run in ascending order.</p>
<p>Likewise, <code class="language-plaintext highlighter-rouge">self.client</code> makes HTTP calls to the various APIs endopint, you can use HTTP methods like <code class="language-plaintext highlighter-rouge">GET</code> or <code class="language-plaintext highlighter-rouge">POST</code>.</p>
<h2 id="2-list-retrive-and-post">2. List, retrive and Post</h2>
<h3 id="list"><strong>List</strong></h3>
<p>Add <code class="language-plaintext highlighter-rouge">@task(2)</code> to list data.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">task</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">list_items</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_api_version</span><span class="si">}</span><span class="s">/products/"</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Authorization'</span><span class="p">:</span> <span class="sa">f</span><span class="s">'Bearer </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">access_token</span><span class="si">}</span><span class="s">'</span><span class="p">})</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_api_version</span><span class="si">}</span><span class="s">/products/comments/"</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Authorization'</span><span class="p">:</span> <span class="sa">f</span><span class="s">'Bearer </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">access_token</span><span class="si">}</span><span class="s">'</span><span class="p">})</span>
</code></pre></div></div>
<h3 id="retrive"><strong>Retrive</strong></h3>
<p>Likewise, we iterate and retrieve data in one endpoint point using for loops. Similarly, we can also import Python’s time <code class="language-plaintext highlighter-rouge">time.sleep()</code> to stop for each retrieval process. To retrieve each product, we write <code class="language-plaintext highlighter-rouge">@task(3)</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">task</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">view_products</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">product</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">50</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_api_version</span><span class="si">}</span><span class="s">/products/</span><span class="si">{</span><span class="n">product</span><span class="si">}</span><span class="s">/"</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Authorization'</span><span class="p">:</span> <span class="sa">f</span><span class="s">'Bearer </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">access_token</span><span class="si">}</span><span class="s">'</span><span class="p">})</span>
<span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.3</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="post"><strong>POST</strong></h3>
<p>We refresh old token and generate new one.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">task</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">refresh_token</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">new_refresh_token</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">base_api_version</span><span class="si">}</span><span class="s">/auth/token/refresh/"</span><span class="p">,</span>
<span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s">"refresh"</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">refresh_token</span><span class="p">},</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'Authorization'</span><span class="p">:</span> <span class="sa">f</span><span class="s">'Bearer </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">access_token</span><span class="si">}</span><span class="s">'</span><span class="p">})</span>
<span class="k">print</span><span class="p">(</span><span class="n">new_refresh_token</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="3-result-analysis">3. Result Analysis</h2>
<p><img src="https://vijaypathak.com.np/assets/image/2020/locust-dashboard.png" alt="Locust Dashbaord" class="full" /></p>
<p>After running the test, the results are viewable in different visual means. In dashboard home, it includes the statistics like the number of requests made, failure rate, the average time taken for various endpoints.</p>
<p>The visualizations include;</p>
<p><img src="https://vijaypathak.com.np/assets/image/2020/number_of_users.png" alt="Locust number of users" class="full" /></p>
<p>The numbers of users simulated in a different time interval and their bounce rate.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2020/response_times.png" alt="Locust response time" class="full" /></p>
<p>The response time in (ms) in real-time or after test completion, includes the comparison of percentage and the median time taken.</p>
<p><img src="https://vijaypathak.com.np/assets/image/2020/total_requests_per_second.png" alt="Locust Dashbaord" class="full" /></p>
<p>Finally, the most important - failure rate can be viewed alongside RPS (Current requests per second).</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Hint:</strong> All the raw data can be also exported in a CSV file. It is very helpful when you want to keep records of performance over time.</p>
<p>Around 100-150 users were simulated in total to tests various endpoints in the above examples. The actual users that could use the system could easily crash a single server.</p>
<h3 id="swarming-the-system"><strong>Swarming the system</strong></h3>
<p><img src="https://vijaypathak.com.np/assets/image/2020/locust-overload.png" alt="Locust Dashbaord" class="full" /></p>
<p><img src="https://vijaypathak.com.np/assets/image/2020/locust-cpu-usages.png" alt="Locust Dashbaord" class="full" /></p>
<p>Next, we simulated around 5k-7k users to use the API in total. This results - API cannot handle the majority of requests and start throttling. Moreover, even when code is optimized - CPU power will give up eventually. And we will get a warning like CPU usages is high or API cannot process too many requests.</p>
<h1 id="how-to-fix-api-not-able-to-process-requests">How to fix API not able to process requests?</h1>
<h2 id="load-balancing-with-nginx">Load balancing with Nginx</h2>
<p>One of the ways we can resolve the APIs slow down or frequent requests failure is to implement HTTP load balancing - we will use <a href="https://www.nginx.com/">Nginx</a>. This technique allows to optimize and maximizing the resource utilization while reducing latency and ensuring fault-tolerant configurations by running multiple instances of the same application as highlighted in the official <a href="http://nginx.org/en/docs/http/load_balancing.html#nginx_load_balancing_configuration">documentation</a>.</p>
<h2 id="what-is-load-balancing">What is load balancing?</h2>
<p><img src="https://vijaypathak.com.np/assets/image/2020/load-balancing.jpg" alt="Load balancing with Nginx" class="full" /></p>
<p>Load balancing is the process of distributing incoming traffic across a group of servers. The load balancer sits in front of servers and routs client requests across all servers which are capable of fulfilling those requests efficiently.</p>
<h2 id="writing-configs">Writing configs</h2>
<p>The default Nginx config files are located at <code class="language-plaintext highlighter-rouge">/etc/nginx/sites-available/</code>. You need to open your current active configure file that is serving the main application and include <code class="language-plaintext highlighter-rouge">upstream</code> and <code class="language-plaintext highlighter-rouge">location</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http <span class="o">{</span>
upstream test_api <span class="o">{</span>
<span class="c"># replace localhost with instance internal IP</span>
server host2.localhost.com<span class="p">;</span>
server host3.localhost.com<span class="p">;</span>
server host4.localhost.com<span class="p">;</span>
server host5.localhost.com<span class="p">;</span>
<span class="o">}</span>
server <span class="o">{</span>
listen 80<span class="p">;</span>
<span class="c"># enable for HTTPS i.e. letsencrypt</span>
<span class="c"># listen 443 ssl; </span>
<span class="c"># ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME/fullchain.pem;</span>
<span class="c"># ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME/privkey.pem;</span>
<span class="c"># include /etc/letsencrypt/options-ssl-nginx.conf;</span>
<span class="c"># ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;</span>
server_name localhost.com<span class="p">;</span>
location / <span class="o">{</span>
proxy_pass http://test_api<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the above example, we see many instances of the same application allocated with the same upstream name. Reverse proxy work in both HTTP and HTTPS alongside other protocols like Memcache.</p>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> The upstream name and proxy_pass name <code class="language-plaintext highlighter-rouge">(test_api)</code> must be same. It is best to use the internal IP for best security and performances in the <code class="language-plaintext highlighter-rouge">server</code>.</p>
<p>The Nginx uses <code class="language-plaintext highlighter-rouge">round-robin</code> mechanism by default which balances the request made in each server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> upstream myapp1 <span class="o">{</span>
least_conn<span class="p">;</span> <span class="c"># load balancing mechanism type</span>
server localhost:3000<span class="p">;</span>
server localhost:5000<span class="p">;</span>
server localhost:6000<span class="p">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>For example, the <code class="language-plaintext highlighter-rouge">least_conn</code> load balancing type will not try not to overload a busy application server instead distributes it to a less busy server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> upstream myapp1 <span class="o">{</span>
server localhost:3000 <span class="nv">max_conns</span><span class="o">=</span>200<span class="p">;</span>
server localhost:5000 <span class="nv">max_conns</span><span class="o">=</span>900<span class="p">;</span>
server localhost:6000<span class="p">;</span>
queue 160 <span class="nb">timeout</span><span class="o">=</span>80<span class="p">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If one of your instances is not powerful compare to another, you can also assign max connection to that particular server using <code class="language-plaintext highlighter-rouge">max_conns</code>. When the max connections are reached it is placed in a queue for further processing.</p>
<p>Likewise, there are different supported algorithms like IP hash, Least time, Session persistence, Weighted load, Health checks that are designed to specific use cases - read more in official <a href="https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method">doc</a>.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>In this guide, we learned how to load test APIs, analyse endpoints that are taking more time to load when a high volume of requests are generated. We learned serving the application using load balancing will help on solving the load time/frequent APIs crashes issues.</p>
<p>Here is the Locust script <a href="https://github.com/hbvj99/locust-test-api/tree/master">repository</a> if you want to test things out.</p>
<p>If you have any questions or maybe alternative ways to tackle the issue, please let me know.</p>Vijay PathakA guide on testing API load time, view performance report using Locust/Python, handling HTTP load balancing with Nginx.Complete guide on deploying Django application with Chaussette, Circus in AWS EC2 Ubuntu2020-10-18T00:00:00+00:002020-10-18T00:00:00+00:00https://vijaypathak.com.np/2020/10/complete-guide-on-deploying-django-application-using-chaussette-circus-in-aws-ec2-ubuntu<p>A guide on deploying Django application using Chaussette WSGI Server, Circus as process and socket manager in AWS Ubuntu 20 LTS EC2 instance.</p>
<h1 id="what-is-chaussette">What is Chaussette?</h1>
<p><a href="https://chaussette.readthedocs.io/en/1.3.0/">Chaussette</a> is a WSGI server that you can use to run your Python WSGI applications; for this article, we’ll run Django application. It can bind a socket on a port or run in already opened sockets.</p>
<p>We will be using <a href="https://circus.readthedocs.io/en/latest/">Circus</a> to manager socket/process manager.</p>
<h1 id="what-is-circus">What is Circus?</h1>
<p><a href="https://circus.readthedocs.io/en/latest/">Circus</a> is a Python program which can be used to monitor and control processes and sockets.</p>
<h1 id="setup-ssh-on-local">Setup SSH on local</h1>
<p>Once you download the PEM key after creating an instance from AWS console, move it to the <code class="language-plaintext highlighter-rouge">/home/user/.ssh</code> and add config for quick connect;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span><span class="o">></span>config
Host forum <span class="c"># name you prefer</span>
Hostname 0.0.0.0 <span class="c"># your server IP</span>
User ubuntu <span class="c"># user</span>
IdentityFile ~/.ssh/aws-n-virginia-ubuntu.pem <span class="c"># file name</span>
</code></pre></div></div>
<p>Let’s connect to the server using SSH using: <code class="language-plaintext highlighter-rouge">ssh forum</code></p>
<h1 id="server-setup">Server Setup</h1>
<p>The first thing after login is to upgrade everything.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo apt-get update && sudo apt-get -y upgrade</code></li>
</ul>
<h2 id="install-tools">Install tools</h2>
<p>Install useful tools that we will be using;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo apt install python-dev build-essential links python3-pip python3-cffi python3-wheel python3-setuptools libffi-dev libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev virtualenv git</code></li>
</ul>
<h2 id="nginx">Nginx</h2>
<p>Nginx is a reverse proxy, load balancer, mail proxy and HTTP cache - generic TCP/UDP proxy server.</p>
<p>Install Nginx, enable, setup Firewall.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo apt install nginx</code></li>
<li><code class="language-plaintext highlighter-rouge">sudo ufw app list</code> # view the list</li>
<li><code class="language-plaintext highlighter-rouge">sudo ufw allow 'Nginx HTTP'</code> # enable port 80</li>
<li><code class="language-plaintext highlighter-rouge">sudo ufw allow 'Nginx HTTPS'</code> # enable port 443</li>
<li><code class="language-plaintext highlighter-rouge">sudo ufw allow 'Nginx Full'</code> # optional</li>
<li><code class="language-plaintext highlighter-rouge">sudo systemctl enable nginx && systemctl start nginx</code></li>
</ul>
<h2 id="postgresql">PostgreSQL</h2>
<p>PostgreSQL is an open-source object-relational database system. Let’s install Postgres and setup.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo apt-get install postgresql postgresql-contrib</code></li>
<li><code class="language-plaintext highlighter-rouge">sudo systemctl enable postgresql && sudo systemctl start postgresql</code></li>
</ul>
<p>Setup database;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo -i -u postgres</code></li>
<li><code class="language-plaintext highlighter-rouge">psql</code></li>
<li><code class="language-plaintext highlighter-rouge">CREATE USER DB_USER WITH PASSWORD 'YOUR_PASSWORD';</code></li>
<li><code class="language-plaintext highlighter-rouge">CREATE DATABASE DB_NAME;</code></li>
<li><code class="language-plaintext highlighter-rouge">GRANT ALL PRIVILEGES ON DATABASE DB_NAME TO DB_USER;</code></li>
</ul>
<h2 id="setup-git-repository-in-server">Setup Git Repository in Server</h2>
<p>There are many ways on pushing local or remote code to the deployment server i.e. CI/CD, FTP. We’ll be setting up the remote server using Git.</p>
<p>I learned this great approach working for some company - this will only push the latest HEAD changes from your local dev environment. This method is useful when you don’t want to directly pull the changes from Git repository into the server.</p>
<p>Let’s name our origin as <code class="language-plaintext highlighter-rouge">server.git</code> and set up the standard folder structure. Appending .git in a directory name helps to quickly identify it as git repository.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">mkdir backend && cd backend && mkdir server.git app conf logs media static && cd server.git</code></li>
</ul>
<p>Next we setup a <code class="language-plaintext highlighter-rouge">server.git</code> as bare repo and configure.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">git init --bare && git --bare update-server-info && git config core.bare false && git config receive.denycurrentbranch ignore && git config core.worktree /home/ubuntu/backend/app/</code></li>
</ul>
<p>Add hooks for post-receive so that we checkout file every time when we do push.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> hooks/post-receive <span class="o"><<</span><span class="no">EOF</span><span class="sh">
#!/bin/bash
git checkout -f
source ../venv/bin/activate
circusctl reload
deactivate
# other scripts path here
</span><span class="no">EOF
</span></code></pre></div></div>
<p class="notice--info text-justify"><i class="fa fa-info-circle"></i> <strong>Info:</strong> <code class="language-plaintext highlighter-rouge">post-receive</code> is invoked when doing git push and updates. You could customize it to run other commands or notify user. Although anything that might take longer time is better to avoid as suggested in offical <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">doc</a>.</p>
<p>and don’t forget to make it executable using;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo chmod +x hooks/post-receive</code></li>
</ul>
<h2 id="server-configs">Server configs</h2>
<h3 id="virtual-env">Virtual Env</h3>
<p>Create a virtual environment Python 3 environment and activate</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">virtualenv venv -p python3 && source venv/bin/activate</code></li>
</ul>
<h3 id="circus-and-chaussette">Circus and Chaussette</h3>
<p>Install it using;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">pip install circus chaussette</code></li>
<li><code class="language-plaintext highlighter-rouge">cd conf && cat>circus.ini</code></li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>watcher:webapp]
cmd <span class="o">=</span> chaussette <span class="nt">--fd</span> <span class="si">$(</span>circus.sockets.webapp<span class="si">)</span> forum.wsgi.application <span class="c"># Django setting path</span>
<span class="nv">uid</span><span class="o">=</span>ubuntu
<span class="nv">endpoint_owner</span><span class="o">=</span>ubuntu <span class="c"># server user</span>
numprocesses <span class="o">=</span> 3
use_sockets <span class="o">=</span> True
copy_env <span class="o">=</span> True
copy_path <span class="o">=</span> True
virtualenv <span class="o">=</span> /home/ubuntu/backend/venv/ <span class="c"># virtualenv path</span>
stdout_stream.class <span class="o">=</span> FileStream
stdout_stream.filename <span class="o">=</span> /home/ubuntu/backend/logs/app.log
stderr_stream.class <span class="o">=</span> FileStream
stderr_stream.filename <span class="o">=</span> /home/ubuntu/backend/logs/app_err.log
<span class="c"># optionally rotate the log file when it reaches 1 gb</span>
<span class="c"># and save 3 copied of rotated files</span>
stdout_stream.max_bytes <span class="o">=</span> 1073741824
stdout_stream.backup_count <span class="o">=</span> 3
stderr_stream.max_bytes <span class="o">=</span> 1073741824
stderr_stream.backup_count <span class="o">=</span> 3
<span class="o">[</span><span class="nb">env</span>:webapp]
<span class="nv">PYTHONPATH</span><span class="o">=</span>/home/ubuntu/backend/
<span class="o">[</span>socket:webapp]
host <span class="o">=</span> 127.0.0.1
port <span class="o">=</span> 8085
</code></pre></div></div>
<p>Refer <a href="https://circus.readthedocs.io/en/latest/for-ops/configuration/">here</a> for other configs.</p>
<h2 id="config-circus-daemon">Config Circus Daemon</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">circusd --daemon circus.ini && circusctl reloadconfig && circusctl reload</code></li>
</ul>
<h3 id="config-nginx">Config Nginx</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">cat>nginx.conf</code></li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>upstream django_project <span class="o">{</span>
server 127.0.0.1:8085<span class="p">;</span> <span class="c"># for a web port socket see circus.ini [socket:webapp]</span>
<span class="c"># server unix:///path/to/your/mysite/mysite.sock; # for a file socket</span>
<span class="o">}</span>
<span class="c"># Redirect www.forum.vijaypathak.com.np to forum.vijaypathak.com.np</span>
<span class="c"># server {</span>
<span class="c"># listen 80;</span>
<span class="c"># server_name www.forum.vijaypathak.com.np;</span>
<span class="c"># return 301 https://forum.vijaypathak.com.np$request_uri;</span>
<span class="c"># }</span>
<span class="c"># Configs for server</span>
server <span class="o">{</span>
listen 80<span class="p">;</span> <span class="c"># server port</span>
server_name forum.vijaypathak.com.np<span class="p">;</span> <span class="c"># domain/IP name</span>
charset utf-8<span class="p">;</span>
<span class="c"># Enable when required</span>
<span class="c"># access_log /home/ubuntu/backend/logs/nginx.access.log;</span>
<span class="c"># error_log /home/ubuntu/backend/logs/nginx.error.log;</span>
<span class="c">#limit_conn conn_limit_per_ip 100;</span>
<span class="c">#limit_req zone=req_limit_per_ip burst=100 nodelay;</span>
<span class="c"># robots.txt path</span>
<span class="c">#location /robots.txt {</span>
<span class="c"># alias /home/ubuntu/backend/static/robots.txt;</span>
<span class="c">#}</span>
<span class="c"># Favicon path</span>
<span class="c">#location /favicon.ico {</span>
<span class="c"># alias /home/ubuntu/backend/static/img/favicon.ico;</span>
<span class="c">#}</span>
<span class="c"># Static file</span>
location /static/ <span class="o">{</span>
<span class="nb">alias</span> /home/ubuntu/backend/static/<span class="p">;</span>
<span class="o">}</span>
<span class="c"># Media file</span>
location /media/ <span class="o">{</span>
<span class="nb">alias</span> /home/ubuntu/backend/media/<span class="p">;</span>
<span class="o">}</span>
<span class="c">## Deny other host names i.e your domain/IP here</span>
<span class="k">if</span> <span class="o">(</span><span class="nv">$host</span> <span class="o">!</span>~<span class="k">*</span> ^<span class="o">(</span>forum.vijaypathak.com.np<span class="o">)</span><span class="nv">$ </span><span class="o">)</span> <span class="o">{</span>
<span class="k">return </span>444<span class="p">;</span>
<span class="o">}</span>
location / <span class="o">{</span>
proxy_set_header X-Forwarded-For <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
proxy_set_header Host <span class="nv">$http_host</span><span class="p">;</span>
proxy_set_header X-Forwarded-Proto <span class="nv">$scheme</span><span class="p">;</span>
proxy_redirect off<span class="p">;</span>
<span class="c"># proxy_set_header X-Forwarded-Proto https; #enable ssl</span>
<span class="c"># proxy_pass http://webapp;</span>
client_max_body_size 50m<span class="p">;</span>
add_header Strict-Transport-Security <span class="s2">"max-age=63072000; includeSubDomains; preload"</span> always<span class="p">;</span> <span class="c"># 2 years</span>
add_header X-Content-Type-Options nosniff<span class="p">;</span>
add_header X-XSS-Protection <span class="s2">"1; mode=block"</span><span class="p">;</span>
add_header X-Frame-Options SAMEORIGIN<span class="p">;</span>
<span class="c"># Enable when required</span>
<span class="c"># CSP allowing popular third party integrations</span>
add_header Content-Security-Policy <span class="s2">"default-src 'self' ; script-src 'self' 'unsafe-inline' 'unsafe-eval' adservice.google.com adservice.google.com.np pagead2.googlesyndication.com d31qbv1cthcecs.cloudfront.net www.google-analytics.com cdn.ravenjs.com connect.facebook.net platform.twitter.com apis.google.com www.google.com www.gstatic.com maps.googleapis.com; connect-src 'self' googleads.g.doubleclick.net fonts.gstatic.com wss: securepubads.g.doubleclick.net d5nxst8fruw4z.cloudfront.net sentry.io maps.gstatic.com www.google-analytics.com certify.alexametrics.com; img-src 'self' data: certify.alexametrics.com maps.gstatic.com maps.googleapis.com d5nxst8fruw4z.cloudfront.net www.google-analytics.com stats.g.doubleclick.net ssl.gstatic.com csi.gstatic.com www.facebook.com syndication.twitter.com www.gravatar.com pagead2.googlesyndication.com; style-src 'self' 'unsafe-inline' fonts.googleapis.com; font-src 'self' data: cdn.vijaypathak.com.np fonts.gstatic.com; frame-src googleads.g.doubleclick.net www.youtube.com accounts.google.com content.googleapis.com www.facebook.com staticxx.facebook.com platform.twitter.com; manifest-src 'self'; worker-src 'self' fonts.gstatic.com"</span><span class="p">;</span>
<span class="o">}</span>
<span class="c"># Deny all hidden files and directory being served</span>
<span class="c"># Append in bottom</span>
location ~ /<span class="se">\.</span> <span class="o">{</span>
access_log off<span class="p">;</span>
log_not_found off<span class="p">;</span>
deny all<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Read more on Nginx configs <a href="https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html#configure-nginx-for-your-site">here</a>.</p>
<h3 id="add-nginx-config-to-etcnginxnginxconf">Add nginx config to /etc/nginx/nginx.conf</h3>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo nano /etc/nginx/nginx.conf</code></li>
</ul>
<p>Include Nginx config inside http;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>include /home/ubuntu/backend/conf/nginx.conf;
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo nginx -t && sudo systemctl restart nginx</code></li>
</ul>
<h3 id="migrations">Migrations</h3>
<p>Run migration to generate table and relations in the database</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">python manage.py makemigrations && python manage.py migrate</code></li>
</ul>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Note:</strong> We are using Chaussette for serving our Application hence running the server manually like in local dev environment is not required.</p>
<h3 id="ssl-certificate-with-certbot">SSL certificate with Certbot</h3>
<p>Install Let’s Encrypt SSL (auto-renew/valid for 90days)</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo apt-get install python3-certbot-nginx</code></li>
<li><code class="language-plaintext highlighter-rouge">sudo certbot --nginx</code> # run cerbot to get SSL for domain</li>
</ul>
<p>Finally test auto-renewal of SSL;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">sudo certbot renew --dry-run</code></li>
</ul>
<h2 id="include-this-in-django-settings-for-better-security">Include this in Django Settings for better Security</h2>
<h3 id="use-https-ssl">Use HTTPS (SSL)</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- SESSION_COOKIE_SECURE = True
- SECURE_HSTS_SECONDS = 63072000 # 2 years
- SECURE_HSTS_INCLUDE_SUBDOMAINS = True
- SECURE_HSTS_PRELOAD = True
- SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
- SECURE_BROWSER_XSS_FILTER = True
- SECURE_CONTENT_TYPE_NOSNIFF = True # Prevents Cross-Site scripts
- SECURE_SSL_REDIRECT = True # Include 'django.middleware.security.- SecurityMiddleware' at top in Middleware
- SECURE_REFERRER_POLICY = 'same-origin'
- CSRF_COOKIE_SECURE = True
</code></pre></div></div>
<h3 id="other-headers">Other headers</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- USE_X_FORWARDED_HOST = True
- X_FRAME_OPTIONS = 'DENY' # Include 'django.middleware.clickjacking.XFrameOptionsMiddleware' in Middleware to Prevent Clickjacking
</code></pre></div></div>
<h1 id="local-git-setup">Local Git setup</h1>
<p>Let’s pull a Django application from remote Github to local first using;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/hbvj99/forum.git
</code></pre></div></div>
<p>Make sure that you have separate .env file and add your new secret key, database, SMTP or other credentials here.</p>
<p>Great, next we set up the remote server in local - <a href="https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server">more info</a>.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">git remote add APP_NAME forum@:/home/ubuntu/backend/server.git/</code></li>
<li>
<p>Check if you successfully set up the remote URL with <code class="language-plaintext highlighter-rouge">git remote -v</code></p>
</li>
<li><code class="language-plaintext highlighter-rouge">git push APP_NAME --all</code> local branches will be pushed on server, doesn’t require remote as upstream branch</li>
</ul>
<p class="notice--info text-justify"><i class="fa fa-info-circle"></i> <strong>Info:</strong> <code class="language-plaintext highlighter-rouge">APP_NAME</code> can be any name you prefer. <code class="language-plaintext highlighter-rouge">forum</code> is our ssh name that we set <a href="#setup-ssh-on-local">earlier</a>.</p>
<h1 id="final-things-to-consider">Final things to consider</h1>
<p>Finally, we check if our application ready is ready for production - it shouldn’t detect any issues when running;</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">python manage.py check --deploy</code></li>
</ul>
<p>Also, don’t forget to check official deploy <a href="https://docs.djangoproject.com/en/3.1/howto/deployment/checklist/">check list</a>.</p>
<p>If you made to here; our site is Live now with a custom domain, SSL, SMTP and following decent security measure.</p>
<p>At last, get security analysis of your website on <a href="https://observatory.mozilla.org/">Mozilla Observatoty</a>.</p>
<h2 id="commands">Commands</h2>
<p>After you push the latest changes to the server, you need to run</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">circusctl reload</code></li>
</ul>
<p class="notice--info text-justify"><i class="far fa-sticky-note"></i> <strong>Tips:</strong> You can add above command in git hooks <code class="language-plaintext highlighter-rouge">post-receive</code> to auto run when changes is detected.</p>
<h3 id="folder-structure">Folder Structure</h3>
<p>Here’s the Final folder/file structure if you quickly want to check;</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">.</span>
<span class="err">├──</span> <span class="n">backend</span>
<span class="err">└──</span> <span class="n">server</span><span class="p">.</span><span class="n">git</span> <span class="c1"># repo name
</span> <span class="err">──</span> <span class="n">HEAD</span>
<span class="err">└──</span> <span class="n">braches</span>
<span class="err">───</span> <span class="n">config</span>
<span class="err">└──</span> <span class="n">description</span>
<span class="err">└──</span> <span class="n">hooks</span>
<span class="err">───</span> <span class="n">index</span>
<span class="err">└──</span> <span class="n">info</span>
<span class="err">└──</span> <span class="n">logs</span>
<span class="err">└──</span> <span class="n">objects</span>
<span class="err">└──</span> <span class="n">refs</span>
<span class="err">└──</span> <span class="n">heads</span>
<span class="err">└──</span> <span class="n">tags</span>
<span class="err">└──</span> <span class="n">app</span>
<span class="err">└──</span> <span class="n">YOUR_FILES_HERE</span>
<span class="err">└──</span> <span class="n">conf</span>
<span class="err">───</span> <span class="n">circus</span><span class="p">.</span><span class="n">ini</span>
<span class="err">───</span> <span class="n">nginx</span><span class="p">.</span><span class="n">conf</span>
<span class="err">└──</span> <span class="n">logs</span>
<span class="err">───</span> <span class="n">app</span><span class="p">.</span><span class="n">log</span>
<span class="err">───</span> <span class="n">app_err</span><span class="p">.</span><span class="n">log</span>
<span class="err">└──</span> <span class="n">media</span>
<span class="err">└──</span> <span class="n">static</span>
<span class="err">└──</span> <span class="n">venv</span>
</code></pre></div></div>
<p>If you have any confusions or better approach that helps on improving the guide please comment down below.</p>Vijay PathakA guide to deploy your very own Django application using Chaussette WSGI Server, Circus as process and socket manager in AWS Ubuntu EC2.Working full-time remotely, it is for you?2020-04-03T00:00:00+00:002020-04-03T00:00:00+00:00https://vijaypathak.com.np/2020/04/working-full-time-remotely-pros-cons<h1 id="remote-worker">Remote worker</h1>
<p>With the recent trend, distributed teams are growing rapidly and it seems like more companies are changing, adding work from home option. “Work from home” and “remote work” are different in nature in term of the full-time job though they sound and work similarly.</p>
<p>For instance, a remote worker may not always have a physical office near to their location and team can be distributed all over the world - working on a specific domain i.e. software development. While people usually “work from home” for a certain period of time and they probably have a physical office too.</p>
<p>Now, you can imagine why remote work could be great. I personally had a pleasant experience working full time as a software developer. But there were obviously some of the things I missed comparing to traditional office-based jobs.</p>
<h1 id="things-you-will-enjoy">Things you will enjoy</h1>
<h2 id="work-flexibility-and-location">Work flexibility and Location</h2>
<p>First thing, work flexibility is great! You could be working from your home or a cafe in the morning and take a couple of hours break to meet with your friends, and you can continue later.</p>
<p>The location doesn’t matter although you need to have a stable internet connection since you will be connected with the team most of the time and use tools like Google Meet, Zoom, Asana, Slack. You will get the chance to work together with Clients, Co-workers all over the world in various time zone - the best part, again this will differs slightly to your profession and position.</p>
<blockquote>
<p>In situations where self-isolation is essential like COVID-19, you are already working from home.</p>
</blockquote>
<h2 id="learning-opportunities-and-high-productivity">Learning opportunities and High productivity</h2>
<p>If you are a beginner or the first time, you will love the workforce. The first thing I enjoyed was the team collaboration and pair programming. You will likely get a mentor if you are starting fresh and you might also have a chance to brush your skills through online learnings - usually included.</p>
<p>Likewise, you will make friends that will help you and vice versa. Apart from the work, you will also have an opportunity to travel and join meetups or events happening around the world.</p>
<p>In my opinion, after working for a few weeks, you will soon realize your level of skill. You will likely improve your weakness and look for suggestion and you will work harder, this is because you are directly connected to high skills workers around you. Moreover, it will also help you to increase your soft skills such as communication, time management.</p>
<h2 id="save-costs-and-health">Save costs and health</h2>
<p>It saves you money; you don’t have to spend on the commute, lunch breaks. You can use this time to read a book or play a game or do something that would de-stress you. For me, I started writing a blog.</p>
<p>When you are not travelling and often working from home, you have time to incorporate physical exercises and when you are sick, staying home allows you to take care of yourself while working together- no more junk foods.</p>
<h1 id="things-you-might-not-like">Things you might not like</h1>
<h2 id="social-life-and-technology">Social life and technology</h2>
<p>Let’s be clear here, there are tons of things in your home that might distract you completely while trying to focus on particular issue especially when the deadline is near. You can relocate to a peaceful environment but repeating this is harder since you will not be saving costs. I think ones need to find the best suitable hours for productivity but this will not work if you are pair programming.</p>
<p>You may think working from home is a great chance to balance work/life but it really isn’t, your workplace and home blends it together. You might develop a habit of checking notification and calendar constantly even when you are already off for the day. Likewise, if you are working on technology as a programmer you will likely spend over-hours fixing bugs or many times deciding which is better design and end of the day you might not have time for yourself.</p>
<p>Similarly, working remotely means completely depending upon the technology and having the right tools. If you aren’t familiar or comfortable with video conferencing, the rights tools that are used by your colleagues like team management, you might miss out if your employer doesn’t provide you with a time frame to learn those stuff or you aren’t provided access to those tools.</p>
<p>If you are a type of person who actively loves outdoors activities and meeting new people, you might be anxious not being able to make face-to-face communication, spending more time in one place - I remember, one of my colleagues dropping out because of this very reason. On the other hand, this could be a plus point for others.</p>
<h2 id="language-barrier-and-open-mind">Language barrier and open-mind</h2>
<p>This is the most important of all, proficiency in the second language is a must if your native language is not English. Without right and clear communication with the team members, there is really nothing you can do. I would say slightly above intermediate to fluent is good, you need to also practise listening dialogues, slangs and different accents for improvements.</p>
<p>In spite of working with the right tools, there is a high chance that you will miss something in the team meetings if you don’t ask questions. Having friends, helpful people is also important if you got stuck somewhere - they are the one to help you. You are constantly talking with new peoples when video conferencing, and you need to understand them quickly and brushing English skills is important, listening podcasts, reading books helps a lot. Let’s be honest here, we sometimes don’t understand and get nervous when dealing with clients.</p>
<blockquote>
<p>The ability to precisely redirect the same queries to other team member that he/she could definately answer, that’s the skill we need to master.</p>
</blockquote>
<p>Learning new things is very important, as you practise there is a chance that you will make mistakes, a lot sometimes. Open to the criticism and improvements are extremely essential, this is the reason you need to work as an open-minded person - your team will like you more.</p>
<h2 id="things-to-understand-first">Things to understand first</h2>
<p>There are some of the things that you need to have a clear understanding before involving with any organizations. Most of the remote work is based on a full-time, part-time contract base. You need to first understand the what type of work you will do after joining as an employee, rather than reading reviews ask for opinions from ex-employees - you can find them in Linkedin, might help to crack interviews.</p>
<p>Likewise, when you have a first interview or meeting, always be confident and ask if they are planning to hire for long-time or for a specific period of time and be clear about the benefits that you will get.</p>
<p>Especially, if the organization don’t have a physical office in your city, there is a high chance that you are being recruited by a third party agency to work for them, this is not necessarily a bad thing if you are starting fresh but you need to have a clear idea about further plans - usually pays less, many of such partnership keep changing.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>At last, working remotely full-time could be a enjoyable experience, I tried to display my writing in helpful ways as you might have noticed. I believe there are unique factor varies to every organization depending on principal and transparency. If you are considering working remotely, you should definitely give it a try.</p>
<p>I’m writing primarily from my personal experience if you have a suggestion that I might have missed? Please don’t forget to write them in comments.</p>Vijay PathakWork from home pros and cons - my experience being a full time remote workerInternet Explorer detection using javascript2020-02-02T00:00:00+00:002020-02-02T00:00:00+00:00https://vijaypathak.com.np/2020/02/internet-explorer-detection-using-javascript<h1 id="internet-explorer">Internet Explorer</h1>
<p>Back in the early days when using Dail-up or ADSL connections, Internet Explorer browser was well liked browser offered by Microsoft to the Windows line of operating systems which were started in 1995. It is still present in most of the Windows Operating System like Windows 10 and now replaced by Edge browser which is the sucessor to IE.</p>
<h2 id="modern-applications-and-ie">Modern applications and IE</h2>
<p>Most of the present-day application relays on JavaScript frameworks like Angular or React for better UI/UX handling and to minimize the load time to the server. Also your modern app/website will likely to work in IE, but with the catch that some of the design or features might not operate as your client wish. To fix this issue, we could include Conditional <a href="https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/compatibility/ms537512(v=vs.85)?redirectedfrom=MSDN#examples">comments</a> that only IE can understand and are ignored by any other modern browsers like Google Chrome, Safari, Firefox. However, using such conditional comments is <a href="https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/compatibility/hh801214(v=vs.85)?redirectedfrom=MSDN">not supported</a> by all IE <a href="https://en.wikipedia.org/wiki/Internet_Explorer_version_history">versions</a>, it works in IE <= 9 but version 10 and 11 aren’t supported. This is the reason why our <code class="language-plaintext highlighter-rouge">code</code> targeted to IE browsers doesn’t work.</p>
<p>We need to detect specific features of the browsers to display custom message to each browser. To do this, we could use browser <a href="https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/samples/hh273397(v=vs.85)?redirectedfrom=MSDN">feature detection</a> using JavaScript.</p>
<h1 id="browser-detection">Browser detection</h1>
<p>There are various ways to detect browser through features, user agent or even prefix detection which all have limations that isn’t reliable. For instance, you could disable user agent through browser plugins or extension, detecting WebKit engine version to specify browser isn’t reliable either beacause Chrome, Safari, and Opera are <a href="https://en.wikipedia.org/wiki/List_of_web_browsers">WebKit-based</a>.</p>
<blockquote>
<p>“If it walks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.”</p>
</blockquote>
<p>We could relay on something like <a href="https://devopedia.org/duck-typing">Duck Typing</a> for browser detection which works on all versions of IE, i.e 6-11. The idea is very simple, in duck typing, an object’s suitability is determined by the presence of certain methods and properties. We will detect certian feature that are unqiue to the browser.</p>
<h1 id="development">Development</h1>
<p>To start, we can simply use <code class="language-plaintext highlighter-rouge">isIE = /*@cc_on!@*/false</code>, this will set <code class="language-plaintext highlighter-rouge">isIE</code> to <code class="language-plaintext highlighter-rouge">false</code> in all browser and they will ignore it as a comment beacause only IE detects <code class="language-plaintext highlighter-rouge">!</code> (negation) in the conditional comment. The IE would sees it as; <code class="language-plaintext highlighter-rouge">isIE=!false</code> which the statment will be true, explained in book JavaScript Pattern by Stoyan Stefanov <a href="https://books.google.com.np/books?id=WTZqecc9olUC&pg=PA206&lpg=PA206&dq=isIE+%3D+/*@cc_on!@*/false&source=bl&ots=KcKmYhpNUu&sig=ACfU3U0DcTj7kvOjtp1TaiEJKJ1mQmcTJA&hl=en&sa=X&ved=2ahUKEwiOnc27trPnAhXQyjgGHWbJBBUQ6AEwBHoECAoQAQ#v=onepage&q=isIE%20%3D%20%2F*%40cc_on!%40*%2Ffalse&f=false">section</a>.</p>
<p>We change the <code class="language-plaintext highlighter-rouge">display</code> property to <code class="language-plaintext highlighter-rouge">block</code> and whenever IE browser is detected through duck typing, this way the <code class="language-plaintext highlighter-rouge">div</code> block holding browser message appears in block style.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">function</span> <span class="nx">checkIE</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">isIE</span> <span class="o">=</span> <span class="cm">/*@cc_on!@*/</span><span class="kc">false</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">"</span><span class="s2">detectIE</span><span class="dl">"</span><span class="p">).</span><span class="nx">style</span><span class="p">.</span><span class="nx">display</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">block</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Finally, we create a simple block of html and name the ID <code class="language-plaintext highlighter-rouge">detectIE</code> for the main div container. Then we simply add CSS style <code class="language-plaintext highlighter-rouge">display</code> property to <code class="language-plaintext highlighter-rouge">none</code>, this will disable message for other browsers.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><div</span> <span class="na">id=</span><span class="s">"detectIE"</span> <span class="na">style=</span><span class="s">"display: none;"</span><span class="nt">></span>
<span class="nt"><h2></span>Browser is not supported<span class="nt"></h2></span>
<span class="nt"><p></span>{Your browser is not supported by Energidata Portal (EDP). This may mean that there are important features that you do not have access to. We recommend using one of the following browsers: Google Chrome, Microsoft Edge, Apple Safari, Morzilla Firefox.<span class="nt"></p></span>
<span class="nt"></div></span>
</code></pre></div></div>
<h1 id="at-last">At last</h1>
<p>This might be the most realiable way to detect IE browser if you wish to accomplish small tasks such as displaying alert message for IE browser users.</p>
<p>You can use similar tools like <a href="https://browser-update.org/">browser-update</a> to notify visitors about their web browser versions in order to use your web application.</p>Vijay PathakGuide on detecting IE browser and adding custom warning messageLiquid Whitespaces in Jekyll2019-11-15T00:00:00+00:002019-11-15T00:00:00+00:00https://vijaypathak.com.np/2019/11/liquid-whitespaces-in-jekyll<h1 id="whitespaces-and-liquid-template">Whitespaces and Liquid template</h1>
<p>Whitespaces are common blank spaces that you would see in the HTML document when rendered, it can be horizontal or vertical. You may not have noticed whitespaces when using Jekyll unless you view source, it by default uses <a target="_blank" href="https://jekyllrb.com/docs/liquid/">Liquid</a> templating language to process document like this blog which runs in <a target="_blank" href="https://github.com/mmistakes/minimal-mistakes">minimial-mistakes</a> for Jekyll. Liquid version 4 tags eliminate this behavior.</p>
<p>For example, In <code class="language-plaintext highlighter-rouge">_layouts > home.html</code>, we run simple <code class="language-plaintext highlighter-rouge">{{ content }}</code> tag to output whitespaces in HTML.</p>
<p>Input:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {{ content }}
<h3 class="archive__subtitle">{{ site.data.ui-text[site.locale].recent_posts | default: "Recent Posts" }}</h3>
{% for post in paginator.posts %}
{% include archive-single.html %}
{% endfor %}
{% include paginator.html %}
</code></pre></div></div>
<p>Output:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o"><</span><span class="n">h3</span> <span class="k">class</span><span class="o">=</span><span class="s2">"archive__subtitle"</span><span class="o">></span><span class="no">Recent</span> <span class="n">posts</span><span class="o"><</span><span class="sr">/h3>
</span></code></pre></div></div>
<h1 id="what-causes-this-issue">What causes this issue?</h1>
<p>The liquid version 3 generates white spaces between <code class="language-plaintext highlighter-rouge">{{ content }}</code> tag even when any conditional statements are true or false. There are lots of such statements in <code class="language-plaintext highlighter-rouge">_layouts</code>, <code class="language-plaintext highlighter-rouge">_includes</code> folder which results in occasional white spaces between any tags used.</p>
<h1 id="the-fix">The Fix?</h1>
<p>Liquid 4 has whitespace control to fix this error by updating tags. You are in luck if you already host Jekyll blog in GitHub Pages, its <a target="_blank" href="https://pages.github.com/versions/"> dependency</a> already use liquid version 4.</p>
<p>To fix this issue, Simply update all older liquid 3 tags <code class="language-plaintext highlighter-rouge">{% content %}</code> to <code class="language-plaintext highlighter-rouge">{%- content -%}</code> which is liquid tags version 4 inside <code class="language-plaintext highlighter-rouge">_layouts</code>, <code class="language-plaintext highlighter-rouge">_includes</code> folder. You can use, find in folder option and replace all old tags with new one if your are using Visual Studio Code referring <a target="_blank" href="https://shopify.github.io/liquid/basics/whitespace/"> whitespace control</a> docs.</p>
<p>Push your blog to GitHub pages to see changes or run <code class="language-plaintext highlighter-rouge">update bundler</code>, if your use Jekyll with bundler.</p>
<h1 id="conclusion">Conclusion</h1>
<p>The above simple method upgrades your Liquid templating in Jekyll. Your HTML file size decreases by slightly and you would expect to increase performance although the noticeable difference is smaller.</p>
<p>If you are using <a href="https://github.com/mmistakes/minimal-mistakes">Minimal Mistakes</a> theme like this blog; according to <a href="https://github.com/mmistakes/minimal-mistakes/pull/2608">Michael</a>, it is better to get rid of whitespaces through minifying/third-party plugin. This simple hack won’t get incluced in the <a href="https://github.com/mmistakes/minimal-mistakes/tree/master">master</a> branch anytime soon!</p>Vijay PathakGuide on removing extra whitespaces in JekyllNetTv Box Easy Hack2019-11-04T00:00:00+00:002019-11-04T00:00:00+00:00https://vijaypathak.com.np/2019/11/nettv-box-hack<h1 id="nettv-iptv-box-easy-hack">NetTv IPTV box easy hack</h1>
<p>This is a simple guide to install Android TV OS skin/bypass restrictions on NetTv devices. You will be able to use the Android platform into your television after successfully flashing the <b>proper</b> firmware. NetTv interface is basically a custom or limited software that runs on the top of any Android TV OS skin Box. It requires a device disassemble to find the correct model number.</p>
<p>The below procedure can also be applied to fix any bricked device. You may require to short NAND pins if your device won’t boot. Most of the firmware are available pre-rooted.</p>
<blockquote>
<p><b>Disclaimer: This is a non-proper guide to install android tv firmware to your NetTv box. It is solely done for educational purposes. I’m not responsible for any physical damage or bricking of the devices that you might encounter. Please proceed on your own risk. thank you
</b></p>
</blockquote>
<h1 id="method-1--custom-firmware">Method 1 : Custom Firmware</h1>
<h2 id="how-it-works">How it works?</h2>
<ul>
<li>Understand your device model number, chipset, specs</li>
<li>Download correct firmware</li>
<li>Understand the flashing procedure</li>
<li>Install Android</li>
</ul>
<h2 id="requirement">Requirement</h2>
<ul>
<li>USB 2.0 Type A Male to Male connector</li>
<li>USB burning tool for Amglogic <a href="https://androiddatahost.com/5yaux" target="_blank">chipset</a>, for Rockchip <a href="https://androiddatahost.com/5yaux" target="_blank">device</a>. Visit <a href="https://androidmtk.com/category/drivers" target="_blank">here</a> for different chipset</li>
<li>Firmware for Vianet devices model Amglogic <a href="https://drive.google.com/open?id=1vujacdrzMZI5kcKKqBUzBYT9eidP9g-s" target="_blank">S905x</a> 1GB RAM/4GBROM, get older worldlink firmware for model MXQ <a href="http://firmware.mxqproject.com/index.php/2018/04/04/mxq-4k-rockchip-3229-android-nougat-firmware-update-files/" target="_blank">RK3229</a> 1GB RAM</li>
<li>Windows 7 or plus</li>
</ul>
<h2 id="procedure-for-vianet-amlogic-s905x">Procedure (for Vianet Amlogic S905X)</h2>
<p>(1). Remove Phillips screw hidden under four soft pads.</p>
<p><img src="https://user-images.githubusercontent.com/43197293/65968537-0ce9ac00-e483-11e9-9c01-7d9d746ca94f.png" alt="Screenshot from 2019-10-01 19-30-52" /></p>
<p>(2). Note the device model number. In the below image, the device is using Amlogic S905X. Your device name may vary, in case of different name X, search model X in google and its specs or chipset.</p>
<p><img src="https://user-images.githubusercontent.com/43197293/66922432-cc735c00-f046-11e9-94fe-32286bb61841.jpg" alt="IMG_20190815_114509__01" /></p>
<p><img src="https://user-images.githubusercontent.com/43197293/67307640-858ad800-f518-11e9-8744-b66b5f14a983.png" alt="usb" /></p>
<p>(3). Change language on USB burning tool by navigating to the top right second tab, click and choose the English language. Load the firmware image file by clicking file>import image. The tool will verify the file and click START when completed. In the configuration, choose normal erase and erase bootloader option. <b>Please find the correct firmware image to continue. The file extension should be .IMG</b></p>
<p>(4). Hold the device reset button for about 7 seconds, usually located behind SPDIF port or sometimes AV port. Use a toothpick to hold reset pin and connect Type-A male connector from your device USB port (use bootable USB port) to your PC USB port. Plug the power cord to the device.</p>
<p>(5). Once the device is detected on the USB burning tool, the device ID is shown and the flashing procedure will continue.</p>
<p>(6). Wait for flashing or download system procedure to 100% and burned successfully text is shown (this might usually take 3 to 5 minutes).</p>
<p>(7). The firmware update is completed. The device may take a few minutes to start a fresh new boot in Android TV platform.</p>
<hr />
<h2 id="images-after-fresh-install">Images after fresh install</h2>
<p><img src="https://user-images.githubusercontent.com/43197293/67455257-d7cd1580-f64c-11e9-8702-e0ad8d79a925.jpeg" alt="IMG_20190815_132345-01" />
<img src="https://user-images.githubusercontent.com/43197293/67455258-d7cd1580-f64c-11e9-8341-73aabab508f5.jpeg" alt="IMG_20190815_132430-01" />
<img src="https://user-images.githubusercontent.com/43197293/67455259-d865ac00-f64c-11e9-8707-d4697a377c04.jpeg" alt="IMG_20190815_132921-01" />
<img src="https://user-images.githubusercontent.com/43197293/67455260-d865ac00-f64c-11e9-97fe-28fb19f8025c.jpeg" alt="IMG_20190815_143723-01" />
<img src="https://user-images.githubusercontent.com/43197293/67455261-d8fe4280-f64c-11e9-90a0-2e26dff3d147.jpeg" alt="IMG_20190815_143933-01" />
<img src="https://user-images.githubusercontent.com/43197293/67455263-d8fe4280-f64c-11e9-8720-1433ebc6f303.jpeg" alt="IMG_20190815_144527-01" /></p>
<h1 id="method-2-install-on-the-top-of-offical-nettv">Method 2: Install on the top of offical NetTv</h1>
<h2 id="process">Process</h2>
<p><a href="#method-1--custom-firmware">Method 1</a> is fairly limited and you need to disassemble your device before proceeding, mostly it’s done for testing purposes since your official NetTv services won’t work after.</p>
<p>This is the solid alternative, let me explain. We are going to bypass and access the hidden <a href="https://play.google.com/">Google play store</a> to install other Android apps. This is relatively easy and we aren’t modifying firmware this time so we can install updates from ISPs in future without a problem.</p>
<blockquote>
<p>We can do this because NetTv devices are simply an Android TV OS skin limited by third party software and we are going to bypass this.</p>
</blockquote>
<p>Steps to follow;</p>
<ol>
<li>First, grab the USB mouse and keyboard and attach both to the device backside USB ports. Disconnect the LAN Ethernet port from the device and connect to your wifi by navigating to net tv settings>network.</li>
<li>Next, press Win(Windows logo key)+b from the keyboard, a browser will appear.</li>
<li>Goto <code class="language-plaintext highlighter-rouge">https://play.google.com/store/apps</code> in browser URL and try to install an app, you will be redirected to Google sign-in page.</li>
<li>After signing, install a <a href="https://play.google.com/store/apps/details?id=ca.dstudio.atvlauncher.free">tv launcher</a> from play store, after installing press home key from remote and select the new launcher as default.</li>
<li>Make sure you download some apps externally from the browser (we’re going to download youtube for android tv app since it is incompatible in play store), the default storage location is internal>Download.</li>
<li>Now you can open play store app and install various apps as you may like, but as I have noticed some firmware versions have disabled installing external apps from storage. We’re going to enable this since most of the devices come pre-rooted.</li>
<li>We’re going to install a <a href="https://play.google.com/store/apps/details?id=jackpal.androidterm">terminal</a> from play store first and open.</li>
<li>Next, enable superuser by typing <code class="language-plaintext highlighter-rouge">su</code> as a command.</li>
<li>We are going to enable unknown source apps by typing command <code class="language-plaintext highlighter-rouge">settings put global install_non_market_apps 1</code> This is important for users who cannot access Android settings and manually enable.</li>
<li>Install apps by typing command: <code class="language-plaintext highlighter-rouge">pm install /storage/emulated/0/Download/your_app_name.apk</code> Follow this command to install different apps.</li>
</ol>
<hr />
<h3 id="image">Image</h3>
<p><img src="https://user-images.githubusercontent.com/43197293/83907736-a5a89800-a785-11ea-8cbf-7a5f5ec4e26d.jpg" alt="IMG_20200603_181154__01" /></p>
<blockquote>
<p>Note: If you are having connection issues, you can remove LAN Ethernet port and connect through wifi to run apps that require an active internet connection.</p>
</blockquote>
<p>That’s it. Your apps will now be available in launcher apps. Please feel free to comment down below if you have any questions?</p>
<h2 id="contribution">Contribution</h2>
<p>You can modify the content, optimize the guide by sending pull <a href="https://github.com/hbvj99/nettv-box/pulls">requests</a>.</p>Vijay PathakA guide to install a Android TV OS skin/bypass restrictions in NetTv IPTV devices. Tested in Vianet, WorldLink ISPs.Air Pollution Visualization using Open Data2019-01-27T00:00:00+00:002019-01-27T00:00:00+00:00https://vijaypathak.com.np/2019/01/air-pollution-visualization-using-open-data<p>This is a small project I did when I was in college, the use of open data to create effective air pollution awareness among the community using JQuery. View how the air quality is changing, health effect it offers using chart visualization provided by CanvasJS, save the visualization in image or PDF document.</p>
<h1 id="air-pollution">Air pollution?</h1>
<p>Emissions is the term used to describe the gases and particles which are suspended or emitted in the air through various sources which results in formation of toxics substances which includes carbon monoxide, nitrous oxides, methane, chlorofluorocarbons gases often harmful to human health when consumed indirectly at Hazardous levels - detrimental to human health and the planet, major factor for <a href="https://www.epa.gov/clean-air-act-overview/air-pollution-current-and-future-challenges#limiting">Climate Change</a>.</p>
<h2 id="how-it-works">How it works?</h2>
<ul>
<li>Fetch open-source data available at <a href="https://www.epa.gov/outdoor-air-quality-data" target="_blank">EPA</a></li>
<li>Render in charts using <a href="https://canvasjs.com/" target="_blank">CanvasJS</a></li>
<li>Display health messages based on current air</li>
<li>Visualize AQI, PM2.5 using AQI color code</li>
</ul>
<h2 id="atmospheric-particulates-data-available">Atmospheric particulates data available</h2>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Air_quality_index" target="_blank">AQI</a></li>
<li>PM2.5<a href="https://en.wikipedia.org/wiki/NowCast_(air_quality_index)" target="_blank"> NowCast </a>Concentration</li>
<li><a href="https://en.wikipedia.org/wiki/Particulates">PM2.5</a></li>
</ul>
<h2 id="requirements">Requirements</h2>
<ul>
<li>CanvasJS 2.0, get free license if you are a student <a href="https://canvasjs.com/apply/student/">here</a></li>
<li>Query 3, get <a href="https://jquery.com/download/">here</a></li>
</ul>
<h2 id="how-it-works-1">How it works?</h2>
<p>Lets first create a simple CanvasJS, enable some of the useful features like animation, theme, interactive, tooltip, subtitles inside <code class="language-plaintext highlighter-rouge">chart_data</code></p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">chart_data</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">animationEnabled</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">exportEnabled</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">exportFileName</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Phora Durbar Live AQI</span><span class="dl">"</span><span class="p">,</span>
<span class="na">animationDuration</span><span class="p">:</span> <span class="mi">1400</span><span class="p">,</span>
<span class="na">zoomEnabled</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">zoomType</span><span class="p">:</span> <span class="dl">"</span><span class="s2">xy</span><span class="dl">"</span><span class="p">,</span>
<span class="na">theme</span><span class="p">:</span> <span class="dl">"</span><span class="s2">light1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">interactivityEnabled</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">toolTip</span><span class="p">:</span> <span class="p">{</span>
<span class="na">content</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Date: {label}<br/>Data: {y}</span><span class="dl">"</span>
<span class="p">},</span>
<span class="na">axisY</span><span class="p">:</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="dl">"</span><span class="s2">AQI</span><span class="dl">"</span><span class="p">,</span>
<span class="na">valueFormatString</span><span class="p">:</span> <span class="dl">"</span><span class="s2">#</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">title</span><span class="p">:</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="dl">"</span><span class="s2">AQI</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">subtitles</span><span class="p">:</span> <span class="p">[{</span>
<span class="na">text</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Phora Durbar, Kathmandu</span><span class="dl">"</span>
<span class="p">}],</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Next, we’ll define empty Array <code class="language-plaintext highlighter-rouge">dataPoints</code> in <code class="language-plaintext highlighter-rouge">chart_data</code></p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">data</span><span class="p">:</span> <span class="p">[{</span>
<span class="na">dataPoints</span><span class="p">:</span> <span class="p">[]</span>
<span class="p">}]</span>
</code></pre></div></div>
<p>We are creating a visualization for realtime emission - <a href="https://www.epa.gov/">EPA</a> has physical stations in the various cities to measure air pollution levels and also keep record for historical data. We’ll simply use the <a href="https://www.airnow.gov/index.cfm?action=airnow.global_summary">data</a> available globally, collect the RSS feed available for your city and add URL to object. <code class="language-plaintext highlighter-rouge">window.onload</code> function is used to render the data first whenever CanvasJS is loaded.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">phora_pm2</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">https://dosairnowdata.org/dos/RSS/PhoraDurbarKathmandu/PhoraDurbarKathmandu-PM2.5.xml</span><span class="dl">'</span><span class="p">;</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">onload</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">chartRenderer</span><span class="p">(</span><span class="nx">phora_pm2</span><span class="p">,</span>
<span class="nx">chart_data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">column</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">AQI</span><span class="dl">'</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We also need an AQI references where people would be able to distinguish the air pollution levels. It is divided into different color ranges and health messages.</p>
<p>Function <code class="language-plaintext highlighter-rouge">aqiColor</code> returns color code where AQI is input, add <code class="language-plaintext highlighter-rouge">healthMessage</code> to generate health message for AQI value <code class="language-plaintext highlighter-rouge">y</code>. The value range, color code and health message all can be found in the AQI Basics <a href="https://airnow.gov/index.cfm?action=aqibasics.aqi">doc</a>.</p>
<p><b>Note: You can change the chart type column to supported <a href="https://www.chartjs.org/docs/latest/charts/">Charts</a> by modifying chart_data when window.onload is initiated.</b></p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">aqiColor</span><span class="p">(</span><span class="nx">y</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#fff</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">50</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#34e400</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">51</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">100</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#fcff00</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">101</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">150</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#f77e01</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">151</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">200</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#f61802</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">201</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">300</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#8f3f97</span><span class="dl">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">301</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2">#7e0623</span><span class="dl">"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">healthMessage</span><span class="p">(</span><span class="nx">y</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Error!</div><br/>No data was received from the station</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">50</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Good</div><br/>The AQI value is between 0 and 50. Air quality is satisfactory and poses little or no health risk.</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">51</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">100</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Moderate</div><br/>The AQI is between 51 and 100. Air quality is acceptable; however, pollution in this range may pose a moderate health concern for a very small number of individuals. People who are unusually sensitive to ozone or particle pollution may experience respiratory symptoms.</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">101</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">150</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Unhealthy for Sensitive Groups</div><br/>When AQI values are between 101 and 150, members of sensitive groups may experience health effects, but the general public is unlikely to be affected.<br><b>Ozone:</b> People with lung disease, children, older adults, and people who are active outdoors are considered sensitive and therefore at greater risk.<br><b>Particle pollution:</b> People with heart or lung disease, older adults, and children are considered sensitive and therefore at greater risk.</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">151</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">200</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Unhealthy</div><br/>Everyone may begin to experience health effects when AQI values are between 151 and 200. Members of sensitive groups may experience more serious health effects.</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">201</span> <span class="o">&&</span> <span class="nx">y</span> <span class="o"><=</span> <span class="mi">300</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Very Unhealthy</div><br/>AQI values between 201 and 300 trigger a health alert, meaning everyone may experience more serious health effects.</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">y</span> <span class="o">>=</span> <span class="mi">301</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="dl">"</span><span class="s2"><div class='text-uppercase'>Hazardous</div><br/><b>Health alert:</b> AQI values over 300 trigger health warnings of emergency conditions. The entire population is even more likely to be affected by serious health effects.</span><span class="dl">"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Finally, add function <code class="language-plaintext highlighter-rouge">chartRenderer</code> and include empty Arrays for <code class="language-plaintext highlighter-rouge">dataPoints</code> <code class="language-plaintext highlighter-rouge">lables_</code>. We use JQuery to fetch <a href="https://dosairnowdata.org/dos/RSS/PhoraDurbarKathmandu/PhoraDurbarKathmandu-PM2.5.xml">XML</a> data using <code class="language-plaintext highlighter-rouge">chartRenderer</code> function.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">chartRenderer</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="nx">chartData</span><span class="p">,</span> <span class="nx">chartType</span><span class="p">,</span> <span class="nx">airDataType</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">chart</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">CanvasJS</span><span class="p">.</span><span class="nx">Chart</span><span class="p">(</span><span class="dl">"</span><span class="s2">chartContainer</span><span class="dl">"</span><span class="p">,</span> <span class="nx">chartData</span><span class="p">)</span>
<span class="kd">let</span> <span class="nx">dataPoints</span> <span class="o">=</span> <span class="p">[]</span>
<span class="kd">let</span> <span class="nx">labels_</span> <span class="o">=</span> <span class="p">[]</span>
<span class="nx">$</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="nx">data</span><span class="p">).</span><span class="nx">find</span><span class="p">(</span><span class="dl">"</span><span class="s2">item</span><span class="dl">"</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">$dataPoint</span> <span class="o">=</span> <span class="nx">jQuery</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="kd">let</span> <span class="nx">label</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="nx">$dataPoint</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="dl">"</span><span class="s2">ReadingDateTime</span><span class="dl">"</span><span class="p">).</span><span class="nx">text</span><span class="p">()).</span><span class="nx">toLocaleString</span><span class="p">(</span><span class="dl">'</span><span class="s1">en-US</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">month</span><span class="p">:</span> <span class="dl">'</span><span class="s1">short</span><span class="dl">'</span><span class="p">,</span>
<span class="na">day</span><span class="p">:</span> <span class="dl">'</span><span class="s1">numeric</span><span class="dl">'</span><span class="p">,</span>
<span class="na">hour</span><span class="p">:</span> <span class="dl">'</span><span class="s1">numeric</span><span class="dl">'</span><span class="p">,</span>
<span class="na">hour12</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">minute</span><span class="p">:</span> <span class="dl">'</span><span class="s1">numeric</span><span class="dl">'</span>
<span class="p">});</span>
<span class="kd">let</span> <span class="nx">y</span> <span class="o">=</span> <span class="nx">$dataPoint</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="nx">airDataType</span><span class="p">).</span><span class="nx">text</span><span class="p">();</span>
<span class="nx">dataPoints</span><span class="p">.</span><span class="nx">push</span><span class="p">({</span><span class="na">label</span><span class="p">:</span> <span class="nx">label</span><span class="p">,</span> <span class="na">y</span><span class="p">:</span> <span class="nb">parseFloat</span><span class="p">(</span><span class="nx">y</span><span class="p">),</span> <span class="na">color</span><span class="p">:</span> <span class="nx">aqiColor</span><span class="p">(</span><span class="nb">parseInt</span><span class="p">(</span><span class="nx">y</span><span class="p">))});</span>
<span class="nx">labels_</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">label</span><span class="p">)</span>
<span class="p">});</span>
<span class="nx">chartData</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">type</span> <span class="o">=</span> <span class="nx">chartType</span>
<span class="nx">chartData</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">dataPoints</span> <span class="o">=</span> <span class="nx">dataPoints</span>
<span class="kd">let</span> <span class="nx">health_aqi</span> <span class="o">=</span> <span class="nx">dataPoints</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="kd">let</span> <span class="nx">z</span> <span class="o">=</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">health_aqi</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">y</span><span class="p">)</span>
<span class="kd">let</span> <span class="nx">message</span> <span class="o">=</span> <span class="nx">healthMessage</span><span class="p">(</span><span class="nx">z</span><span class="p">)</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">"</span><span class="s2">health_message</span><span class="dl">"</span><span class="p">).</span><span class="nx">innerHTML</span> <span class="o">=</span> <span class="nx">message</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">render</span><span class="p">()</span>
<span class="p">});</span>
</code></pre></div></div>
<p>If you refer to XML <a href="#xml-tree-structure">structure</a>, we are interested in XML <code class="language-plaintext highlighter-rouge">item</code> element where reading date, AQI and description are found. In <code class="language-plaintext highlighter-rouge">dataPoints</code>, <code class="language-plaintext highlighter-rouge">label</code> represents the time sequence in Month-Day-Time in 24hr format, <code class="language-plaintext highlighter-rouge">y</code> is AQI value in float data type, <code class="language-plaintext highlighter-rouge">aqiColor</code> fetch color code for every AQI, each feed from station may consists of 24 hours of data and interval of 1 hour update time. Later, the slicing is done to get the latest feed and generate health messages. Use <code class="language-plaintext highlighter-rouge">chart.render()</code> to visualize the chart. And don’t forget to add document.getElementById() method for <code class="language-plaintext highlighter-rouge">health_message</code> to render in HTML document.</p>
<h3 id="xml-tree-structure">Xml tree structure</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span>
├── rss
└── channel
└── title
└── item
└── title
└── description
└── param
└── conc
└── nowcastconc
└── aqi
└── desc
└── readingdatetime
</code></pre></div></div>
<h2 id="render-chart-in-html">Render Chart in HTML</h2>
<p>Import JQuery, CanvasJS, visualization_document.js in HTML head tag. CanvasJS only render a chart if you include id, height, and width. Likewise, we will use the append health messages.</p>
<p>To render CanvasJS;</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">id=</span><span class="s">"chartContainer"</span> <span class="na">style=</span><span class="s">"height: 420px; width: 100%;"</span><span class="nt">></div></span>
</code></pre></div></div>
<p>To render health message;</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">id=</span><span class="s">"health_message"</span> <span class="na">class=</span><span class="s">"card text-justify"</span> <span class="na">style=</span><span class="s">"padding: 30px;"</span><span class="nt">></span>Updating...<span class="nt"></div></span>
</code></pre></div></div>
<p>We have successfully created a visualization graph with a health message for any latest AQI data. We are also adding a simple HTML table as a reference to the AQI color range and code to understand air quality better.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><table></span>
<span class="nt"><thead></span>
<span class="nt"><tr></span>
<span class="nt"><th></span>Color<span class="nt"></th></span>
<span class="nt"><th></span>AQI Level<span class="nt"></th></span>
<span class="nt"></tr></span>
<span class="nt"></thead></span>
<span class="nt"><tbody></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #34e400;"</span><span class="nt">></td></span>
<span class="nt"><td></span>0 - 50 Good<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #fcff00;"</span><span class="nt">></td></span>
<span class="nt"><td></span>51 - 100 Moderate<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #f77e01;"</span><span class="nt">></td></span>
<span class="nt"><td></span>101 - 150 Unhealthy for Sensitive Groups<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #f61802;"</span><span class="nt">></td></span>
<span class="nt"><td></span>151 - 200 Unhealthy<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #8f3f97;"</span><span class="nt">></td></span>
<span class="nt"><td></span>201 - 300 Very Unhealthy<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><td</span> <span class="na">style=</span><span class="s">"background: #7e0623;"</span><span class="nt">></td></span>
<span class="nt"><td></span>301 - 500 Hazardous<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"></tbody></span>
<span class="nt"></table></span>
</code></pre></div></div>
<h2 id="at-last">At last</h2>
<p>The final visualization;</p>
<p><img src="https://vijaypathak.com.np/assets/image/2018/chart_visualization.png" alt="Chart visualization" class="full" /></p>
<p>You can find the source code <a href="https://github.com/hbvj99/kathmandu-air">here</a> or view live <a href="https://vijaypathak.com.np/kathmandu-air">demo</a>.</p>
<p>If you got stuck somewhere, find project file structure tree below;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>air_pollution
|
+-- index.html
|
+-- js
| |
| <span class="se">\-</span>- canvasjs.min.js
| <span class="se">\-</span>- live_visualize.js
| <span class="se">\-</span>- jquery.min.js.js
</code></pre></div></div>Vijay PathakVisualize the air pollution in your community using JavaScript and open-data