/categories/aws/Recent content in aws on Devops DiaryHugo -- gohugo.ioen-usTue, 09 Feb 2021 00:00:00 +0000/tech/s3-elasticsearch-snapshots/Tue, 09 Feb 2021 00:00:00 +0000/tech/s3-elasticsearch-snapshots/We use elasticsearch for a lot of purposes within our organisation, and thanks to kibana, we sometimes also use elasticsearch to store some data that isn’t necessarily timeseries data, but just nosql data that we want to visualize. We had a business requirement of having multiple backups for one of these types of data. So, we decided to have a dump of these indices once everyday and store it in an Amazon S3 bucket, in case the worst should happen./tech/logging-at-scale-2/Fri, 12 Jun 2020 00:00:00 +0000/tech/logging-at-scale-2/Continuing from the last post where we created a logging pipeline for storing and visualizing access logs in realtime, we faced some further issues with the said pipeline. In this post, I’ll try to build on that and get rid of some of these issues. An expected unexpected problem With the increase in number of requests during spikes in traffic, we were running quite a few instances. And that’s when we first encountered the AWS API request throttling./tech/logging-at-scale/Fri, 08 May 2020 00:00:00 +0000/tech/logging-at-scale/So I work for an Ad-Tech company, and in this industry we receive a huge volume and rate of requests. At our peak, we serve around 35,000 requests per second, which is actually a bigger number than you initially realize. Over the past 2 years, our business has seen an exponential growth in the amount of requests we serve and the infrastructure we handle. When I joined 2 years back, we were serving like 400-500 qps and handled like 20 servers for different purposes./tech/setup-own-dhcp/Sun, 03 May 2020 00:00:00 +0000/tech/setup-own-dhcp/On a network with a large number of clients, DHCP is a must for dynamically assigning IP addresses to all the connected clients in order to avoid address conflicts and automating other required configurations like subnet mask, default gateway, DNS servers etc. Nowadays, almost all of the routers, even the smaller home routers already have an inbuilt DHCP server within the router that assigns IP addresses to connected clients. Still, why not learn how to set up one for yourself?/tech/blog-setup-part-2/Thu, 16 Apr 2020 00:00:00 +0000/tech/blog-setup-part-2/So, in the last post, we finished setting up the infrastructure for our blog. In this post, we’ll deploy our blog to our server and make our blog live! The idea here would be to do all this within the User Data Script, which is executed when the EC2 instance is initializing. We’ll write a shell script with all the required commands and run it with the User data script. Here are the things that need to be done to make our blog live:/tech/blog-setup-part-1/Wed, 08 Apr 2020 00:00:00 +0000/tech/blog-setup-part-1/So, after a long time and a very little deliberation, I finally decided to stop using Google Blogger and move to a “self-hosted” soultion. The idea was to setup most of the things from scratch and have much more control over my own blog. So, I decided to give “static site generators” a try. I looked at a few different static site generators and in the end decided to go with Hugo, because it was version controlled, fast, secure, easy to understand and had good flexibilty in terms its functionality (not to mention a lot of themes to go with it)./tech/setup-own-vpn/Thu, 06 Feb 2020 00:00:00 +0000/tech/setup-own-vpn/VPN services are pretty popular these days. As days pass, more and more people are starting to use VPN services. There are a lot of benefits for using VPNs, some of them being: Protection from snooping on public networks (like public WiFi hostspots etc.) Bypass geographic restrictions on websites. Access blocked websites and services. Hide your internet activity from your ISP. Avoid network throttling (in some cases) and so much more….