A day in the life of a Rakuten Blockchain Lab DevOps Engineer

This is a blog post detailing a typical day in the life of a Cloud DevOps Engineer working for a global e-commerce giant. Rakuten Blockchain Lab has been fully remote since March 2020. This is a snapshot of a day in November 2020 during the Covid-19 pandemic. 

• 07:55 Woke up, fell out of bed, dragged a comb across my head 
• 08:00 Create a to-do list of what I’d like to accomplish today.  The complete list may be wishful thinking as this job is part reactive. This comes with the territory as a DevOps Engineer in a high performing agile team. As an owner or curator of DevOps tools and practices, I am tasked with driving the DevOps capability direction and strategy.  In Rakuten Blockchain Lab, we have the autonomy of a start-up but backed by a great culture and resources of an e-commerce heavyweight. We are the masters of our own destiny and that means we opt for staying up to date with industry trends and up to date tooling. This makes for a dynamic work environment for software engineers.
• 08:15 Breakfast and morning chaos with the kids. More time spent with my family is a real positive outcome of lockdown, also Weetos will be back on the menu when we get back to the office
• 08:30 Regular check-in call with Japanese Operations Team to enable a greater level of self-sufficiency for Operations activities on Rakuten Wallet, which is a cryptocurrency exchange we launched in August 2019.
  
  

Working closely with the teams in Tokyo and Hanoi on this project has been a great experience and is really interesting to see the culture contrasts and what we can learn from each other. This is an exercise in knowledge transfer, relationship building and process refinement. 

• 09:00 Issue overnight with integration tests.  Deploys from Jenkins to one of our CI clusters seem to be failing.  We exclusively run our platforms on cloud-based Kubernetes and our deployments are managed with the help of packaging tool Helm. Automation and DevOps best practices benefit from standardisation, Helm helps to realise this. We had to migrate from Helm 2 to Helm 3 at the end of last week prior to mid-November deprecation so that we remain in our support contract with Microsoft Azure for helm registry. What have we missed? “But it builds locally” actually might apply in this instance.
• 09.40 Coffee time, my body can't do this on its own 
• 09.50 Root cause of deploy issue turned out to be a merge conflict on a feature branch of failing services and subsequently didn’t have the new chart updates. Helm charts can reduce the complexity and increase the automation opportunities for deployment of microservices.  Running a microservices based architecture offers increased agility due to short build,test and deploy iterations. The chart change in question was around the pod resource allocation, where we use a configuration hierarchy the chart should have received a value that we made switchable to all services. Chart now has the ability to be controlled at a platform level or overridden at a service level.  We have the charts and yaml in line and they are now deploying.  Board is green in time for a release event this afternoon. 
• 10.00 Stand up.  To honour my company’s Japanese heritage. Here is a haiku describing Daily stand-up in lockdown:

                            Highlight todays tasks

                            bookended with great banter

                            Neil, you are on mute 

                                                            - Neil McCaughley, Rakuten Blockchain Lab, 2020

• 10:35 Preparation for Bitcoin-Cash fork. Our Rakuten Wallet platform relies upon interactions with public crypto blockchains which means we react to the cryptocurrency community chicaneries and events.  Today we will be testing an upcoming (Ed: This happened on 15th November 2020) hard fork on the Bitcoin Cash network. The result of a hard fork will likely split the Bitcoin Cash crypto currency into two new blockchains: BCH and BCHA. We have to support and consider both eventualities for Rakuten Wallet.  

We are testing and preparing for the potential fallout out by spinning up various node forking setups and using these against our platform. Every hard fork we encounter has been different and this has already thrown a few curve balls, but it is great to be in the mix and interacting with the community and sometimes founders of the chains. Test fork event happens at 12pm GMT we need to have this ready. 

• 12:02 Doorbell.Hermes and DPD delivery vans arrive simultaneously with a total of 5 parcels, someone needs to revoke my Prime membership during lockdown.
• 12:05 Fork event triggered. Quick Zoom and pass over to QA and Developers to proceed with fork testing
• 12:10 Check in with DevOps tasks.  PR on a screen sharing call with another DevOps Team member.  Channels should always be open for a 10 min chat that should try and bridge the gap of a head over the screen office chat, but this time it’s in tracksuit bottoms. 
• 12:30 Lunch time.  Mrs McC is home for lunch, a welcome break from the home office. Today’s menu is Chicken soup and ‘A Place in the Sun’
• 13:30 We need to spin up and start a new private blockchain in preparation for a new internal product PoC release "Rakuten Donation Chain”.  The aim of this project is to track charity donations to allow donors to view their personal donation history, confirm that the charity has received the donation fund, and receive a thank-you message from the charity.   This private blockchain differs from decentralised public blockchain like Bitcoin or Ethereum, where anyone can download the protocol and read, write or participate in the network. This private blockchain is invitation only but will allow organisations to employ distributed ledger technology whilst still protecting the privacy of the participants.  

Our tasks today include building a Kubernetes cluster, its associated infrastructure then configure and deploy the cluster components.  Luckily this isn’t our first rodeo.  We use custom Terraform plans for IAAS provisioning executed against Azure Cloud. This will be the home for an Enterprise Blockchain framework called Multichain, that was forked from Bitcoin and developed by a company called Coin Sciences. This cluster will be much like the other Multichain environments we host so it’s a well refined automated task. Our deployments and configurations likewise are routine and maintained when updates are required.  Creating environments this easily can lead to cloud provider bills growing rapidly, but we have a mechanism to keep tabs on this.

• 14:25 Review more yaml
• 14:30 Our Donations environment has a new blockchain cluster and is now ready for a release.  We have a set of release scripts that can be ran by anyone (we usually have 1-2 people down for a release).  We have multiple platforms that are governed by different release cadences and can be deployed to with a triggered release or by Continuous Deployment.  The Release process can cater for multiple platforms considering they use shared components.  This is an automated process and as you would expect needs maintenance and care, but this responsibility is shared out among the Engineering team.  I’m there for support but we are looking good
• 14:45 School run.  Why are my kids always out last?
• 15:10 Another deployment issue.  Same as this morning, referred them back to Slack channel where the fix was made available accompanied by a gif of Leonardo Di Caprio pointing at a tv
• 15:30 This afternoon I attended a security workshop over Zoom for nominated Security Champions across Rakuten offices in Europe.  This gives us the chance to keep ahead of the curve around latest security threats; implement best practices; and leverage combined wealth of knowledge to benefits the group.  
  
  Today’s topic is Ransomware.  This is quite applicable for a department that runs a cryptocurrency platform.  There are some great takeaways around how to protect ourselves from the latest industry hacks. Fortunately, all of this is a re-enforcement of what we practice and implement.  Nothing urgent here, I’ll feed this back to the team on a dedicated Slack channel and during next regular security review.  We take security very seriously and is the responsibility of all of the engineering team.
• 16:25 Kids are making an awful racket. Must be fighting over who loves me more.  Zoom spacebar to talk function paying dividends this last year.
• 16:30 Answer a few slack messages and emails and that’s me for the day. Time for a run to decompress before the evening Dadmin starts (tea, homework, taxi service, fortnite, bedtime)….
  
  
• 22:00 Googles ‘How much bitcoin should I invest to buy a villa in Spain in 2040’

Neil McCaughley is a Staff DevOps Engineer working at Rakuten Blockchain Lab for three years.  He has nine years experience working in DevOps Release Engineering and 16 years Software Engineering experience in total working in Banking, Healthcare and Blockchain sectors.