Works, initiatives and policy of the engineers that support the services of Colorkrew.
This is a presentation of their challenges and the mindset that they have when they carry the projects forward.
MSP (Managed Service Provider) Project
MSP Project is a project that deals with Colorkrew’s cloud-management service “Kuramane.”
Kuramane is a full-managed service that provides cloud sales, system environment structuring, and 24/365 surveillance and operation that are arranged and adjusted for each client’s needs and requests.
Our product is characterized by multi-cloud service that deals with three biggest public clouds, AWS, Azure, and GCP, with enterprise companies as the major clients who utilize our services.
These days, due to the increase in the clients which adopts serverless architectures besides the IaaS based on VM, we provide various system designing.
We also have opportunities to catch up with the latest technologies such as attending overseas conferences or giving presentations at exhibitions held by cloud vendors.
Constitution of the project members: 12 design and construction members, 15 operation members, 15 surveillance members, 2 support team members.
Clouds: AWS, Azure, CGP, and other national clouds
OS: REHL, CentOS, ubuntu, Amazon Linux, Windows Server
Middleware: Apache, nginx, php, php-fpm, IIS, redis, memchad, java, mysql, posgresql, SQL Server, fluentd, kibana, Active Directory, LDAP
Program and script languages: Python, C#, ASP.NET, ruby, shell, PowerShell
Monitoring tools: Zabbix, Datadog, Opsramp, mackerel, monitoring tools provided by cloud venders
Other tools: GitHub, terraform, ansible, chef, Jenkins, Docker, Slack, Teams
The team is mainly consisted of the members who have their fundamental experiences as an engineer in the time of on-premises infrastructure and network engineering.
In order to adjust the cloud first policy and put more resources into DevOps, some members with programming background also participate in the project.
The team is subdivided into design, operation, and surveillance teams.
The engineers in design teams hold roles of a pre-sales engineer and accompany sales-persons to visit Kuramane clients.
When orders are made, the engineers continue working in design and construction phase until the operation is handed down to the operation team.
Inside the design team, two or more members become ‘buddies’ and work together trying to make the best of each other’s strengths.
They also give presentations together at exhibitions held by cloud vendors.
The operation team holds the role of system operations for Kuramane clients.
They also provide suggestions for improvement based on the monitoring results and hold regular meetings with the clients.
The regular operations and technical inquiries are handled during weekday business hours.
The trouble-shooting assistance is provided 24/365.
The surveillance team monitors the clients’ systems 24/365.
When any malfunction or error is detected in clients’ systems, the issue is escalated to the engineers in the operation team.
With some systems equipped with manuals, surveillance team may work on the system recovery following the manuals.
Challenges in the Project
My name is Akagawa, I’m leading the MSP (Managed Service Provider) project.
I’m always trying to step into unknown territory, keeping in my mind to update myself with the latest information of cloud area, which progresses and improves every day.
Although the fundamental system of infrastructure is gradually shifted to cloud services in today’s business scenes, in most cases, the reality is that companies just transfer existing structures into cloud servers.
As it is said with the expression “lift and shift”, in the next phase of the following time, the systems need to be shifted to the one with cloud-native structures, and proposals taking advantages of cloud computing such as cost efficiency or adaptability to high-load environment, will be needed.
Depending on situations, we propose “hybrid” system of on-premises and cloud, and for the new projects without precedents, we offer feasibility verification through POC.
To sum up, in the MSP project, the members are asked to catch up with the latest information in cloud area that progresses and improves each day, give proposals that best fit the client’s desires, and have opportunities to give a try in unprecedented situations.
Exodus from traditional, old-fashioned infrastructure engineers
Since “Infrastructure as Code” was advocated, I have been working on utilizing Ansible and Chef, and these days, not only the structure management by CloudFormation coding, but also opportunities for function coding by Python is increasing.
We are trying to be a group of engineers who can do from proposing the system structure that best fits the client’s needs to conducting actual operations such as structure management of infrastructure content and the whole cloud, or even writing event driven codes and organizing deployment environment.
What I am trying to achieve as an engineer
Before joining Colorkrew, I belonged to a support desk of a mobile communications business, or SES company for which I worked at its clients’, that are mostly System Integrator, sites.
At the former job, I had a chance to verify AWS and was greatly shocked when I learned that servers are activated by just few clicks; this motivated me to change my job and work at Colorkrew.
Soon after the entrance to Colorkrew, I was asked, “which project do you want to work in, on-premises one and AWS environment?” and I immediately answered “AWS!” although I didn’t have actual experience of AWS development.
I raised my hand for accompanying sales visits for new projects and participated as a pre-salesperson, although I did not have experience in pre-sales.
Even though it was soon after my child’s birth, I raised my hand for the position of supporting engineers at a subsidiary of our parent company (at that time) in Singapore and transferred AWS skills to the local engineers, although I cannot speak English.
I choose what I do not based on if I have done it before and I can judge if I should continue after I gave a try.
I know this might sound a little reckless, and I am, of course, careful with situations that might have an effect on our services, but this is my idea which I keep in my heart while working.
We often hear people talking about jobs that will be taken over by AI, and I think the demand for infrastructure engineers will be decreased with the development of cloud vendors.
The future cloud vendors want to provide is where “people get rid of unnecessary management and focus on their business”; I don’t have any disagreement and I believe that should be the future.
This might not be a bright future for infrastructure engineers.
However, we cannot stop the change in the world, so I think it is us that need to be changed.
Some instances in the latest situation include coding of infrastructures, automation and reporting of operations that are originally done by hands, system proposals that are free from IaaS.
As an engineer, I want to remain responsive to the changes in cloud situation and society, keeping in mind a challenging spirit, and keep changing and adjusting myself.
The “Mamoru" Project
Mamoru is a symbol of usability and security. Mamoru brand has two services. Mamoru PUSH is an authentication app & service that protects user's important information with a new sensory two-factor authentication.Mamoru Biz is innovating your workstyle by making work exiting and smart with QR codes.
This time, we will introduce Mamoru Biz.
A variety of “small things” are being sold in Colorkrew's office: small snacks and pastries, drip coffee, different types of alcohol at the bar counter, etc... managing this variety of things and prices is very troublesome.
Mamoru Biz lumps these in-company charges together by using QR Code to read barcodes when making purchases.
The purchases are recorded and centralized in the accounting department and their amount is deducted from salary at the end of the month, making it a cashless system.
Members Breakdown: 1 PL and Marketer, 3 Programmers, 1 Infrastructure Engineer, 1 Designer, 1 Back-Office
Back-end: PHP7, MySQL, Redis, Laravel
Infrastructure : Azure (Web App for Container, SQL Database, Redis)
Tools: GitHub, Slack, Docker, engineer's preferences for the edition, etc.
Let's start with the technical side.
The Mamoru brand provides a two-factor authentication service, an intra-company payment service, and a few more services not yet disclosed. These are all developed by a single team.
Since each service of Mamoru brand is still developing, it is very often for a team member to build a new function for service A today, and fix a bug on service B for tomorrow. Therefore, we are trying to have as many members as possible to be able to develop every service.
To put it simple, each member has his/her own confident technical field, and is extending their reach to cover more areas.
When inducing a new technology on Azure, it will be mainly members who are skillful with Azure to get involved, but other members will also help when building on the existing environment. Members who do not write codes on daily basis will also send PullRequest to GitHub on occasions. After someone finishes the Code Review by merge request, the service will be released in the internal staging environment, then released in the production environment.
Not only digging deep into one's own confident skill, but also widening their technical range. While one side of the story would be member redundancy, it also shows what Colorkrew is treasuring – the opportunity to "Grow".
On the business side, measures are designed with a focus on Project Leaders who have plentiful Domain knowledge in each service. They have many connections with the industry as well. Just recently, we were thankful to have an external person, who is leading a famous security group, to hold a seminar for us. A unique asset for Mamoru Team may be the opportunity to gain these valuable experiences.
The challenges of this project
My name is Akiyama, I'm the person in charge of infrastructure.
As a technical challenge, we have introduced Docker to the production environment with Mamoru Biz.
The reason of the introduction
The infrastructure used to run Docker is “Web App for Container”, a PaaS Service of Docker developed by Microsoft Azure.
The reason for using Docker in the Mamoru Biz development is that we could use it to make the development environment and the performance environment similar and get a portability that was not locked it on a specific cloud, assuming that Mamoru would have had an over-all usage of Azure.
Even though Web App for Container has just been officially released in September 2017, I was already using it for the customer environment in the infrastructure operations of another project different from Mamoru, so there was not much opposition to its introduction.
You can find an article posted in Colorkrew's blog (Colorkrew Blog) on the link below about the other service that uses roughly the same structure (in Japanese):
“LAMP のシステムを Azure の PaaS に移行しました”
Obstacles to the introduction
A primary obstacle to the introduction was that the team did not have much experience in Docker.
However another Member, who was developing Mamoru Biz with me, was really enthusiastic about it,
so we made it through safely.
“Web App for Containers” is a PaaS that is specialized in operating one Docker container through the web. The Dockerfile could be created by the developer and I only reviewed it.
The deployment flow used Docker Hub's private repository and GitHub continuous integration. From the beginning the flow was simple and over the time the team could learn more and more about Docker.
There is also an article on Colorkrew Blog about the deployment flowchart (in Japanese):
Engineer at Colorkrew
When carrying on projects like this, there are many cases where I will explain to programmers how to use basic Azure service and the important points before getting them used to use it by themselves.
Many people are also carrying other projects, so our time working simultaneously on a single project is limited.
Even while sharing knowledge, it is essential for each engineer to be "self-driven" at Colorkrew.
It might be suitable for those who enjoy large discretion.
Goals as an engineer
I was originally a programmer and now I'm doing infrastructure engineering.
DRY and YAGNI principles do not mean only service functions, even in project activities, we always keep them in mind at work every day.
For example, while Azure is 2nd on the cloud market just behind AWS, engineers that have experience of having worked at AWS are overflowing not only at Colorkrew but all over the world.
There were not many engineers pursuing Azure in Colorkrew when I joined the company, so I thought that activity opportunities may increase by opening up that area, and I took the initiative to deepen my understanding of Azure.
Also, there weren't any resistance to writing programs among the infrastructure engineers, it feels like there aren’t so many people who have a sense of what they are pursuing in infrastructure through service development.
I think as an engineer who has both infrastructure and programming perspectives, I can provide feedback to Colorkrew.
Also, personally, I manage to keep hacking a little bit.
Rails, Electron, React, RaspberryPi etc., caught my interest at that time.
In the IT world, the technology you’ve been playing with will be used in your work 1 year later which is what happens to Zara.
It’s important to keep light footwork in mind.
I’m very happy to be able to feel the moment when “someday the things I did will come back to me”.
The “Goalous” Project
Goalous is a company-oriented internal SNS service designed to make people reach goals together as a team.
It adopts the concept of “GKA”.
A “Goal” is set as the large objective in a project, and “KR (Key Results)” are the necessary indicators to achieve it. Everyone can post and share the “Actions” that they make on a daily basis to get close to the goal, along with a picture. Making your activity openly visible creates a succession of new collaborations.
Goalous is a teamwork service that encourages the invigoration for not only one project but for the whole organization.
Members breakdown：4 web engineers, 1 PL
Back end: PHP7, CakePHP, MySQL, Redis, Nginx
Front end: React, Redux, Angular, jQuery, webpack
Infrastructure: AWS (EC2, ALB, OpsWorks, RDS, ElastiCache, S3, etc)
Tools: GitHub, TravisCI, JIRA, Confluence, Slack, Vagrant, Chef, etc
If an American engineer that can barely speak a few words of Japanese suddenly joined your team, what would you do?
Ask to be transferred, be troubled, start doing weight training, calm down your mind with sutra chanting… every person will probably have a different reaction.
At that time, everyone in the team was Japanese, and no one was really good at English.
However, if the team keeps communicating in Japanese, he will end up being left out and useless. But English is difficult… so after discussing about it with everyone, we came to a decision.
"OK! From now on, basically, let's use English!!!"
Since then, we have conducted all the morning meetings, Slack, Git comments, source comments, JIRA&Confluence... all in English, until now.
When we started to switch to English, frankly speaking, the development efficiency decreased and the team members were feeling some stress. The reason why we kept using English despite that is because we already understood that English was indispensable to work as an engineer.
If we look at the bright side, we can have business conversations in English with an American native speaker on a daily basis, there is no environment as privileged as this.
Even if it is not perfect yet, now we are able to have technical discussions in English.
In addition, a Brazilian engineer and an Indonesian engineer have joined, making the team even more international and fun.
English did not bring only communication with the new members.
We are now able to read a large amount of technical documents written in English and ask technical questions to other engineers all around the world through the Internet.
This is definitely one of the ways to broaden our scope as engineers.
The challenges of this project
My name is Yoshida, I am the lead engineer of Goalous.
Some time ago the performance of all the services on Goalous have been greatly enhanced, and to achieve this we switched over to HTTP/2.
The reasons for switching
One of the main characteristics of HTTP/2 is that it allows the browser to create one stream of multiple requests (multiplex). I wanted to use it to increase the speed of the website.
With HTTP/1.1, the network creates requests individually.
Most browsers try to work around this by sending multiple requests at the same time, but there is a limit of the number of requests that can be sent at once. This causes a new request to be “blocked” until an old one is completed if there are too many (“Head-of-line blocking”).
An example that showcases this problem is the high number of images that take a large place in Goalous. It had a negative effect on the performance as the browser had to wait on old images to download before it could receive new images.
However, HTTP/2 solves this issue of HTTP/1.1 by setting up a stream, allowing multiple requests on a single TCP connection (stream multiplexing) which minimizes “blocked” requests. This results in less time for the user to wait on the browser to download files, which is what I wanted to achieve by switching to HTTP/2 as one of the measures to increase Goalous’s performance.
Difficulties that we have faced
We use a development tool called Chef to configure our application. AWS provides a resource called OpsWorks that allows us to operate Chef configurations in a cloud setting. Updating both Chef and OpsWorks to use HTTP/2 proved to be complex.
With HTTP/2, we needed to change a setting in AWS’s Load Balancer. The original setting, ELB, needed to be changed to ALB. While we were quite comfortable with working on ELB through the console, there were challenges when trying to prepare ALB’s settings.
But after a lot of hard work and testing, we were successful in updating the Opsworks environment to HTTP/2.
This was “the first step”
Honestly, we cannot consider that we are making the best use of HTTP/2’s merits yet.
From now on we plan to keep improving by exploring all its possibilities gradually, as it allows stream priority, flow-control, server push…
However, switching over to HTTP/2 was a good preparatory challenge to make Goalous an even faster and better system.
Indeed, I think that it is important to take care of this kind of groundwork from the start to ensure the future of a service.
There is also an article on Colorkrew’s blog (Colorkrew Blog) about this switch to HTTP/2 (in Japanese):
Goals as an engineer
I think that the important thing is “am I working in a good mood?”.
It means “am I doing a challenging work?”, but also “how can I turn boring tasks into something fun?”, or “how can I get rid of the boring tasks?”.
The tasks that need to be done even though they are boring are overflowing.
For example, doing the boring manual procedures that have become a routine, or correcting the parts of an application that have become a hindrance… this is the kind of tasks that maybe no one wants to do, right? (smile)
Therefore, we think of improvements consisting of how to reduce the boring manual procedures as much as possible so that we can focus on the interesting tasks.
One of the ways to achieve this is the “automation”.
Adding automated testing to our deployments, Travis CI for server testing and Selenium for browser testing, helped lighten the burden of important, yet mundane tasks.
Automating and getting rid of boring tasks in that way then allow you to focus on more interesting tasks.
Of course, there are some things that just cannot be automated, but I think that it is always necessary to search for the origins and the causes when an unnecessary task comes up.
Eagerness at work and performance are directly linked with each other. This is the reason why from now on I want to keep being aware of all the things that could go against my motivation at work.
Engineers need to have the driving force to keep pursuing spontaneously the techniques that evolve every day.
However, if they limit themselves to that, they might stay partial and narrow-minded. So, what should they do?
This is a presentation of how Colorkrew organize open knowledge exchanges.
Regardless of whether you are a new or recent graduate or even an experienced professional, Colorkrew is focused on recruiting talented members all year long.
If you’re passionate about creating services that make the world more fun and also looking to improve your skills, then you should apply with us.