#9YearsBeingWild – Beyond the Backend: a constant evolution

Acting behind the scenes, Wildlife’s backend team plays a critical role in the performance of our games.

This article is part of a content project that will celebrate Wildlife’s 9th anniversary by sharing inspiring stories from the Wilders that helped build our company. #9YearsBeingWild

The Backend area has curious features. As its name suggests, engineers dedicated to this type of work operate behind the scenes. It’s their responsibility to ensure that the various systems developed and used by the company operate with maximum efficiency. If all goes well, our performance is hardly perceived by the end consumer who, in our case, is the player. However, if something goes wrong, the problems quickly become apparent. In other words, the less our work is noticed, the better.

Though it may sound like an onus, I consider it to be one of the most thought-provoking and challenging aspects of my work. It was the willingness to face challenges and solve problems that attracted me the first time I set foot in the Wildlife office in 2015. I studied electrical engineering with emphasis in telecommunications at the Polytechnic School of USP (Brazil) and dreamed of doing research in the area of signal processing.

Unlike many colleagues here at the company, I’ve never been a gamer. During my childhood, I played a few games on a Nintendo 64 with my sister, but in a casual manner, without much attachment to the games. My first professional contact with them was during an internship I did in France, in a small company which developed audio tools for games. This was also my first professional contact with the field of programming. Back in Brazil, I worked for about a year in a company that developed educational software, until I received an invitation for a job interview at Wildlife.

The shift from research to computing was greatly driven by my taste for problem-solving and building things. That was one of the first things that caught my attention in Wildlife’s interview right from the start. At that moment, I was presented with technical challenges related to the day-to-day work of the company and for which I had to propose solutions. I had to figure things out based on what I knew and give my best shot!

The company’s selection process and ambitious goals for growth captivated me. I was the third person hired to work with the Backend team that, at the time, was still structured as the responsible for offering various services to the company.

AN INTENSE START

Back then, like I said, there were three people on the team. As small as it seems (and it was!), it’s important to remember that the whole company had approximately 50 people. We had some critical services under our responsibility, such as purchases validation, and storing game events.

The other activities involved a series of small systems for our games, which at that time were mostly offline and single player. In Sniper 3D, we took care of the account synchronization system, for example. In addition, we also took care of the office infrastructure, such as Wi-Fi maintenance and our network.

It was a time of hard work in which the number of projects exceeded, by a lot, the number of engineers on the team. We all had different projects under our responsibility. It was intense, but it was a valuable time, because we learned a lot from the variety of projects we had on our hands and especially with colleagues, since our culture of exchanging knowledge and skills has always been very strong.

TIME TO TAKE THE LEAP

In early 2016, we started making multiplayer and online games, which made the Backend work even more relevant (and interesting) for the team. With online matches, the proper functioning of the servers was critical. If they didn’t work well, the matches just wouldn’t happen. With our responsibility increasing, the expansion of the team was inevitable. With six people now, our focus was to build the infrastructure needed to build amazing games that had a great experience for our players.

It was a time when we built a lot of things. For each demand, we created libs, services, systems, and frameworks that could be reused. Until 2017, our main contributions were the servers for the games Castle Crush and War Machines. We built critical services that are used to this day, such as clan systems, leaderboards, matchmaking and account synchronization, for example. It was also a challenge to develop the systems needed to adapt original single player games, such as Bike Race and Sniper 3D, to synchronous multiplayer versions.

At the beginning of 2018, I was already working in a leadership position it the team, with activities focused on talent recruitment and improving our software development processes, when I was promoted to manager. During this period, the Backend area underwent its first restructuring. We divided it into three sub-teams: Infrastructure, Games and Services. One of the goals was to pass on to our game engineering teams the skills and knowledge to make the necessary corrections and adaptations for the proper functioning of the servers for each of the games they work with.

This aspiration had very important objectives: to give more autonomy to the teams, allowing them to make end-to-end decisions about the operation of a game, and, mainly, to expand the scale of games in production, something that would only be possible by sharing knowledge and skills among more teams and focusing the work of the Backend team in creating new technologies used in different games and projects.

A CHALLENGE CALLED PITAYA

While we were in this period of transitioning knowledge regarding the operation of servers for the games teams, acting as consultants during the process, we split our Backend projects into two fronts: creation of services and systems for games, and development of solutions to improve the work of other internal teams. A very prominent project in this sense was Pitaya, our framework for game servers, which is used to this day. It has brought improvements to our games as well as interesting technical challenges.

The framework has many responsibilities, including making the servers recognize and communicate with each other. We must explain that a game has different types of servers, for example: servers for the execution of the matches, others for what we call metagame, which runs all the logic of what happens outside the match itself, such as cards, chests, rackets (in the case of Tennis Clash), elements that vary according to the type of the game, among others. We need all these servers to scale out horizontally, in view of the very large demand which oscillates with special dates, feature releases and at different times of the day. Until then, the framework used to articulate this whole operation was Pomelo.

For a while, it worked very well, but the system was not kept up to date by its creators, so we ended up getting many bugs. In addition, it used a rather obsolete Node.js version. We made the adjustments that we considered necessary, sending modifications that were included in the original project but unfortunately it didn’t meet our demands at the scale we needed.

The issue of language also posed another problem. As we were in the process of moving the development of servers to game developers, there was a “language mismatch”. These engineers were more used to strongly typed languages such as C# and C++, and working with languages such as Javascript, introduced a number of unwanted challenges (and bugs ).

Imbued with the spirit of one of our values, which stresses innovation from research, we extracted what worked from Pomelo and set out to create a new framework, aligned with our needs.

We have inserted a number of improvements related to monitoring and managing errors and authentication, creating an integrated and customizable solution, according to the demands of each game. We also invested in a more structured architecture for servers, facilitating the future development of games.

We opted for using Go, which was already used by the Backend team and that would facilitate the transition of game developers to the world of servers.

As for the name, I confess that our imagination was not so good. Because Pomelo is a fruit, we decided to follow the same idea – but at least we chose a beautiful fruit. Names aside, the important thing is that Pitaya (Dragon fruit) has brought more stability to our games and created a structure that allows us to keep on growing.

THE HUB COMES INTO PLAY

Another solution created in 2018 and worth mentioning is HUB. While thinking of a way of facilitating the work of internal teams, we created this platform that integrates the entire operation of our games. Among the different features, one of the main contributions was the creation of an important bridge between the Product and Data Science teams. Prior to HUB, the reports used by the Product team needed to be requested by email, and the consolidation of this data was time consuming for the data scientists. With this platform, there are a number of parameterized reports, which can be generated and consumed by the Product team itself, as needed.

NEW TIMES

Currently, the Backend itself, as an area, no longer exists. In the middle of last year, Wildlife was restructured and separated into two business units. Game Studio, responsible for creating and maintaining games—hence the importance of that knowledge transfer I spoke about before. The other unit, called Platform and Services, aims at making the company grow by providing a platform that leverages the best of our teams when creating, operating and distributing our games. Today we work in different squads that have professionals with different profiles, including backend engineers.

I often say that I’ve never had a boring day at Wildlife. We are always surprised by new challenges that require us to seek more knowledge. As with our games, I feel that we are continually facing more complex and thought-provoking phases.

Being in an environment with smart, curious colleagues who have different approaches to solving problems is a great help in this process of constant evolution and adaptation. Mistakes happen, we can’t deny it, but they in turn become valuable learning experiences. The important thing is that we are at the forefront, so we always have to keep our thirst for discovery alive.