LinuxMacPC GamingTechnologyWindows

How some Developers went remote in the time of COVID-19

It’s April 2020 and COVID-19 has begun to reshape the video game industry. While game studios are not suffering the damage afflicting the travel, live sports, or concert businesses, convention cancellations and retail limitations have begun to rewrite how the industry does business.

But while the future lies uncertain, what’s become clear early in the crisis is that video game studios and publishers have been remarkably successful at moving their workforces to remote operations. This has occurred both in conjunction with government lockdowns of businesses and large gatherings and sometimes, in advance of them.

Though game developers all over the world are now working at their home desks, learning more about their coworkers’ cats and kids over video calls, there is still a long road ahead for studios organized around remote work. Here are the stories of a few studios that made the transition, what went well for them, and what they’re looking at going forward.

The Move

According to available reports, the SARS-CoV-2 virus made landfall in the United States and in South Korea on January 20. Its rapid progression from there shaped how companies all around the world began their remote work responses.

First, its regional spread in Asia began influencing companies based in or those with heavy ties in the region. Of the studios we spoke to, Blizzard Entertainment, Ubisoft, Netmarble, Hyper Hippo Games, and others began taking formative steps to move their companies to remote setups.

According to Blizzard Entertainment’s VP of HR Jesse Meschuk, along with chief information and security officer Mark Adams and chief legal officer Claire Hart, the company began organizing an internal task force as part of a drive to help its Asian offices move to remote work. Ubisoft Toronto managing director Alex Parizeau and Netmarble USA president Simon Sim said their parent companies did the same. These developers relied on their task forces to create remote work practices that could benefit all their audiences, and adapt to local laws, regulations, and needs.

The virus’ spread in China had bigger ripples too. Hyper Hippo CEO Sam Fisher said that word from the company’s Asian partners began shaping their decision-making ahead of other companies. Tequila Works CEO Raul Rubio said COVID-19’s impact was apparent at the Las Vegas D.I.C.E. Summit in February.

From there, these largely Western developers began to ready precautionary remote work strategies, hoping that the “test days” they were planning wouldn’t be necessary. For some, those test days became day zero of their employees’ work-from-home transition.

According to Phoenix Labs VP of operations Jeanne-Marie Owens, the Dauntless developer, which has offices in Seattle among other locations, began taking action after Washington State implemented its stay-at-home order on February 29, but its remote work practices became the norm for the rest of the company a few weeks later. Moving the company remote branch-by-branch had the benefit of helping Phoenix Labs figure out exactly what needed to be done, and in Owens’ words, the staggered move over the course of a week was “enough time” to move the mid-sized company.

Across the Atlantic, Failbetter Games and Tequila Works joined their Canadian cousins at Hyper Hippo Studios in going full work-from-home on Friday the 13. En Masse Entertainment’s Stefan Ramirez confirmed their studio acted around the same time. Of the studios we spoke to, South African developer 24 Bit Games was the one to act the slowest, but per CEO Luke Lamothe, the company began implementing its remote work protocols on March 16, 11 days before a nationwide lockdown would begin on the 27. (For context, the country’s first case appeared on March 1).

Though the Northern Hemisphere was exiting winter during this transition, tales of these transitions sound a lot like preparations for a snowstorm. Blizzard Entertainment sent its employees home with a slew of goods like toilet paper and paper towels that were flying off the supermarket shelves, while Ubisoft, GameHouse, and other companies sent home basic office supplies like chairs, headsets, and plants under the expectation they wouldn’t return to the studio for some time. ​

Once employees returned to their home offices, IT administrators and system admins had to ensure the next step–that employees could work as effectively from home as they could in the studio.

The Technical Challenges

While most of the studios we spoke with didn’t describe the remove to remote work as overly arduous, the transition did expose unique inadequacies normally managed by an office environment. While many studios have had practices in place for years to support occasional employees doing work from home, moving every employee to that status was another story.

Different developers have discussed different remote work setups that adapt to their team’s needs. Blizzard and Ubisoft discussed two of the more complicated setups we heard. Blizzard has implemented multiple layers of remote work strategy, ranging from cloud applications that can be used on personal machines to advanced graphics-accelerated desktop infrastructure to handle frame rate- and latency-sensitive tools.

Parizeau says Ubisoft Toronto has largely favored a VPN remote desktop workflow supplied by Citrix. Instead of moving more devices home, this approach means that studio employees are interacting with their work machines over a long distance.

Phoenix Labs’ Owens said the immediate challenges revolved around build-syncing, devkits, and software licenses for remote work tools. Syncing builds in particular was still a thorn in the studio’s side when we spoke with her.

“The way that we do our builds, we use a piece of software that allows to use the PCs across the office to distribute the load of building faster,” she explained. “So if you were in the office and you were thinking to build it was able to use all of the other computers in the office to do that faster.

“That doesn’t work well in a remote distributed environment and having to use PCs from other people’s homes or back in the office.”

Since Phoenix Labs ships Dauntless on all major platforms, Owens said the company had to reach out to its partners at Microsoft, Sony, and Nintendo to adapt the terms of their devkit licenses so the devices could follow developers home. And after its Seattle office began the transition, the company realized it didn’t have enough individual licenses secured for its VPN software and other tools.

At Hyper Hippo, Fisher reported that the studio had to adapt its patching and update cycle for games like AdVenture Capitalist to deal with Apple and Google’s sudden changes to their build review process. According to him, both companies let developers know that it would be adjusting staffing and moving their review teams to remote work, a process which would delay some patches.

Fisher said, this meant moving AdVenture Capitalist’s weekend-structured events to be midweek-structured, though this occurred in conjunction with feedback from their players that this would support their new lifestyles too.

Above: Even Hyper Hippo Games’ promotional images for AdVenture Capitalist have adapted to life under quarantine.

Failbetter Games producer Stuart Young said that while his company’s transition was fairly smooth, they’ve been surprised by changes to the tools they use that have emerged as part of the outbreak. Though video chat service Zoom has been soaring in popularity, the much-needed security changes the platform had to implement in the wake of “Zoom bombings” became a slight roadblock for a studio used to just hopping on calls with one another.

Though cloud computing software has reduced the number of physical components needed for a studio on-site, some developers we spoke to discussed the need to keep some staff in the office (at least initially) to help everyone at home. At Tequila Works, Rubio volunteered for the task himself, helping grapple with a surge to Madrid’s power grid that took the company’s desktops online. “There was a power peak during the night that disabled an entire wing of the building,” he explained.

“Even if UPS [systems] kicked in and the automatic switch-off protocol worked, I had to restart the power switches because that’s how power panels work!”

During 24 Bit Games’ transition, Lamothe said the company needed to keep some staff on site—the plan was to have four engineers on staff, with two “floating” team members on call if need by. With an empty office, they were able to keep three meters apart and support developers who needed access to dev kits, test PCs, etc. These engineers would join their comrades at home after South Africa’s lockdown order went into effect.

All of these changes have put a large burden on different information technology teams, who’ve had to implement the new procedures conceived of by COVID-19 task forces and company leadership. At ProbablyMonsters, the new studio from former Bungie boss Harold Ryan, system administrator Alex Zimmerman described how the transition hit his team.

“This endeavor was pretty heavy workload-wise,” he said. “Basically you do support through the day. Then as things start to dwindle off in the evenings, that’s when you can actually make progress on improving the processes or improving the experience for employees for the next morning. Then you rinse and repeat.”

“I would say the first week was pretty brutal in the sense of starting early before everyone got online, making sure everything was in working order from the changes you made the night before and all of it later in the evening. Now I think we’re in a pretty good state.”

Zimmerman added that during this time, Ryan himself jumped into the trenches with the IT team and helped respond to individual work tickets.

Though these various developers’ tech teams have worked frantically to ensure a sense of “normalcy” to working from home, many studio heads have begun to recognize that mandated long-term remote work had created a new kind of normal, with its own unique challenges. Even with many of the tech solutions under control, the impact of COVID-19’s spread goes beyond the physical relocation.

And for some, those challenges are only just beginning. For more click here.

 

Comment here