Career Progression, Skill Matrices, and Peer Review on Data Teams
Introduction
One of the most important jobs of any team leader is ensuring that career advancement within the team is a viable option. As part of demonstrating viability a set of evidence should be present. First, there should be a history of existing team member promotions. Second, the team leader needs to openly discuss career advancement with team members. Third, there should be well-thought-out documentation describing the career paths available and what advancement on these paths requires.
When I began leading teams I quickly encountered talented employees eager to know what the next step for their career was and exactly how they could get there. Early on, my responses were the same vague statements I was being given by my manager: “we need to see excellent performance”. However, coming from a scientific background and working in an analytics field, this subjective half-answer drove me nuts. What defines excellent? How exactly is performance measured? If technical skills are evaluated shouldn’t we gather input from others who possessed these technical skills? How can we reduce subjectivity? How can we be better servant leaders?
While we are far from having flawless answers to these questions, the Science and Analytics team at GoGuardian is implementing an approach I’ve been working on for several years. First, we shift our thinking of some career paths from parallel to series. Second, we provide clear descriptions of career paths for technical contributors. Third, we define the skills necessary for technical career path advancement in skills matrices that are scored by peers.
To be clear, these are not novel solutions. Competency or knowledge management as a concept within a business has been around for decades [Nonaka 1995, wikipedia/competence], and the idea of quantifying competency using skills matrices has been around for many years as well [Solnet, et al. 1993]. Diving in organizational psychology or people analytics publications [https://medium.com/the-drill-down, https://medium.com/@josh_bersin] can help one learn more.
Parallel Not Series
At many companies, there is a short runway of career advancement for technical contributors followed quickly by management positions. This creates all sorts of problems. It implies that leading others is the ultimate skill desired of all employees and promotes the loss of technical talent to the time suck of management. Instead, we want career paths that allow a conscious choice as to whether team members want to advance as technical contributors, team leaders, or both.
To accomplish this, we began framing advancement on a technical contributor path and a management path as parallel processes rather than series. Advancement on each path is independent; a team member can advance on the technical path without it bringing them any closer to advancement on the management path and vice versa. Technical contributors, no matter how far they advance, have no responsibility to manage or lead others, and managers may continue advancing on the technical contributor path if they wish.
Figure 1 - A traditional serial career path from technical contributor to manager compared to a parallel career path where technical advancement is separate from advancement as a people manager.
It’s admittedly a subtle change, but it brings clarity to conversations about career advancement with team members possessing different goals and ambitions. Conversations with some can focus purely on moving from Technical Contributor III to IV. Conversations with others can be about remaining at Technical Contributor II and beginning to focus on becoming a Manager and then a Senior Manager. Because both paths are available independently, the choice of which to pursue becomes a conscious decision around one's personal time allocation.
Parallel paths do introduce some complexities. For one, many people on the team have two titles and are free to use either one. For example, we have a team member who currently holds Data Scientist III and Data Science Manager titles. He tends to reference the Data Scientist III title (and it’s public-facing version “Senior Data Scientist”) more than the Data Science Manager title. In contrast, our Business Intelligence Analyst III tends to use her Business Intelligence Manager title more often.
Parallel paths might also increase the likelihood of managers with lower salaries than their direct reports. For example, a new data engineering manager leading a team with a Data Engineer V might very well be paid less than their highly talented and experienced direct report. Unfortunately, there isn’t a simple solution. One approach that appears to help is promoting from within often and early. Promote new managers within the team by having them manage more junior colleagues instead of immediately placing the entire team under them. This can leave the most experienced technical contributors reporting directly to a more experienced manager.
In the end, establishing parallel technical and leadership paths is mostly about paving the way for a more quantitative and peer-focused approach to advancement on the technical contributor path. Next, we discuss some documentation and processes implemented to make the technical contributor path relevant and as unambiguous as possible.
Technical Contributor Path Documents
Technical contributor career path (TCP) documents detail the achievements expected for each promotion on the technical contributor path. I’ll provide some examples of my version of these documents here, here, and here. To most managers and human resources departments, these should feel familiar and be relatively easy to produce. They are similar to a standard role description document, but by referencing a skills matrix the TCPs waste little space describing skill levels and instead focus on accomplishments and responsibilities.
While some teams probably already have similar documents I want to emphasize a few subtle but important attributes of the TCPs. First, titles on the TCP are devoid of decoration. For example, the TCP description for data scientists on our team consists of the titles “Data Scientist I”, “Data Scientist II”, “Data Scientist III”, etc. I suggest avoiding decorators like “Senior”, “Lead”, or ‘Principal” as these all imply some authority over others which is not the point of progressing on the technical contributor path. If an employee wishes to have more say in what, when, and/or how their teammates perform their duties they are encouraged to inquire about the management career path.
A second important attribute is the use of numbers whenever possible: Years of experience in the field (broadly speaking), the number of major projects completed, counts of contributions to open-source projects, and specific scores on the skills matrix are clearly stated. Also, I’d urge teams to develop agreed-upon definitions for words like “project”, “success”, and “completed” along with any other critical descriptives found in the TCPs. The primary goal of these documents is to remove as much ambiguity as possible and provide employees with a challenging but clear path forward in their careers.
The Skills Matrix
At the core of the technical contributor path is the skills matrix. Skills matrices are not new, and a quick search brings up numerous examples, software tools, and business school links. Broadly speaking, they are a document describing the array of skills important to a role. For those working in data, I have some example skills matrixes for data scientists, business intelligence analysts, and data engineers. Each skill is described in a few sentences and grouped into broader domains.
Figure 2 - A portion of a skills matrix implemented in a spreadsheet
The granularity and grouping of skills in a skills matrix seem critical. In some implementations, the skills are very broad and apply to any role within the company. While many skills are applicable across all roles, for the skills matrix to be useful as an evaluation of a technical contributor the skills listed should be refined without being too specific. It helps to frame the skills matrix as a list of areas in which an employee might grow their knowledge to be more valuable to the company. Going too broad might result in technical contributors with insufficient technical depth in skills of use to the company. Going too specific increases the likelihood the skills matrix will select for extreme specialists who might become irrelevant as the company needs change. Here are a few examples attempting to balancing broad and specific skills:
Too Specific
Knowledge of Github, their APIs, documentation, use of tags, and repository metadata
Skills in utilizing PowerBI including creating custom mapboxes, writing and reviewing DAX, visualizing financial data, and integrating with AWS Redshift
Knowledge of maintaining enterprise data center infrastructure consisting of RackSolutions server cabinet enclosures and Dell PowerEdge servers. As well as knowledge of provisioning, patching, and release management of Ubunutu server 20 LTS systems.
Too Broad
Knowledge of coding best practices
Skills in making charts
Data infrastructure skills
Just Right
Knowledge of best practices, tools, and techniques in managing and versioning large volumes of code shared across multiple developers
Skills in utilizing BI tools to connect to and query stored data to build effective interactive data visualizations
Skills in the initial set-up and ongoing maintenance, both physical and software-related, of on-premise data infrastructure
If you check out the example skills matrices, you might be surprised by the number of skills that aren’t technical. For example, there are skills such as “Business operations/functions” and “Data Presentation”. There is even an entire domain of skills I call (for lack of a better term) “Intangibles” that consists of things like “Attitude” and “Interpersonal skills”. Even when evaluating technical contributors, ignoring soft skills or general business knowledge is foolish, and we view these intangible skills as critical especially for higher levels of the TCPs.
The overall idea of the skills matrices is that someone scoring perfectly would be an ideal employee in that role. This imaginary person would be an impressive expert in all the technical areas related to their job, understand the business, know how to organize and manage large complex projects on their own, be driven to learn, and pleasant to work with. In reality, no one person maintains flawless expertise in all these areas throughout their career. The goal is to promote progress and hopefully the skill matrices allow us to measure that progress and reward it proportionately. By providing skills matrices we hope to remove ambiguity around phrases like “meets expectations” or the use of overly-broad evaluation criteria like "Technical skills" when performing reviews. Instead, we aim to make career progression a transparent and quantitative process detailing where one excels and where one can improve.
Combining the Skills Matrix and TCP Documents
Being scored on the skills matrix is a requirement for anyone wishing to advance on a TCP. Indeed, expected scores on the skills matrix are baked into descriptions for every level on the TCPs. So the obvious next question should be “Where do scores on the skills matrix come from?”. On our teams, the answer is peer review.
I strongly prefer allowing data scientists to evaluate the skills of other data scientists, BI analysts to evaluate the skills of other BI analysts. This is a clear holdover from my time in academic science where someone with expertise in your field was supposed to be the ultimate judge of your work. Therefore, each team member on the technical contributor career path must anonymously complete a skill matrix for every other team member on the same career path. This is certainly easier said than done.
One challenge is the difficulty of scoring peers with which one might not interact much. I’ve taken to weighting scores by the amount of time teammates have overlapped which seems to help ease this concern. I also remind participants that a lack of evidence for proficiency in a skill is possibly informative. Technical contributors shouldn’t be islands. They need to solicit code review, give presentations about their work, share documentation, and discuss ideas with other team members. Through these interactions, peers should gather enough information to evaluate one another.
I should point out that qualitative feedback is still a critical part of reviews on our team. Assessments from peers on a skills matrix are useful, but we still want to hear thoughts that can’t be captured in numbers. We currently solicit non-anonymous qualitative feedback from within and outside the team, from managers, and peers. The qualitative feedback acts to enrich the information from a skills matrix, though the two sources are amazingly congruent so far in all our reviews.
Conclusions
Despite the challenges, I still see substantial value in the utilization of skills matrices on technical teams. They allow one to begin moving beyond the frustratingly vague assessments often seen in employee reviews. Couple the skill matrices with anonymous peer review and another frustration begins to resolve: technical contributors being assessed by others who don’t actively utilize the skills being assessed. When we add numeric and objective career pathing documentation such as the TCPs we make additional strides towards a career progression system that is clear, fair, and hopefully has had several potential biases removed. Finally, by presenting career progression on a technical path and a leadership path as parallel we no longer force technical contributors to try and lead in order to progress, and we don’t require decades of isolated technical contributions from emerging leaders before allowing them to try their hand at leading teams.
Creating skills matrices is a lot of work. Making career path descriptions that are as quantitative and clear as possible is also time-consuming. Facilitating anonymous peer review on the skills matrices also adds a lot of work to each review cycle. However, career progression is a major factor in employee retention, and retaining talented employees should be a priority for any company. For me, focusing on team career progression is always worth the time and effort. I hope other servant leaders reading this feel the same and can use the resources and knowledge in this post to improve career progression on their teams.
Appendix
What this all looks like in practice
I spent a lot of words describing just a few concepts. I wanted to provide a simple list of how this all works on our current team.
TCPs and skills matrices are available to anyone via a shared Google drive folder
Skills matrix scoring is conducted every 6 months
I create a copy of the relevant skills matrix for each team member on a TCP. This copy can be viewed only by myself and the team member.
Each team member fills in their copy of the skills matrix providing raw scores for all other employees on their same TCP
I collect the raw scores into a central document only I can access
The raw scores for each employee are weighted and averaged, providing 4-5 numbers that represent each team member’s final scores for that review cycle
Individual meetings are scheduled between all managers and direct reports
Skills matrix scores are discussed along with qualitative feedback from teammates, other colleagues, and their manager
Discussion continues with a review of the TCP, where their current achievements and skills matrix scores place them on the TCP, their future ambitions, and how we can get them there
Hopefully, at the end of the process, there are team members who met the criteria for a promotion (my favorite part of my job!). If so, the manager is now armed with quantitative and qualitative information with which to lobby human resources, finance, and their boss and say “this employee is highly appreciated by their teammates and colleagues, has completed X number of impactful projects, and their technical peers rated them at the Data Engineer III level. I’d like to give them that title and an appropriate change in compensation”.
Not Discussed - The management path
This is still a work in progress. I’ve arrived at the basics of a “skills matrix” for managers, but am not at a point where I’m ready to share this document. I do think what is good for the goose is good for the gander and if technical contributors have their technical skills regularly evaluated, leaders should have their leadership skills regularly evaluated.
Not Discussed - Where to place skills matrix cutoffs in the TCPs
This takes some finessing for sure. I feel strongly that moving the bar to be promoted should be avoided if possible, making this even more important to get right. That being said, I have certainly found myself needing to revise skills matrix requirements in TCP descriptions. Typically, I realize I’ve set a requirement too high. Other times I’ve come to find a particular skill even more important to a role than I initially expected and need to raise the score.
Not Discussed - How quantitative reviews help facilitate company planning
This might seem like a bit of a leap initially. However, I’ve found quantitative career pathing pairs well with quantitative planning and prioritization methods like Objectives and Key Results (OKRs). I’m a huge fan of OKRs and have advocated for their use at multiple companies. Without going into too much detail, the OKRs system states that team member objectives should be moonshots, impossible to fully complete in the time allotted! To compensate for this ambition, the OKR system dictates that scores on OKRs can NOT be part of compensation discussions. This sounds easy at first, but come review time I’ve found managers asking “Well if scores on their projects aren’t part of their review, how do I review them?”. My answer of course: skills matrices! The combination of OKRs and skill matrices essentially creates a team that is 1) aiming to accomplish a ton without fear of failure and 2) constantly learning and expanding their skill set to advance their careers. It’s a winning combination.
Citations
I. Nonaka.; H. Takeuchi (1995). “The knowledge creating company: how Japanese companies create the dynamics of innovation”, New York: Oxford University Press, p. 284, ISBN 978-0-19-509269-1
R. Solnet, M. Poponiak, G. Young and L. Wasson (1993). "The technical professional advancement program. . . strengthening IBM, strengthening America," Proceedings of the Tenth Biennial University/Government/Industry Microelectronics Symposium, Research Triangle Park, NC, USA, 1993, pp. 249-252, doi: 10.1109/UGIM.1993.297062.
August 22, 2020