July 2

Reaching your ceiling in data science or software development: how to keep growing as an Individual Contributor

0  comments

"Well, in our country," said Alice, still panting a little, "you'd generally get to somewhere else—if you run very fast for a long time, as we've been doing."

"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!" 

Lewis Carroll. Through The Looking-Glass.

Meet Fiona.

Fiona is a data scientist at company Z. She lacks a traditional background in math and computer science, so the position has been challenging from the beginning. Fortunately, company Z had faith that Fiona could learn on the job. She started doing relatively simple, even routine work of the data-analyst type: running simple SQL queries, compiling simple reports, making exploratory data analyses into PowerPoint presentations and presenting them to her colleagues on a large screen. Sometimes she would train a model or two — but always without any real insight into the model or its performance. 

Years kept flying by, yet Fiona was still at company Z: running simple queries, compiling simple reports, and presenting her exploratory data analyses. This was not due to lack of effort: Fiona is very diligent and hard-working. She has often stayed at work until 7pm. She has followed all the helpful and not-so-helpful recommendations given to her by her manager and her more senior colleagues: use version control. Use a good IDE instead of jupyter notebooks in the browser. Check out free data science talks on the Internet. Use logging statements liberally. Have your teammates review your code. Above all, have faith that you will get better over time.

It is this faith that she has started to lose. It should interest the reader to know that Company Z is a progressive one. It is distinctly ahead of the pack, when it comes to internal mentorship. In Fiona’s case, it did not seem to make a dent. Every now and then, perhaps every few weeks, one of the senior data scientists would take a look at what she was doing. Then after a 10 or 15-minute chat, the senior colleague would offer a tip or two:

Here this loop is too clumsy. You may want to use list comprehension with a lambda function.

The result is not statistically significant, but here it doesn’t matter.

Use a neural network in this case. It will perform well.

Fiona just could not grasp this kind of advice because she lacked the foundation. She was good enough with lists, but lambda functions made no sense to her whatsoever. Why would this kind of code even compile?! Statistical significance was another mystery to her: if an A/B test has shown that A is better than B, how could someone claim otherwise while keeping a straight face? Is it some esoteric knowledge learned by the chosen few in graduate school? And do not even get me started on the neural network one. Once Fiona had the courage to ask, why indeed a neural network was better in a particular case. Her difficulty was that she could not even grasp what “better” meant for a machine learning model. Imagine her frustration when the senior colleague went deep into the nuances of convolutional networks and how those, after all, have “some similarity” to boosted trees. He felt good about himself, and Fiona was too embarrassed to admit that she could not understand a thing.

Some readers would advise her to go over such complex explanations on her own time, at home. That she had tried too, but memorizing all the relevant details proved impossible. Even when she could find a published article explaining a particular concept, it was of no use. No amount of staring at the printed text seemed to help: data science remained as elusive as when she had started the job a few years prior.

Other readers would suggest online courses — and sure enough, she had tried that too! Last year she took a 10-hour course “Machine Learning in Python” on one website and a 10-hour course “Communicating effectively with your Data Science Manager” on another one. She did a pet project plotting income distributions in her city and even wrote a guest blog post about it for a personal blog of an Australian Director of Data Science. That guy is working at a startup neither of us has ever heard about. Fiona is dutifully subscribed to two IEEE journals and owns a Pomodoro timer, chunking her time into 25-minute working intervals interlaced with 5-minute breaks.

Nothing. 

She started to feel that there is no hope for her.

She started to feel that she will never get better.

She started to feel that her senior colleagues attempting to mentor her were wasting their time.

She thought of working for another company, but what has she learned over her last five years at Company Z? Here at least she knows how everything operates; she can keep adding a little bit of value, hopefully justifying her modest paycheck. But who else would want to hire her? 

She feels stuck in a hopeless situation and dreads going to work every day. Make no mistake: Company Z has a meaningful mission, and Fiona is treated well. She just feels useless. 

What has happened to Fiona may be all too common, but what if she just doesn’t have what it takes to succeed in the tech industry? In our quest for answers we will travel to the other coast and take a look at a big financial company F. 

Inside of this company we will find Jacob.

Jacob comes from an upper-middle class family. He holds mildly conservative views, largely due to his upbringing. If he could do it all over again, he would have become a lawyer or a politician. And yet, after going to a good college he somehow ended up with a degree in Computer Science and Engineering. 

Being conservative, Jacob decided to start his career at a big company B. He has worked hard and put in the same number of overtime hours as Fiona — not forgetting to get his regular after-work glass of beer with a few of his coworkers. Being a person who has been handed privileges from birth, he did not need to take a lot of initiative. Company B is not particularly progressive when it comes to internal mentorship either, not even close to Company Z. However, due to his love of sports and beer, senior team members liked hanging out with him. They would explain little bits of tech to Jacob every now and then. Having been through enough college courses — and enough experienced tutors paid for by his parents — he would understand and pick up at least some of the bits he was being shown. His senior colleagues attributed it to his drive or natural potential, and kept helping him improve further.

A couple years later Jacob moved on to another big company F. While hopping jobs was not in his character, his new employer has a better 401(k) plan, and is involved with trading stocks, which has a lot of potential, — explains Jacob. New responsibilities prompted a new cycle of learning and higher pay. Eventually Jacob got pretty good at C++ and financial analytics, and settled comfortably into his cushy, stable role of a Senior Quantitative Developer. 

This is where his growth came to a halt. 

Five years forward, Jacob is still at Company F, in the same role. He is not really learning on the job any more. Every now and then, perhaps every few weeks, one of the quants will take 15 minutes to explain a particular trading concept to Jacob. Lacking a foundation in finance, he finds it impossible to understand any of the finer points being thrown at him. The quants attribute it to Jacob’s lack of enthusiasm or lack of natural aptitude for finance.

He keeps using the same language he has been using for the last several years, C++, and calls himself an “expert”. True experts among the readers will disagree: Jacob knows nothing about the internals of the language, such as how templates are actually implemented under the hood. He does not know or care about the differences between various C++ compilers, and ten years from now he will not know either. He has never attempted to create a binding to wrap his favorite C++ subroutines into a python library. Jacob’s knowledge is confined to a rather narrow subset of the language constructs and libraries that he has become comfortable with. 

This was not due to lack of effort: much like Fiona, Jacob is very diligent and hard-working. He has followed all the excellent and not-so-excellent suggestions of the quants, his more senior developer colleagues and his several bosses: attend a conference per year. Read useful blog articles every now and then. Heck, he has even taught C++ to some kids at a local nonprofit sponsored by a local church! The course has lasted for 6 months, and he ended up giving 14 lectures and grading 3 homework assignments submitted by 12 different kids!

Some readers would advise Jacob to try and get a deeper understanding of finance as he is working for a financial company. That too he did, even completing a 20-hour course online called “Foundations of Banking”. Now every time the Fed changes interest rates, Jacob has a great topic for a morning coffee chat with the quants. He has also purchased 3 ounces of gold to build his financial intuition, as per the recommendation of one of his bosses, but does not have time to keep following the gold prices. Most days he is sufficiently tired after work and wants nothing more than to spend time with his family or to catch up on his favorite TV show. After all, life is about work-life balance, isn’t it?

Other readers would suggest doing a side project. He did try that too, creating a tool to price stock options. To be sure, the essential code came from a blog that he had googled in the first 30 seconds, but the part downloading the necessary data from a public API and the interactive visualization plots, that was Jacob’s. He has even presented the project at a local meetup. 

None of this seemed to help. Every time he reflected on his career, it looked exactly the same: same company, same tasks, same people, same pay. After running out of other ideas Jacob even bought a Pomodoro timer slicing his time into 25-minute work periods interlaced with 5-minute breaks, in the hope that it will “improve his productivity” and “propel him to a whole new level”, or so the product description said.

Nothing.

He started to feel that there is no hope for him.

He started to feel that he will never get better.

He started to feel that the quants explaining trading concepts to him were wasting their time.

He thought of changing companies, but would he get the same kind of salary? At company F he knows everything, it is like his second home, so he can keep adding enough value to justify his fat paycheck.

Jacob considered a lateral move into data science, as he is curious about working on a user-facing project. But that would come with an almost 3X pay cut, as those companies are not as lucrative as the ones in the financial sector, and he is not that skilled when it comes to python and data science either. 

He feels stuck in a less-than-ideal situation and as time goes by, begins to dislike going to work. Make no mistake: Company F is successful, and Jacob is treated well. He just feels stuck and unable to realize his potential.

Fiona’s and Jacob’s situations appear very different, yet both of them have reached their ceilings and stopped growing. They have exhausted the possibilities of what came naturally to them. Jacob has started from a better place due to sheer luck, but neither of them was able to overcome the limitations of the hand they’ve been dealt. 

Jacob may have seemed more talented or more motivated when he was swiftly picking up bits of computer science at his first job at company B, capitalizing on his good college education and his expensive tutors. Yet he was just as unable to learn finance from the little bits thrown at him by quants as Fiona was unable to integrate little nuggets of data science shared by her senior colleagues. 

Neither can you. If you have an area of weakness, it will remain your area of weakness for life, unless you put sustained, dedicated effort into correcting it. And once you have reached your ceiling, in your career or otherwise, you must improve one of your weaknesses in order to progress further. Put differently, today you may still be reaping the harvest from the seeds that had been planted into you by your parents or your teachers in school or college. Even if all this potential ends up being realized, you must plant new seeds if you are to go further.

Schools are one way of planting seeds, and going to college, getting a PhD or an MBA remain valuable ways of acquiring knowledge. New educational options spring up every day, including online universities and countless standalone online courses scattered over the World Wide Web. Both Fiona and Jacob could break through their respective ceilings by doing additional coursework. 

An astute reader may disagree: isn’t going to college the worst way to learn? Everyone will tell you that the best learning happens on the job, and they do have a point. It has indeed been my experience that people from Academia struggle to convert their theoretical understanding into practically useful outcomes. Universities teach abstract concepts disconnected from reality; a “real job” requires you to engage with this “reality”. 

The idea of the futility of going to college should not be taken too far. The author has experience interviewing data scientists who have just graduated from a traditional 4-year college with a degree in Computer Science. The author also has experience interviewing bootcamp graduates with many years of experience in the “real world”: delivering pizza, working as a barista or as a retail sales associate at a department store. In terms of their ability to do the job, the first group wins, hands down. People in the second group tend to be positive, social, enthusiastic, but they cannot do the job quite yet. Do you really believe that once hired, they will miraculously pick up the skills that they had not picked up during 6 months in a coding bootcamp? Quite the opposite: a job does not give you opportunities to learn what you have not yet learned, only to solidify what you already have.

The principal exception is sales. When you work in sales, most of the learning happens on the job, while you are trying to sell. The commission-based nature of most sales jobs affords employers the luxury of hiring untrained employees. A new salesperson, no matter what they are selling and to whom, does not have to be paid a lot of money, sometimes nothing at all, if the position is 100% commission-based. There is no harm if the new person keeps trying to sell: eventually they will either learn or quit. It works the same way with recruiters and real estate agents. Those fields thus represent an incredible opportunity for the right person: skill acquisition can be done on the job!

In most other fields, the basics must be learned prior to getting the job. In order to transition from one role to another, the basics of the new role must be learned prior to transitioning. If Fiona wanted to do complex, interesting modeling, she would have to level up her skills in addition to and outside of her job at company Z. If Jacob wanted to start creating quantitative trading models himself, he would have to level up his understanding of finance in addition to and outside of his job at company F.  The essential learning can not and does not happen on the job!

C’mon, but our company has a culture of mentorship. 

So does Company Z where Fiona works. While her senior colleagues do have a lot of knowledge to share, they are not professional teachers or coaches. They are not doing it enough to be good at it, nor is Fiona receiving enough mentorship for it to really lift her up to a new level. This amount of mentorship is still better than nothing, so over time Fiona will learn something. Compare this kind of mentorship to an index fund that grows at 10% per year: it is nice when you have to preserve wealth, but it is not a good way to create wealth starting from nothing. 

Have you ever seen a pianist learning a new piece during a concert? Obviously not. A piece is learned and rehearsed dozens or hundreds of times prior to a performance. There is a certain learning that will take place the first time a piece is performed live — but it is not the only kind of learning. It is easy to overestimate the importance of learning on the job. After all, if we have pianist Alice who has performed a piece in front of a live audience and another pianist Bob who has only rehearsed this piece in a studio, other things being equal, Alice will have a deeper understanding of that piece. But it is absurd to conclude from it that the most important learning happens on the job. Compare Alice and Bob to Charlie who has only performed the piece (to his friends), without having practiced it. Who do you think will play it better, Bob or Charlie? Bob — hands down. The most important learning takes place before the job, not on the job. To better appreciate this insight, it helps to notice how it also applies to basic skills such as speaking English, and to even more basic skills such as recognizing faces or being able to find your way from one room to another. Good luck passing an interview while you are still learning to speak!

A meticulous reader is sure to point out that for a professional pianist, practice is considered an integral part of the job. The same objection may apply to data science and software development, but only in a wider sense. If you see constant growth — reading books, attending conferences, doing side projects — as an intrinsic part of your job, whether or not your employer pays for, encourages or knows about those activities — then technically speaking, you are learning on the job. This is a semantic nuance, and it obscures the key insight: performing a skill well — which you paid for— is distinct from improving it — which you are not paid for. Unfortunately, one cannot take the place of the other.

Consider a few more examples: have you ever seen an architect, figuring out the laws of geometry while planning a new building? Have you seen a police officer learning how to fight while on duty? A new driver deducing the traffic rules by observing other cars on the road? No, no and no! People have a long tradition of using exercises in virtually all fields of human endeavor. If your job is “the real thing”, then improving your performance requires “homework” or “exercises” separate from and outside of it. 

It is sometimes possible to have an organization that is completely committed to lifelong learning of its members and has engineered the work to include lots of exercises ensuring constant growth of the employees. Navy SEALs, American Ballet Theatre and the New York Mets come to mind. Those are elite organizations with elite members: frogmen, ballet dancers and Major League baseball players are in it for the long haul. It makes sense to invest in people when they are staying with your organization for the next 5 or 10 years, and every single one of them is highly motivated to improve. Even then, the cost of such continuous learning for the organization is high, and so is the expertise of the mentors required to enable such growth. If you think that American Ballet Theatre is profitable — think again, and SEALs are paid by the taxpayers. Professional sports are an outlier due to the sheer profits generated. 

Tech companies with products used by millions do have the benefit of scale, as do proprietary trading firms. Still, the majority of data scientists and software developers employed by them and not creating enough value, are not sufficiently committed to their craft or are not sufficiently committed to their employer to justify setting up a system for constant growth of all employees. And what if you do find yourself in one of those truly exceptional organizations where everyone is indeed constantly learning? Celebrate! Of course, there will still be a ceiling: getting ahead and assuming a more prominent role would still require you to acquire additional strengths or correct remaining weaknesses not yet naturally taken care of in the course of belonging to that excellent environment. 

Ultimately, your growth has to be your responsibility and not your company’s.

This is the only way to consistently get good results.

And here quantity becomes important. Remember, Jacob and Fiona did try to improve themselves such as by attending a conference or doing a small pet project. The main reason they failed is that such attempts are simply not enough. The amount of time Jacob and Fiona have deliberately spent on their growth as individual contributors pales in comparison with the amount of time they have spent doing regular work. For most data scientists and software developers, it is good if they can invest 40 hours per year into their professional growth, such as by attending a few talks, doing a course online and having a half dozen chats with more experienced colleagues. We can call it learning by accident. The best employers out there take steps to encourage it though, much like Company Z, prefer the euphemism mentorship culture. It does work sometimes — lucky accidents do happen — though most students of this method end up like Jacob: doing well on the outside, yet a far cry from the person he could have become.

Let us do the math: a full-time job adds up to 2,000 hours per year. Using the education-by-accident approach, 2% of your work time is spent on growth as an individual contributor. In college, you are learning full-time for 4 years. This means that it would take you 200 years of the typical learning-by-accident to get the equivalent of a college degree’s worth of knowledge. This is comparing apples to apples: while college is not the best way to learn, Jacob and Fiona can attest that learning-by-accident is not the best approach either. Conference talks and little bits of advice from senior coworkers can be just as useless as college classes.

“This is all very good — a patient reader would interrupt me — but how then should one go about finding time for one’s growth as an individual contributor?”

If at all possible, find a job with a built-in learning component. Assuming you do not want to change your career to recruiting, sales or real estate, look for an exceptional organization such as D.E.Shaw or Microsoft Research. If you can get in, this is a solid personal growth strategy.

The second best option is to simply put in hours in addition to your job. 

Here you might want to get creative. Some people can find a way to work less, such as by taking a pay cut and reducing the workload accordingly. The freed up time can then be used for extra learning, leading to higher pay in the future. It may also work to simply ask the employer, if X% of each work week can be allocated for your own learning, to keep growing as an individual contributor; this kind of request is effective if you are a top performer. But the most reliable strategy is to simply allocate time before or after work. If you are hitting your ceiling and want to find a way to grow further, allocating time is the absolute best thing you can do. Once you have time, you will find a solution. 

It is also possible to learn to extract more learning from the experiences you are already having at your current job. An “external” approach would be to get involved into more aspects of your company’s operations: in addition to just writing code see if you can do some interviewing or mentor a junior employee (by giving them what you have never received yourself). Spend time explaining data science to folks in sales or marketing. Generally look for ways to add value, without compromising your current work or creating a perception that you are compromising it. In contrast, an “internal” approach would involve a structured reflection on your current activities including planning at the beginning of a project and debriefing at the end. In the quest for mastery, learning from your own experiences is a must. It is amazing, how many people pay for conference talks, books, and private lessons to learn from other people’s experiences while ignoring the lessons life is handing out to them.

Lastly, there is the option to quit your job, dedicate yourself to full-time learning, then find a better job using the newly acquired skills. Such full-time learning need not involve formal education, and it need not even be paid! It can be as simple as staying at home, learning to build neural network models or  React websites for 8 hours a day, for a few months. 

As the examples of Fiona and Jacob have shown us, there is no way around putting the hours in if you want to grow beyond what will happen “naturally”, given your current knowledge, skillset, situation or environment. 

If you want to break through your ceiling, make it a priority. 

At this point, a disappointed reader may interrupt me: 

But what exactly should I be doing? I understand the part about allocating time, but should I be contributing to open-source? Playing with new libraries? Taking online courses about neural networks? Building pet projects of my own?

That would be missing the point entirely: you have to start before you know the direction. However, if you are that person who needs exact, specific steps, here they are.

STEP 1:

To find out exactly how you can grow as an individual contributor, whether a data scientist or a software developer, start by doing a simple Google search. Compile a list of at least 10 books, audio books, blog posts, and other resources with specific advice as to what you should do. 

STEP 2:

Read as many or as few of those resources as you wish, and discuss what you have learned with a few colleagues. Make a list of the six (6) most promising ideas.

STEP 3:

Choose your first action towards growth as an individual contributor. It can be “contribute to a popular open-source data science library”. If you are still in doubt, consult with a mentor. If you do not have a mentor, roll a dice. Conveniently, it also has six (6) faces.

STEP 4:

Take action. Rinse and repeat.

If you want outstanding or at least satisfactory results, take responsibility for your own growth — and even for the direction of this growth. Do not expect a quick answer as to what you should do because the correct answer will vary depending on your unique background and circumstances. Of course, if you feel like contributing to an open source library, go ahead and contribute. If you feel like going to that conference, go ahead and attend…

...just remember not to rely on those isolated activities to take you where you want to go, as that would be learning-by-accident. You need a plan in order to keep growing as an individual contributor or in any other way. When you start reaching your ceiling, or when your growth slows down, take the time to understand your situation and priorities, and plan your next steps accordingly. Even the best teacher or mentor would need hours to come up with a good course of action for your particular situation. If you are doing it all by yourself, expect it to take much longer. 

Most importantly, once you have made a decision to keep growing, stick to it. It is your life, and it is your responsibility to your past self: you have not come so far just to give up at this point, have you? Perhaps it has been smooth sailing until now, as it has been for Jacob — then now may be the time for you to step up and face the challenge. Just keep putting in time, one hour after another, and the answers will come.

See you on the other side of that ceiling!


Tags

career advice, data science, software development


You may also like

Miscommunication Etude #4

Miscommunication Etude #3

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}
>