'AlphaCode' - better than 54% of competitive programmers

[LetterRip]

So, I'm thinking lots of low quality programmers could be out of work in the near future (10ish years or so).

https://www.theverge.com/2022/2/2/22914085/alphacode-ai-coding-program-automatic-deepmind-codeforce

54th percentile on a competitive programming site vs people who regularly compete, is probably in the 70th-80th percentile of people who program for a living.

Programming isn't just novel program implementation but it is a crucial part of the programmer skill set.

[Fenring]

Whether it happens sooner or later, it seems to me inevitable that AI and/or automation will eliminate the need for a great many jobs. I thought it would happen by around 2030 but it may take a bit longer than that.

[TheDrake]

That's interesting, but I don't think coders have much to fear - yet. These contrived tests have a few factors that will lead to these numbers, IMO.

1. The specifications are precise and complete. This almost never happens. AI will have to learn how to go back to the "architect" and say "what about this?"

2. The pool being graded against are the people willing to spend time doing "codeforce" challenges. I could see that being highly selective, but not sure of the impact. How many of the 54% are experienced? Who has better things to do? How many of them never played more than 5 matches?

3. While the AI might be able to blow an intern out of the water, part of the reason you hire people at that level is to groom them into experienced people.

4. It might displace some of the tasks, this is already happening, but as a tool without reducing the workforce. This is what generally happens, like when spreadsheets came out it didn't really put a lot of accountants out of work.

Still an interesting article, and it is interesting tech.

[LetterRip]

2030 wouldn't surprise me, not would 2050, timelines are difficult to predict about both the technical advances and adoption.  I tend to think it will happen far faster than the pessimists predict, but probably not as fast as the optimists.

[LetterRip]

TheDrake,

the number of professional programmers who can't do FizzBuzz is pretty astronomical.

Like me, the author is having trouble with the fact that 199 out of 200 applicants for every programming job can't write code at all. I repeat: they can't write any code whatsoever.

[...]

Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Most good programmers should be able to write out on paper a program which does this in a under a couple of minutes. Want to know something scary? The majority of comp sci graduates can't. I've also seen self-proclaimed senior programmers take more than 10-15 minutes to write a solution.

https://workplace.stackexchange.com/questions/164156/are-most-applicants-really-unable-to-implement-fizzbuzz-in-an-interview

[LetterRip]

TheDrake,

they also list the competitions that it entered and its rank in each, it did its best in

https://codeforces.com/contests/1623 (top 21th percentile)

https://codeforces.com/contests/1591
https://codeforces.com/contests/1608
https://codeforces.com/contests/1613
https://codeforces.com/contests/1615
https://codeforces.com/contests/1617
https://codeforces.com/contests/1618
https://codeforces.com/contests/1619
https://codeforces.com/contests/1620
https://codeforces.com/contests/1622
https://codeforces.com/contests/1623

looks like it was '54th percentile' rank on average, so slightly worse than the average competitor, not slightly better.

Ranks are listed in table 4 of this paper,

We evaluated our best system on all Codeforces competitions from 2021/12/01 to 2021/12/28 with
more than 5,000 participants per contest, a total of 10 competitions. The system was an ensemble
of 41B and 9B models with clustering, which performed best on our validation set but turned out
to be slightly worse than using the 41B model alone with clustering (see Appendix C.1 for more on
ensembling). For each contest, we simulated running AlphaCode live, generating samples for each
problem, filtering with example tests, and then clustering to get candidate submissions. We submitted
these selected candidates to the Codeforces platform,6 and computed AlphaCode’s placement in each
contest. After the first run, we repeated this procedure two more times to measure variance and
performance with more than 10 submissions. Sources of variance include problem distribution, model
training, sampling, filtering, and clustering. See Appendix D for the exact evaluation procedure, and
Table A5 and Table A6 for full results.

[...]

Our 10 submissions per
problem result corresponds to an estimated Codeforces Elo of 1238, which is within the top 28%
of users who have participated in a contest in the last 6 months (a small and selected subset of all
programmers).

[...]

With up to a million samples per problem, we can solve 34.2% of
problems in our validation set; and with one hundred thousand samples, we solve 31.8% of problems
in our validation set, and 29.6% of problems in our test set.

https://storage.googleapis.com/deepmind-media/AlphaCode/competition_level_code_generation_with_alphacode.pdf

[LetterRip]

More interesting bits from the paper,

Our models are relatively better at problems that deal with bitmasks, sorting, maths, and greedy
algorithms, but notably worse at dynamic programming (DP) and constructive algorithms.

(a)
shows that when given a simplified description of the problem (not available in real evaluations), the
model solves it at a much higher rate. (b) shows that the solve rate goes down dramatically when
given related but different problems, and is not very affected by different ways of describing the same
the problem. (c, d, e, g, h) show that the model is largely unaffected by changes that do not seem
significant (like replacing words with synonyms or removing some type details), but responds more
to larger changes (like deleting words or making the problem ill-posed). Notably, in Appendix E.3.3,
we see that the model deteriorates relatively more in response to lower quality descriptions as model
quality increases, indicating that better models are more capable of paying attention to subtle but
important description changes. Finally, (f ) shows that the model relies on different sections of the
description, and particularly on the specification. This makes sense because the specification describes
how to read the input, and otherwise the model would have to guess the input format.

[TheDrake]

TheDrake,

the number of professional programmers who can't do FizzBuzz is pretty astronomical.

Like me, the author is having trouble with the fact that 199 out of 200 applicants for every programming job can't write code at all. I repeat: they can't write any code whatsoever.

[...]

Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Most good programmers should be able to write out on paper a program which does this in a under a couple of minutes. Want to know something scary? The majority of comp sci graduates can't. I've also seen self-proclaimed senior programmers take more than 10-15 minutes to write a solution.

https://workplace.stackexchange.com/questions/164156/are-most-applicants-really-unable-to-implement-fizzbuzz-in-an-interview

I hate these questions. They are made up brain ticklers that have nothing to do with the actual job. I'm guessing the reason people take so long isn't to get the correct output algorithmically, is that they are worried about their code being elegant or optimized. Just look at all this foolishness in the link below, giving the"right" answers prone to difficult testing and debug.

The simple algorithm is to seed the list with content equal to index, then walk by 3 and fizz it. Walk a third time by fives and buzz the contents.

Voila.

But then some joker will weep uncontrollably and tsk you for walking the array three times when you only had to do it once, or because you are generating memory read operations when you didn't need to.

At which point, our poor interviewee tries to point out that this likely can't matter because of the size of the array.

Then the interviewer says, what I'd there were a million entries?

Well that wasn't the design you asked for, and I think I'm done here I withdraw my application.

https://wiki.c2.com/?FizzBuzzTest

[jc44]

TheDrake,

the number of professional programmers who can't do FizzBuzz is pretty astronomical.

Like me, the author is having trouble with the fact that 199 out of 200 applicants for every programming job can't write code at all. I repeat: they can't write any code whatsoever.

[...]

Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Most good programmers should be able to write out on paper a program which does this in a under a couple of minutes. Want to know something scary? The majority of comp sci graduates can't. I've also seen self-proclaimed senior programmers take more than 10-15 minutes to write a solution.

https://workplace.stackexchange.com/questions/164156/are-most-applicants-really-unable-to-implement-fizzbuzz-in-an-interview

I hate these questions. They are made up brain ticklers that have nothing to do with the actual job. I'm guessing the reason people take so long isn't to get the correct output algorithmically, is that they are worried about their code being elegant or optimized. Just look at all this foolishness in the link below, giving the"right" answers prone to difficult testing and debug.

The simple algorithm is to seed the list with content equal to index, then walk by 3 and fizz it. Walk a third time by fives and buzz the contents.

Voila.

But then some joker will weep uncontrollably and tsk you for walking the array three times when you only had to do it once, or because you are generating memory read operations when you didn't need to.

At which point, our poor interviewee tries to point out that this likely can't matter because of the size of the array.

Then the interviewer says, what I'd there were a million entries?

Well that wasn't the design you asked for, and I think I'm done here I withdraw my application.

https://wiki.c2.com/?FizzBuzzTest

If you are a programmer and hope that the client will ask you for what they actually want then you are sadly deluded - at least thinking what extensions might be required will normally save you time later and if interviewing I would look for that attitude in a candidate.

But going back to interview, the obvious easy and efficient answer to the question is: "I construct a constant string which conforms to the spec and then print it".

[Mynnion]

All of this swings us back to a discussion on basic income.  It's easy to say get a job but what happens when the job market drops drastically?  Coding is certainly not the only market that will be taken over by AI.

[TheDrake]

Throughout history, jobs get eliminated and new ones show up. Automation drastically reduced the number of agriculture and manufacturing jobs. But we don't have more unemployment than 50 years ago. Oh no, what will the coopers do for a living now??!!

As for the actual job, the vast majority of coding is assembly. Understand the problem, search on the internet for existing code, configure modify and patch it in to the system.

The best possible answer to fizzbuzz is, "I'd google it and use the best option for my environment."

[NobleHunter]

Just because new jobs have always shown up doesn't mean they'll always do so.

I'm not sure we don't have more unemployment but we do have lower wages (after inflation and relative to productivity) which implies a surplus of labour.

[TheDrake]

Sure, on a long enough timeline we could basically become the Elohim. But at that point, economics stop mattering do they not? Compensation by merit would be a relic - assuming that the machines don't phase us out entirely.

Healthcare isn't going anywhere. Neither is entertainment. Unlikely that education goes automated. Real estate. Hospitality. Public safety. Most store clerks (the amazon model isn't likely to replace the clerk at 7-eleven). Pilots.

[Fenring]

Throughout history, jobs get eliminated and new ones show up. Automation drastically reduced the number of agriculture and manufacturing jobs. But we don't have more unemployment than 50 years ago. Oh no, what will the coopers do for a living now??!!

I'm not even sure you can measure this. There is this thing now when people leave the workforce, they are no longer counted. How many have done so? That's just one thing. There's also type of employment. Many people are "employed" but in ways that historically would have been seen as not having a real job. Some of it is because there aren't enough real jobs to go around, some of it is corporate practice to keep people on part-time.

Let's not even get into AI, automation, and self-serve business methods, which have no historical precedent. But definitely let's look at 1870's agrarian economics or 1950's vacuum cleaner culture to figure out how things will work in 2030!

[TheDrake]

Throughout history, jobs get eliminated and new ones show up. Automation drastically reduced the number of agriculture and manufacturing jobs. But we don't have more unemployment than 50 years ago. Oh no, what will the coopers do for a living now??!!

I'm not even sure you can measure this. There is this thing now when people leave the workforce, they are no longer counted. How many have done so? That's just one thing. There's also type of employment. Many people are "employed" but in ways that historically would have been seen as not having a real job. Some of it is because there aren't enough real jobs to go around, some of it is corporate practice to keep people on part-time.

Let's not even get into AI, automation, and self-serve business methods, which have no historical precedent. But definitely let's look at 1870's agrarian economics or 1950's vacuum cleaner culture to figure out how things will work in 2030!

You'd have to assume that this is the LAST transformation and that humans no longer have anything to offer to believe that this transition would be any different than the rest. I just don't see any evidence of that, although I do believe that "truck driver" will not be an occupation in fifty years (or sooner).

In fact, the BLS forecasts faster-than-average job growth in many occupations that AI is expected to impact: accountants, forensic scientists, geological technicians, technical writers, MRI operators, dietitians, financial specialists, web developers, loan officers, medical secretaries, and customer service representatives, to name a very few. These fields will not experience job growth in spite of AI, but through it.

That report by Carl Frey and Michael Osborne is a fine piece of work, but readers and the media distorted their 47 percent number. What the authors actually said is that some functions within 47 percent of jobs will be automated, not that 47 percent of jobs will disappear.

Frey and Osborne go on to rank occupations by “probability of computerization” and give the following jobs a 65 percent or higher probability: social science research assistants, atmospheric and space scientists, and pharmacy aides. So what does this mean? Social science professors will no longer have research assistants? Of course they will. They will just do different things because much of what they do today will be automated.

The intergovernmental Organization for Economic Co-operation and Development released a report of their own in 2016. This report, titled “The Risk of Automation for Jobs in OECD Countries,” applies a different “whole occupations” methodology and puts the share of jobs potentially lost to computerization at nine percent. That is normal churn for the economy.

article

[NobleHunter]

Sure, on a long enough timeline we could basically become the Elohim. But at that point, economics stop mattering do they not? Compensation by merit would be a relic - assuming that the machines don't phase us out entirely.

Healthcare isn't going anywhere. Neither is entertainment. Unlikely that education goes automated. Real estate. Hospitality. Public safety. Most store clerks (the amazon model isn't likely to replace the clerk at 7-eleven). Pilots.

But can those industries absorb all the people freed up by automation? Especially when automation continues to increase productivity even if the automation doesn't eliminate the need for a person doing the job.

And you want to have provisions in place for people who can't find a job before they put a demagogue in power to ease their "economic anxiety." 

[TheDrake]

Sure, on a long enough timeline we could basically become the Elohim. But at that point, economics stop mattering do they not? Compensation by merit would be a relic - assuming that the machines don't phase us out entirely.

Healthcare isn't going anywhere. Neither is entertainment. Unlikely that education goes automated. Real estate. Hospitality. Public safety. Most store clerks (the amazon model isn't likely to replace the clerk at 7-eleven). Pilots.

But can those industries absorb all the people freed up by automation? Especially when automation continues to increase productivity even if the automation doesn't eliminate the need for a person doing the job.

And you want to have provisions in place for people who can't find a job before they put a demagogue in power to ease their "economic anxiety."

Did somebody cancel the social safety net when I wasn't looking? I know Republicans keep trying to dismantle it, but I don't think they've succeeded yet.

[Fenring]

Did somebody cancel the social safety net when I wasn't looking? I know Republicans keep trying to dismantle it, but I don't think they've succeeded yet.

There is no social safety net, not even in social democratic countries. If you have no job you're screwed, across the board. Why do you think slave wage earners go into work when they're sick? Because they know they'd be fired if they didn't and they can't afford that. Even the two weeks it might take to find another crap job is something many people can't afford.

[NobleHunter]

Did somebody cancel the social safety net when I wasn't looking? I know Republicans keep trying to dismantle it, but I don't think they've succeeded yet.

It certainly isn't good enough to keep people forced to rely on it from being fodder for any yahoo with a lust for power.

[Mynnion]

People have moved from agriculture and Manufacturing into the service industry (many medical) and development support such as programming and design.  Some medical and many other service level jobs will be replaced.  I work automating hospital labs and providing data management systems.  Medical Technologists that used to manage a few dozen assays a day are now able to manage thousands.  This is probably a good thing since Med techs are facing work shortages because of low pay and recognition (at least as compared with doctors, nurses, rad techs....). 

Transportation and related industries employ about 9% of jobs.  How many of these will be replaced by AI/automated systems?

Where do you see future employment?  Not everyone will be capable of the more complex jobs that can't be replaced with AI.  Large numbers of dissatisfied unemployed do not equal a stable nation.

[LetterRip]

Throughout history, jobs get eliminated and new ones show up.

That is because automation took extreme amounts of capital and time to design for very specific jobs, and there were tons of jobs that were difficult to automate.  Human brains and human dexterity have allowed for rapid retraining and focusing on complicated tasks that aren't automatable and it would take a long time for the AI to move into new niches.  Near future it might be almost as fast to train the AI to do the task, and the AI would cost on the order of 1% of what a human could do it for, do it drastically faster, and do it 24 hours a day 7 days a week.  Dexterity jobs that have a variety of tasks - plumber, carpenter, electrician, janitor; and empathy/caregiver jobs; entertainers; and perhaps some creative and intellectual - are likely to be the last things automated - but it is unclear what percentage of jobs will be out of reach of AI and robotics for the next 20-50 years in the future.  I think only a small fraction of the population will be employable within 50 years.

[TheDrake]

Throughout history, jobs get eliminated and new ones show up.

That is because automation took extreme amounts of capital and time to design for very specific jobs, and there were tons of jobs that were difficult to automate.  Human brains and human dexterity have allowed for rapid retraining and focusing on complicated tasks that aren't automatable and it would take a long time for the AI to move into new niches.  Near future it might be almost as fast to train the AI to do the task, and the AI would cost on the order of 1% of what a human could do it for, do it drastically faster, and do it 24 hours a day 7 days a week.  Dexterity jobs that have a variety of tasks - plumber, carpenter, electrician, janitor; and empathy/caregiver jobs; entertainers; and perhaps some creative and intellectual - are likely to be the last things automated - but it is unclear what percentage of jobs will be out of reach of AI and robotics for the next 20-50 years in the future.  I think only a small fraction of the population will be employable within 50 years.

You convinced me. Humanity is doomed. None of us will figure out anything to do with ourselves. We haven't even done away with most call center jobs at this point, and the ones that are automated are more likely to have me pressing numbers on a keypad than doing speech recognition that has become trivial quite some time ago for voice assistants. But I'm sure that will all change soon.

[LetterRip]

You convinced me. Humanity is doomed. None of us will figure out anything to do with ourselves. We haven't even done away with most call center jobs at this point, and the ones that are automated are more likely to have me pressing numbers on a keypad than doing speech recognition that has become trivial quite some time ago for voice assistants. But I'm sure that will all change soon.

Google's Contact Center AI is fairly recent,

https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-contact-center-ai-2021-highlights

https://cloud.google.com/solutions/contact-center

I doubt that most call centers are even aware of it, let alone have had time to consider migrating to it.

[yossarian22c]

...
You convinced me. Humanity is doomed. None of us will figure out anything to do with ourselves. We haven't even done away with most call center jobs at this point, and the ones that are automated are more likely to have me pressing numbers on a keypad than doing speech recognition that has become trivial quite some time ago for voice assistants. But I'm sure that will all change soon.

It isn't a matter of finding something to do. Its a matter of finding something that is economically beneficial to other people who have the resources to pay you. Arts, learning, community volunteering and interactions. Plenty of things for people to do. Its a matter if we allow the transition to a more automated society to happen in a way that doesn't produce an oligarchy of the the people who "own" the AIs. If plumber is the last job available, the market will be full of applicants pretty soon. And wages will go down from a good middle class living to the new McDonalds.

[TheDrake]

I'm still waiting for Duplex to show up. It was supposed to be everybody's personal assistant. Google was real excited to tell us about it three years ago. The hamburger flipping robot showed up at that time as well. Boston Dynamics has a robot that can perform backflips, but here's a fun fact. For a whole lot of tasks the human is cheaper per hour. Maybe that changes, maybe it doesn't.

What about pure information that doesn't require physical capital, delivery, maintenance and repair? Not sure, but AI is very bad about judgement. That's why we still have content review human drones, because AI still isn't quite sure which is pornography and which is the venus de milo. And image recognition is something that AI is particularly good at.

Meanwhile, have you noticed how really awful AI is at targeting advertisements? It resorts to showing you ads for stuff you already bought, or only got interested in once. How about YouTube? If you watch one floor repair video, it will offer you more for weeks at a time because it thinks you love them.

And those are applications that companies have actually been trying to do for years.

[LetterRip]

I'm still waiting for Duplex to show up.  It was supposed to be everybody's personal assistant. Google was real excited to tell us about it three years ago.

The Contact Center AI is essentially a more advanced version of Duplex.  They decided it would be more profitable selling it to businesses.

The hamburger flipping robot showed up at that time as well. Boston Dynamics has a robot that can perform backflips, but here's a fun fact. For a whole lot of tasks the human is cheaper per hour. Maybe that changes, maybe it doesn't.

Boston Dynamics stuff is fairly trivial.  They are doing a mocap of a person with a backpack (to simulate the weight distribution) then doing virtual training to tweak to mocap to the robot model.  It looks impressive but isn't.  More interesting is their depalatizing robots.

Robotics costs are due to the low volume of production.  Baxter was only selling about 2,200 units a year and had one of the highest sales volumes for robots, so all of the engineering and management overhead is recovered on that tiny volume, and they can't purchase parts in large enough volume to get good prices.

[LetterRip]

Also you can use Duplex,

Google Duplex works in 49 US states, according to Google’s support site. Only Louisiana is holding out.

[...]

Google Duplex was first launched for owners of Google’s Pixel, Pixel XL, Pixel 2, Pixel 2 XL, and Pixel 3 and 3 XL phones. Now, though, the feature should be available on any device with the Google Assistant app and access to Search or Maps.

https://www.androidauthority.com/what-is-google-duplex-869476/

How to use it is mentioned in the article.