As organizers, the speaker selection process is one of the hardest and yet funnest tasks that we face.
Fun, because it is simply joyful to read through the proposals we receive. They have been a continuous testimony to the creativity, thorough research, and deep knowledge that our community is capable of. Hard, because we value every single proposal. We are fully aware of the emotions and expectations that go into submitting a talk and we hate that, in the end, talk selection also means that we have to decline many of these due to the limited number of speaker slots available.
Every year, when we notify applicants that their talks have been declined, we receive a few disappointed replies.
Our talk selection process is something we’ve put a lot of thought into and which we take very seriously. We’ve spent a lot of time discussing and refining it and we appreciate any feedback and constructive discussions on the matter. In fact, we learned a lot from other organizers who shared their insights. We want to give back by sharing our own learnings and making our talk selection process as transparent as possible.
We hope that this extensive post on the subject will be insightful to both speakers submitting to conferences and conference organizers interested in talk selection strategies.
There is no such thing as a perfect talk selection process
The perfect talk selection process does not exist. There is no one-size-fits-all solution, and we cannot make a universally satisfying ranking from the best to the worst talk. This process is designed to find the really great talk proposals for CSSconf EU. In order to filter out the talks that will eventually culminate in an amazing line-up, we first need to be clear about what we want the conference to be.
We find the following set of steps and principles really helpful:
1. Understanding the Nature and Goals of the Conference
CSSconf EU is a community-driven conference for professionals in the web development industry. We are not an academic event, we are not a standards group event, we are not a user group. This defines the opportunities and limitations of our event.
The conference attracts web developers and designers with varying professional backgrounds and levels of experience. Our goal is to provide them with talks which:
- showcase technical excellence, latest research and cutting-edge technologies,
- provide hands-on knowledge that is both useful for their job and which helps them become better designers and developers,
- entertain, inspire, and encourage them to become active members of a great community.
2. Agreeing on Criteria for Talks
With the above in mind, we agree on the criteria which guide our first round of voting. These criteria, together with explicit details, helpful tips and resources, are also spelled out in our Call for Speakers:
- relevance of the topic for the CSS community
- coherence and clarity of the proposal
- novelty/originality of the topic
- can the topic be reasonably covered in the allocated time
3. Bias and “Blind Voting”
The lack of diversity in tech conference speakers is a problem, and a problem that we at CSSconf are well aware of and do our best to tackle. Anonymizing talk proposals is one of the ways in which we try to hack the unconscious bias that curators have against (or in unfair advantage of) personal attributes of the submitter. This is commonly referred to as “blind voting”, but we prefer the non-ableist term anonymized voting (hat tip to Noah Slater).
4. The First Round: Voting on Anonymized Proposals
The talk proposals we receive are typically short. They often contain no information about the research (or lack thereof) that went into a talk. Sometimes they leave open questions regarding what would actually be covered. The most common comment we add to our voting is “could be great, if done right”. We can only speculate whether a talk will live up to its 5 sentence description. We have observed that the clarity and care of the written proposal are usually indicative of the final talk, and this is one thing we take into account when unsure whether a proposal will deliver what it promises. For this first round we always err on the good side: If a talk “could be great, if done right”, we assume it will be done right.
The Shortlist
We start our process by going through the talk titles and descriptions and voting based on these only, on a scale of 1 to 5. We calculate the average of our votes, and also highlight posts where our votes are in disagreement. So far, so simple. We make a cut and end up with a shortlist of talks, all of which are already talks we really want to see at CSSconf EU. This process allows us to neutrally approve the content of a proposal with no knowledge of the author.
5. The Second Round: Ensuring Speaking Skills and Expertise
Once we have the shortlist, we de-anonymize and look at the details that the speakers on our shortlist provided. In our talk submission form we ask for links to github, personal blogs or websites, twitter, and videos of previous talks. We review these links and videos, or google around for more. We might even email a submitter and ask for additional information or a quick skype call. This is to ensure that the speaker has the necessary skill to present their talk.
In most cases, we are excited to see how much the submitters have already contributed elsewhere, and how great they are at presenting their thoughts. In the past, we have also taken the risk of bringing on first-time speakers, not knowing entirely what will happen on stage. Looking back, we couldn’t have been luckier to have invited these people, and to have witnessed them excel at their very first talks. Having first-time speakers make it to the final selection is also one of the key plus points of anonymized voting.
6. The Third Round: Talk Curation
The first two rounds are only the beginning of a complex process: the actual curation. Being a one-day, single-track event, we want to present to our audience the perfect combination of talks. The considerations that guide us in this round of the selection process have much more to do with how well the talks complement each other. These considerations include:
Varying Levels
We want talks that cater to different levels of knowledge: Some should be at minimum accessible to beginners, others should challenge even the experts.
Talks Beyond the CSS World
We want to hear talks that look beyond the CSS developer world. For example talks that (also) discuss our closely related discipline of design.
Giving the Stage to New Faces
Sometimes, we can reduce the list immediately if a speaker we already featured last year made it into the list; we avoid inviting the same speaker two consecutive years.
Finding a Well-rounded Combination of Topics
One of the toughest situations is to have 2 or more perfect talks covering the exact same topic. Or a really amazing talk that covers a topic we heard last year. Or 5 talks that feature really exciting and still experimental features, but we can only have a few of these. Or 5 talks that deal with a really interesting keynote-type topic, but we also can have only so many of these.
There is a fair amount of shuffling around and discussing various combinations in the shortlist that goes on before we make a decision. So far, this has always been a decision that’s left us feeling a little bit disappointed about all the great talks we simply won’t be able to fit in.
It is also worth mentioning that…
5. We Do Not Simply Discard the Talks That Got Voted Out
The anonymized voting is an imperfect hack to simmer the mass of proposals down to a shortlist. But, hackers that we are, of course we hack the “outvoted” talks again! We re-visit all talks that show a high deviation of our votes, i.e. where we are in disagreement. These are usually surprisingly few in number. This also means that we then re-consider our votes, and the proposal might still end up in the shortlist.
6. Kicking Submitters Out Of The Selection
Yes, we do that too. Having participated as organizers or volunteers in various events, we are aware that harassing, abusive behavior is a reality in our community. We do our best to ensure that no one known to be a harasser or violator makes it to our event, let alone our stage.
7. “Favorites” and “Doing Favors”
The CSS scene is small, and among the people who submit a talk are good friends, colleagues, conference buddies, former speakers, befriended organizers, people who have invited us as speakers before. Some of them ask for advice before submitting, some keep pushing and sweet-talking, some assume they will get in because of a connection, some never tell you. As organizers, this is tricky for us to navigate, and it’s very uncomfortable to reject a friend, a sponsor, an employer. We protect ourselves against the expectation of “doing favors” by making our criteria as clear as possible, and pulling ourselves out of the voting process as soon as we identify a personal “favorite”. Having a team of three organizers makes this easier and helps us even out the biases of each individual.
8. Invited Speakers vs Call for Speakers
At CSSconf EU, we directly invite a very small number of 1-2 speakers early on, even before our Call for Speaker opens. Naturally, one of our goals is to bring accomplished, excellent speakers on stage – Speakers who are able to deliver an inspiring keynote, who excite our audience and have years of experience and knowledge to share. And who often require an earlier notice to plan their conference schedule. All remaining speaking slots are reserved for submissions to our Call for Speakers.
9. No Sponsored Talks
We do not sell speaking slots. We also do not favor speakers whose travel expenses are paid by their company over independent speakers.
10. Constantly Improving: Doing More, and Still Falling Short
- We are constantly considering new strategies to make the CSSconf EU selection process more open, transparent, and respectful of our submitters’ time and work.
- We offer support and feedback along the application process. Via email, twitter, or on our blog.
- We encourage and nudge people to submit talks (overly simplified: to improve the “pipeline” problem).
- We run free CSS workshops and meetups in Berlin to help develop the local scene and offer a practice ground for aspiring conference speakers.
We would love to do even more. If we had the resources, we’d like to:
- provide personal feedback and non-standardized rejection emails for each proposal. A constructive feedback email takes us ca. 30-60 minutes to write and agree on. Thus, we currently only do that upon request, and opted to have more time to work on other aspects of the event, e.g. our scholarship programme.
- We would love to send out rejections quicker and notify people earlier, but in our experience, the selection process is time-consuming if done right.
We empathise with the disappointment that a rejection email can cause. If you were rejected, we hope that these insights into our process are helpful, and we invite you to reach out to contact@cssconf.eu should you have further queries.
All that said, we have been extremely happy with how every single talk turned out at CSSconf EU in the past. In 2015 too, as every year, we are anxious to fulfill the expectations of our audience. Every talk we selected only made it into our line-up because we are 100% excited about it. We can’t wait to hear our speakers present their thoughts, to see them dive into the details of their work that got us so curious, and to be entertained, to be inspired to try out new things, to have great discussions afterwards. We hope that you are, too!