Introduction
Earlier this year, we gathered in Ginkgo’s Boston HQ for Hack Week: an opportunity for the whole team to gather, to meet remote colleagues face to face, to work on passion projects, and to see just how much we can get done in a single focused week. No top-down planning and prioritization, no meetings, no tickets – just a Slack channel to coordinate, desk space, coffee, and a view of Ginkgo’s robots.
The rules were simple:
- Meetings are canceled. Regular work is paused.
- Everyone is free to work on anything Ginkgo-related.
- Teams are allowed and encouraged.
- Demo by Friday afternoon.
In this post, we’ll share some highlights of what was accomplished; some thoughts on why events like Hack Week are important; and some implementation tips to try this with your team
The Results
At the end of the week, 36 projects by teams from 1 to 12 people shared their accomplishments. The projects ranged from small quality of life improvements to existing software, to exploration of new services, to experiments with exciting new technologies. A number of the “hacks” were such clear wins that they went straight to production while others informed ongoing development and roadmaps.
To give you a taste of what the team built, here are a few brief descriptions:
- An exploration of technologies like NFC and QR codes to streamline equipment reservations and logins.
- Quality of life improvements by extracting code into common libraries, speeding up CI/CD pipelines, upgrading dependencies and creating bots to automate upgrades.
- Our tools are wide-reaching, complex, and fast-moving; several projects improved the way we communicate updates to our scientists by automating the delivery of targeted, personalized messages.
- Acceleration of the default analyses we run on every NGS sample – thousands of analyses per day – reducing the runtime by 42%!
- A prototype for easier examination of Omics data that our HTS team creates.
- Use of ChatGPT to create chat interfaces to complex APIs, lowering the costs and barriers to entry for exploring our data with powerful libraries.
- Improving access to internal knowledge by leveraging “Retrieval Augmented Generation” (RAG): using a stock LLM paired with an searchable database of internal documents to answer questions.
We also took advantage of the whole team being together in Boston to have some fun with team dinners and games of giant chess.
A post-hackweek survey of the Digital Technology department showed that the event was an unmitigated success, with 100% of the respondents calling it a good use of time (“agree” or “strongly agree” on the Likert scale).
Why Hack Week Is Important
Some folks might raise an eyebrow at an entire department spending a whole week working on… who knows what, honestly. There is no project planning, no deliverables, no scope of work! Are Hack Weeks a ZIRP?
There are 3 main reasons we believe this is a great use of our time:
- Hack Week makes us faster.
- Hack Week provides an opportunity for outsized returns on investment.
- Hack Week makes us a stronger team.
Hack Week Makes Us Faster
Hack week makes us faster by fixing two kinds of friction:
- Friction in the products and services we provide
- Friction in tools and processes we use to develop and deliver those products and services
This is super important, because for a company like Ginkgo, speed is a compounding benefit.
Friction in Products and Services
In every organization, the folks doing the “hands-on” work – engineers, designers, help desk support agents, and others – know a ton of small problems that never quite make it into the prioritized list of issues and ranked list of projects that come out of planning processes. These might be documented annoyances, tech debt, or just that thing you saw and filed away in your brain hoping to fix that one day.
The usual way to address this is to tell teams to allocate some % of total estimated effort to fixing things like that. In practice, this is necessary but not sufficient. Some issues need sustained focus, and a few hours here and there are not enough. Some issues don’t quite feel bad enough at the moment – even if they are major time-sucks over time – to work on when there are other pressing deadlines. Some folks just feel uncomfortable going off-plan, for a variety of reasons.
Hack Week removes all that. Just go fix it! Need focus time? Have a whole week. Need prioritization? You wanting to fix it is priority enough.
Every decent-sized engineering org has people who are highly motivated by making things work better. They will fix problems whose true impact is much larger than we realize. But we must get out of their way and let fixers fix and make the tools more usable, processes more automated and make everything just a bit faster.
Friction in Tools and Processes
We aim to pull out all the friction during Hack Week – just put your headphones on, work on your thing, and ship (to a demo environment). Every time we do this, we find out that there are aspects of our setup that cause friction.
Certain operations can only be done by a few specific people, or require multiple handoffs, or occur on rigid schedules. Some things just require too much busy work and oddly specific knowledge.
Every year, we’ll find this kind of stuff – clear signs of what slows us down every week, but become obvious in Hack Week, when we are all trying to go from start to finish on a tight timeline. So we apply pressure to get things done quickly, and observe what slows people down. We take these learnings, and we improve our tools and processes. Templates get created, automation and bots put in place, conventions and status quo rethought.
We become faster at doing our jobs – in all weeks, not just Hack Weeks – as a result.
Speed is a Compounding Benefit
There’s a lot of digital ink spilled on the benefits of speed for companies in general – speed of execution, speed of innovation, speed of response to customer needs.
For straightforward end to end project delivery speed, it’s a no-brainer. There’s simple linear math here: if you can make and sell 10 widgets in the time it takes someone else to make and sell 8, you are making 20% more in the same amount of time. But it gets better. If you are 20% faster than the competition, everything else being equal, you get way more than a 20% increase in your share of the market. You might even grow the market, through speed alone, because projects that were not feasible because of the timelines become possible. Turnaround speed is a critical feature; this is why Amazon invested in their delivery network, for example.
But it gets better. Ginkgo is fundamentally not in a widget-making business, we are in an innovation and applied research business. The speed with which we iterate and learn is crucial. Faster processes and tools mean faster OODA loops and faster DBTAL cycles. Faster loops and cycles mean shorter time to learning and insights. That means we don’t just create the end product faster, we learn how to get better and faster, faster. That’s a compounding benefit.
The time we spend in Hack Week making everything work smoother and faster has a huge return on investment.
Hack Week is a Shot at Outsized Returns on Investment
Most of the year, we create project plans, interview stakeholders, write down requirements, break down and estimate tasks, set goals – the whole thing. We do this for risk mitigation and predictability. We need to know that the things we are planning to do stand a decent chance of delivering. We’ve got people depending on us! Promising more than we can deliver is bad. Running into unexpected problems and blowing deadlines is bad. Discovering half way through that you need another team to have done something, while they are completely focused on something totally different, is bad.
It’s important to do this stuff, it’s what keeps the balls in the air. Most of the time.
Planning and estimation reduce risk and improves predictability by removing variance in work outputs. Which is mostly a good thing! The problem is that reducing variance not only removes the lows, but also trim the highs. It is an approach built to minimize investments in big, risky moves, because failure is expensive.
Hack Week is the opposite. We encourage big, risky moves, and we celebrate failure. Got an idea that will make us 4x faster or easier? Go for it! It might fail. It’ll probably fail. That’s ok. It might succeed! Or it might give us great information, by failing, for how it might succeed next time!
Hack Week encourages high variance work. High variance work gives us a shot at completely changing the trajectory.
Hack Week Makes Us a Stronger Team
The hacking, learning, and social activities we organize during the week make us a stronger team, which translates to the other 51 weeks a year. By spending time working and having fun together, we are able to form stronger cross-team bonds. This makes communication and collaboration easier. By providing time to learn, experiment, and teach others, we are creating space for learning and growth, investing in the team, in our skills and success. Through social activities and mixers we put faces to names and create opportunities to see each other as real, three-dimensional people, not just slack handles and Zoom rectangles. This improves our social cohesion and greases the workings of the digital technology machine.
Rolling Your Own
Here are some tips and tricks we use to pull off Hack Weeks.
Schedule and Organization
You don’t want to introduce too much process for how people choose what to work on, but a little bit of structure helps. We create a wiki page for every hack week with a table to which people can add short idea blurbs, links to further explainers, and a list of who is planning to work on the idea and what help they need. We also create a temporary Slack channel for idea discussions a month or two before the event.
And the most important thing: we remind everyone, especially team managers, to cancel all meetings.
On Monday, we kick the week off with a quick get together to have coffee and pastries, go over the timeline, and give folks a last-minute chance to find teammates and ideas they want to pursue.
The middle of the week is spent hacking. We also organize learning opportunities – discussions on technical topics, “office hours” with internal and external experts, whatever people are into. Since we are a distributed team, and Hack Week is our one annual get-together, we also encourage folks to plan optional team activities like dinners and other outings.
On Thursday, some fraction of people always want to stay late and finish up their projects. We don’t require or expect folks to do that, but since many do, we support them by ordering pizza and feeding anyone still in the office late. Great conversations and early demos happening over Thursday Pizza are a Hack Week hallmark.
Friday afternoon is Hack Fair. We share out all the project presentations through Slack, put up posters and demos, and invite the whole company to check out what we’ve been up to. Many great discussions happen around those posters!
Riding high on the excitement of doing demos, seeing what amazing things our colleagues have accomplished, and all the stimulating conversations, we head off to an offsite and the sunset.
Presentations
One of the few Hack Week requirements is that teams do their best to present their work by Friday. To make this easier, we created a standard template for slides that can be distributed for remote viewing, or printed out and glued to poster boards for the Friday Hack Fair. The template is intentionally simple, with a slide for each of the following:
- Project Name & list of team members
- Big bold succinct problem statement
- Problem context – whom the problem impacts, why it matters
- What’s your bright idea?
- What did you actually do?
- State of the hack: where you got to, how can we play with it.
- What’s next?
We found that having a standard template is good both for the hackers, who spend less time wondering what and how they are supposed to present, and for Hack Fair attendees (including other hackers!) who can read & understand standardized presentations faster.
Of course, we allow – and encourage! – live demos to go with the posters and slides.
Voting
We use Confluence templates to generate pages for each project – hosting the project presentation and any links or narrative descriptions teams want to add. Folks are invited to vote for their favorite projects during Hack Fair, and during the week after. They can vote for as many projects as they like. Voting is simply hitting a thumbs-up emoji on the relevant page. Some enterprising individuals have taken to generating QR codes and putting them on their posters, to make voting even simpler.
Keeping It Tidy
A bunch of people hacking on lots of projects can leave a lot of debris around. To minimize this, we provide several resources.
Hack Week GitLab Group
A dedicated GitLab group provides a convenient landing spot for most greenfield projects, and separates them nicely from more “permanent” work. A GitHub org or even a dedicated git repo with a folder per project can work just as well. It’s nice to be able to instantly know if the code you are looking at is from a hack week project or “for real”.
Dedicated Sandboxes
Ginkgo uses AWS accounts to enforce tight access controls and segregate application domains. Developers generally have access to resources in accounts relevant to their team, but require special approvals to gain access to other accounts. In cases where leveraging a shared Kubernetes infrastructure makes better sense, we use namespaces to achieve the same goal.
To simplify Hack Week collaboration on net-new projects for folks from different teams, we create a dedicated AWS account and Kubernetes namespace, and grant all Hack
Week participants access. This way developers can work on sandboxed prototypes without running into access restrictions, and also give us a convenient method for terminating all running processes and deleting test data after hack week is over.
Automation Joy
There’s nothing quite like going from 0 to demo in five days to highlight where a bit of automation would help us focus productively on the actual problem, rather than undifferentiated heavy infrastructure lifting. It’s a real pressure test for your developer platform!
Developer portals like Backstage are great to expose scaffolding templates and “golden paths”, but you can also make do with something as rudimentary as a Jenkins job or even a shell script. Spinning up databases and servers pre-instrumented according to company conventions, scaffolding RPC services and front-ends, hooking up to your observability tools, basic CI/CD setups to make it trivial to ship the code to a staging environment, applying your company’s color palettes and design standards – automating any and all of this away for the common cases speeds up Hack Week work, and makes regular work faster and easier, too.
To Sum Up
Hack Week is a fun and exciting event where teams get to work on cool and creative projects that can lead to big rewards for the company.
Hack Week makes us faster because it gives us a chance to fix problems that needed to get addressed for a while, but have not gotten the attention they deserve. It improves our products and services. It also helps us identify what’s been holding us back and gives us a chance to fix it. The freedom to take on “high variance” work sets this week aside from all other weeks, and provides a counterweight to more sure, lower variance work we do normally.
Lastly, it’s a time when people can come together, break down barriers, and work on projects they’re passionate about. Plus, it’s a great way to bond with your colleagues and build stronger relationships.
We had fun with this one; hope you give it a try!
(Feature photo by Caspar Camille Rubin on Unsplash)
