Hey everyone, Leo here! I’m the Systems Administrator of the shiny MacBook Pros and Apple products that we use at Ginkgo. My team is “IT Engineering” and I work on configuration management, automations, DNS changes and some scripting. Big shout outs to my awesome colleagues Connor & Alex for all the hard work they have done on so many projects!
So, what’s IT Engineering about?
IT Engineering is like the glue of the Digital Tech team. We bounce between a large number of Enterprise SaaS products that we manage, as well as supporting a number of the products that Ginkgo employees use.
A large part of our typical day is supporting our Helpdesk Engineers and setting them up so they are prepared for common issues they may encounter in a typical day. Security and compliance are a big topic, and everything we do is geared toward that. Of course, we always take user experience into consideration.
My time at Ginkgo is approaching 2.5 years so I have much to share! I am the team lead for IT Engineering. Much of what we do is behind the scenes and I’d like to give you a glimpse into what my day-to-day is like.
I add value to my team and the company by owning various administrative tasks such as:
- Managing vendor relations;
- Working on software solutions that get us to our desired future state (think OKRs);
- Sourcing IT hardware; and
- Managing financial IT purchases.
What am I working on this week?
Yesterday I spent the better part of the day standing up our Jamf Pro Sandbox environment. Jamf Pro is Apple’s answer to Microsoft’s Active Directory. Essentially if there is a way to manage or automate Windows devices via Active Directory, then Jamf Pro can do it for the Macs.
Much of our time is spent developing automations that allow us to quickly onboard and offboard users and services, for applications and for directory services. Our work also includes supporting Identity and Access Management as we try to leverage single sign-on (SSO) and technologies like SCIM provisioning (where we can quickly onboard users to our applications as well as reduce password fatigue).
Other areas I will dive into this week will be onboarding a new application in our SSO solution, and some configuration and management of our virtualization environment. I will also be working on educating the IT HelpDesk team and building runbooks on how to assist users with Zoom meetings, Zoom Webinars, Zoom Rooms, and Zoom Phones. Migrating the company over to Zoom from another solution was a big project for our team, and it was a big success!
Let’s talk about MacBook Pros!
Past State
Everything, literally everything, that was done to create and manage the Macs was done manually, either by the Helpdesk or IT. This was incredibly time consuming and tedious. Hundreds of clicks and steps. Not fun.
Current State
A few hundred automations later and dozens of scripts (mostly in Bash or Zsh), and we are in a much better place. Now software is automatically deployed to end user endpoints.
Furthermore, Apple has baked-in security into the OS. This is a great thing but at the same time, a technical challenge I have been working to overcome via various extension attributes, kext, Team ID, bundle IDs etc.
What does any of that mean? It means we want to automate away any and all user and HelpDesk manual interactions, via code and configuration management files.
Example: Configuring our Macs with our Antivirus software was not as straightforward as one would assume! Apple requires that applications be manually given permissions, such as full disk access. This is because Apple believes in an Opt-In policy, not Opt-Out like Facebook.
In sum, Apple requires the Admin or the user to consent for an application, Opt-In, to be able to access all the files on your system, instead of automatically giving the app that access and then requiring you to Opt-Out. This is particularly good for microphones and webcams, as it would be creepy if an Admin or App could gain access to your mic or webcam without your consent.
This definitely needed an automated solution! So how do I do this? Well, that depends on the version of MacOS–which is why I pushed hard to standardize our version of MacOS!
For example, on MacOS Mojave I used a series of terminal commands to extract critical information such as the Team ID, bundle ID, and Kext–and then input those identifiers into our scripts. Then I modified the script to provide the correct level of permissions for each app. I’m simplifying a bit here, but let’s say many hurdles have been jumped to figure it all out!
Then all of that changed with Catalina, and even more with Big Sur. That’s why we now tightly control MacOs upgrades, to keep all the automations in alignment. Once we have redone the automations we verify that our new automations work with minimal impact to our users before an OS upgrade is made available to Ginkgo users.
Desired Future State
100% automation, I want to take a newly arrived Mac out of its box with no configuration done by the HelpDesk, and hand it to the user on their first day. The user then only needs to sign on with their credentials and boom! All the configurations push along with all of the apps. That’s it. Basically no user or HelpDesk interaction is required.
Jamf Connect auto-signs the user into our IDP (Identity Provider), which gives the user access to all their assigned SaaS apps. All the locally installed apps are deployed with Jamf and permissions are provided by Jamf Pro.
Note: The only exceptions so far are mic and webcams, due to Apple requiring consent explicitly from the user, But, as I said above, this is a good thing.
As I also discussed earlier, I manage operating systems upgrades and updates with Jamf Pro. I do this by making upgrades available with the click of a button, and having smaller updates automatically install (but require the user’s consent to take effect or reboot the machine). I also restrict software we do not allow running in our environment; that said, we are always on high alert to not negatively impact our users’ productivity.
I’ll show a super simple example how I would do a small fraction of one of these steps via Terminal via the codesign
command to provide the Unique Team Identifier variable:
Terminal(hostname):~ username$ codesign -dr- /Applications/nameofantivirus.app Executable=/Applications/nameofantivirus.app designated => identifier "nameofantivirus.app" and anchor apple generic and certificate 1[field.1.3.941.193434.1145.6.2.9] /* exists */ and certificate leaf[field.55.3.941.193435.1145.6.2.11] /* exists */ and certificate leaf[subject.OU] = DDVYEPZVPR
I then take the Team ID and use that in our scripts to identify the app I am declaring. This is one of multiple ways I do this. Depending on how the developer made the software, this method may or may not work.
Other more recent configurations forced me to make custom modifications to Jamf’s AMP policies where I had to make a custom profile that translates into custom code, just for AMP, so that the new network socket module could operate correctly without user or HelpDesk interaction.
Current Projects in the works for IT Engineering
- Zero Touch Deployment for the MacBook Pros: This is the short way of referring to what our desired future state is for the Apple Products in our environment.
- Management of MacOS upgrades and updates via Jamf Pro
- Cloud imaging solution for Windows 10: This will allow us to image devices at all our future sites, regardless of geographical location. Our goal is to have one source of truth for all Windows imaging, thus reducing issues caused by old images.
- Smart IT vending machines + Smart lockers (fully automated vending machines): This allows people to authenticate into the vending machine with a FOB, and allows us to automatically track its inventory and have it re-stocked, manage costs, and control who can take what.
- Zoom Phones: Complete our migration.
- Education and training: We are spending a lot of energy in keeping our IT support team trained and keeping our runbooks up to date. We are growing quickly and solid processes and procedures will help us scale and solve user issues quickly and efficiently
Closing Note
Ginkgo’s IT Engineering is a complicated and fast-paced work environment. There is never a dull moment. I look forward to continuing to implement the various solutions for Ginkgo moving forward so we can more efficiently enable our users to make biology easier to engineer.
(Feature photo by Daniel Korpai on Unsplash)