Muck Rack

Imagine reading and cataloguing every English-language article published on the web. That’s what public relations platform Muck Rack does on a daily basis. Every story — including this one — is automatically scraped by its platform, categorized and pinned to the corresponding author’s profile. To make that happen, Muck Rack’s engineering team needs to bring order to the chaotic data generated by an industry where every publication has a different content system, page layout and text format. This summer, the New York-based company hired Matt Dennewitz as its new VP of engineering. Dennewitz has spent the last few months digging into Muck Rack’s codebase to learn how it works and what makes it tick. One of the first things that stood out to him was how clean it was — which makes sense, considering that it’s built to make sense of utter chaos. Writing clean code saves onboarding time. First off, it helps new hires get a handle on what each part of your codebase actually does. It also helps engineers working in a new language get up to speed. Build a culture around clean code. From the onset, Muck Rack CTO Lee Semel encouraged engineers to prioritize writing clean code, creating a legacy that’s passed down as new hires join the team. Provide tools and create rules around clean code. As a policy, Muck Rack uses linting tools like eslint, black and isort to check code for errors before it’s committed. The team also conducts pull reviews and evaluates new hires based on their ability to write clean code. In order to quickly scrape millions of websites for news stories, Muck Rack needs to identify what a news story looks like, differentiate between headlines and the story itself, and sort every story by the person who wrote it. It’s a tall order that requires navigating custom-built content management systems and websites that block bots. And once you have the data, you need to appropriately sort it to make it useful. Muck Rack CTO Lee Semel built Muck Rack’s platform in the Python-based Django framework because of its emphasis on readability and clarity in its code structure. He had previously written in PHP, but the language proved to be difficult to maintain a clean structure in. When the scraping engine finds a news publication, it strips the story down to its text level and looks for metadata tags identifying headline, byline (or author information), dateline and text. If there are no tags, the company’s algorithm is trained to identify common patterns for what these elements typically look like, and where they are usually placed. From there, the platform scans the story for patterns like author name, publisher and content to match it with the correct author. This is how it knows a story is published by say, Stephen A. Smith, noted ESPN hot take analyst, rather than another writer named Stephen Smith. “You can’t always avoid author collisions, but you can set up smart rules.” “We can infer associations based on publisher or recency of other publications,” Dennewitz said. “You can’t always avoid author collisions, but you can set up smart rules.” When Dennewitz looked into modules inside the Muck Rack code base, one of the first things that stood out to him was how clean the code was. He saw well-formatted Python types annotated, variables annotated, thorough documentation and obvious naming that makes everything easy to read and understand. The emphasis on clean code stemmed from the culture CTO and co-founder Lee Semel started 10 years ago, Dennewitz said. Semel’s initial work in the codebase served as the example for all new hires to follow, and later informed the documentation new engineers receive today. The company also makes clean code a priority in hiring. Every candidate receives a test project, and clean, understandable code is one of the primary skills they’re measured on. Before any code is written, engineers are encouraged to reach out to the product team to discuss the feature they’re requesting. Bec