Views and announcements

Signifyd: Scaling systems and teams to achieve high-speed growth

  • Written by Michael Liberty, CPO and co-founder of Signifyd

    If you’re the sentimental (and geeky) sort, you no doubt know that this year we’re celebrating the 53rd anniversary of Conway’s Law.

    And even if you’re not sentimental you might want to tip your hat to Melvin Conway, who in writing about systems and organizations in 1967 made an observation that should be regarded as the “Golden Rule” of building effective Product teams. What Conway said was, “Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organisations.”

    Conway was a computer scientist, but his observation was sociological. He recognized that programming is a social activity with many collaborators contributing to the development of any significant system. Thus, how you develop good software over an extended period of time is as much a sociological question as it is a technical one.

    Conway’s Law (from Giulio Santoli)

    I’ll propose a corollary to Conway’s law which reinforces its sociological nature: “Organizations which design systems… are constrained to have communication structures that are supported by their system designs.” In other words, communication structures and system designs are mutually reinforcing and tightly coupled. In the case of a software application, you cannot decide how to architect that application without deciding implicitly how the people building that software will operate. The question of design and communication are inextricably linked.

    Implications for teams that want to be “agile”

    “Agile Development” has nearly religious devotion and universal adoption among modern software companies. Declaring that you follow a “Waterfall” development methodology is a surefire way to kill your recruiting pipeline. Despite this devotion, too many companies ignore the implications of Conway’s Law for their Agile Development teams. First let’s briefly review what the Agile Manifesto actually says,

    …[W]e have come to value:

    • individuals and interactions over processes and tools.
    • customer collaboration over contract negotiation.
    • responding to change over following a plan.
    • working software over comprehensive documentation.

    It’s reinforced with 12 principles, including, “Working software is the primary measure of progress.”

    Although it’s not explicit in the principles, “small teams function better than large ones,” is certainly implied.  It’s simply harder for large teams to abide by, “Businesspeople and developers must work together daily throughout the project.” Or, “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” 

    Like everyone else, we at Signifyd consider ourselves practitioners of Agile Development, but not because we’ve adopted Scrum or some other agile methodology.  Rather, consistent with the spirit of the Manifesto, we believe teams perform better when each member has the full context of their project, as well the responsibility and capability to bring products from nascent idea to production service. And we believe that’s easiest to achieve with small teams.

    Given that desired communication structure, what would Conway’s law tell us is the best system architecture? For us, the answer has changed over time, but today it is unequivocally a Microservice Architecture, defined as “designing software applications as suites of independently deployable services.”

    When and why to adopt to microservices

    Embracing a Microservice Architecture is not a free lunch -- it comes with significant extra technical overhead relative to a monolithic application, and it’s certainly possible to adopt it prematurely. However, at a certain scale, the overhead is worth it because of the intense clarity of purpose that microservices enable.

    Consider Signifyd’s life story, so far, as an example. When the company launched in 2011, it was my co-founder Raj Ramanand and me. We knew, practically on a minute-by-minute basis, what needed to be done, who would do it and when it was finished. That applied to code, funding, hiring and ordering lunch for that matter.

    A few years into our startup journey, I’d made a few thousand commits and added a million lines of code — and deleted around 900,000 lines. There’s no doubt that level of churn was due, in part, to my inexperience at the time in developing a SaaS application, but it also reflected that we were still trying to figure out what our application and business would be.  We had to react quickly to our perception of customer needs, and often those perceptions were wrong. Raj, myself and our early employees were so tightly linked, that the communication structure of our organization could accurately be described as monolithic. And so was our tech stack. Our public API, web user interface, webhooks, fraud scoring, reporting, etc., it was all one deployable application.

    As a startup, you have just one advantage over incumbents: speed.  They have more money, more employees, more customers, more everything.  The only correct answer to the typical VC question, “Couldn’t Google do this?” is “Yes, but we’ll do it faster.” All you have is speed, and that speed comes from intense focus and limited overhead.  Decisions, engineering or otherwise, are made very fast.  Our monolith was not a marvel of engineering (far from it!), but it did let us move fast without spending our limited resources on orchestrating lots of microservices.

    With that being said, every startup hits a wall where its communication structure must evolve, when it’s untenable to continue as a monolith. When that happens, you must evolve your systems with the organisation, or they will block your progress.  For us, I think we hit that wall at about 20 people in product and engineering, give or take.

    RELATED: Belfast helps Signifyd become one of US' fastest-growing innovation companies

    Don’t resist the inevitability of Conway’s Law

    Twenty-person agile teams aren’t really a thing, so it was obvious we needed to break down into smaller feature teams.  We had, and wanted to keep, a “build it, deploy it, monitor it” ethos in our engineering organization. A monolith is inherently mismatched with this structure.  No matter how clean and decomposed your code is, if the application is a monolith, ultimately teams are building, deploying, and monitoring everything, not just their component. That’s a significant cognitive burden — if, when you go to deploy your team’s bug fix, you’re worried about inadvertently deploying another team’s major feature, you’re constantly distracted.

    I think we at Signifyd were fortunate enough to understand the inevitability of Conway’s Law, so we were able to work with it rather than fight it. Doing so pointed us squarely at microservices. And lucky for us, the tools enabling a microservice architecture had advanced immensely since we founded the company in 2011— version 1 of Docker, Kubernetes, and gRPC hadn’t even been released back then. Nevertheless, the upfront cost of doing that migration was meaningful, and there were significant non-obvious challenges we encountered when doing that switchover. For example (and these are just a few),

    1. Debugging, Logging, and Cascading failures become far more challenging.  When you’re dealing with a single application that’s just one deployable artifact, debugging is pretty straightforward.  You can solve most problems by looking at stacktraces.  What happens, on the other hand, when your application crosses multiple network boundaries? Suddenly, a failure in one microservice can cascade to the whole system with almost no visibility into the root cause. Recall, we have a “build it, deploy it, monitor it” ethos. That becomes very frustrating if you’re constantly being paged and can’t determine root causes. Tools like Opentracing and Jaeger are key to addressing this challenge.
    2. End-to-end testing becomes more complicated. Multiple microservices need to work together to provide a useful application. In order to test an application’s performance, all the microservices required to drive that application need to be run together. In a complex system like ours, no developer is going to understand all the microservices that are required to support their application, and even if they did, spinning them up manually would be tedious.  Dependency management takes on a new level of importance and needs ongoing care and attention. We’ve chosen to manage our microservice dependency graph within our build tool, Gradle, because we can integrate it into our integration test suites.  Other tools like Docker Compose can help here too.
    3. Again, reinforcing the technical and sociological nature of these problems, when teams are developing microservices, each can theoretically develop their own standards, practices, and culture as well as use any technology. While that freedom is at times appealing, it can also be damaging to team members in subtle ways. There is a learning curve associated with any new technology, and if there are vast differences from team to team, engineer mobility will be limited and team membership can become ossified. At Signifyd we practice “constrained freedom,” the idea that teams operate autonomously within a rather broad set of boundaries — boundaries just narrow enough to ensure that we don’t succumb to practices so divergent that it's as if we are no longer part of a larger organization.  It’s similar to Dan McKinley’s well-known “Choose Boring Technology” philosophy.

    For the past five years, and the past three in particular, we at Signifyd have been extracting pieces from our monolith and spinning up new teams that develop collections of microservices. The most important result of our work has been scalability — but not scalability in the traditional way engineers think of it.

    RELATED: Pushing Code to Production on Day Two: My Experience Joining Signifyd

    Engineers normally think about scalability in terms of the number of requests or the amount of data you can handle. Microservices help there, but what I’m most interested in is organizational scalability. We want the ability — and we’ve achieved it through a microservice architecture — to add more people to our team and see our output grow linearly.

    Can your organisation scale horizontally?

    If you anticipate rapid growth for your company in the years ahead, a question you must ask is, “Is my team horizontally scalable?”  Do my system and organizational APIs, so to speak, align? Can you add to a team without becoming mired in the bureaucratic overhead that the Agile Manifesto was designed to combat: documentation, rigid development plans, etc.?

    Signifyd has been growing rapidly, particularly in the last two years. Our global R&D centre in Northern Ireland, which opened just over a year ago, is building products and services essential to Signifyd's overall offering. In the last year alone, we’ve added 60 team members in Belfast, on our way to more than 150 in the next few years.

    The idea from the start was to have the teams in Belfast operate autonomously with communication to, but no gatekeeping by, our US teams in Silicon Valley. That means “build, deploy and monitor” their own applications. That is not something we could do without our microservices architecture.

    And while we didn’t make the move to microservices as part of our expansion into Northern Ireland, the success that our teams in Belfast are enjoying and the innovation that they are producing is a clear confirmation that both Conway’s Law and Signifyd are on the right track with the philosophy.

    RELATED: Raj Ramanand, Signifyd CEO and Co-Founder, on Belfast’s building culture

    About the author

    An article that is attributed to Sync NI Team has either involved multiple authors, written by a contributor or the main body of content is from a press release.

    Got a news-related tip you’d like to see covered on Sync NI? Email the editorial team for our consideration.

    Sign up now for a FREE weekly newsletter showcasing the latest news, jobs and events in NI’s tech sector.

Share this story