(calm piano music) – This is a short tour of visualizing, documenting, and exploring software architecture And we’re gonna start here Sorry, imagine we’ve invented teleporters and I teleport you here, where are you? France? Close, but not quite So yeah, it says La Rue de something This is what happens when we drop new joiners into a code base We kind of drop them straight into the middle of it and they’re a bit lost How do we solve this problem? How do we figure out where we are? We zoom out, we use technology, we open up the maps on our phone and start zooming out Any better? Not zoomed out enough Well, you can now see the name of the road, but we’re not quite sure still where we are We zoom out one more time Yeah, there we go So that shows you that’s where I live There’s a lot of cluttering on this map, isn’t there? It’s not very, very clear to see what’s going on So with things like Google Maps, we can reduce the amount of information Like that Gets a bit clearer, now we can start to see some of the place names and where some of the bits and pieces are If you’ve never heard of Jersey before, this is still kind of useless so you have to pinch to zoom out one more time And again And again So now there you go, there’s Jersey It’s a small island just off the coast of France Now, if you come and visit Jersey, which you should because it’s lovely, when you come through the airport you’ll get a map And this is a traditional Visitor Map thing and it shows you information like this The map’s broken up into four quarters and it kind of tells you enough information to get around It tells you enough information to go and find the major sites and that sort of thing It doesn’t tell you everything though It’s not listing every street, it’s not listing every building on every street either There’s a zoomed in part for town, the main city, and we’ve got a little bit more detail on it, but it’s a representations, it’s not quite accurate So this thing here, this blue bit, the tide actually goes out there So if you pull into Jersey and you boat and you park there, you will find it dry at some point And also, this bathing pool here It looks like it never gets refreshed with water, but it does, which is useful because I’m sure kids pee in it in the summer. (audience laughs) The thing these maps have in common is that they show points of interest So what do I really want to go and see when I visit Jersey? Contrast this with like an ordnance survey map This is a very, very detailed representation of Jersey We’ve got contours and different types of land and all sorts of things going on here And you need a bit of intelligence and some help to get started interpreting this thing Both maps are showing selected highlights as well And that’s a really interesting thing here, selected highlights So when you come to Jersey next year on your summer holiday, you’ll probably drive past this big castle It’s called Elizabeth Castle, it’s on the south coast from the sea And it’s a 16th century castle, some beautiful granite blockwork, and it was basically built to stop the French and English taking Jersey over And when you walk around this castle, you kind of get to the top in the middle and things start to change And you’ve got this beautiful granite stone thing with this weird concrete thing stuck on top for some reason And you’re like, “That looks really weird “What’s all that about?” And history’s important here, so when you’re walking around the castle, it tells you that in World War Two when the Germans occupied Jersey, they refortified some of the existing fortifications So that’s what that big concrete thing is on the top and those sorts of gun replacements and so on and so forth Of course, all of this stuff is described in guide books and there are lots of them out there and of course, if you want to be a bit hipster, you can use the website We have the Internet now So that’s my quick introduction and sales pitch for Jersey, I guess I’m gonna come back to the same thing Right, so visualization One of the things I do with software teams around the world is I run a visualization workshop and we give some people some requirements So we say, “Right, you have 90 minutes in groups “to go draw some pictures.” And these are the types of pictures people actually draw I promise I have not made any of these up I’m not sure I could make these things up Do these look like diagrams you see on your whiteboards? Yes (audience laughs) I have another 15 gigabytes I could show you, but I won’t Sometimes when I’m running this workshop, I actually hear people say this Right, so they’re drawing a box or a shape and they say, “This doesn’t make sense, but we’ll explain it later “when we do our presentation or something.” And that’s fine, but we don’t always present our diagrams, do we?
Now, if we’re drawing a picture out of our system, we don’t always stand there and present it And a nasty trick I like to do is I like to have two teams swap diagrams And guess what happens? Because they weren’t part of that conversation creating those diagrams, they have no idea what’s going on and they’re at this kind of Wat moment. (audience laughs) And they’re mad about the color coding and the shapes and the lines and basically none of the notations on it makes any sense whatsoever When you ask people, “Was this an easy exercise?”, they say, “Well, actually yeah it was.” And then you say, “Well, why are your diagrams such a mess?” and they go “Oh yeah, good point.” We don’t really know what to draw, we’re not sure what sort of diagrams we should draw, the levels of details, shapes, notations, should we use UML? Who here uses UML? That’s a really small number, like four or five people So what’s everybody else doing? Something else. (laughs) I’ve asked this question around the world and UML is massively falling out of fashion I have no evidence to back any of this up, this is all completely anecdotal, but I’m seeing more and more teams who have no UML skills in the teams anymore I do use UML, but I use is sparingly for small parts for a software system What I want to show are very small parts of say, a class hierarchy or something Google doesn’t help If you go to Google and ask it what it thinks a software architecture diagram looks like, you get this Page after page after page of essentially pretty-colored block pictures, the sorts of things you can do in a Visio or PowerPoint And we laugh at these things, but these are exactly the types of diagrams I see when I go and visit organizations You open up their conference pages or whatever, bang, you get this sort of stuff And half the pictures just don’t make any sense at all This is hard I think I’ve run this workshop for about 10,000 people now, all around the world, and nobody does it sufficiently well the first time around, believe it or not And the irony here is Agile So one of the great things Agile has done is it’s made us more visual Whenever I go and visit Agile organizations, you’ve got their camera boards and the storywalls and the information dashboards So we’ve gotten awesome at visualizing, essentially processing the way we work We’ve totally forgotten how to draw pictures of the things we’re building And this is just about good communication If you want to move fast as a team, if you want business agility, then you need to communicate well, that’s it So there’s some really simple tips here around notation that I could give you And just stuff like put titles and pictures, make sure your arrows are annotated, make sure your arrows point one way, and so on and so forth And the notation around drawing architecture diagrams is really, really easy to fix One of my key points is this one here, responsibilities We often joke that naming is hard in software, so it doesn’t make sense that most of our architecture diagrams are essentially just a collection of named boxes because the diagrams contain a huge amount of ambiguity So my simple tip is add more text to your diagrams And here’s a really simple example This is the same diagram, the version on the right has more text And that additional text allows us to see things like the responsibilities of those building blocks There’s a lot more information there In terms of content, when we’re drawing architecture diagrams you can’t show everything on a single picture And this is why people do talk about things like views and viewpoints and perspectives And there are lots and lots of different ways to do this I would mention Philippe Kruchten’s 4 + 1 model as a book that details a whole bunch of ways to do this and there are more The thing is all of those view catalogs have this logical view of a system separate to the development view of the system So the logical view is often either the functional, logical, conceptual building blocks and then there’s something else that refers to how we’re building these things Why? Why do we do this? I don’t get it Because whenever I go to organizations and I see their nice, fluffy logical architecture diagrams, they never match the code And that’s kind of the point here If I’m drawing architecture diagrams, I need them to match the code Otherwise, they’re just lying to me George Fairbanks is sitting in the back He calls this the model-code gap in his awesome book When we’re having architecture discussion, we’re using abstract concepts like modules and components and services, but we don’t have those same things in our programming languages Quick show of hands, who’s a Java developer here? Right, so in Java is there a layer keyword? No In Java is there a component keyword? No, but we create components and layers by assembling classes and interfaces and packages and things together So what I’m trying to get to is a set of diagrams that actually reflect the code there That’s my ultimate goal Before we can even attempt to solve this problem,
It’s showing us all of the code level elements and all of the relationships between them and it’s hard to really pick out what the important parts of this code base are And this is a really, really small app, like less than 1,000 lines of code What happens if you point this at 100,000 or one million lines of code? You just get crazy, crazy diagrams And the reason is that these diagramming tools see code, not components These diagramming tools are usually unable to zoom up to show you bigger abstractions It’s essentially the model-code gap thing again And we can trace this problem right back in time There was a paper about this problem in the 90s The first two opening paragraphs basically say if you ask an engineer to draw a picture of their software system, you get a nice high-level view If you reverse engineer a diagram from the code, you get something totally, totally different The reverse engineered diagram is super accurate, of course, but it’s not how the engineer thinks about the world And it all comes back to this question Well, what is a component? If I want to draw a component diagram, I need to understand what a component is And if we go back to my little class diagram here, these two boxes I’ve highlighted are really what I consider to be the components in this little API app I’ve built There’s a Java server handling API requests and there’s a workspace component that’s dealing with these structure of the workspaces You’ve heard of server-less, this is framework-less (laughs) This is like the simplest implementation you could possibly write And all of these other code things are just parts of those two components And again, this is what the code structure looks like So I’ve got two major components with a bunch of helper code, essentially Now, we have to say that the code is the single point of truth, the code is the final embodiment of all the architectural ideas Can we get that information back out of an existing code base? And the answer is not really So if you were to give me your code base, could I generate something like a context diagram automatically by looking for references to people and software systems in your code base? And the answer is no because they don’t really exist We don’t have this information in our code base much of the time The same with containers Can I get a list of the containers just by scraping data from your code base? There’s some information in there, but it’s kind of hard to find When I get down to the component level, this is really the level I want to generate automatically because it’s the most volatile and it changes the most frequently George Fairbanks to the rescue George Fairbanks says we should adopt an architecturally-evident coding style Anybody doing this? You should, it’s really a tool that’s fantastic So it’s a really, really simple technique and it’s simply about embedding information into your code base so that your code base reflects your architectural ideas and intent This sounds kind of high-level and wobbly In concrete terms, it’s just stuff like naming conventions So if you have a logging component on your architectural diagram, make sure there’s something in your code base called logging component Maybe it’s about a namespacing or packaging convention One fold or one namespace, one package per box on the diagram Or maybe it’s machine-readable metadata, annotating stuff to be this is important, this is a component for example And by using this we can then extract really useful information from a code base and supplement it where that information’s not possible And I want to move away from drawing diagrams in Visio There’s something called an architecture description language I’m guessing no one’s ever heard of it and no one’s ever used it And that’s because it’s never really entered mainstream industry, but an architecture description language is essentially a textual description of say, the static structure of a software system There are a bunch of them out there, Darwin or Koala for example, but the syntaxes are horrible and you have to teach developers another weird looking language just so they can describe the piece of software that they’re building But this is a fantastic concept because we’re not dealing with pictures, we’re dealing with text And as developers we like text We can diff text, we have tooling to support text So let’s take all of these things, chuck ’em in a pot, stir it around, and come up with an architecture description language using code, using general purpose code, the same code we’re using to build our systems And that’s the other piece of Structurizr, so there were two Open Source libraries and basically they are a very small implementation of the C4 stuff Again, there’s a bunch of classes in both, one for Java and one for .NET, and they let you create people and software systems and containers and components and wind them together to describe your software architecture So that’s how we got using this for describing my little API thing So as a user, a person The person, the software developer’s using my Structurizr product and Structurizr uses the API to store information locally So again, we can just write up some code
to create that little model And then I can create a system context view by adding the appropriate things to my diagram And the net result is very simply you get a picture like this from code like this This is a very, very simple way to describe the high-level structures of a software system We go down to containers, it’s just the same deal So from a container perspective, essentially all I have is a little API server, it’s a web app, it’s a Java web app storing information on a file system So again, we can create a couple of containers, we wind them together just using some method calls, and we can create some diagrams So basically you write code, you get pictures This is great for the high-level stuff Once you get down to components, you don’t want to have to do that So this is why the Open Source libraries have some component finder things in it And the component finders find components See, naming’s not hard after all Now, the question becomes how do you find components and the answer is it’s up to you because every code base is different And this comes back to the architecturally evident coding style thing So if you have a naming convention that your team’s adopted, you can go and find components based upon that naming convention If you use a framework like Spring, you can go and find Spring annotations and call them components and so on and so forth So lots of different strategies you can plug in to find components This is the code I use to find the components in my little API application Got a couple of different strategies here I want to find the things ending with the word servlet and I want to find the things that I’ve annotated with my own special app component annotation Find them, wind them together And actually there’s some logic behind the scenes in the framework that goes and finds the inter-component dependencies And that’s the sort of diagram you get from that So that’s the API servlet component that it found and that’s the workspace component and it’s identified the relationship between them You might be wondering how did that text get on there? And the answer is you have to add some more metadata to your code base to do that This is really about creating a model, so I want to get people away from using diagrams and back to using modeling as an approach for describing software And because once you have a model, you can do lots of really interesting things like generate diagram keys automatically. (cheers) No more horrible notation we don’t understand Because this is a model, we can do things like hyperlink the model to the code So if you go online and you find these diagrams, you can double-click any of the components and it takes you right to GitHub, it takes you to the exact implementation of that thing And again, it comes back to the maps thing Diagrams are maps This is a funny tweet 388,000 pixel wide picture My reason for including this tweet is basically, does such an approach scale? And the answer is no if you use it naively So this is what happened when I threw my own tooling at one of my own web apps All of these things are web app controllers and these things are components My goodness, it’s horrible I don’t think the code’s horrible, the diagram’s horrible, and of course making the diagram canvas larger does not help one bit because you still have the message, just spread out But because this is a model, you can do some interesting things like don’t show me everything, show me a slice through the system For example, we can say what’s a slice? Well, maybe it’s a slice starting from a web app controller, an entry point to your system And show me that slice until you drop out of the bottom of the app So basically you can create a larger number of much, much simpler pictures So that’s my approach for dealing with complexity in scale Once you have a model, you can throw it into lots of other types of tooling So if you’re a Graphvis fan, part of the Java Open Source library is a Graphvis exporter so it creates a DOC file, you put the DOC file into Graphvis, it auto-generates diagrams for you If you hook all of this stuff up to your build process, guess what happens? Your documentation, your diagrams remain up to date when your code changes and that’s ultimately what I’m trying to get to here So the title of this talk was Visualize, Document, Explore so we should talk about documentation really quickly A lot of people are no longer documenting anything I know that sounds a bit extremist We have our friend, the Agile Manifesto to thank for that It people’s misinterpretation of what the Agile Manifesto says about documentation And of course, if I did drop you into a project that’s unfamiliar to you, a code base that you’ve never seen before, you do get that kind of, “Where am I on this leafy lane?” and you have to start zooming around exploring and try to figure out where you are And this takes time That lane I dropped you in right at the start, you could keep walking until you met somebody and asked them, “Where am I? “Where’s the nearest X?” for example But it’s just gonna take some time
Of course, once you’re exploring the code base, the code base actually doesn’t tell you everything you want to know The code doesn’t tell you the entire story, especially around things like rational and intent Why were these decisions made? That’s often missing from the code base And often, teams have lots of tribal knowledge kicking around as well, specialists and experts in particular parts of the code base And that’s fine until those people leave or they get run over by the proverbial London bus The thing is, the bus factor as we call it, is not just about buses Imagine you have a small team here, one does get run over by a double-decker bus, bye-bye Someone goes into sabbatical for a year and we have to fire somebody because they’re useless And now we have a smaller team and the smaller team starts saying things like, “You know that thing there “that we have to run every week, what is it?” Oh, and again, that’s a kind of extreme thing, but I have seen situations like that occur So how do we fix our documentation problems? Well, we write some documentation And this is where the SAD comes into play And it’s called the SAD because it makes you sad And there are lots of templates out there for this sort of thing Every consulting company I have ever worked at has created their own Rupp has one, there’s lots of them out there And these sort of software architecture documents normally include some really interesting, insightful information How do we get to the design, what were the design decisions, what is the architecture, how do we look after it? These documents tend to be just horrible Hundreds of pages Out-of-date totally, useless So how do we fix this? Well, it turns out naming is our friend What I do is I just rename the document and all the problems go away. (audience laughs) And I call it a guidebook Like the Jersey guidebook thing I talked about at the start If you come to Jersey, you buy a guidebook, it has a bunch of maps to help you navigate an unfamiliar environment, it shows you the sites and the itineraries, the points of interest, the things you really should come and see It talks about the history and the culture How did Jersey get to where it is today? And there’s all the practical stuff Again, if you take that metaphor and apply it to a software system, you get a really, really interesting set of documentation So maps, the diagrams, show me what the code looks like, how do we explore the code base, which parts of the code base are useful to understand, which parts are important because most stuff isn’t How did the code base evolve to where it is today and how do we look after it? This is my single tip for documentation, just describe what you can’t get from the code base That’s it, it’s as simple as that Knock it up a level of abstraction And make these things small Don’t have hundreds and hundreds of pages of stuff that becomes out-of-date Make these things as small, lean as possible They’re really kind of living, breathing, evolving style documents This isn’t the big up front design, we must create a SAD up front approach This is a supplementary piece of documentation that sits alongside the code base And it’s a product-related document A software system, a document, a supplementary document This doesn’t mean you can’t do product documentation as well, but every software system should have a user guide essentially How does the software work? (sighs) You choose, I don’t care It’s the tooling thing and what tooling do you recommend for documents? A lot of people still use Word, that’s fine SharePoint if you really have to Lots of people using Conference, but I’m seeing more and more teams at tops of things that I ask to dock a mark down And they’re creating a nice little documentation files and they’re sticking the documentation files next to the source code in source code control And then they do stuff at build time, generate HTML, uploading to websites and Wikis and so on So there’s a lot of opportunities for using different tooling here Again, that’s something I’m trying to do with Structurizr I want to create a software architecture model that contains both the model, the visualization stuff, and the documentation So this is the code I wrote to document my little API application It’s a bunch of simple mark down files, you upload it as part of the model, and some documentation’s basically generated for you, that’s it So again, I’m trying to keep all of this stuff in one place And you can also embed diagrams in your documentation So again, these things are together where they belong There are some other approaches to documentation So this guy’s saying lots of really interesting stuff around living documentation, he’s got a whole bunch of little tools, Open Source and GitHub, that you can use for creating documentation from code, for example And these guys in Germany have a software architecture document they call arc42 and it’s a really, really nice, lightweight, lean approach to documenting software systems It’s very similar to the approach I particularly take And there’s lots of content, lots of free content that you can grab on arc42 People say, “How long should a document be?” and that’s the wrong question to ask