This was an odd book It s really 3 things an internal training manual a recruiting tool and an external guide to Google s testing practices The first two really got in the way of what would have otherwise been a very interesting book As a budding uality Assurance Analyst at a relatively small software company I decided to give this book a read to get an understanding of how a big company like Google approaches software testing I found it to full of extremely helpful information and it opened my eyes to how the rest of the tech industry approaches software testing It goes into detail about how Google ensures that its product line is of the highest uality featuring the various types of positions in the Engineering Productivity department at Google the Software Engineer in Test the Test Engineer and the Test Engineering Manager The skills and responsibilities for each position is fleshed out in great detail and this book did an excellent job of describing Google s software testing practices in an easy to understand manner If you re in software testing and are curious about this topic you won t be disappointed In my day to day job as a software engineer I constantly encounter new and interesting problems in the realm of testing software So when I found How Google Tests Software I was very excited to find out what techniues Google employs to tackle some of these thornier problems at scaleThe opening chapters of the book do a really great job of describing the ethos of testing at Google and very clearly defining what their goals are and how each of their roles and tools will contribute to that overall goal Reading through these chapters I found myself chomping at the bit to find the nuggets of wisdom that I could apply in my own workHowever past the opening chapters for me the book really started to wane in terms of technical details and I found myself working harder and harder to find the lessons that I could take away While I appreciated the author s descriptions of the tools that Google uses internally the overviews weren t enough for me to determine exactly what about the tools solved their testing problems and it wasn t enough to really understand how to build a similar infrastructure at my companyThe latter half of the book focused heavily on interview transcripts with testing leaders at Google While I did appreciate their war stories and takeaways I felt the interviews didn t have uite enough useful information to justify the amount of space that they took up in the book Even the last chapter felt like of an anti climax rather than a solid wrap up of the book as a wholeOverall I thought the book had a lot of potential but the pacing issues with the size of each chapter and the dearth of useful technical details really took a lot of the wind out of its sails I learnt about uite a few useful tools especially pyAuto for driving Chrome automation as well as Protocol Buffers for creating class definitions and serializing data easily However most of the content in this book is management than technical focused and would probably benefit someone who is already familiar with the testing scene as it currently is than new readers trying to find ways to improve their software uality The best book on testing or facilitating engineering productivity as it s nown within the book and among the google SETs software engineers in testing I ve ever read though most of this will be old hat to anyone who s digested the lessons of test driven development read on the advice of my new manager at google Must read for everyone whose work related to software testing and uality assurance When I found out about the book How Google Tests Software it didn t take long until I had ordered a copy I find it uite fascinating to read about how Google does things whether it is about their development process their infrastructure their hiring process or in this case how they test their software I am a developer at heart but I have worked for a few years as a tester so testing is also dear to meIt s uite an interesting book and it makes some great points about the future of testing However despite the phrase Help me test like Google on the cover it is not as useful as I had hoped when it comes to improving your own testingThe book starts off by describing the ey roles at Google SWE Software Engineer SET Software Engineer in Test and TE Test Engineer Briefly the SWE builds features for Google s products the SET develops testing infrastructure and larger scale automatic tests and the TE tests the products from a user s perspective After the introductory chapter there is a chapter each on the SET and TE roles and there is also a chapter on the TEM Test Engineer Manager role The final chapter is about the future of testing at Google and in generalSoftware Engineer in Test SETAs the different roles are explained in the respective chapters there is also uite a bit of detail on how the testing is done at Google The most interesting part in the chapter on the SET role is the part about the infrastructure There is of course extensive support for running tests automatically There is common infrastructure for compilation execution analysis storage and results reporting of tests Tests are categorized as small medium large or eno. Google faces the world's largest toughest software testing challenges To get it right Google is pioneering the future of testing and automation You also need to get it right Learn from Google In How Googl.
Rmous Small tests are basically unit tests where everything external is mocked out and they are expected to execute in less than 100 msMedium tests involve external subsystems and can use database access but generally run on one machine use no network services and are expected to run in under a second Large and enormous tests run a complete application including all external systems needed They can be nondeterministic because of the complexity and they are expected to complete in 15 minutes and 1 hour respectively A good way to summarize them is that small tests lead to code uality and medium large and enormous tests lead to product uality The common test execution environment for running the tests has been developed over time and has several nice features It will automatically ill tests that take too long to run thus the time limits mentioned aboveIt has several features to facilitate running many different test concurrently on a machine it s possible to reuest an unused port to bind to instead of a hardcoded port number that could clash with another test writing to the file system can be done to a temporary location uniue to the current test and private database instance can be created and used to avoid cross talk from other tests Further their continuous integration system uses dependency analysis to run only tests affected by a certain change thus being able to pinpoint exactly which change broke a certain test This system has been developed by Google for many years and has become uite capable and tailored to their way of workingTest Engineer TEThe most interesting part in the TE chapter is the description of the process used for developing the test plan for a product The test plan s purpose is to map out what needs to be tested for the product and when it is done it should be clear what test cases are needed It can be a challenge to find the right level of detail for a test plan but it seems like they have found a good balance at GoogleThe Google process for coming up with the test plan is called ACC which stands for Attribute Component and Capability Attributes are the ualities of the product the The Complete Idiot's Guide to Detoxing Your Body key selling points that will get the people to use the product The examples given for Chrome include fast secure and stable There won t typically be that many attributesNext the Components are the major subsystems of the product around 10 seems to be a reasonable number to include Finally there are the Capabilities which are the actions the system can perform for the user Whereas there are relatively few attributes and components there can be uite a number of capabilities The capabilities lie at the intersection of attributes and components It is natural to create a matrix with attributes along one axis and components along the other axis Then each capability will fit in at the given coordinates Aey property for a capability is that it is testable and each capability will lead to one or test cases to verify its functionality Thus the matrix is an aid in enumerating all the test cases that are neededThe matrix allows you to look at what capabilities affect a certain module If you look along the other dimension you will see all capabilities supporting a certain attribute The matrix is also useful in risk analysis and when tracking testing progressIn the same chapter there is also a good story about a 10 minute test plan James Whittaker did an experiment where he forced people to come up with a test plan for a product in 10 minutes The idea was to boil it down to the absolute essentials without any fluff but still being useful Because of the time constraint most people just made lists or grids no paragraphs of text In his opinion and I agree this is the most useful level it is uick to come up with and doesn t need a lot of busy work filling out sections in a document template and still it s a useful basis for coming up with test cases The common theme in all cases was that people based the plan on capabilities that needed testingToolsThere are other interesting testing tools described in the book too One such tool developed at Google is BITE Browser Integrated Test Environment When testing a browser based app like Google Maps and something went wrong there was a lot of information to extract and put into the bug report For example what actions lead up to the bug what version of the software was running how the bug manifest itself etc The BITE browser extension eeps track of all the actions the tester made in the application and supports filing a bug report by automatically including all the relevant information It also has support for easily marking in a screen shot where the bug appearedAnother interesting tool is Bots It involves automatic tests where many different versions of Chrome fetch the top 500 webpages on the web The resulting HTML is compared and detailed diff reports are producedTipsThere was also a sprinkling of interesting ideas that can definitely be of use in any test organization throughout the book Here are the ones that stuck in my head When asking people to estimate a value for something for example the freuency of a certain failure scenario use an even number of values eg rarely seldom occasionally often That way you can t just pick the middle value you re forced to. E Tests Software legendary Google testing leader James Whittaker and two top Google experts reveal powerful lessons Google has learned about testing software at unparalleled scale and speed and offer brea.
Think about it carefullyAnother example in the same area If you want people s opinion of how likely a certain failure scenario is you could just ask them about it But another techniue is to assign a value yourself and then ask what they think Then you have given them something to argue against Often people have an easier time to say what something isn t then what it isThere is also a uote from Larry Page that is referred to several times in the book for example regarding the relatively few testers at Google Scarcity brings clarity and later on Scarcity is the precursor of optimization Worth thinking aboutAs well as describing how the testing is done and which tools are used there are also a number of interviews with various people in the test organization The chapter on TEM Test Engineer Manager in particular consists almost entirely of interviews 8 in total Most interviews in the book were interesting to read but many of them weren t that useful in terms of tips or ideas to use in your own testingThe Future of TestingFor me the best chapter in the book was chapter 5 Improving How Google Tests Software It is the last and shortest chapter only 7 pages In it James Whittaker shares some profound insights about testing at Google and testing in general One of the flaws he sees with testing is that testers are testers They are not part of the product development team Instead they exist in their own organization and this separation of testers and developers gives the impression that testing is not part of the product it s somebody else s responsibility Further the focus of testing is often the testing activities and artifacts the test cases the bug reports etc not the product being tested But customers don t care about testing per se they care about productsFinally a lot of the testing mindset we have today developed in a different era When you released a product that was it There was no easy way to upgrade it and users had to live with whatever bugs slipped through However these days so much of the software can be fixed and upgraded without a lot of fuss In this environment it makes less sense to have testers act as users and try to discover what bugs they might run into Instead you can release the software and see what bugs the actual users encounter Then you make sure these bugs are fixed and that the new release is pushed out uicklySo his opinion is that testing should be the responsibility of all the developers working on the product It should be their responsibility to test the product and to develop the appropriate tools with some exceptions for instance security testing Whether you agree or disagree with this it is definitely food for thoughtConclusionInitially when I had just finished reading the book I felt a little disappointed It was interesting to read but there didn t seem to be that much to take away from it and apply to your own testing Pretty much all of the techniues and tools are tailored for Google and their needs which is just as it should be But that means that they may not be applicable to your own situationHowever as I am going through it again while writing this review I realize that there are uite a few good ideas in it they just have to be adapted to your specific situation So while not directly applicable the ideas in the book serve as inspirations for how testing can be organized and executed This is not a typical book It has both good and bad The good the level of technical detail the clarity and the straightforward presentation of the authors The book painted a picture where SET software engineer in test was the best technical job and TE test engineers are the best generalist job At least from a reader s point of view It certainly is encouraging and enlightening particularly for google engineersThe bad1 the writing style Oh my god the authors have almost no writing style The whole book reads like a training handout it has verbatim interviews interspersed in the chapters Short articles written by SETs TEs inserted into the text Long code segments interspersed with cultural discussions Verbatim copies of blog posts There is no consistent flow of thoughts The book seriously lack readability improvement2 the vision itself All three authors have left Google after this book was published I cannot help but asking does the vision stand good with time or it is just an wishful thinking of the group of engineers at the peak years of the testing effort of Google How does Google test software how it should be tested with developers doing most of their own testing saving the manual testers for the things that are best done by a human I like the philosophy This book provides interesting views into how an organization Google worked to introduce uality into their software development efforts I particularly enjoyed the interviews with the team tasked with rolling out a new approach to testing within the organization Tasking a small group of experts to drive a new program new mindset new approaches into an organization is tough And getting exposure to how other s have faired is valuable Speaking of mindsets in the spirit of travelling light and delivering value I also enjoyed the coverage of the Attribute Component Capability approach to figuring out and communicating test plans. Kthrough best practices you can use even if you're not uite Google's sizeyet Using these techniues you can transform testing from a bottleneck into an accelerator and make your whole organization producti.