When it comes to data analytics, it pays tothink big. PySpark blends the powerful Spark big data processing engine withthe Python programming language to provide a data analysis platform that can scaleup for nearly any task. Data Analysis with Python and PySpark is yourguide to delivering successful Python-driven data projects. Data Analysis with Python and PySpark is a carefully engineered tutorial that helps you use PySpark to deliver your data-driven applications at any scale. This clear and hands-on guide shows you how to enlarge your processing capabilities across multiple machines with data from any source, ranging from Had oop-based clusters to Excel worksheets. You'll learn how to break down big analysis tasks into manageable chunks and how to choose and use the best PySpark data abstraction for your unique needs. The Spark data processing engine is an amazing analytics factory: raw data comes in,and insight comes out. Thanks to its ability to handle massive amounts of data distributed across a cluster, Spark has been adopted as standard by organizations both big and small. PySpark, which wraps the core Spark engine with a Python-based API, puts Spark-based data pipelines in the hands of programmers and data scientists working with the Python programming language. PySpark simplifies Spark's steep learning curve, and provides a seamless bridge between Spark and an ecosystem of Python-based data science tools.

(This book is available at a reduced price for course adoption when ordering six copies or more. Please contact [email protected] for more information.)

The purpose of Experimentation in Software Engineering: An Introduction is to introduce students, teachers, researchers, and practitioners to experimentation and experimental evaluation with a focus on software engineering. The objective is, in particular, to provide guidelines for performing experiments evaluating methods, techniques and tools in software engineering. The introduction is provided through a process perspective. The focus is on the steps that we go through to perform experiments and quasi-experiments. The process also includes other types of empirical studies.

The motivation for the book emerged from the need for support we experienced when turning our software engineering research more experimental. Several books are available which either treat the subject in very general terms or focus on some specific part of experimentation; most focus on the statistical methods in experimentation. These are important, but there were few books elaborating on experimentation from a process perspective, none addressing experimentation in software engineering in particular.

The scope of Experimentation in Software Engineering: An Introduction is primarily experiments in software engineering as a means for evaluating methods, techniques and tools. The book provides some information regarding empirical studies in general, including both case studies and surveys. The intention is to provide a brief understanding of these strategies and in particular to relate them to experimentation.

Experimentation inSoftware Engineering: An Introduction is suitable for use as a textbook or a secondary text for graduate courses, and for researchers and practitioners interested in an empirical approach to software engineering.

Applicable to any problem that requires a finite number of solutions, finite state-based models (also called finite state machines or finite state automata) have found wide use in various areas of computer science and engineering. Handbook of Finite State Based Models and Applications provides a complete collection of introductory materials on finite state theories, algorithms, and the latest domain applications. For beginners, the book is a handy reference for quickly looking up model details. For more experienced researchers, it is suitable as a source of in-depth study in this area. The book first introduces the fundamentals of automata theory, including regular expressions, as well as widely used automata, such as transducers, tree automata, quantum automata, and timed automata. It then presents algorithms for the minimization and incremental construction of finite automata and describes Esterel, an automata-based synchronous programming language for embedded system software development. Moving on to applications, the book explores regular path queries on graph-structured data, timed automata in model checking security protocols, pattern matching, compiler design, and XML processing. It also covers other finite state-based modeling approaches and applications, including Petri nets, statecharts, temporal logic, and UML state machine diagrams.

Property-based testing helps you create better, more solid tests with little code. By using the PropEr framework in both Erlang and Elixir, this book teaches you how to automatically generate test cases, test stateful programs, and change how you design your software for more principled and reliable approaches. You will be able to better explore the problem space, validate the assumptions you make when coming up with program behavior, and expose unexpected weaknesses in your design. PropEr will even show you how to reproduce the bugs it found. With this book, you will be writing efficient property-based tests in no time. Most tests only demonstrate that the code behaves how the developer expected it to behave, and therefore carry the same blind spots as their authors when special conditions or edge cases show up. Learn how to see things differently with property tests written in PropEr. Start with the basics of property tests, such as writing stateless properties, and using the default generators to generate test cases automatically. More importantly, learn how to think in properties. Improve your properties, write custom data generators, and discover what your code can or cannot do. Learn when to use property tests and when to stick with example tests with real-world sample projects. Explore various testing approaches to find the one that's best for your code. Shrink failing test cases to their simpler expression to highlight exactly what breaks in your code, and generate highly relevant data through targeted properties. Uncover the trickiest bugs you can think of with nearly no code at all with two special types of properties based on state transitions and finite state machines. Write Erlang and Elixir properties that generate the most effective tests you'll see, whether they are unit tests or complex integration and system tests. What You Need Basic knowledge of Erlang, optionally Elixir For Erlang tests: Erlang/OTP >= 20.0, with Rebar >= 3.4.0 For Elixir tests: Erlang/OTP >= 20.0, Elixir >= 1.5.0

Software Engineering for Science provides an in-depth collection of peer-reviewed chapters that describe experiences with applying software engineering practices to the development of scientific software. It provides a better understanding of how software engineering is and should be practiced, and which software engineering practices are effective for scientific software. The book starts with a detailed overview of the Scientific Software Lifecycle, and a general overview of the scientific software development process. It highlights key issues commonly arising during scientific software development, as well as solutions to these problems. The second part of the book provides examples of the use of testing in scientific software development, including key issues and challenges. The chapters then describe solutions and case studies aimed at applying testing to scientific software development efforts. The final part of the book provides examples of applying software engineering techniques to scientific software, including not only computational modeling, but also software for data management and analysis. The authors describe their experiences and lessons learned from developing complex scientific software in different domains. About the Editors Jeffrey Carver is an Associate Professor in the Department of Computer Science at the University of Alabama. He is one of the primary organizers of the workshop series on Software Engineering for Science (http://www.SE4Science.org/workshops). Neil P. Chue Hong is Director of the Software Sustainability Institute at the University of Edinburgh. His research interests include barriers and incentives in research software ecosystems and the role of software as a research object. George K. Thiruvathukal is Professor of Computer Science at Loyola University Chicago and Visiting Faculty at Argonne National Laboratory. His current research is focused on software metrics in open source mathematical and scientific software.

From fundamental concepts and theories to implementation protocols and cutting-edge applications, the Handbook of Mobile Systems Applications and Services supplies a complete examination of the evolution of mobile services technologies. It examines service-oriented architecture (SOA) and explains why SOA and service oriented computing (SOC) will play key roles in the development of future mobile services. Investigating current service discovery frameworks, the book covers the basics of mobile services and applications developed in various contexts. The first section provides readers with the required background in mobile services architecture. Next, it details of middleware support for mobile services. The final section discusses security and applications of mobile services. Containing the contributions of leading researchers and academics from around the world, the book: Introduces a new location-based access control model Unveils a simple, yet powerful enhancement that enables Web services to locally manage workflow dependencies and handle messages resulting from multiple workflows Examines an event-based location aware query model that continuously aggregates data in specific areas around mobile sensors of interest Addresses the problem of location-based access control in the context of privacy protection Presents a layered architecture of context-aware middleware Considers the development of assistive technology solutions for the blind or visually impaired Discussing architecture for supporting multi-mode terminals in integrated heterogeneous wireless networks, this book addresses the network availability constraint to serve all mobile services originating from a single-user terminal. It examines QoS protocols and their enhancements in supporting user mobility. Analyzing mobile services security vulnerabilities, it details security design best practices that mobile service developers can use to improve the security of their mobile systems.

This book shows how to identify potential design errors and modify procedures in the design process to mitigate design-induced error. Real life examples are used to demonstrate the points being made. Many of the concerns raised in the book have come from a worldwide study conducted with designers, managers, and end-users.

The reliability of software is becoming increasingly important to a large range of industries that rely on complex computer systems and machinery with computer control. The reliability of a system depends on both the hardware and the software that comprise the system. Although faults in design can continue to give problems, the issues and the techniques for meeting severe reliability requirements in hardware have been understood for some time. In the case of software both the techniques and a positive attitude of software developers to the achievement of reliability are much less well established. They are particularly crucial in the development of software dependent safety-critical systems. There are genuine differences between software and hardware: software has no physical existence; software includes data as well as logic; few software quality metrics exist; it is deceptively easy to introduce changes into software; effects of software changes propagate explosively; software developments have traditionally made very little use of pre-existing components; software may have much higher complexity than hardware (to the extent of attempting to implement functionality of a complicity that would be unacceptable for hardware). However, in many ways software development is like hardware development and can be managed and controlled using very similar techniques to those used in any other engineering discipline. The differences listed above are the very factors which make a genuine engineering approach essential for the successful development of reliable software. Improvements in the technology of software development have reached the point where the major issues have been identified and considerable progress has been made in addressing them. Methods for technical software development are well defined and practical working tools to support improved software production are commonly available. Experience shows that application of the right methods can bring significant improvements in quality and productivity, and this handbook brings together the various aspects of software development into an integrated coverage of the subject of developing reliable software. This handbook represents a unique reference work which provides: complete and comprehensive subject matter - based on real experience rather than hypothesis; straightforward reading - a reference point for technical staff, managers and customers; extensive references - for experts who require more in-depth subject coverage; appendices - containing technical material supporting the subject of software reliability. The Software Reliability Handbook provides an essential contribution to the successful production of reliable software systems. Sponsored by the Alvey Directorate, this book will be an essential tool for those involved with the increasing reliance placed on computer software in control systems of aircraft, power stations and vehicles, especially where life is at risk.

The definition and parameters of teacher education have recently been changed by the concept of mentoring. Supporters of the concept maintain that it is an effective technique for inducting and retaining new teachers, but who and what are mentors, and what attributes do they possess? Previous research has identified collaboration, enthusiasm, emotional commitment, and sensitivity as the necessary traits of an effective mentor. It has also been found that mentors are available, give immediate feedback, listen attentively, and collaboratively solve problems with mentees. Epistemological structures that best serve the mentor-novice relationship, gender issues between mentors and proteges, and moral development orientations within the mentor-intern relationship have also been investigated. Thus, mentoring is a complex activity that involves a mentor, an intern or novice, and a process within a predetermined structure. The articles in this special issue validate the fact that the terms "good mentor" and "effective mentoring relationship" often elude easy definition. However, a common theme emerges: The mentor-mentee relationship is a transformative one that can change the course of one's life.

In many of nowadays web-based environments for electronic marketing and commerce, that present large multimedia product and service catalogues, it becomes more and more difficult to provide naive end users, such as private consumers or commercial business partners, with intuitive user interfaces to access the large multimedia collections describing the presented products and services. The same holds for marketing managers and other employees responsible for managing and maintaining the large and constantly changing set of multimedia information chunks and fragments contained in these collections. As a consequence, many efforts are devoted to improve the quality of the interaction between users and databases. Virtual Reality (VR) techniques are a promising interaction paradigm particularly suited to novice and/or occasional users. The users are facilitated in the database navigation since the system proposes them an environment that reproduces a real situation and gives the possibility of interacting by manipulating objects that have a direct correspondence with known objects.

A developer's knowledge of a computing system's requirements is necessarily imperfect because organizations change. Many requirements lie in the future and are unknowable at the time the system is designed and built. To avoid burdensome maintenance costs developers must therefore rely on a system's ability to change gracefully-its flexibility. Flexible Software Design: Systems Development for Changing Requirements demonstrates the design principles and techniques that enable the design of software that empowers business staff to make functional changes to their systems with little or no professional IT intervention. The book concentrates on the design aspects of system development, the area with the most flexibility leverage. Divided into four parts, the text begins by introducing the fundamental concepts of flexibility, explaining the reality of imperfect knowledge and how development participants must change their thinking to implement flexible software. The second part covers design guidelines, stable identifiers, stable information structures, the Generic Entity Cloud concept, and regulatory mechanisms that give business staff control over system modifications. Part three relates strategic information systems planning to flexible systems. It examines the elicitation of requirements and the relevance of agile methods in a flexible systems environment. It also discusses practical aspects of stable identifier design and compares the testing of traditional and flexible software. In part four, the book concludes with details of the flexible UniverSIS system and an explanation of the applications and extensions of the Generic Entity Cloud tools. The combination of smart design and smart work offered in Flexible Software Design can materially benefit your organization by radically reducing the systems maintenance burden.

This book gathers chapters from some of the top international empirical software engineering researchers focusing on the practical knowledge necessary for conducting, reporting and using empirical methods in software engineering. Topics and features include guidance on how to design, conduct and report empirical studies. The volume also provides information across a range of techniques, methods and qualitative and quantitative issues to help build a toolkit applicable to the diverse software development contexts

This work provides a comprehensive overview of research and practical issues relating to component-based development information systems (CBIS). Spanning the organizational, developmental, and technical aspects of the subject, the original research included here provides fresh insights into successful CBIS technology and application. Part I covers component-based development methodologies and system architectures. Part II analyzes different aspects of managing component-based development. Part III investigates component-based development versus commercial off-the-shelf products (COTS), including the selection and trading of COTS products.

Software Quality Assurance: Integrating Testing, Security, and Audit focuses on the importance of software quality and security. It defines various types of testing, recognizes factors that propose value to software quality, and provides theoretical and real-world scenarios that offer value and contribute quality to projects and applications. The practical synopsis on common testing tools helps readers who are in testing jobs or those interested in pursuing careers as testers. It also helps test leaders, test managers, and others who are involved in planning, estimating, executing, and maintaining software. The book is divided into four sections: The first section addresses the basic concepts of software quality, validation and verification, and audits. It covers the major areas of software management, software life cycle, and life cycle processes. The second section is about testing. It discusses test plans and strategy and introduces a step-by-step test design process along with a sample test case. It also examines what a tester or test lead needs to do before and during test execution and how to report after completing the test execution. The third section deals with security breaches and defects that may occur. It discusses documentation and classification of incidences as well as how to handle an occurrence. The fourth and final section provides examples of security issues along with a security policy document and addresses the planning aspects of an information audit. This section also discusses the definition, measurement, and metrics of reliability based on standards and quality metrics methodology CMM models. It discusses the ISO 15504 standard, CMMs, PSP, and TSP and includes an appendix containing a software process improvement sample document.

This book is an introduction to graph transformation as a foundation to model-based software engineering at the level of both individual systems and domain-specific modelling languages. The first part of the book presents the fundamentals in a precise, yet largely informal way. Besides serving as prerequisite for describing the applications in the second part, it also provides a comprehensive and systematic survey of the concepts, notations and techniques of graph transformation. The second part presents and discusses a range of applications to both model-based software engineering and domain-specific language engineering. The variety of these applications demonstrates how broadly graphs and graph transformations can be used to model, analyse and implement complex software systems and languages. This is the first textbook that explains the most commonly used concepts, notations, techniques and applications of graph transformation without focusing on one particular mathematical representation or implementation approach. Emphasising the research and engineering methodologies used, it will be a valuable resource for graduate students, practitioners and researchers in software engineering, foundations of programming and formal methods.

"The situation is good, but not hopeless" (Polish folk wisdom) The text is devoted to the Software Analysis and Testing (SAT) methods and s- porting tools for assessing and, if possible, improving software quality, specifically its correctness. The term quality assurance is avoided for it is this author's firm belief that in the current state of the art that goal is unattainable, a plethora of "gu- anteed" solutions to the problem notwithstanding. Therefore, the rather awkward phrase "improving correctness" is to be understood as an effort to minimize the number of residual programming faults ("bugs") and their impact on the software's behavior, that is, to make the faults tolerable. It is clear that such a minimalist approach is a result of frustration. Indeed, having spent years developing software and teaching (preaching?) "How to do it right," I still do not know how to go about it with any degree of certainty! It appears then I probably should stop right now, for who with a modicum of common sense would reach for a text that does not offer salvation but (as will be seen) hard work and misery? If I intend to continue, it is only that I suspect there are many professionals out there who have similar doubts. And they are the intended audience of this project. The philosophical underpinning of the text is the importance of sound engine- ing practices in software development.

Information Systems Development: Reflections, Challenges and New Directions, is the collected proceedings of the 20th International Conference on Information Systems Development held in Edinburgh, Scotland, August 24 - 26, 2011. It follows in the tradition of previous conferences in the series in exploring the connections between industry, research and education. These proceedings represent ongoing reflections within the academic community on established information systems topics and emerging concepts, approaches and ideas. It is hoped that the papers herein contribute towards disseminating research and improving practice

* Targests readers with a background in programming, interested in an introduction/refresher in statistical hypothesis testing * Uses Python throughout * Provides the reader with the opportunity of using the book whenever needed rather than following a sequential path.

This book provides techniques for offshore center managers and head office managers to motivate and manage globally distributed teams, which are spread across the offshore center and the head office, and thereby achieve higher productivity. Readers learn how to integrate the offshore center with the head office to make the offshore team an extension of the head office. While integrating teams with the head office, offshore center managers can still retain independence and authority to meet team aspirations. The book provides insight into devising new organizational structures to balance the authority and responsibilities of offshore center and head office managers. Head office managers responsible for managing globally distributed projects learn how to achieve a higher success rate on their projects and be better rewarded for their efforts in offshoring. Head office managers also learn techniques to make more significant contributions in their expatriate assignments to the offshore center. This book guides both the offshore center managers and the head office managers to fully realize the potential of the offshore center, which can result in higher revenues and profitability.

Going where no book on software measurement and metrics has previously gone, this critique thoroughly examines a number of bad measurement practices, hazardous metrics, and huge gaps and omissions in the software literature that neglect important topics in measurement. The book covers the major gaps and omissions that need to be filled if data about software development is to be useful for comparisons or estimating future projects. Among the more serious gaps are leaks in reporting about software development efforts that, if not corrected, can distort data and make benchmarks almost useless and possibly even harmful. One of the most common leaks is that of unpaid overtime. Software is a very labor-intensive occupation, and many practitioners work very long hours. However, few companies actually record unpaid overtime. This means that software effort is underreported by around 15%, which is too large a value to ignore. Other sources of leaks include the work of part-time specialists who come and go as needed. There are dozens of these specialists, and their combined effort can top 45% of total software effort on large projects. The book helps software project managers and developers uncover errors in measurements so they can develop meaningful benchmarks to estimate software development efforts. It examines variations in a number of areas that include: Programming languages Development methodology Software reuse Functional and nonfunctional requirements Industry type Team size and experience Filled with tables and charts, this book is a starting point for making measurements that reflect current software development practices and realities to arrive at meaningful benchmarks to guide successful software projects.

In the decade since the idea of adapting the evidence-based paradigm for software engineering was first proposed, it has become a major tool of empirical software engineering. Evidence-Based Software Engineering and Systematic Reviews provides a clear introduction to the use of an evidence-based model for software engineering research and practice. The book explains the roles of primary studies (experiments, surveys, case studies) as elements of an over-arching evidence model, rather than as disjointed elements in the empirical spectrum. Supplying readers with a clear understanding of empirical software engineering best practices, it provides up-to-date guidance on how to conduct secondary studies in software engineering-replacing the existing 2004 and 2007 technical reports. The book is divided into three parts. The first part discusses the nature of evidence and the evidence-based practices centered on a systematic review, both in general and as applying to software engineering. The second part examines the different elements that provide inputs to a systematic review (usually considered as forming a secondary study), especially the main forms of primary empirical study currently used in software engineering. The final part provides practical guidance on how to conduct systematic reviews (the guidelines), drawing together accumulated experiences to guide researchers and students in planning and conducting their own studies. The book includes an extensive glossary and an appendix that provides a catalogue of reviews that may be useful for practice and teaching.

Software is one of the most important products in human history and is widely used by all industries and all countries. It is also one of the most expensive and labor-intensive products in human history. Software also has very poor quality that has caused many major disasters and wasted many millions of dollars. Software is also the target of frequent and increasingly serious cyber-attacks. Among the reasons for these software problems is a chronic lack of reliable quantified data. This reference provides quantified data from many countries and many industries based on about 26,000 projects developed using a variety of methodologies and team experience levels. The data has been gathered between 1970 and 2017, so interesting historical trends are available. Since current average software productivity and quality results are suboptimal, this book focuses on "best in class" results and shows not only quantified quality and productivity data from best-in-class organizations, but also the technology stacks used to achieve best-in-class results. The overall goal of this book is to encourage the adoption of best-in-class software metrics and best-in-class technology stacks. It does so by providing current data on average software schedules, effort, costs, and quality for several industries and countries. Because productivity and quality vary by technology and size, the book presents quantitative results for applications between 100 function points and 100,000 function points. It shows quality results using defect potential and DRE metrics because the number one cost driver for software is finding and fixing bugs. The book presents data on cost of quality for software projects and discusses technical debt, but that metric is not standardized. Finally, the book includes some data on three years of software maintenance and enhancements as well as some data on total cost of ownership.

In business, driving value is a key strategy and typically starts at the top of an organization. In today's digital age, driving software value is also an important, and often overlooked, key strategy. Executives, and the corporate board, need to expect the highest level of business value from the software the organization is developing, buying, and selling. In today's digital transformation marketplace, it is imperative that organizations start driving business value from software development initiatives. For many years, the cost of software development challenged organizations with questions such as: How do we allocate software development costs? Should these costs be considered an overhead expense? Are we getting the most value possible for our investment? A fundamental problem has been built into these questions - the focus on cost. In almost every other part of the organization, maximizing profit or, in the case of a not-for-profit, maximizing the funds available, provides a clear focus with metrics to determine success or failure. In theory, simply aligning software spending with the maximizing profit goals should be sufficient to avoid any questions about value for money. Unfortunately, this alignment hasn't turned out to be so simple, and the questions persist, particularly at the strategic or application portfolio level. In this book, Michael D.S. Harris describes how a software business value culture-one where all stakeholders, including technology and business-have a clear understanding of the goals and expected business value from software development. The book shows readers how they can transform software development from a cost or profit center to a business value center. Only a culture of software as a value center enables an organization to constantly maximize business value flow through software development. If your organization is starting to ask how it can change software from a cost-center to a value-center, this book is for you.

While emerging technologies create massive opportunity, especially for investors and companies that seek more adaptable forms of economic growth than currently available, value is held inert by traditional approaches, patents, and other closed systems. Yet, open data, content, and information may be the key to mass innovation for future technologies, although they bring difficult challenges to private-industry models that depend on the established ideas of intellectual property. It is from this foundational observation that OpenXFORM (a blending of the words Open and the engineering abbreviation for Transformation) was developed and is explored and described in this book. The intent of the model design is to synthesize an approach to the process of innovation, inspired by natural systems and human-centric design processes. OpenXFORM describes how an open system of innovation can adapt to the unregulated world of information, data, and content; can decompose its own information to release to the open world; and can discover ways to find the points of synergy among the studied and tested methodologies that put human relationships first. This book presents an explicit innovation process that shows how to move from a breakthrough idea through a process that encourages innovative thinkers to test their assumptions, validate hypotheses, and tune and tweak their ideas, not only to drive solutions for users but also to meet the strategic goals of their companies. The anatomy of innovation through OpenXFORM contains the process for moving ideas from a flight of fancy to an explicit concept that is ready to produce.

Software analysis patterns play an important role in reducing the overall cost and compressing the time of software project lifecycles. However, building reusable and stable software analysis patterns is still considered a major and delicate challenge. This book proposes a novel concept for building analysis patterns based on software stability and is a modern approach for building stable, highly reusable, and widely applicable analysis patterns. The book also aims to promote better understanding of problem spaces and discusses how to focus requirements analysis accurately. It demonstrates a new approach to discovering and creating stable analysis patterns (SAPs). This book presents a pragmatic approach to understanding problem domains, utilizing SAPs for any field of knowledge, and modeling stable software systems, components, and frameworks. It helps readers attain the basic knowledge that is needed to analyze and extract analysis patterns from any domain of interest. Readers also learn to master methods to document patterns in an effective, easy, and comprehensible manner. Bringing significant contributions to the field of computing, this book is a unique and comprehensive reference manual on SAPs. It provides insight on handling the understanding of problem spaces and supplies methods and processes to analyze user requirements accurately as well as ways to use SAPs in building myriad cost-effective and highly maintainable systems. The book also shows how to link SAPs to the design phase thereby ensuring a smooth transition between analysis and design.