The Exploding Endpoint Problem: Why Everything Must Become an API
The future consumer will expect to access all human knowledge and services on-demand, using devices that fit in their pocket, and all for the price of a meal. This requires a digital experience that can connect anything to anything and continuously adapt to changing data and services.
Quite simply, every future app may need an approach to Program the World, a way of connecting, accessing and controlling every digital and physical asset on the planet.
No one can envision what the next digital experience will be. But for it to exist, tomorrow’s programmers will require secure access to the world’s data and systems. We must enable every form of digital asset to become programmable, transforming everything into an Application Programming Interface (API). Everything must — and will — become an API.
Here are a few of the problems we must solve:
- Massive scale: There are billions of digital assets: legacy apps, SaaS apps, APIs, SDKs, IoT devices, databases and identities. Yet most of these assets are not yet accessible nor programmable.
- Data diversity: We are generating 2.5 quintillion bytes of new data a day, and this rate is accelerating exponentially. Most of this data is at rest, not accessible to apps-in-motion, and at best weakly accessible — with response times poorly suited to on-demand reactivity.
- Hapless heterogeneity: The discovery, access, interface, and format of APIs are not standardized leading to fragmentation of programming languages, frameworks and developer consumption models such as containers and serverless as a way to deal with complexity. It’s no wonder that 70% of IT managers have an IT skill gap in their organizations.
- Dizzying demand: Use is expected to dramatically increase as digitization of systems-of-record become the world’s future competitive advantage (see Andreesen’s Software is Eating the World). Global Internet users are growing 10% year-over-year to more than 3.5B (source: Mary Meeker 2017 Internet Trends report), and consumer usage shows solid growth with the average consumer spending 5.6 hours/day online (source: eMarketer 2008-2016).
When APIs are network accessible by others, they become an endpoint. Whether it is data, apps or serverless functions, everything is becoming a programmable endpoint.
We already have billions of endpoints, and the growth is exponential. In 20 years, the world will have 100 million programmers; combined with vendor offerings that can create an API-on-demand for the duration of a single request, this is moving us toward a model where countless micro-endpoints are being deployed as part of everyday activities. And when developers can regularly create APIs and operate them on-demand with a minimal cost, the incentives to create even more endpoints by splitting tasks leads to an explosion of endpoints.
In less than 20 years, the world will grow from billions to more than 1 trillion programmable endpoints. We are facing an Exploding Endpoint Problem.
Almost all Enterprises Will Feel the Pain
The future apps we write will increasingly be programmed across these diverse (and sometimes short-lived) endpoints. The resilient communication between endpoints is the discipline of integration. It isn’t easy. Challenges include financial compensation (for those who build the endpoints), transaction management and event sequencing (between endpoints), circuit breakers (governing endpoint use), service discovery and protocol handling (of new endpoints) and service mediation (across systems of endpoints). These are hard problems that every software project must address when they leverage even a few endpoints.
Today, there are scant efforts to standardize how endpoints are managed. Instead, the API integration market is fragmenting, creating integration problems that will be an order of magnitude larger than programmers experience today.
This exploding endpoint problem increases the challenges for enterprises, which are struggling to keep up with line-of-business needs, supplier integration, market intelligence, system integration from M&A and the enablement of locked assets as they strive to become a digital native organization and live up to Andreesen’s software-eating-the-world vision.
The primary causes of this pain include:
- Most enterprise digital assets are still inaccessible, tied-up into large monolithic applications- and have not yet been converted into programmable endpoints.
- The types of assets to convert into endpoints have fragmented, driven by IoT device diversity, by every custom application having a unique API, and by the NoSQL movement enabling data schema variety.
- Integration vendors that promise solutions have failed to consider the diversity, dynamism and heterogeneity of the problem — and ultimately deliver integration offerings that are not resilient nor easily substitutable.
- Digital asset governance is statically-driven, not designed around real-time surveillance, ensuring change control becomes unpopular and costly. It is always multiple steps behind the fast-moving reality.
- The splitting of tasks into numerous micro-endpoints is leading the industry to the rise of cross-cloud integration: the need to write resilient transactions over a network that spans multiple endpoints. There are hundreds of programming frameworks that offer to simplify this problem, but all still require solutions that include custom logic, hidden edge cases and painful hours of debugging.
- Service desk and monitoring-driven SLAs are lowering their debugging value as IT learns that observing digital assets at rest cannot pinpoint root causes. A rethink is underway as Application Performance Management (APM) vendors race to disrupt themselves by replacing monitoring with full-stack, transaction-in-motion observability.
Increasing Demand Is Resulting in Disaggregation
Rarely is building large monolithic applications ever considered anymore, since they’re widely recognized as fragile, high-maintenance and prone to scalability issues. Instead, massive customer demand has led to the rise of companies like Google, Uber and Amazon that have instead architected disaggregated, componentized systems to scale to meet customer demand.
This is a continuation of a trend that we have seen over five decades. Massively disaggregated approaches including microservices, serverless and APIs are simply becoming the norm for software design. It is now common belief that application disaggregation is the only way to maintain rapid market agility with unlimited scalability. In fact, developers are embracing this disaggregation by being API-first, with enterprises doubling internal endpoint count every 18 months.
Integration Is Suddenly the Hottest Market
The application integration market is massive at nearly $25 billion in software and services sold each year. As disaggregated architectures drive the increase in programmable endpoints, integration will become a problem for every organization and every developer.
After Salesforce acquired Mulesoft for a whopping $6.5 billion earlier this year, Gartner’s VP and Fellow, Massimo Pezzini said, “Integration, the reportedly uncoolest thing in IT, ultimately turns out to be the hottest!”
Open Source Solutions Are Essential for Integrating Disaggregated Architectures
With the ever-growing diversity of integrating endpoints, open source software and vendors are the only viable way to ensure technology keeps pace with market needs.
Closed source, open core, and iPaaS vendors are not equipped to embrace this challenge. The protocols, data formats and APIs of endpoints change so frequently that only a community, collaboration, and shared approach available through open source can address the future needs of integrators.
Open source helps customers and community contributors to engage more deeply — and ultimately maximize integration solutions and overall value over disaggregation. Open source can be more reactive allowing rapid introduction of new technologies to connect more diverse digital assets to reach even more developers.
