A year has passed since we have published our middleware thesis which was focused heavily on Pocket Network. In the meantime, the middleware space has evolved, growing exponentially. In this piece we explore middleware and infrastructure again, from a bottoms-up perspective. Attempting to connect the dots in this rapidly evolving space.

Worth noting, this thesis is heavy on Zee Prime bags. We have done this deep dive while looking inward at our own portfolio, projecting individual theses forward and combining them into one middleware meta-thesis.

We apologize for including buzzwords such as composability, Web 3, etc.. It is hard to paint a picture without relying on some of the memes. Memes play a role.

The Middleware Thesis Vol. 2

As development of higher level applications on blockchains continues to grow beyond early DeFi primitives, the need for robust infrastructure and middleware also expands. We at Zee Prime have been vocally bullish on this “category” for years now. Our first middleware piece – Infrastructure Lego The Middleware Thesis – highlighted the importance of robust data relaying in the development of sustainably decentralized platforms and the economics associated with it.

This time, we seek to broaden our thinking to the middleware ecosystem that has emerged, particularly in the context of enabling decentralized applications. While something like Dfinity is definitely the turn key, end to end solution for all your troubles, until that day of mass adoption, we will have to improvise and build this independently piece by piece.

The narrative is a tough sell these days

Through the blossoming of app complexity (yay, real apps!) we have seen demands for supporting infrastructure explode. Beginning in the early days of DeFi, a stream of data networks, indexers, access controls and other middleware tools have begun to pop up – all critical pieces of glue for the next generation of apps.

The landscape can be difficult to map. Middleware is a nebulous concept much like Web 3. The lines are often blurred. Fundamentally what we mean when we say middleware is any project supporting other higher level applications. Last time we discussed the topic, we noted it would emerge as connectedness from the middle, however mapping the web of interactions to a 2D plane is messy.

Is this a map of Web 3 middleware?

In this refreshed middleware thesis, we will describe additional categories of middleware we previously had not covered, while also articulating their use cases for builders and the rationale for using them. This is by no means an exhaustive list.

Storage/Data

One of the first critical elements in a decentralized application stack is also the most foundational to basic computing: storage. Web 3’s Cambrian explosion of activity and complexity also requires storage solutions beyond a simple record of account states on a base layer. Decentralized applications are increasingly seeking technology stacks that lack centralized points of failure or the censorship associated with Web 2 solutions.

Every application will need, in some form, the services offered by Web 3 middleware. A challenge to this, however, is that dev-ops is hard and not every buildooor has the expertise to implement Web 3 middleware in their project. As a result, we need abstraction layers that enable much easier assembly of these infrastructure legos into new projects.

This mirrors the challenges facing crypto adoption more broadly. Wallets, seed phrases, and gas for transactions are all inherently very unfriendly to the user, and until greater levels of abstraction exist, adoption will face headwinds. We must assume that the average user will not deal with the complexity we see today.

Storage networks such as Arweave and Filecoin have been around for some time now, offering distributed matching systems for excess storage and storage demand – this likely needs no introduction. Different solutions offer varying degrees of permanence and censorship resistance. They are foundational blocks to any decentralized technology stack. Storage can be categorized into two types of protocols, foundational storage layers, and aggregators that serve as scaling solutions and facilitate wider adoption.

Banyan is a piece of the puzzle. We view it as an important aggregation layer for storage networks. Focused on brokering storage and improving the economic incentives of existing bridges into storage protocols, Banyan ensures applications can utilize Web 3 storage solutions in a network agnostic manner with guaranteed provenance.

Banyan is also a potential bridge for Web 2 applications that would like to begin integrating novel web3 services such as decentralized storage solutions. Currently, implementation of these storage solutions is highly complex and Banyan’s abstraction layer and marketplace enhances the accessibility of these legos.

Similarly, Spheron also aims to serve as a sort of abstraction layer for middleware solutions more broadly. The protocol aims to be a one-stop-shop for deploying and automating Web 3 projects. It features an “app store” like interface where non Web 3 natives can easily select from decentralized infrastructure offerings – much like Digital Ocean of Web 2.

Data Models and Validity

Blockchains are state machines that are constantly producing data in the form of state transitions whilst executing computation. As this continues through time, the amount of accounts, states, and smart contracts grows rapidly. This can create a variety of problems, from indexation to initial node synchronization and back-up durability. These affect scalability and security of the underlying state machine.

KYVE takes a more granular approach to specific data validity issues, leveraging the Arweave network to provide critical support to apps and protocols. KYVE is a decentralized, data lake (raw unstructured data in one pool) solution for storing, verifying, and retrieving data streams.

KYVE’s initial product market fit importantly aids the node synchronization issue. By providing easy to retrieve, verified archival state data, initial node synchronization times can be greatly reduced, ensuring new validators can be added and network security can be preserved. In an alternative scenario, with extremely long (and growing) initial node synchronization times, network security could be risked if validator numbers diminish and new ones experience difficulty onboarding.

While we have covered where and how data might be stored, we also need to think about models and interpretation of this data. For the applications built on top of these state machines, data production from their activity will require flexibility of storage and computation beyond account balances.

Ceramic, at its most basic level, is a decentralized data model network. Why is this important infrastructure to decentralized applications? To put it another way, a “Layer 1” (L1) stores the state of account balances (accounting data). Where KYVE seeks to provide a data validity solution to L1 state transitions, Ceramic seeks to store the state and models of applications’ data beyond base layer account information. Users of the network are able to create collections of IPFS data (streams) allowing for static data (such as that on Filecoin or arweave) to become higher-order mutable content.

Further to this, Ceramic then layers on a marketplace for open source data models allowing for composability of these data models. You might often see Ceramic noting that this brings to the data the same level of composability that ERC20s brought to DeFi, proposing a sort of a standard on data and making it reusable across applications. Money legos meets data legos. This is almost a return to open APIs living on top of social networks like Facebook.

Kwil takes an SQL compatible approach to enabling web 3 data models. The biggest advantage to a model like this, is the vast number of developers knowledgeable about SQL. Kwil uses a network of nodes to maintain the relational databases. These databases (called data moats) are maintained by a subnetwork of the nodes and stays up to date by scanning new writes, and querying other nodes within the same moat. Most interestingly, the nodes can run an Advanced Request Gateway that effectively enables logic execution for database interactions.

Indexers

With the proliferation of data being produced by the apps and networks, interpretation layers are required. As with the early days of the web, people had to remember addresses by hand, maintaining books of IP addresses. DNS and search engines helped provide a human readable indexed layer to this.

As the internet progressed, centralization of the indexed data grew with economies of scale and facilitated more user-friendly querying of this data. Similarly, across L1 blockchains, and storage networks, indexation is vastly important. Due to the nature of the distributed systems, the data can be siloed across various locations and difficult to retrieve. Indexation layers help expedite the query process and create standardization procedures.

A case study in indexation is Zee Prime portfolio company Subsquid. Subsquid takes a multilayered approach to indexation of on-chain data in a decentralized manner. Their ultimate goal is to support Web 3’s next generation of APIs. Supporting both Substrate and EVM ecosystems, the protocol defines types, schema, and definitions of on-chain data and subsequently enhances retrievability of the newly indexed data by switching it to their API based call solution instead of RPCs.

The layering is composed of two types of nodes: Squids categorize and support subsequent api querying of the data, while Archives continuously ingest the raw data from the underlying state machines and save it to the database.

Similarly, SolanaFM is also an indexer, and block explorer that serves Solana’s ecosystem by processing raw blockchain data into a queryable format much like the other ecosystem indexers. The immediate use case SolanaFM provides is using their APIs to power the DeFi applications on Solana. These solutions may sound familiar if you have ever come across Graph and Subquery. Both solutions are targeting a variety of end markets.

Glitter tackles another problem altogether: decentralized storage. Think of it as indexing-as-a-service for developers. As Web 2 applications look to move into Web 3, they are bringing a colossal amount of data to the new world. While the populating data helps grow Web 3, it also puts both the developers and the community in front of the daunting task of storing and indexing that data.

Glitter created a win-win solution for developers and the community, by providing hassle-free indexing service in exchange for crowdsourced data. The model has already been proven effective in collaborations with several social applications that store data on Filecoin.

Access Controls

One of the most important and historically underrepresented missing pieces of Web 3 app infrastructure are access controls. Who can see what is on the internet? It’s an important philosophical question, and of growing importance in the face of security concerns across state/corporate/individual sovereignty considerations. The semantic nature of public blockchain/Web 3 technologies enables us to better differentiate which users should be able to access what and how. Despite the inherent openness of these systems, an access control framework would allow for encryption/decryption based on a specified provisioning framework.

Lit Protocol aims to solve this question via its usage of threshold cryptography. At its core, the network can provide access to resources and content across the web based on some public credential (such as an NFT inside of a wallet). The protocol operates a network of nodes which validate attestations and approve handshakes. This network can verify both an attestation that is provided, and that such an attestation meets the previously set access control condition all within secure computational enclaves. Once validated, requisite content can be utilized. In a way, some view Lit protocol as the read solution to Ceramic’s write.

Guild.xyz is also attempting to address the problem of access controls from a different angle. Focused initially on creating token permissioned discord environments, the project has expanded into focusing on a multichain access control portal based on similar principles.

Integration Platforms

To further integration and fitting of blocks together that we imagine in our 3D bridge world, Polywrap (formerly known as Web3api) is pushing integrations of Web 3 protocols to much higher degrees of efficiency. While Web 3 protocols are open, and technically composable, actually achieving this composability in practice is much more difficult than it is in Web 2. This is because each protocols requires specific business logic to be run within the application, often times bundled as an SDK in a specific language

Integrating all of these different SDKs is extremely inefficient because of a lack of standardization. Additionally they are language specific, meaning that protocol developers often publish duplicate SDKs in more than one language, creating a maintainability nightmare.

Polywrap’s solution eases this burden by leveraging standardized schemas & WebAssembly. Instead of pre-bundling the various protocol’s SDKs into your application, a Polywrap integration can simply make invocations against an easy to read schema (think REST API). This will cause the wrapper to be downloaded, and executed within the application at runtime. In plain terms, this means any application equipped with Polywrap can gain access to any Web 3 protocol.

User experience remains relatively high friction with web 3 applications. As we highlighted earlier, entering gas presents user friction. Through the integration of Biconomy’s APIs, an application can enhance this user experience. Biconomy’s platform provides a palette of tools to enable gasless transactions, enjoy quicker confirmation times, payment in ERC20s, and instantaneous cross chain transacting.

Gasless transactions are made possible via usage of meta transactions (via ERC2771) and some clever forwarding designs. The cross chain feature is enabled by pools of liquidity on the supported layers/chains. Off-chain servers (executor nodes) are then used to monitor the pools for incoming transactions and subsequently “release” the other side of the transaction.

These kinds of tools are critical for smoothing out the user experience for the next billion adopters of crypto. Our goal should be to constantly strive for a more seamless flow of interactions across web 3 systems.

While not explicitly fitting into one category neatly, Sepana is building the search engine for web 3. Whether DeFi, Social, DAO activity, or NFTs, Sepana’s solution will host a full text search engine that will enable users to browse the entirety of web 3. By leveraging the work of indexers such as those described above, as well as its own indexation of web 3 applications and data, the protocol will serve as a gateway to the broader ecosystem. Moreover, Sepana’s transparent and open source algorithms can be used to drive other applications such as social media feeds based on social graphs stored in a database solution like Ceramic or Kwil.

The long term vision here has very interesting ramifications. Imagine a world where you could tune your social media experience (i.e. your feed) for specific emotions or outcomes (such as reducing polarization as Sam Harris has often discussed or happiness more broadly). This could all be possible with open sourced algorithms that platforms can adjust accordingly or put tuning in the hands of their users.

How Does It All Fit Together?

Most modern technology companies and applications can be distilled to some form of the following business model: Data production/ingestion, models on top of the aforementioned data, and control/distribution over the data/models. Smooth UX and dopamine farming of modern network applications require these elementary processes.

As a consequence of such workflows, we would expect middleware solutions to propagate from the middle of these technological needs and support them in a Web 3 context. With the projects/types of middleware described earlier, we can see this is clearly the case. Every piece fits to support the generalized workflow we described:

While these categories above work as a broader generalization, in reality many stretch across multiple buckets. In 2022 it is hard to precisely define the categories because of these overlapps.

To make this a bit more practical, let’s take a commonly used example, such as a social media network and expand such a mental model to the broader Web 3 middleware stack.

Our hypothetical social media network will be aptly named twatter. In this practical application we can watch the flow of the platform’s product – a social media experience – flow through a middleware stack leveraging components pictured above. Note that we do not think that the social Web 3 is “Twitter-but-decentralized”. We imagine Social Web 3 to be more of an emergent phenomenon, potentially even referencing Web 2 apps for attestation through something like Sismo (if they decide to open their APIs).

Starting at its rawest form, all the data from the platform (usernames, profile pictures, historical activity, social graph, etc.) can be stored and indexed on one of the storage networks in IPFS format. The data models that give this structure and meaning are stored on Ceramic or Kwil, the Twatter account on the database solution would have models for each piece of the platform previously mentioned.

If for instance the platform requires you to have a free to mint NFT to access the platform (spam reduction mechanism or something), then users would connect their wallet to the platform and an access control protocol would conduct the attestation validation and handshake before displaying the platform to the end user. Integration platforms can be included at the app level to enable other web 3 services natively, while Sepana’s algorithms can be used to design feeds based on the social graphs.

Flows could look like this

Most interestingly, while writing this article we stumbled upon Orbis Social, which has effectively built the social network described above, using almost the same stack we described. Next generation apps are under development and in coming months we hope to see more idiosyncratic use cases.

An important distinction to make here is that the farther to the right you go on the diagram, the more chain agnostic it is. Pieces of this structure can be interchangeable with other competing products, however many of these business models lend themselves to the network effects of monopolies. Conversely to Web 2 monopolies, these platforms end up redistributing the compounding value of such pseudo-standardization back to the users of the platforms.

The Next Gen of App Buildooors & Web 3 Middleware

Whilst the tools of Web 3 continue to emerge, we must continue to ask ourselves why? Are they really offering any benefits to Web 2 counterparts?

Web 3 middleware should be built on the same foundational principles as earlier crypto forefathers. Teams should be choosing Web 3 middleware on its merits alone, because it enables their application to achieve its goals. Whether it is robustness from a security, durability, or censorship resistance perspective, the merits of web3 middleware should stand by themselves. There are probably many new features that will be unlocked as a result of some of these inherent characteristics that we can not even imagine.

These infrastructure legos can enable deeper levels of integration typically associated with semantic web – Tim Berners-Lee’s vision of an open and composable internet – and offer cheaper solutions to hosting and computation than Web 2 counterparts. As Dennis Nazarov pointed out, a complex computational system can be represented by a healthy garden requiring modularity and specialization, and in the Web 1/2 world, users gave up managing state for the ability to enable connectedness. Web 2 giants keep valuable state information private, because of the compounding value that can be captured as a result of having it.

Public state machines allow this model to be reversed, state is maintained in an open manner and insertion of a token based economic model can enhance alignment to better outcomes on both sides of the table. This is the nature of reflexive assets.

The Zee Prime Thesis

In a lot of ways middleware is the B2B segment of crypto. As a result, good middleware solutions tend to be both highly technical and not intuitive to typical end users (and this is okay, not the target audience). Instead of constantly looking at the new DeFi protocol, NFT project, or GameFi studio, we also view being on the factory floor helping the sausage get made as critically important to continued development of novel applications.

In summary, these infrastructure legos (and future ones too) will do the following:

• Increase censorship resistance

• Promote positive sum economic games

• Improve efficiency

• Enable novel business models

One additional potential impact of such interchangeable infrastructure blocks, and abstraction layers is that applications will:

• Be increasingly farther from the base layer; and,
• Be more chain-agnostic as a result

This is not a refutation of the fat protocol thesis (value accruing to the base layer hypothesis), but more of a flagging of ramifications of such continued advancements. In principle this can be viewed as lowering switching costs. The earliest on-chain applications (primarily DeFi) feature extremely high proximity to the base chain (i.e. financial products built on financial accounts).

More complex, non-financial applications will have a looser affiliation to such chains, lowering the cost of switching (free to mint NFT access control is extremely easy to port to a new chain and wallet). Applications are already drawing users across chains.

We are firm believers that adding the transmission of value to the transmission of information is a meaningful step change, but to achieve that potential, and the kinds of applications and user experiences that would facilitate it, a great deal of infrastructure legos is required in between.

Value capture in this “segment” is one of the most debated topics when discussing middleware investments. In one sense, a truly critical piece of middleware looks alot like a public good, although one could argue this will also apply to certain successful future apps as well (Twitter wanting to be a public good).

As a result people might expect margins/fees/revenue to converge towards the lowest possible bound. While we believe that this will be the case to some extent, it is more reasonable to assume a convergence towards a publicly acceptable toll rate.

While a seemingly unattractive quality at first glance, in the context of an asset/resource light business amongst the world’s first truly global technological revolution, this scale easily reaches into the billions of dollars of value capture for these legos. As these pieces of middleware execute a specific function for applications, their TAM is agnostic of the chains below them, or the apps above them at any given time. It is rather the function they are providing more broadly.

While middleware and DeFi do share the self-referential nature of token based economic models in common, they ultimately differ in their ability to return value. Middleware projects often benefit from clear supply and demand drivers for their tokens (such as network nodes), for the delivery of the service they provide. Conversely, most DeFi projects have less clear demand drivers for tokens and regulatory concerns over distributing cash flows make the picture even murkier.

It is for these reasons that we continue to search for new middleware solutions that will enable more compelling next generation applications for the continued adoption of crypto. We believe in a new generation of apps that will unbundle finance and online commerce/activity at scale. The a16z-esque way to put this: we do not want skeuomorphic apps but native apps.

If you are a founder, builder or have an idea for a critical piece of infrastructure that needs to be built to aid this, you know where to find us.

Glossary

Web 3 – referring to the Beyond the Buzzword Article

Apps – we are not using dApps (decentralized apps) because we think that “how d is your app?” is a spectrum. It’s possible that apps will be powered by decentralized and also centralized solutions. Does that mean they are not dapps?

WASM – short for WebAssembly can be thought of as a stack-based virtual machine that executes in binary format, but can be compiled in various languages such as rust, C, Python etc.

SDK – Stands for Software Development Kit. It is the set of software building tools for a given platform, often in the form of libraries, frameworks, building blocks or debugging tools.

IPFS – The InterPlanetary Filing System. It is a distributed system for storing and accessing information using content addressing.

Skeuomorphic – Official definition as follows: a skeuomorphic design includes features which make a new thing look older or more familiar. In practicality it means new ideas that are easy to digest. Skeuomorphic apps as an opposite to native apps.

API – Application Programming Interface. This is a piece of software that can be used to leverage other software.

RPC – Remote Procedure Call. In the context of public blockchains, this is basically when one part of the computer calls a procedure (think an action) in another part of the network (i.e. another computer/application).