Proposed by:
Requested amount:
18.69K DOT≈$92,535.3

#618 · Subsquid Migration to ArrowSquid & Indexing Upgrades

We are excited to introduce our proposal for the Substrate Subsquid Migration to ArrowSquid & Indexing Upgrades. This project targets enhancing the performance and functionality of Subsquid, a vital component providing access to on-chain data within the Substrate ecosystem. Through this proposal, we aim to improve the speed, efficiency, and indexing capabilities of Subsquid, ultimately benefiting developers and users across various chains such as Polkadot, Kusama, Moonbeam, and Acala.

Subsquid currently provides the community with access to on-chain data via GraphQL gateways, serving as data sources for APIs and on-chain data exploration tools. The main objective of this project is to migrate the existing Substrate Subsquid ecosystem to the ArrowSquid processor as well as enhance its indexing capabilities. This migration will enhance the speed and efficiency of all 50+ public squid endpoints.

Currently, all Giant Squid Main squids utilize the previous Substrate processor, Firesquid. The recent upgrade to the ArrowSquid processor introduces significant improvements, such as the ability to ingest unfinalized blocks directly from a network node, resulting in near real-time APIs with subsecond latency. However, this upgrade to the processor resulted in breaking changes and all squids that want to use it have to be adjusted to conform to the breaking changes.

Additionally, Substrate squids currently lack support for indexing Staking and Crowdloans pallets, which contain critical information for various chains in the Substrate ecosystem, including Polkadot, Kusama, Moonbeam, and Acala. The migration will also improve the performance of Polkascan as it depends directly on the data and performance from GiantSquid

The project aims to address these challenges by:

  • Migrating the existing giant-squid-main codebase to utilize the ArrowSquid processor, improving speed and efficiency across all supported chains.
  • Enhancing indexing capabilities by implementing indexing of calls and events from the Staking and Crowdloans pallets, enabling access to vital on-chain information.

The project will be executed in two milestones. The first one is focused on migrating giant-squid-main to ArrowSquid, which includes upgrading the dependencies, configuration, and endpoints adaptations, as well as testing and deployment. The second milestone will be focused on indexing Staking and Crowdloan pallets calls and events.

For a detailed description of our proposal, please visit: Polkadot Treasury - Subsquid Upgrades Proposal

Companies Background:

Subsquid is providing the community with access to on-chain data in the form of GraphQL gateways and it is doing it completely for free for the time being. Such endpoints are being used both as data sources for complex APIs and as an on-chain data exploration tool. The purpose of the proposal is to ask for funds to cover the cost of running the infrastructure, given the utility they provide to the community.

LimeChain is a software development company founded in 2017. Positioned to handle end-to-end product development with the capability to manage projects from conceptualization and design to the complete implementation of a certain product. LimeChain specializes in blockchain technology, with an emphasis on developing blockchain-related and infrastructure solutions. In the context of the Polkadot ecosystem, the expertise is focused on developing infrastructure implementations, including Gosemble, a framework for building Substrate compatible Runtimes in Go, Fruzhin, a Host implementation in Java, a framework for runtimes in AssemblyScript, a framework for runtimes in AssemblyScript. On top of that, Limechain has also implemented a Parachain Validation Conformance Testing suite.

Read more
StatusExecuted
98%Aye
Aye (231)
31.22M DOT
Nay (24)
442.44K DOT
Decision28 / 28d
Confirmation4 / 4d
0.0%0.71%
0.20%Support Threshold
0Support Threshold
Support(0.57%)
≈ 7.93M DOT
Issuance
≈ 1.40B DOT
Vote
HristoGergovSep 26, 2024

Hello all,

With this post, we want to share that we have successfully completed the Substrate Subsquid Migration to ArrowSquid & Indexing Upgrades

This update achieved two major goals: migrating the existing giant-squid-main codebase to the latest ArrowSquid process, and enhancing the indexing capabilities of Substrate squids by incorporating calls and events from the Staking, Parachain Staking, and Crowdloans pallets.

We’ve documented these achievements, along with our future roadmap, in a detailed report that outlines our deliverables and upcoming plans. You can view the report here:

REPORT: Substrate Subsquid Migration to ArrowSquid & Indexing Upgrades

Looking ahead, we are now reaching some ideas about additional features aimed at increasing adoption of the Giant Squid. Among the ideas being explored are extensions like the Conviction Voting pallet, Nomination Pools pallet, and XCM pallet—promising candidates for the project's future growth.

Kind regards,

LimeChain team

jimmytudeskiApr 17, 2024

[Deleted]

jimmytudeskiApr 17, 2024

The detailed plan to implement indexing for Staking and Crowdloans pallets will provide invaluable data that developers and stakeholders can utilize to optimize their strategies and engagement within the ecosystem. This clearly indicates LimeChain’s commitment to delivering tangible benefits through meticulous planning and thoughtful integration of cutting-edge technology.

Moreover, continuing to provide these services as a public good, free of charge, aligns perfectly with the Polkadot ethos of openness and community support. Initiatives like these empower users and enhance transparency, which is fundamental for the growth and stability of any decentralized ecosystem.

I'm excited about the potential of having near real-time API access through the new processor, drastically reducing latency and improving the overall user experience. This kind of forward-thinking and investment in infrastructure is precisely what we need to maintain Polkadot’s competitive edge.

Supporting this proposal is not just a vote for a technical upgrade—it's a vote for a stronger, more connected future for Polkadot and its related chains. Thank you, LimeChain, for your continued effort to push the boundaries of what our technology can achieve.

Best of luck. I look forward to the successful implementation of this project and to the many benefits it will undoubtedly bring.

Apr 12, 2024

Hello from Polkadotters. Subsquid is very important for indexing in the Polkadot ecosystem. The upgrade to ArrowSqwuid and introducing of new features such as staking or crowdloans pallets are beneficial for the ecosystem. We are in favor of this referendum. AYE.

HristoGergovApr 10, 2024
Helikon

Hello @chrislime,

Thanks for the proposal. I need to clarify a few things before I can vote.

Subsquid seems to have received a total of ~$796K for 3 quarters of 2023, presumably for both the maintenance of the codebase and providing a publicly available data access infrastructure. These fundings were as below.

  1. $224K for Q1 2023
  2. $230K for Q2 2023
  3. $343K Q3 2023

There doesn't seem to be a funding request for Q4 2023, and the next proposal for funding is this one at $176K. Can you please explain why there is no request for Q4 2023, and how this proposal relates to it? Are the public data access endpoints still available for all the chains listed in the Q1-Q3 2023 proposals? If so, why is the amount so different than the previous proposals?

This proposal seems to be adding Staking and Crowdloan pallets support for Polkadot, Kusama, Moonbeam, Moonriver and Acala. This is good. But just to clarify, are the existing deployments (I'm assuming all other chains are still publicly available) going to be upgraded to ArrowSquid processor as well?

Could you please also share the public access endpoints to all of the available Substrate chains? Or even better, could you direct me to the relevant documentation, so that I can test the endpoints myself?

Looking forward to your response. I abstain in the mean time.

Regards, kukabi | Helikon

Hello @🏔 HELIKON 🏔 

Thanks for the feedback and the raised questions on our proposal. We are sending you our feedback below. If you need any further clarifications or details on those, we will be happy to provide them.

Are the public data access endpoints still available for all the chains listed in the Q1-Q3 2023 proposals

> FireSquid APIs have been switched off except for Astar in preparation for the ArrowSquid migration. After the migration all public API will be back online. You can find Astar’s API here - https://squid.subsquid.io/gs-explorer-astar/graphql

Can you please explain why there is no request for Q4 2023, and how this proposal relates to it. > Subsquid will submit a retroactive proposal for Q4 2023 and it will be the final one.

If so, why is the amount so different from the previous proposals? > This proposal has development costs and the running of giant squid costs covered. In the previous proposals, it included the costs of running the Archives (the biggest cost by far) which are the current data source. After migrating to ArrowSquid these archives will no longer be used and be deprecated which will save costs in the future.

But just to clarify, are the existing deployments (I’m assuming all other chains are still publicly available) going to be upgraded to ArrowSquid processor as well? > Yes, all supported chains will be upgraded to ArrowSquid. After that Polkadot, Kusama, Moonbeam and Acala APIs will be further upgraded to have Staking and Crowdloan data.

Could you please also share the public access endpoints to all of the available Substrate chains? Or even better, could you direct me to the relevant documentation, so that I can test the endpoints myself? > You can find all the supported chains listed here: https://polkadot.polkassembly.io/referenda/372. Please, have in mind that those are down at the moment, one of the outcomes of the proposed migrations will be that the endpoints will be brought back online.

Kind Regards, LimeChain team

HelikonApr 8, 2024

Hello @chrislime,

Thanks for the proposal. I need to clarify a few things before I can vote.

Subsquid seems to have received a total of ~$796K for 3 quarters of 2023, presumably for both the maintenance of the codebase and providing a publicly available data access infrastructure. These fundings were as below.

  1. $224K for Q1 2023
  2. $230K for Q2 2023
  3. $343K Q3 2023

There doesn't seem to be a funding request for Q4 2023, and the next proposal for funding is this one at $176K. Can you please explain why there is no request for Q4 2023, and how this proposal relates to it? Are the public data access endpoints still available for all the chains listed in the Q1-Q3 2023 proposals? If so, why is the amount so different than the previous proposals?

This proposal seems to be adding Staking and Crowdloan pallets support for Polkadot, Kusama, Moonbeam, Moonriver and Acala. This is good. But just to clarify, are the existing deployments (I'm assuming all other chains are still publicly available) going to be upgraded to ArrowSquid processor as well?

Could you please also share the public access endpoints to all of the available Substrate chains? Or even better, could you direct me to the relevant documentation, so that I can test the endpoints myself?

Looking forward to your response. I abstain in the mean time.

Regards, kukabi | Helikon

saxembergMar 31, 2024

Thanks for the proposed improvements in data amount like staking and crowdloan pallets and improved performance under Arrowsquid and infrastructure support. We look forward to see the quality of the squids delivered and if such efforts are positive we hope we can see more detailed data being integrated in the future.

We have voted AYE in the meantime, and we will change it only if there is any significant development that will make us change our mind.

HristoGergovMar 28, 2024
none

Why 18,694.00 DOT?

Hello @none  ,

For detailed effort estimates and cost breakdowns for each task, please refer to sections 4.3, 4.4, and 4.5 in the proposal document here: https://docs.google.com/document/d/148ZnlnG7nrDRVhS-VQQ-Ld21u78qaXWqcVsV366EPn4/edit#heading=h.8tzungqebig6

If you need any further details, we will be happy to provide those.

Kind Regards, Hristo

noneMar 28, 2024

Why 18,694.00 DOT?

G
Georgi_PMar 27, 2024

Big AYE for me 👍

dzlzvMar 19, 2024

Dima from Subsquid here. We are very happy to see Giant Squid getting decentralized and community owned thanks to LimeChain. We've been running Giant Squid APIs for almost two years for now, but it became evident that it has to be run as a public good, and we should have no more say then any other in the Polkadot data community.

Powered by Subsocial