Zyte Extract Summit 2026 (Austin): A Recap
Our impressions from the first chapter of Zyte’s annual web scraping conference.
Adam Dubois
- Published:
The first chapter of Zyte’s Extract Summit has concluded, but the memories (or, for most – video recordings) will remain. As always, it was a fascinating event to follow the latest web scraping developments. It is also a lot to digest in one sitting.
Don’t you worry, though: we’ve once again made a recap covering the presentations in digestible bites. They’re all very much worth viewing, but we believe it’s helpful to get familiar with their contents in advance.
The second chapter of Extract Summit takes place in Dublin on November 10-11.
You'll find our coverage of earlier summits and other major industry events here.
Organizational Matters
Zyte prefers to run conferences live. Same as last year, this Extract Summit has two chapters: Austin in October and Dublin in November. Organizing two events is a huge undertaking, especially considering their scope. Austin alone featured 10 presentations – plus a legal panel, each taking half an hour on average.
Leading up to the Austin event, Zyte obviously tried to steer attendees toward the meatspace. The sales of tickets, which cost up to several hundred dollars, were promoted front and center, with online attendance deep in the background. The live event was also longer – it had one more day dedicated to workshops.
Due to time zone differences, we weren’t able to experience the conference live, but it likely included the usual setup: a livestream with a box for asking questions. The Discord channel saw no activity this year – it was apparently abandoned by the organizers.
Main Themes
It’s AI. Surprised? Thought so.
In all seriousness, artificial intelligence really did dominate the discourse. The use of AI for data parsing carried over from last year and remains relevant. In addition, we’re starting to see more and more agentic uses. In particular, AI has taken up the role of a mechanic: Konstantin used AI to fix parsers on the go, while Joao employs a whole team of agents to do more thorough investigations. While still more experimental than robust, LLMs are also starting to tackle website blocking challenges.
At the same time, the industry is getting more paranoid about logging and observability. For proxy providers like Massive, this helps to keep bad actors at bay. For AI startups buying data, provenance ensures better model training. For lawyers, it helps create a defensible data strategy in a climate where not much is certain but things are certainly moving fast.
The Talks
Extract Summit’s Austin chapter included 10 presentations and one legal panel. Use the links to navigate:
- What AI Actually Changes in Web Scraping
- The Antiban Arms Race and the Open Web
- How Compliant Residential Proxy Node Sourcing Works
- Who’s Actually Buying Web Data Now
- TCP IP Fingerprinting – How It Works and How We Beat It
- The Naive AI Trap
- Legal Panel
- How One Custom-Made Agent Steers Our Other Engineering Agents
- The Spider That Rewrites Itself
- The Multi-Armed Bandit and Web Data
- Beyond the Scrape: Ops, Costs and Building Trust
Talk 1: What AI Actually Changes in Web Scraping
The main event kicked off with a presentation from none other than Shane Evans, Zyte’s CEO. Well, this is how most of such conferences work, but intros usually last around five minutes. Shane went ahead and delivered a full half-hour talk, with slides and all.
Shane began by demonstrating a parser he built with the help of AI. Ten times faster and less buggy than existing software, it demonstrated how good LLMs have become. However, Shane cautioned against blindly believing that the same holds true for everything AI touches. He gave examples from three areas of web scraping: development, data reliability, and web access. All have transformed: but while the development time of feeds has halved, the situation with, say, data validation remains hairier.
In the end, though, Shane made it clear that AI is helping, and that the help will only get more pronounced. One of his more interesting pieces of advice was to no longer consider historical data as a reliable estimate – estimate as you go. All in all, it was a brilliant way to start the day.
Talk 2: The Antiban Arms Race and the Open Web
According to Akshay Pilar, the VP of engineering at Zyte, the invention of barbed wire upended the lifestyle of cowboys as we know them. Outside of spaghetti westerns, it’s not like we know cowboys that much. But we too are going through disruptive changes, with legal and technical barriers springing up and new modes of transportation emerging.
Akshay recounted how much more sophisticated the process of web scraping has become in a decade: the same latency budget now gives us 10x larger decision trees to navigate before opening a page, at 10x more resource use. This doesn’t really work in the agentic age, where AI opens hundreds of pages to satisfy one query.
Ultimately, the law put the use of barbed wire in check (though it did take ten years to do so!), and we’re seeing measures like Web Bot Auth emerge. But they may not always benefit us or the open web; therefore, the most important thing right now is to find a place at the decision-making table. Or so Akshay thinks. His presentation was persuasive and surprisingly rich in detail, going as far as to show how individual JA4 handshakes differ. We recommend watching it.
Talk 3: How Compliant Residential Proxy Node Sourcing Works
Jason Grad from Massive opened his talk with a bold proposition: “hey, other proxy providers, please steal our ideas!” In any other circumstances, we’d consider his request eccentric at the very least. But when Massive’s ideas involve ethical residential IP sourcing, and when just months ago the FBI shut down a major proxy server provider from Israel, we can’t help but eagerly nod.
Jason’s talk had three structural parts. He briefly listed the main residential proxy sourcing methods: botnets, apps, and the same apps with consent. He then showed some examples of dark patterns. Finally, Jason taught viewers how proper consent and monitoring should be done. To prove that his company actually had skin in the game, the presenter gave a concerning data point: 69% of Massive’s potential customers got turned away.
The takeaway is that by choosing to sacrifice short-term revenue, proxy providers ensure their long-term growth. It would really help to have some kind of standards, but that’d require someone to rise to the occasion. Either way, ethics and transparency in the proxy server space are becoming less negotiable every year.
Talk 4: Who's Actually Buying Web Data Now
Andrew Harris, former engineering lead at ZoomInfo and now his own boss, shone a light on how the AI age has changed the requirements for buying web data. The gist of his talk was that data outcomes now trump volume or infrastructure robustness, and that a missing row is no longer just a rounding error you can shrug off.
The main reason is that AI companies, which have become main data buyers, care deeply about the output their ingested data produces. Andrew distilled four main criteria in order of importance: coverage, freshness, provenance, and reproducibility. AI companies need to know what’s missing, treat freshness as a percentile distribution, care about where the records come from, and are much more sensitive to parser changes than other industries.
If you’re supplying context for models or are thinking of doing so, Andrew’s talk can help you better understand where the market stands. For us who don’t really sell data, its findings went very much against conventional wisdom. And that’s fascinating.
Talk 5: TCP IP Fingerprinting – How It Works and How We Beat It
A nerd’s delight – Byteful’s Joseph Dye dug into the guts of TCP/IP fingerprinting, going as far as to show how the fingerprints differ at the bit level. It’s a highly specialized technical talk and so perhaps all the more interesting for it.
If you’re unfamiliar with the concept, the TCP/IP fingerprint is used to determine the client’s operating system. This is achieved through certain IP and TCP headers which operating systems handle slightly differently. Joseph showed exactly how and using which tools. He then demonstrated how to beat the fingerprinting by implementing a service in the user-space.
The presenter even went the extra mile by sharing other possible fingerprinting methods via TCP/IP. An overachiever, indeed.
Talk 6: The Naive AI Trap
Reviewing the slides for the recap of this talk put us in doubt. It seemed like a solid presentation, made by a highly credentialed and experienced speaker, about a relevant topic, in a niche where experiences can rather easily be generalized.
And somehow, it wasn’t a good presentation, from a rather weak delivery to a lackluster Q&A. Even the presenter’s shirt conspired against him, messing with the camera. The timing was also unfortunate; Pravin Khandke covered the topic of validating financial data just after the first decision models appeared, so it didn’t include Jev. In the end, the conference’s host aptly summarized: “Hopefully, the people building high-level financial stuff took away some stuff.”
We are by no means trying to be rude. And if you want to learn how to stop AI from failing silently and build (relatively) bulletproof extraction agents, you may take away valuable insights. Just skim the slides first.
Legal Panel
The legal panel brought together Zyte’s Chief Legal Counsel Sally Anne Hinfey and Mintz partner Scott Lashway to give EU and US perspectives on current issues in the legal world. Zyte’s senior counsel Victoria Vlahoyiannis moderated.
It turns out that the two legal systems differ in significant ways. The EU relies on legislation much more, at the same time lacking concepts like fair or transformative use. It also protects facts in databases and gives way more power to robots.txt than Sally would like it to have. The US, on the other hand, relies heavily on case law – the problem here is that lawyers are rarely technologists, and that the laws they’re applying may not always be in tune with reality.
When it comes to preventing data collection, documents like terms of service remain relevant; however, both panelists agreed that technical boundaries like paywalls or login walls have pulled ahead, andthe law is barely catching up. They also touched upon new access methods like the pay-per-crawl initiative without dwelling on them too much.
Finally, the panelists discussed relevant cases like the Suno lawsuit and Bartz vs Anthropic. The main conclusion here was that companies should really document their data acquisition diligently and put a defensible data strategy in place.
Panel discussions are always something we recommend – no exception here. Watch it, learn it, and scrape safely.
Talk 7: How One Custom-Made Agent Steers Our Other Engineering Agents
Joao Xzibit (respectfully nicknamed by us) Drumond from Crawly demonstrated how he manages an orchestrator agent that manages a team of specialist agents that keep his crawlers running. In particular, the presentation focused on fixing broken scrapers with minimal human oversight.
Joao’s agents run a pipeline in a Kanban-like board that functions as a state machine. The Product Owner and Tech Lead write the spec, the Engineer builds the code, while the QA with the Verifier make sure it doesn’t break. The manager agent called sirineo oversees the whole process, stepping in where necessary. The human only confirms the code merge.
Joao demonstrated a real example of how this process works. It took around two and a half hours to complete the fix, and usually, no intervention is required. Joao plans to eventually (actually, within weeks) remove the human element altogether. The talk had beautiful Claude-made slides and overall made for an entertaining watch.
Talk 8: The Spider That Rewrites Itself
Zyte’s Head of R&D Konstantin Lopukhin addressed the same topic of self-healing but from a different angle: what if a crawler could heal on the go, taking minutes instead of hours or even weeks like in the olden days of the humans? You know, similarly to those action movies where the hero wriggles out on top of a fast-moving car to fix it.
To verify his hypothesis, Konstantin built a tool called scrapy-phoenix that plugs into Claude. It stepped in only after the crawler’s parser broke and tried to get it back up. The tool required some accommodations, such as limiting its scope and structuring the code to notice layout mismatches. It also had other limitations.
However, an agent did manage to fix the parser successfully, often pausing the crawler just for several minutes and costing less than a dollar. As a bonus, it ensured great reproducibility. We could say that Konstantin’s experiment succeeded, and he considered tackling unblocking challenges next.
Talk 9: The Multi-Armed Bandit and Web Data
Logan Harless, the CTO and Co-Founder of String, spoke about choices – in particular, the best way to find an optimal configuration for unblocking websites. It was a little theoretical: Logan comes from finance, and those guys love inventing algorithms, especially when they involve probabilities.
Why complicate your working life like this? Well, web scraping has gotten pretty darn complex: between fetching methods, proxies, headers, request volume, and other variables, you can get up to 90 billion (with a B!) unique configurations to choose from. The speaker demonstrated different scenarios, applying various approaches to find the optimum. In brief, you’ll likely want to try out Thompson + forgetting, but don’t forget to prune your options first.
If you have half an hour to spare, you may appreciate choosing this arm of the bandit to spend it on.
Talk 10: Beyond the Scrape: Ops, Costs and Building Trust
The last talk had Giancarlo Morillo from LexisNexis on the stage. Giancarlo decided to go against the grain and represent the buyer’s side of the equation, showing the concerns they had to address on their way to data.
This basically boiled down to five questions: Build or outsource? Legal approvals? Which types of scrape? Cost at scale? And finally, malicious content? The classic dilemma of whether to outsource, of course, hinges on budget and in-house expertise; but the suggestion to measure cost per accepted record was good advice. The presenter’s team also implements AI for legal review in an interesting way, which may be worth hearing.
Giancarlo’s presentation was eerily well-rehearsed, so it lasted only about fifteen minutes. It was a good way to wrap things up.
Conclusion
That’s all, folks! Thank you for reading. We still have one more conference to go, and that’s Extract Summit Vol. 2, taking place this November in Dublin.
Adam Dubois
Proxy geek and developer.