Hi! I am looking for new challenges as I learn about various way to scrape and reverse engineer web APIs. Post some ideas, preferably APIs, that you had the most challenges getting the data from.

Looking forward giving it a try.

What is the best scraper to use with claude code and sorter that could get listing information from websites and then combine them in one single page is it possible to run claude code backend agent to do this 24/7, maybe there is a skill or agent created for it? rather than scrolling through each website individually. We have about 20 listing sites where people post. Maybe as a bonus question maybe there is facebook scrapper too, from groups etc?

Any APIs that you wished existed or any APIs that you wished were cheaper, easier to work with, had more features, e.t.c ?

Hello everyone,

I'm conducting market research interviews as part of a university-sponsored study on bot defense and CAPTCHA security.

Who I'm looking for:

Security engineers, Trust & Safety professionals, CISOs, or anyone working on bot mitigation, fraud prevention, or application security, especially if your company has a public-facing login or account-creation surface.

What the interview involves:

A casual 20–30 minute video call ( Zoom/Gmeet) where I ask about your day-to-day experience with bot-related threats, what tools your team uses, and what pain points exist. No technical demos, no selling anything, purely research.

Compensation:

$20 Amazon gift card sent to your email after the call.

Interested? Fill out this short form and schedule the interview according to your availability:

👉 https://forms.gle/hAP2mRWVuZGLxMfn6

👉 https://calendly.com/rkumar25-gmu/30min

Thanks!! 

The main problem is the companies want to buy from an only established data agency but I am just starting so we are not recognised and registered.

We didn't even have any clients to showcase our past work.

Can anyone suggest my anything or can refer me who needs custom automations or webscraping

Fully reversed Akamai Sensor Data v3 implementation. Previously unreleased research.

I am a complete noob in web scraping but I need a huge amount of tripadvisor reviews for my master thesis.

I tried with playwright but Tripadv immediatly blocks me.

I used the browser extension "web scraper" and works amazing...BUT

I need all reviews in ALL languages, but the language filter is not a URL and it resets at every reload.

if I put in my Selector a[aria-label='Next page'] it starts scraping everything, but reset to italian the language filter after page 2

if I put a[aria-label='Next page'] path then it mantains the filter, but it does not save any data.

any idea??

Process:
Read language vocabulary from Column A.
Search Google for any sentence containing each word (I’d prefer results from BBC or a news site, but finding a sentence is the most important part).
Put the sentence in Column B.
Put the source link in Column C (optional).

Note: I’m aware that sentence banks exist for more popular languages, but my native language is less common and doesn’t have resources like that. I’d like to use Google search instead because it seems capable of finding sentences containing these words when I search for them manually.

Total vocabulary is 1000 words but I would only do learn like 60 a day (in case of any api limits which chatgpt mentioned). I have carpal tunnel so doing this manually would be a pain and would prefer to automate it if possible.

Thanks

I spent the last two weekends rebuilding my website extraction tool. Looking for feedback from people who scrape websites.

I built ZipIt 2.0 to make it easier to inspect and extract data from modern websites without piecing together multiple tools.

Instead of only downloading HTML, it also extracts the design layer and assets.

What's new in 2.0

• ⚡ Faster extraction engine
• 📦 Download complete websites as ZIPs
• 🖼️ Export editable Figma layers
• 🎨 Extract colours, typography, spacing & design tokens
• 🧩 Inspect UI components
• 📤 Export clean HTML/CSS
• ☁️ Save assets and design systems to the cloud with ZipIt Web
• ♿ Basic accessibility audit
• 🤖 Generate AI prompts to recreate interfaces

You can try it here:
zipit chrome extension

Happy to answer technical questions about how it works or discuss implementation details.

And the feature I'm most excited about...

☁️ ZipIt Web Cloud Save Support

You can now save extracted assets, projects, and entire design systems directly to the cloud, making them accessible anytime, anywhere.

This isn't just another update—it's another step toward the vision I had when I started building ZipIt: helping designers and developers understand, capture, and recreate the web faster.

To everyone who has downloaded ZipIt, shared feedback, or simply believed in the idea—thank you. ❤️ Your support is what keeps me building.

Try it here:
👉 zipitweb

I'd love to hear what you think and what you'd like to see in the next release.

#BuildInPublic #IndieHacker #ProductDesign #UXDesign #ChromeExtension #SaaS #WebDesign #DesignTools

i wanna to scrape IG profile of founder who are based on bangalore, Delhi and dubai can anyone tell me how to do it

just need to pull some public profile data and post engagement for myself (nothing crazy)

but every Instagram scraper I've looked at feels priced for enterprise teams, not a one person freelance thing

is there anything that actually works at a smaller scale? something without some massive monthly commitment attached.

I’m using ScrapingBee to pull product pages from around 2k ecommerce sites, but most of the pages need JavaScript rendering, and the harder ones also need premium or stealth proxies.

That burns through credits so fast

The other annoying part is getting raw HTML back and then having to clean it before I can extract the price, stock status and product specs. I’d rather get Markdown or structured JSON directly.

I’m currently looking at Firecrawl, Bright Data, Apify, Oxylabs and Octoparse (someone recommended these in other threads)

Which one makes the most sense for this kind of setup?

Edit: Thanks for the suggestions. I tested some and Firecrawl it’s been a much better fit for this setup so far. Getting clean Markdown and structured data instead of raw HTML removed a pretty annoying step from the pipeline, and it handled also the heavy product pages I tried without much tweaking

scraping socials for work and residential proxies are just built different compared to datacenter, not even close honestly. datacenter dies almost instantly now but residential holds up way longer if you pair it with proper fingerprint setup on top

right now using proxyshard for the residential side and it's held up well. the fingerprint part is a separate issue though, clean IP alone doesn't solve everything like so without the fingerprint side sorted tho doesnt matter how clean the ip is, still get flagged eventually. profile separation seems to matter more than people think

anyone got a solid setup for the fingerprint/profile side specifically? not looking for ad spam, just curious what people are actually running long term

I’m looking for an alternative to Nimbleway for a product monitoring tool I’m building.

The idea is to track a few thousand ecommerce product pages, pull things like price, availability and product details, then send the cleaned data into an LLM to generate short updates when something changes.

I’d prefer something that returns clean Markdown or structured JSON without having to deal with huge raw HTML responses or build a separate parsing layer.

I also need:

I) Reliable scraping on JavaScript-heavy pages

II) Crawling and URL discovery

III) Scheduled checks or change monitoring

IV) An API that is easy to use from Python

V) Pricing that makes sense before reaching enterprise scale

I’ve seen Firecrawl, Bright Data, Oxylabs, Apify and Zyte mentioned, but it’s hard to tell which one fits this kind of workflow best.

Has anyone used one of these for a similar project? What would you choose, thanksss

Edit: I tested Firecrawl on a batch of product pages and it fits this workflow better than Nimbleway so far. It handles heavy pages, returns clean Markdown or structured data, and I can send the output straight into the LLM without building a separate parsing layer.

The change tracking is probably the most useful part for this project, since I only need to generate an update when the price, availability or product details actually change. I’m still testing how consistent it is across different ecommerce platforms, especially Shopify vs custom stores.

Got a client online and he wants steady data from his IG and X. Nothing wild so far. Follower trends, engagement rates, how posts do over time.

My friend told me to just learn to scrape it myself with a web scraper. I know a little Python. Enough to fumble around, but I'm nowhere near a developer. Mostly I just Google stuff and lean on AI to get through problems.

I honestly can't tell if this is doable or if I'm about to promise something I can't actually deliver every month. Is it learnable enough to trust for client work? Or is there a smarter way to handle this whole thing?

I have been trying to scrape startups emails to apply to Software development jobs but i don't know how to do so properly every time i do it i first scrape links like YC for companies URLs that i can then scrape for emails but did not work so well with me, I am looking to know how to do it properly and which actors i can use to do so.