sent out more creds to those who requested.
show/hide sold out is currently disabled because i just noticed there was a bug with it when hiding specific shop results.
Hey, I'd love access! Props to you for you putting in the effort, can't wait to see it.
Also, are you in SF with some relation to Satori? I may be misremembering but I think I picked that up somewhere along the way. I think we used to skate together occasionally when I lived in SF in the early 2010's - I used to skate the flat spot at 6th & Fulton in Golden Gate Park a lot and then Waller when it was first appearing there. Anyway, maybe that's not even you. In any case, stoked on what you're doing here!
i am located in sf and did work for satori. first as a in house sales rep, then as the tm / photographer / videographer / editor / web person. haven't been with them for many years though.
definitely skated 6th and fulton back in the day and still skate waller.
Are you doing this with Playwright or something? Regarding server resources, how bad does it get even when running it headless? I’ve also heard you could get page data by making a CURL request or something similar that just gets the page’s HTML content which you can then parse through. What else have you tried to get those nested results like shoe sizes?
python handles the requests, php parses the data through regular expressions, and javascript assists in the display. no browsers (headless or not) needed. then again, maybe that whole flow could be considered a headless browser.
you can use curl to request data. it's a great tool.
as for nested results, some shops return shoe size in their main search results page. others don't and getting it would require another request.
server resources are a concern, especially when the site is available to the world. i added rate limiting to hopefully prevent abuse, and also spent a good amount of time attacking the server while monitoring server resource usage.
there is server side caching, which helps with speed if people are searching for the same thing within a 12 hour time frame.
I actually just used Shop lurker yesterday to make a purchase. I was looking for a particular pair of shoes, and accepted that I'd pay full price, just had to find them in my size.
if you chose to expand it'd would be cool to sort shops by region, closer shops could be faster and sometimes cheaper on shipping. As you add more shops, filtering by region, could result in shorter search times.
nice. more work could be done to improve searching by size, but i've yet to tackle that issue.
adding more shops and sorting by region / locating shops near you is part of the plan.
unfortunately speeding up the search time is difficult because shop lurker makes 1 request per shop chosen. so if you are searching 5 shops at a time, that's 5 total requests. there's also some stuff happening on the back end that ensures the requests actually succeed, which slows things down a bit.
this project has turned into bit of a never ending rabbit hole. thanks again to everyone who has taken a look and provided feedback.
Topic: Shop Lurker (Read 18392 times)