I am trying to aggregate some data for a one off piece of work that I’m doing but I keep getting 500 errors after about 200-500 queries. There is no specific reason as to why it’s happening. I do have about 20,000 queries to do in total but I have split them into ~7000 record chunks and I limit the requests to maximum of 2 per second which keeps it safely under the 600 per 5 minutes limit. Anyone help me out here? It’s getting really frustrating waiting for ~20 minutes only to get a 500 error and having to start the entire query again. Thanks
EDIT: Now getting 502 Bad Gateway errors now. Doesn’t complete more than 5 queries…
EDIT 2: Also getting 504 Gateway Timeout Error…