Ask HN: How would you queue and process 10K+ long running jobs

阿新 • • 發佈：2019-02-08

Hey guys wanted to ask a question about what technologies & methodologies you would architect together if you needed to constantly be queuing up 10K+ jobs, distributing the work out and then reporting that it was completed. It would be required that you should never be able to schedule a duplicate job when one is queued/running, and you need to ensure that each job only gets picked up by 1 worker and run once. These jobs could last anywhere (in run time) from 30 seconds to 1 hour.

I've tried googling but my fu is failing me. Would love to hear the thoughts from people who have maybe solved similar problems.

Thanks!

Ask HN: How would you queue and process 10K+ long running jobs

Hey guys wanted to ask a question about what technologies & methodologies you would architect together if you needed to constantly be queuing up 10K+ j

Ask HN: How do you track and manage your (personal) investments?

With the launch of services like Robinhood, access to investment in the public markets has never been easier. If you are currently investing in anything (s

Ask HN: How would you approach creating modular HTML templates for non-techies?

I am an intern developer at a company and I've been asked to put together some templates that can be used within their CMS to create basic pages. The syste

Ask HN: How are you building large apps with react and GraphQL

What patterns and tools are you using to build large apps with react and GraphQL?With Apollo, do you use cache or link-state? Are you using redux (I love r

Ask HN: How do you discriminate science and pop

It's not that simple. Language may be an issue here.There are people who study how we know stuff. After all the way you "know" that chairs hold you up is d

Ask HN: How are you organizing your accounting?

Have heard different things such as using different online tools and building own excel sheets. What's your method?

Ask HN: How do you deal with complain rate on SES

Hi all,I run a news letter with AWS SES and the complaint rate is reaching 0.05%. They expect 0.1% complaint rate.I have received zero complaints email fro

Ask HN: How do you stay fresh on the basics of computer science?

I had an uncomfortable experience last week where I realized I'd forgotten the basics of condition variables (in particular, how to prevent the thundering

Ask HN: How do you spend time during compilation?

I feel it's a significant problem when compilation lasts for more than 10s>. To not get bored, I am jumping into random articles during that time, loos

Ask HN: How can I improve and make my CV more effective?

Hi All, I am looking for constructive feedback about my CV. I am a software engineer turned manager with a long history in system integration consulting co

Ask HN: What would you like to learn in a programming school in your town?

I've been thinking on a way to help my local community, of beginners or really anyone that wants to learn how to be a programmer, to grow.Opening a small s

Ask HN: How do you measure your funnel?

We are working on a tool to help people measure their conversion funnels. For example:Unique visitors to website -> User Signups -> Paid CustomersTh

Ask HN: How do you handle user money that you have to return

I am thinking about building a system where customers would put money into their account and they can use there balance to purchase things. Whats the best

Ask HN: How do you become the best at something?

I realized that while I'm fairly competent at some things, I'm not really top percentile in anything save my ability to communicate. It's really weighing o

Ask HN: How do you market yourself at your mid

Hello folks,I have a weird situation and if you don't mind, I would like to ask for your advice.I'm having hard time getting hired, both onsite and remotel

Ask HN: How do you organize your stuff?

So, I'm having an issue right now that is accruing rapidly. I have a lot of documents, images, pdfs, videos, code snippets, files, etc, etc that I have tri

Ask HN: How do you notify your readers when you blog?

In the past I have allowed readers to provide their email address to receive new-post notifications, however I discontinued this as I don't want the respon

Ask HN: How do you manage a security policy for your small team?

My company is a small team operating in the digital health space that's recently grown to 12 in the past year. Although we don't deal with a ton of PHI, s

Ask HN: How Are You Curing Your Computer Eye Strain?

Is their a miracle eye drop or combo of ointments and techniques that can sooth our sore eyes?I find the next morning still a bit sore. I guesss after all

Ask HN: How do you deal with papers behind paywalls?

I'm beginning a self-study quest on computer vision. Unfortunately, I've learned that a lot of the less popular papers are not available for free download