A programming task out of the ordinary

[delanvital]

- Imagine two different sites which hold information you use.
- You can not merge them entirely, because both sites are external and accessed via a subscription you pay.
- You have the right to extract all the data needed for this scenario.
- One site is a CRM-system. The other is a large database with supplementary information to that.

Can you do a script which logs into the first site, extracts an identifier, say a number, logs into the second site, looks up that same identifier, then grabs the additional information listed on the second site relating to that identifier and finally adding that to the first site's database.

There is no support from each site making this happen, but there is nothing against doing it.

There is a total of say 7500 unique lines in the first database, each with say 5-10 related columns which needs to be updated with information from the 2nd database. We could settle for 1000 unique entries and 5 related fields. We only have regular web access with password, not sure if a clever fella could find some SQL-like login from that.

We plan on updating the sites twice a year using this script or whatever, if it could be done. If it can, we are willing to pay for it.

Can it be done?
If so, what would it cost?

[b00n]

From what you've said I can't see any problem with doing this.  If there are only 7500 rows it would probably be easiest to write a script that dumps all the data that might be required from the second database so that you have a local copy which can then be processed and added to the first database as required.

If you only have web access and there is no other way to get at the data then it will need to be parsed from the page code, but that should not be very difficult.

Doesn't sound like a huge job and shouldn't cost much, less than three figures I would think.

[delanvital]

From what you've said I can't see any problem with doing this.  If there are only 7500 rows it would probably be easiest to write a script that dumps all the data that might be required from the second database so that you have a local copy which can then be processed and added to the first database as required.

If you only have web access and there is no other way to get at the data then it will need to be parsed from the page code, but that should not be very difficult.

Doesn't sound like a huge job and shouldn't cost much, less than three figures I would think.

Can it be set up, so that we can run the task ourselves, when we see fit? What would it require for us to have to run this script etc?

Are you interested in the job?

[b00n]

Yes, once the script was written you'd be able to run it as needed.  The only issue that springs to mind would be if the data has to be parsed from the web pages and either of the sites changes the layout of its pages significantly, which might break the parsing and would need tweaked to work with the new pages.

I would be interested in the job, but it depends on when you need it.  I'm finishing a uni project at the moment so I'm not going to have any free time till the end of the month.

If I were doing it, I would probably write the script in python which would allow it to be run on any OS that you can install python on, which is pretty much all of them.

[delanvital]

Yes, once the script was written you'd be able to run it as needed.  The only issue that springs to mind would be if the data has to be parsed from the web pages and either of the sites changes the layout of its pages significantly, which might break the parsing and would need tweaked to work with the new pages.

I would be interested in the job, but it depends on when you need it.  I'm finishing a uni project at the moment so I'm not going to have any free time till the end of the month.

If I were doing it, I would probably write the script in python which would allow it to be run on any OS that you can install python on, which is pretty much all of them.

We have time. We have been moaning for years

How about I contact you in a month's time. We can then arrange for you to go over the two systems and give an estimate of how many digits the job will cost? One is very modern, just released in a new version, the other (the CRM) is rather old and crappy...

[b00n]

Sure, any time after 1st September will be fine.

[delanvital]

Cool - I will get back to you. I also have access to some offline data, a  cd-rom, which could be an alternative to the second server instead.