administrator
Posts: 2702
Joined: Mon May 17, 1999 5:11 am

Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 7:19 pm

Hi guys,

First let me start by apologizing to the photographers that had photos lost during the weekend, about 300 entries had to be removed due to database problems. It should all be sorted out now and the effected photographers have been notified to upload again with priority screening. We have taken measures to see to that this does not happen again.

I would like to tell you about a small change to the screening process. Photos that are accepted by the screeners are not added immediately to the database but are queued and added in bulk when there's 10 or more shots waiting to be added. This is why some of you have reported photos as "lost" when you cannot find them on the photo stats page nor in the database. I have added info about this queue to both upload stats pages under MyPhotos. Should any of your photos be queued for addition to the database, it will be mentioned on the stats pages.

This change serves to purposes:
1) Simplify the screening script. During high load, the screening process can take a very long time as the script has to insert the new entries one by one, copy all relevant photo files to all three servers (one main + two backup) and send out emails to photographers. This has sometimes caused timeouts and other problems. By moving the inserts into the database, the copying of photo files and the emailing of photographers to a server based program, I am hoping the situation will improve dramatically.
2) By collecting at least 10 photos before making an insert into the main database, we save precious resources as we can make one big insert with all photos waiting instead of doing 10 or more individual inserts. This has very positive effects on server load and query cache.

I could also mention another small change I did recently - extended the photo ID number to include seven digits instead of six. A small change it might seem but there's a heck of a lot of work behind it. All scripts and databases on the site that deal with photos (there's a lot of those!) had to be checked and some re-written to handle the longer ID field. As you can see, all photo ID numbers now start with a zero but that will not last long. As of writing this, we are 700 photos away from the magic ID of 1,000,000. This is naturally why this change had to be made. If I had not, we would not be able to move beyond ID 999,999.
Note that this change does not effect links to photos. They can be linked to with or without the leading zero. For instance, http://www.airliners.net/open.file?id=0999111 and http://www.airliners.net/open.file?id=999111 works equally well.

Finally, I am happy to say that the new server is now installed and in a semi-test production phase. It is currently serving photos under photos.airliners.net and doing some SQL tasks. I will continue to test it for speed and stability for a few more weeks before moving it to its intended role as the main webserver.

Thank you for showing your work on Airliners.net - you're what makes the site what it is!

Best Regards,
Johan
Working on the site from morning 'till night that's livin' alright (1997-2007)
 
andrewuber
Posts: 2142
Joined: Tue Jul 22, 2003 10:45 am

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 7:22 pm

Thanks for the update! Glad to see almost a million photos on airliners.net!!

Drew
I'd rather shoot BAD_MOTIVE
 
Gary2880
Posts: 1856
Joined: Wed Aug 18, 2004 8:52 pm

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 7:34 pm

so.... we have to get 10 photos accepted before they appear on the database..
Patriotism is the last refuge of the scoundrel :- Samuel Johnson
 
viv
Posts: 2953
Joined: Tue May 31, 2005 5:17 pm

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 7:35 pm

It still takes about 9 days to get a shot accepted.
Nikon D700, Nikkor 80-400, Fuji X Pro 1, Fujinon 35 f/1.4, Fujinon 18 f/2
 
IL76
Posts: 2237
Joined: Mon Jan 05, 2004 5:43 am

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 7:47 pm

Quoting Gary2880 (Reply 2):
so.... we have to get 10 photos accepted before they appear on the database..

No, 10 photos (not just yours) need to have been accepted. If I screen a set of 10 photos, and as a result 5 photos have been rated HQ 3 times, these 5 will be added when at least 5 others have also been accepted.
So additions are done in batches of at least 10 photos to reduce the load on the servers. It's a technical thing, it has nothing to do with the screening process.

[Edited 2006-02-06 11:49:08]
 
Rotate
Posts: 1448
Joined: Tue Feb 11, 2003 5:52 pm

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 8:07 pm

Quoting Administrator (Thread starter):
10 or more shots waiting to be added



Quoting IL76 (Reply 4):
So additions are done in batches of at least 10 photos to reduce the load on the servers

as long as we dont have 100s accepted at once I am fine with that ...

Robin
ABC
 
dc10tim
Posts: 1380
Joined: Mon Jan 03, 2005 12:21 am

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 9:49 pm

Quoting Administrator (Thread starter):
It should all be sorted out now and the effected photographers have been notified to upload again with priority screening.

Hi Johan,

Further to your email received on Saturday evening, I re-uploaded one of the lost photos and left a note for the screeners and also emailed them, yet nothing has moved so far. I appreciate things don't happen like magic, but in the past when I've asked for a priority screen, the shot has been shown as being "in screening" within a couple of hours. It's not as though my life depends on it, but it is a little frustrating given it's nearly two weeks since it was first uploaded.

Regards,

Tim.
Obviously missing something....
 
frippe
Posts: 185
Joined: Sat Sep 17, 2005 4:24 pm

RE: Small Change To How Photos Are Added To The Db

Mon Feb 06, 2006 11:29 pm

Hello,
It is good news to hear that problems now seem to be solved. But I would like to know if we have to do anything special to have our lost photos screened with priority screening.

Maybe I ought to know how to fix a priority screening, but all my photos hitherto have never been near such an honor, so I simply don't know how to fix that.

Maybe the staff when things have calmed down a bit will pick up the lost and reuploaded ones from the queue and add them with priority?

Thanks in advance for any clarification,
Frippe
 
Gary2880
Posts: 1856
Joined: Wed Aug 18, 2004 8:52 pm

RE: Small Change To How Photos Are Added To The Db

Tue Feb 07, 2006 3:49 am

Quoting IL76 (Reply 4):
No, 10 photos (not just yours) need to have been accepted. If I screen a set of 10 photos, and as a result 5 photos have been rated HQ 3 times, these 5 will be added when at least 5 others have also been accepted.
So additions are done in batches of at least 10 photos to reduce the load on the servers. It's a technical thing, it has nothing to do with the screening process.

i see, cheers. thought it was a photographer to photographer thing.

at the rate im going if i had to wait till i had 10 accepted before they showed up in the database it would make a nice christmas present.
Patriotism is the last refuge of the scoundrel :- Samuel Johnson

Who is online

Users browsing this forum: No registered users and 4 guests